Seasonal to decadal predictions are inevitably uncertain - these uncertainties can be accounted for using ensemble techniques. In a perfect system, each ensemble member would represent a potential realization of the true evolution of the climate system, and the predictable components in models and reality would be equal. However, we show that the predictable component is sometimes lower in models than observations, especially for seasonal forecasts of the North Atlantic Oscillation (NAO). In these cases the forecasts are under-confident, with each ensemble member containing too much noise. Consequently, most deterministic and probabilistic measures under-estimate potential skill and idealized model experiments under-estimate predictability. However, skilful and reliable predictions may be achieved using a large ensemble to reduce noise, with results shown here for the NAO.