I noticed Anthony Watts and Judith Curry getting quite excited by a new paper from Emma Suckling and Leonard Smith called An Evaluation of Decadal Probability Forecasts from State-of-the-Art Climate Models. The paper was basically a comparison of an ensemble of dynamical climate models (GCMs) and empirical models. The basic conclusion was that empirical models are, statistically, better at decadal forecasts that dynamical climate models.
I was initially quite positively inclined towards this paper, but the more I’ve thought about it, the more critical I’ve become. Having said that, it’s always possible that I’ve missed some subtlety or misunderstood some aspect of the paper. Hence, I’m happy to be corrected by anyone who knows more than me. The core figure is probably the one I show below. The left-hand panel shows decadal projections for the ensemble of dynamical climate models, while the right-hand panel is for the empirical models. Forecasts are launched every 5 years and so the two panels on each side are simply to split the forecasts as they would overlap if they were in the same figure.
Decadal forecasts from dynamical climate models (left-hand panel) and empirical models (right-hand panel) (credit : Suckling & Smith 2013).
If I understand the rest of the paper, it is essentially an analysis that indicates that – when considering decadal forcasts – the empirical models outperform the dynamical climate models. So, what are my issues with the paper? As far as I understand it, dynamical climate models are extremely complicated. They consider the oceans, land, atmosphere, polar regions, can consider both regional and global climate, and they can do more than simply consider surface temperatures. All this study seems to have done is compare global surface temperatures from these two types of models. Unless I misunderstand something, these empirical models can do virtually nothing else. It’s not really a like-for-like comparison. They’re not really considering two models that can do the same things. They’re comparing a very simple model that can do only one thing, with one aspect of a very complicated model.
Another issue is that if I consider the right-hand panel in the figure above, then it appears that if you were to overlay the top and bottom panels, there would be quite sharp discontinuities at a number of the 5-year launch points. There appears to be an element of this for the dynamical climate models, but it does not appear quite as severe. This would seem to indicate that the empirical models would do a very poor job if used to forecast more than a decade. Additionally, most of the papers cited when discussing the empirical models were written in the 2000s. The comparison, however, starts in 1960. Given that one would expect these empirical models to have been developed based on past knowledge, it would remarkable if they didn’t do a very good job of forecasting the period from 1960 to 2000. You might argue that that’s true for dynamical climate models and there, presumably, is some merit to this suggestion. Dynamical climate models are, however, constrained by the laws of physics. Empirical models, I believe, are not. Given that there are no such constraints on empirical models, it would be pretty amazing if people developing empirical models in the 2000s did not ensure that they were particularly good at forecasting the period prior to 2000.
The paper concludes with a few interesting comments
It also calls into question the extent to which current simulation models successfully capture the physics required for realistic simulation of the Earth system and can thereby be expected to provide robust, reliable predictions (and, of course, to outperform em- pirical models) on longer time scales.
I find this a little bit of an odd statement. Unless I’m mistaken, empirical models have no physics and so they seem to be concluding that dynamical climate models may not have captured all the physics needed because they’re outperformed by models with no physics at all. It may well be that dynamical models do not have all the necessary physics, but it’s not clear why such a comparison is needed to know this. A comparison with actual observations would tell you this. Also, this study has only considered one aspect – surface temperatures – of dynamical climate models. Dynamical climate models are also used for more than just making forecasts. They’re being used to try and understand the climate and how it evolves, and also to consider different future emission pathways. Empirical models, I believe, can do none of this.
The paper then says
The blending (Broecker and Smith 2008) of simulation models and empirical models is likely to provide more skillful probability forecasts in climate services, for both policy and adaptation decisions.
This may well be true. There may well be policy decisions we might want to make based on decadal forecast and so empirical models may well play an important role here. So, I’m certainly not suggesting that empirical models have no role, just that I’m unclear as to the value of the kind of comparison done in this paper.
I have been rather critical of the paper so, again, if someone thinks I’ve misunderstood it, or missed something important, feel free to point it out through the comments. Something I haven’t touched on is how other’s have interpreted it. There is already some evidence that some interpret this as implying the empirical models are better than dynamical climate models. Given that dynamical climate models do much more than simply consider surface temperatures, this interpretation is – in my opinion at least – incorrect. I also find it interesting that these type of papers (i.e., statistical analysis of climate models) often seem to come from people associated with economics, rather than from climate modellers themselves (I commented on something similar a while ago). I appreciate that the authors of this paper have physical science backgrounds, but I’d be fascinated to know what climate modellers actually think of these papers. Do they find them useful and interesting, or do they – secretly – find it frustrating that some think that the way to assess climate models is through statistical analyses rather than through checking how well they satisfy the fundamental laws of physics? Anyway, there’s probably more that could be said but I’ll stop there.