1. The Broad and Provocative Interpretation

In more provocative moments, the authors claim to have established that the combination of nonlinear mathematical models with structural model error is a “poison pill” that “pulls the rug from underneath many modeling endeavors” (Frigg et al. Reference Frigg, Bradley, Machette and Smith2013a, 479). Since most mathematical models of interest in science are nonlinear, and few are free from structural model error, it is supposed to follow from their argument, interpreted broadly, that any “probabilities for future events to occur” or “probabilistic forecasts” (479) derived from such models cannot be trusted. Still, all is not lost, because the authors are willing to concede that “not all the models underlying these forecasts are useless” (Frigg et al. Reference Frigg, Bradley, Du and Smith2014a, 57). A model that has been shown to be “maladaptive” for “quantitative prediction” (which is presumably what their argument establishes) may still be “an informative aid to understanding phenomena and processes” (48). In other words, mathematical models can be qualitatively informative in spite of the fact that they are not quantitatively trustworthy.

Taking this conclusion seriously would have dramatic implications for contemporary climate science. Though a precise assessment would depend on careful parsing of the qualitative versus the quantitative predictions of models, it is safe to say that much of what climate scientists claim to know would have to be regarded as untrustworthy. The authors seem aware of this risk because in the very paragraph where they consider whether or not “science [is] embroiled in confusion,” they include a footnote with the reassuring claim that this “casts no doubt on the reality or risks of anthropogenic climate change, for which there is evidence from both basic physical science and observations” (Frigg et al. 2014a, 48).Footnote ¹ According to the Intergovernmental Panel on Climate Change (IPCC), establishing the reality of anthropogenic climate change requires both ‘detecting’ and ‘attributing’ climate change. Detecting changes in climate, based on weather observations, requires determining that “the likelihood of occurrence by chance due to internal variability alone … is small” (Bindoff et al. Reference Bindoff, Stocker, Qin, Plattner, Tignor, Allen, Boschung, Nauels, Xia, Bex and Midgley2013, 872). This requires an estimate of internal variability, generally derived from a “physically based model” (873). Furthermore, attributing the detected change to a specific cause (e.g., human activity) involves showing that the observations are “consistent with results from a process-based model that includes the causal factor in question, and inconsistent with an alternate, otherwise identical, model that excludes this factor” (872). The IPCC is quite clear that “attribution is impossible without a model” (874).

2. The Narrow and Modest Interpretations

While it would certainly be earthshaking if this argument had managed to establish that most of contemporary mathematical science was “embroiled in confusion” or “maladaptive,” the sheer implausibility of such a result suggests looking for a more modest reading of the scope of the argument. In places, the authors characterize the scope of their argument more narrowly, but they do this in two distinct ways. Sometimes they infer from the precision and locality of a prediction that it must be subject to doubt on the basis of their argument, but in other cases they suggest that the applicability of their argument depends on how a probabilistic prediction was generated from the underlying mathematical model. Ideally, the class of modeling projects subject to doubt by this argument identified by the details of the content of their predictions and the class subject to doubt in virtue of their method of generation would line up, and it would be obvious why they line up. Whether this is so is something that will be considered after these two ways of characterizing the scope have been explicated.

The United Kingdom Climate Projections 2009 project, or UKCP09, which aspires to produce decision-relevant “high-resolution forecasts of twenty-first-century climate” (Frigg et al. Reference Frigg, Smith and Stainforth2013b, 887), is the example repeatedly used by the authors to establish that their argument applies to actual climate science. From the fact that UKCP09 forecasts “the hottest day in August in a particular year” (Frigg et al. Reference Frigg, Bradley, Machette and Smith2013a, 490) or “the temperature on the hottest day in central London in 2080,” the authors infer that their argument is “not just a hobbyhorse for academic philosophers” (Frigg et al. Reference Frigg, Bradley, Du and Smith2014a, 50). Evidently, because the UKCP09 project assigns probabilities to climate variables in relatively small regions far off in the future, the authors infer that this modeling project falls within the scope of their argument. Regional climate projections are typically subject to more uncertainty than their global counterparts. However, if these authors are right, there is, in principle, a new argument establishing the impossibility of the kinds of projections that UKCP2009 hopes to produce.

The second, narrow way of characterizing the modeling projects subject to doubt by this argument involves identifying a particular method of generating probabilistic predictions of a certain sort from a mathematical model and then claiming that predictions of that sort, generated in that way, from a nonlinear, imperfect model are subject to doubt. The methods and predictions are characterized by working through a sample dynamical system whose time evolution is given by the logistic map. In their example, the logistic map relates the population of fish in a pond to its population in the previous week. This system is subject to both the butterfly effect, where arbitrarily close initial conditions will follow very different trajectories, and what the authors call the hawkmoth effect, where small structural differences between two nearby time evolution functions can result in very different trajectories even with identical initial conditions. Consequently, the model can produce “fundamentally misleading” probability distributions over final states. The authors show that in this case a particular strategy for managing the epistemic uncertainty coming from the butterfly effect will not work in the presence of the hawkmoth effect.Footnote ² If a system is subject to the butterfly effect (and initial conditions are uncertain), it is not, generally, possible to make informative point predictions about the state of the system at future moments in time. It is, however, possible to manage the epistemic uncertainty resulting from the butterfly effect by evolving a probability distribution over the initial states of the system into a probability distribution over the final states of the system at some future time. By following this strategy, though one is no longer able to predict the precise future state of the system, one can instead produce a probability distribution over possible future states. Unfortunately, as the authors demonstrate in this case, if the true time evolution is a slightly perturbed version of the logistic map, then the probability distributions over final states at some future time obtained by using the imperfect model can differ substantially from the “true” probability distribution over those final states. This shows that the suggested strategy for managing the uncertainty due to the butterfly effect will not work in this context. The suggested strategy gives rise to enormous second-order uncertainty about how much uncertainty there is due to initial condition error plus the butterfly effect. It is this second-order uncertainty that undermines the predictions generated by the model.

3. Generalizing from the Base Case and the Application to Climate Science

Granting everything the authors hope to have demonstrated about the implications of the hawkmoth effect for their example, assessing the scope of their argument requires considering the conditions under which other modeling projects might—because of their similarity to the base case—also be subject to concerns about the hawkmoth effect. For this purpose, a modeling project can be regarded as having four dimensions across which it can be compared to the base case. First, there is the time evolution function itself, which may or may not behave in the way the logistic equation does in response to small deviations from the ‘true’ time evolution. Only if the time evolution of a modeling project is subject to ‘structural instability’ in a manner relevantly similar to the logistic equation might the predictions of that model be undermined by the hawkmoth effect.Footnote ³ Second, there is the timescale of the desired prediction. Like the butterfly effect, the hawkmoth effect degrades certain sorts of predictive accuracy, and it may indeed do it at a very fast rate, but it still takes time, and so the degree of epistemic danger that it poses will always depend on how far out into the future one wants to predict. Third, there is the manner in which predictions are generated from the dynamic system, which may or may not be relevantly similar to the ‘default’ strategy for generating predictions that the authors employed in the base case. The default strategy was a way to manage the butterfly effect, which was, in turn, shown to be undermined by the hawkmoth effect. Unless the strategy for managing the butterfly effect employed by a modeling project is similarly undermined by the hawkmoth effect, that project would not be relevantly similar to the base case. Finally, there is the kind of prediction that one hopes to obtain from a modeling project.Footnote ⁴ The authors considered probability distributions over complete states of the system and how they evolve as time advances. But there are many other kinds of predictions that one might make about a dynamical system. One can make predictions regarding the statistics that a system will exhibit over an infinite life span, or regarding those it exhibits over a finite period that is long compared to periods we associate with weather prediction. Thus, the kind of prediction generated in a modeling project must be relevantly similar to the distributions over final states considered in the base case for the authors’ concerns about the hawkmoth effect to apply to that project. Overall, a modeling project would be plausibly undermined by the hawkmoth effect based on this argument, if its time evolution, method, and type of prediction (including timescale) are relevantly similar, in turn, to the logistic equation, the default strategy, and probability distributions over final states of the system at some (sufficiently) future time.

The authors do provide reasons for thinking that the mathematical models scientists are typically interested in are not ‘structurally stable’. And they contend that not being ‘structurally stable’ is the feature a model must have if it is to be relevantly similar to the logistic map. Thus, insofar as they are right, the default strategy for managing the butterfly effect won’t work in these models either. Furthermore, the authors propose that if scientists hope to deal with the butterfly effect, the burden of proof should be on them to establish that their model is structurally stable and therefore that the hawkmoth won’t undermine their modeling project. It is important to note that this argument does not in anyway address the other three dimensions—the method, the type, and the duration of a prediction—along which a modeling project would have to be similar to the base case in order for the concerns about the hawkmoth effect to apply to it.

The importance of these other three dimensions can be brought out by considering the implications of this argument, interpreted narrowly, for contemporary climate science. If the scope of the argument is taken to be just those modeling projects that are relevantly similar to the base case, we must consider how many modeling projects in climate science deal with the butterfly effect by evolving probability distributions over initial conditions into probability distributions over the states of the model at future times (hereafter: using the “default strategy” to generate point predictions). These are the cases that, if the rest of the analogy holds, will have been unproblematically shown by the argument to be subject to concerns about the hawkmoth effect. The first thing to note is that the probabilities produced by the UKCP09 project are not produced by the default strategy, as the authors are aware. They say, after describing the default strategy, “UKCP09 probabilities are formed in a more complicated manner, combining outputs from multiple (imperfect) models using Bayesian methods” (Frigg et al. Reference Frigg, Bradley, Machette and Smith2013a, 483). There is no straightforward identity between the scope of their argument characterized by the content of the predictions (localized probability statements about the future) and the scope characterized methodologically. While the UKCP09 projections are relatively precise and local, neither were they produced using the default strategy nor are they point forecasts about the atmosphere at some future time. Perhaps any probability statements about the climate of localized regions in the distant future must, ultimately, depend on attempts to manage the butterfly effect by the default strategy, or be subject to similar worries due to the hawkmoth effect. But the authors provide no reason to think this and merely suggest that it is unclear why the strategy used in UKCP09 would make the concerns about the hawkmoth effect “go away.” Moreover, nowhere do they attempt to show that the modeling projects of the UKCP09 are relevantly similar to the base case, and so they provide no reason, in the first place, for thinking that concerns about the hawkmoth effect are relevant to the sorts of probabilities that are generated in this study.

4. Two Questions

There are two follow-up questions we should ask about the strength of this argument by analogy as it applies to the modeling project in climate science: (1) How strong is it for modeling projects that are using something like the default strategy to generate “decision relevant predictions” appropriately analogous to those in the base case? (2) How strong is it for modeling projects that generate the kinds of local and precise projections generated by the UKCP09 project?

The first question is more difficult. Frigg et al. (2013a, 2013b, 2014a, 2014b) suggest that if we think we know the precise form that the best model of a system should take, and that model happens to be chaotic, then it is overwhelmingly likely that the model is ‘structurally unstable’. Thus, if it is employed in something like the default strategy, the model will be maladapted to quantitative prediction of any kind. But this is less than clear from the mathematics. Whether a model is chaotic is the more straightforward question. But whether or not it is ‘structurally stable’ in the relevant sense is more complicated, and it depends, crucially, on what other models one is comparing it to—on the universality class one takes it to belong to. Otherwise, there is no well-defined notion of a nearby model. Therefore, the strength of the analogy for casting doubt on the capacity of a broad range of mathematical models to generate “decision relevant predictions” when those modeling efforts use the “default strategy” is indeterminate insofar as there are far too many open questions about climate modeling. We simply do not have a clear grip on what the right universality class is for climate models, nor on what the relevant metric of similarity ought to be.

But even if it were to turn out that climate models are structurally unstable, in the relevant way, it is unclear what the relevant timescales would be for such instabilities to manifest themselves. Just as chaotic models with initial condition error can make good predictions in the short run, structurally unstable models with small model error can make good precise predictions in the short run.Footnote ⁵ And with climate models, even if there are good a priori reasons for thinking that the relevant family of models are structurally unstable (or that the burden of proof should be to show otherwise), such an a priori argument would not allow us to determine what the relevant notion of “short run” is. In some empirical contexts, the default strategy might still be a viable one, even when the relevant models are chaotic or even structurally unstable—if the prediction task we are interested in does not get derailed very much by small model errors over the time frame that interests us.

When we move outside the domain of toy models and toy contexts where clean mathematical results are available, the best we can do is experiment with our models and see what small perturbations do to our predictions. Luckily, when climate scientists do study, empirically, the effects of small perturbations to their models on the class of predictions that interest them, they don’t find dramatically divergent results.

The answer to the second question is simpler and more fundamental: climate science rarely uses the default strategy. This is because climate scientists do not usually engage in forecasting probability density distributions over precise states of the atmosphere for precise future times, but that is the output of the default strategy. The sort of predictive task that climate modeling would have to be engaged in for the analogy to the prediction of the base case to hold would be something like creating a distribution for the probability of the earth being in a tiny region of massively high dimensional configuration space that defines the earth’s weather at 3:45 p.m. on September 23, 2098.

We noted that Frigg et al. often argue for the applicability of their argument by analogy, not on the basis of the fact that the “default strategy” has been used, but on the basis of the “precision and locality” of a prediction task being tackled. We already pointed out that despite the fact that the UKCP09 makes what are, for Frigg et al., canonically “precise and local predictions” (or rather projections), it does not employ the “default strategy.” Here we pursue that point in more detail by considering some distinctions that are made in a chapter of the latest IPCC report entitled “Near-Term Climate Change: Projections and Predictability.” The point of the chapter is to assess recent attempts to make use of “observation-based initial conditions” (Bindoff et al. Reference Bindoff, Stocker, Qin, Plattner, Tignor, Allen, Boschung, Nauels, Xia, Bex and Midgley2013, 960) in order to predict near-term climate. The contrast is with what had been the main focus of assessments of future climate in previous IPCC reports: climate projections. Climate projections are “climate change experiments with models that do not depend on initial conditions but on the history and projection of climate forcings” (958). Simulations run in a climate projection are “not intended to be forecasts of the observed evolution of the system but to be possible evolutions that are consistent with the external forcings” (959). That is, they are projections, rather than predictions, in the language of both the IPCC and the UKCIP. The IPCC statements about climate change up to this point have been largely based on experiments of this sort. Insofar as the IPCC has made probabilistic claims about the future climate, then, it doesn’t seem that those probabilistic claims have been generated using the “default strategy” (i.e., by evolving probabilities reflecting uncertainty about initial conditions through their imperfect models); those would be “predictions.” Instead, it seems that the predominant strategy for dealing with uncertain initial conditions has been to ask questions of the models that do not depend, at least very much, on initial conditions, or even on probability distributions over initial conditions—that is, to produce climate “projections.”

Unfortunately, it is very easy to equivocate between a prediction being precise and local, in the sense that these authors use the term, and a prediction being dependent on initial conditions. Even very precise, local, and relatively short-term (order of a decade) climate projections are not weather predictions. Even a precise projection of what one should expect of the highest temperature that the city of London will reach in the summertime in the decade beginning in 2010, while highly “local” in some intuitive sense, could also be a climate projection (a forcing experiment), which is independent of knowledge of the present conditions. It is true, of course, that what you are actually going to get over a relatively short period of time (one sort of precision and locality) is dominated by internal variability, and a precise and local forecast of what you will get might need to consider initial conditions.Footnote ⁶ But it doesn’t follow from this that one can’t also make a short-term projection of what you should expect over a short time frame, independently of initial conditions. This is an important tripartite distinction that Edward Lorenz (Reference Lorenz1975) made between weather prediction and climate predictions of the first and second kind. Weather predictions are predictions concerning what are you are going to get at some future time. Climate predictions of the first kind are the average value the system will exhibit over its lifetime conditional on a forcing. But climate predictions of the second kind concern the expected value of the system over a relatively shorter time (Lorenz allowed for the timescale to go down to a decade) conditional on a forcing, but independent of initial conditions. These expected values about the climate are independent (as a matter of definition) of initial conditions even when the initial conditions turn out to be more important than the forcing in determining the actual weather.

Many climate modeling strategies, as we have seen, involve methods that avoid dependence on a particular set of initial conditions. Thus, it is only predictions (as opposed to projections) that might depend on the “default strategy” in order to manage the initial condition uncertainty. So, only an equivocation between predictions on the one hand and highly local and precise projections on the other would lead one to think that all precise and local forecasts depend on the default strategy and hence (setting aside our doubts about the answer to question 1) are subject to worries related to the hawkmoth effect. This is why even the UKCP09,Footnote ⁷ despite the fact that it makes highly “precise and local” projections, needn’t rely on the default strategy.

The authors flirt, at one point, with dealing with the kind of distinction we are drawing here, but they insist that many of what they call the “predictions” (which are in fact projections) of the UKCP09 are “not so different from weekly predictions in the fish model” (Frigg et al. Reference Frigg, Bradley, Machette and Smith2013a, 489), and so the distinction they discuss “does not help circumvent the difficulties we describe.” (490). But the relevant point of comparison between the two cases of prediction/projection is not whether they seem different or not so different, but the degree to which they depend on initial data and concern the determinate state at some time after the time to which the initial conditions are indexed. The fish prediction does, and the UKCP09 projections do not. And in that respect, the projections of the UKCP09 are not at all like the weekly predictions in the fish model. One is a point forecast, and the other averages over massive numbers of degrees of freedom in the model. Put crudely, a weekly prediction in the fish model is a weather prediction, not a climate projection.

Thus, while we certainly do not want to insist that only synchronic predictions ever fall under the scope of the hawkmoth effect, we do want to caution that whether or not structural instability of some family of models (even when it does in fact exist) “pulls the rug out” from under some field of inquiry will depend a great deal on the complex interplay between the nature of the predictive task to which the model is being put and the timescale of the prediction.

The relevant question for climate science, then (and for any science that depends on nonlinear modeling), is this: how do we decide whether a predictive enterprise is undermined by the possible existence of the hawkmoth effect? Here we repeat a point we made earlier: that the relevant questions for climate science lie outside the domain of the study of toy models, and that the best we can do is to run experiments with our models and see which predictions are stable under the perturbations that we think ought to concern us. The likelihood of the question being settled by mathematical results on toy models is small.

If we want to get a good grip on the limits of the predictive accuracy of our best models, we are better off looking carefully at the actual relations that we believe they bear to the real world than we are looking at their abstract mathematical properties. When we do look at the former set of features of our best models, they reveal many good reasons for skepticism about overly fine-grained projections. If our best models are currently unsuitable for making the fine-grained projections of the kind we find in, for example, UKCP09, we believe that the reasons have to do with the fact that some of the features of our climate system are poorly understood or poorly parameterized. Regional projections are particularly difficult because of the coarse graining of the globe’s topography in global simulations and other domain-specific features of global climate models. It is not, as far as we can tell, because the models need to be infinitely precise in order to do this job well. We do not think that there is any good reason to believe that last claim. And the factors that do in fact make climate modeling so difficult do not comprise a “poison pill.”