1. Introduction

It is now widely accepted that global warming is real and in large part due to human activities.Footnote ¹ But the big picture of increasing global average temperature reveals few specifics about how climate change will impact human lives and natural ecosystems at a local scale. Regional climates are changing, and will continue to change, in their own unique ways. Residents of one region may experience more summer droughts, while others on the same continent experience fewer. One city may face higher storm surges; another, more severe heat waves; a third, reduced monsoon rains.

To some degree, we can adapt to these changes by aligning our behavior, institutions, and infrastructure with the new reality. Farmers can plant different crops, foresters can adjust their management plans, a water company can add a new reservoir, a city might strengthen its flood protection, centers for disease control can prepare for migrating vectors, city heath departments can build cooling centers for vulnerable residents, and insurers (and reinsurers) can recalculate their prices.

Adapting, however, requires insight into where the local climate is headed. Is it worth investing in a new irrigation system? That depends on whether recent dry conditions are the new normal or just a streak of bad luck. Longer-term plans that are expensive to change once implemented would better be informed by projections further into the future. Construction of Britain’s new high-speed train infrastructure, for instance, is set to begin in 2017. We would like to design the tunnels to avoid overheating during late twenty-first-century summers, projected to be warmer and drier than today’s and with more intense hot weather events (High Speed Two Limited 2013). Several coastal sites in the United Kingdom are currently being evaluated for new-build nuclear power stations; the choice of site, physical design, and adaptive management plan are sensitive to long-term projections of local sea-level rise (Wilby et al. Reference Wilby, Nicholls, Warren, Wheater, Clarke and Dawson2011). And similar questions arise, of course, in other parts of the world.

Our topic is how climate science can best inform these local adaptation decisions. We begin by reviewing a standard approach to local climate change projections (sec. 2) and considering some reasons for caution (sec. 3). An alternative emerges from a sideward look at how scientific knowledge has been incorporated into decisions on issues such as dam safety and regulation of toxic substances. The key idea is to introduce a process of structured expert elicitation (sec. 4). We discuss specific issues that may arise in implementing such a procedure for local climate change adaptation decisions (sec. 5) and then draw some conclusions (sec. 6).

2. Answers Offered

Science-based information and advice that aim to inform local adaptation decisions go by the general name climate services. The goods on offer include data sets and climate forecasts (sometimes called products), as well as guidance on how to approach and think about adaptation planning (decision support). A variety of organizations currently develop and deliver climate services, including private consultancies, non-governmental organizations, universities, and government agencies.Footnote ²

While climate services products vary in what exactly they offer and in how the offerings are presented, the methodology behind them is, in broad strokes, often the same: start with output from general circulation models (GCMs), which simulate the whole climate system at a coarse resolution; then use one or more downscaling methods, which interpolate the GCM output to give finer details over the region of interest; finally, downscaled projections may be combined with local historical data. Methods diverge in their selection of GCMs, their approach to downscaling, and how they manage and communicate the uncertainties that pile up along the way.

An example of such a product offered by a government agency is The United Kingdom Climate Impacts Program’s UKCP09 project. The project offers a set of quantitative projections for local variables such as the change in temperature of the warmest summer day in London or the change in total precipitation during the wettest winter season in Tewkesbury.Footnote ³ These projections, at the scale of a 25 km grid, can be directly accessed on UKCP09’s website in the form of probabilities for different timescales (2020s/2050s/2080s) and alternative emissions scenarios (high/medium/low). The projections are advertised as relevant for practical decision support, and “worked examples” show how they should be used to tackle concrete problems such as energy use and sustainability in school buildings, overheating risk for buildings, and flood management policy (Met Office Reference Office2009).

Projects of this kind are not restricted to the United Kingdom. A larger project called weADAPT provides small-scale information based on the CMIP5 archive of GCM runs at particular station locations and using a statistical downscaling method.Footnote ⁴ The weADAPT portal describes some in-depth case studies for use and interpretation of the data. For example, one case study looks at Bagamoyo, Tanzania, and offers an answer to the question, “What do current trends in temperatures, rainfall and mean sea level, and medium- and long-term projections, suggest are the most serious climate risks that government officials in Bagamoyo district need to address?” (Besa Reference Besa2013). Cal-Adapt offers adaptation information for California, Climate Wizard for the entire United States, ClimateImpactsOnline for Germany, and the list could go on.Footnote ⁵ An ever-increasing number of private consultancies also offer similar climate change products, usually based on publicly available data.Footnote ⁶

In sum, questions about future local climate are, today, mostly answered by using state-of-the-art GCMs combined with downscaling.

3. Do the Answers Provide Actionable Information?

Its popularity notwithstanding, this method has questionable credentials as a provider of actionable local information. This conclusion can be reached from at least three different directions. We comment only briefly on the first two: detailed analysis of individual projects, and recognition of actual predictive failures. Our main focus here is a third path to the same conclusion: making explicit the implications of the most recent report from the IPCC.

Each climate projection project is based on a set of methodological assumptions, and these can be subjected to scrutiny. Frigg, Smith, and Stainforth (Reference Frigg, Smith and Stainforth2013, Reference Frigg, Smith and Stainforth2015) analyze the foundational assumptions of UKCP09 and argue that the methods used do not warrant treating model outputs as decision-relevant projections. Similar in-depth analyses could be carried out for other projects. Needless to say, we can only speculate about the outcomes of such an exercise, but given the arguments we present below, we would not expect the verdicts on other projects to differ significantly.

Regarding the success of predictions, Mearns (Reference Mearns2014) reports telling results from a study evaluating models for the purpose of predicting the future regional climate of North America. Mearns and colleagues compared a number of variations on the methodology described above, in each instance combining one of four GCMs with one of six regional climate models (simulation-based downscaling methods). They evaluated each model pair by comparing its simulation of 1980–2005 with the weather data of that period. Two such model evaluation exercises focused on the North American Monsoon in the Southwest United States and Northern Mexico, and summer precipitation in the Northeast. A surprising result was that the model pair that performed best on the North American Monsoon performed worst on Northeast precipitation, and few model pairs performed respectably on both.

The result suggests that local climate projections are more sensitive to the choice of models than previously supposed. The problem is compounded by the difficulty of identifying any factor (or set of factors) in the models that explains the differences in performance. Furthermore, past success is not necessarily a guide to future success (Reifen and Toumi Reference Reifen and Toumi2009), making it difficult to learn from a model’s success in reproducing past data. This suggests that interpreting regional model outputs as decision-relevant projections is more problematic than previously assumed. In particular, it raises doubts about using any single model, or even the same ensemble or ensemble weighting, for general-purpose national climate projections where multiple regions, variables, and timescales are of interest.

Our main argument, however, is based on the most recent IPCC report.Footnote ⁷ The report includes global mean temperature projections for 2100 with a clear assessment of their reliability. They give, for instance, a projection of 2.6°C–4.8°C of global mean temperature rise relative to the late twentieth century, supposing the RCP8.5 scenario, which describes an illustrative “possible future” in which greenhouse gas emissions continue to rise unabated. The IPCC quantifies uncertainty in a projection using standardized language referring to likelihood intervals, for example, “virtually certain” (>99%), “very likely” (>90%), or “more likely than not” (>50%) (Stocker et al. Reference Stocker, Qin, Plattner, Tignor, Allen, Boschung, Nauels, Xia, Bex and Midgley2013, Box TS.1, Technical Summary). The 2.6°C–4.8°C projection is qualified as “likely,” which means that there is a 66%–100% chance of the real-world outcome falling into that range, given the scenario.

The range 2.6°C–4.8°C is arrived at by looking at the CMIP5 ensemble of climate models: a set of state-of-the-art models that have each run the RCP8.5 scenario and supplied a value for the projected temperature change at 2100. This set of single values is then used to fit the parameters (mean and standard deviation) of a Gaussian probability distribution. The range 2.6°C–4.8°C is the interval that symmetrically spans 90% of that distribution. In the “model world,” therefore, this range is an estimate of the interval into which model runs chosen from some underlying distribution will fall 90% of the time. In IPCC terminology, this interval would be referred to as a “very likely” range.

The IPCC authors’ next step is to note that even state-of-the-art climate models share systematic biases (Knutti et al. Reference Knutti, Furrer, Tebaldi, Cermak and Meehl2010). All models are limited by the same computational constraints, they are based on the same limited set of parameterizations, and they are calibrated to reproduce features of twentieth-century climate that may be less relevant for twenty-first-century developments. For these reasons, when translating model-world findings into real-world projections, the IPCC authors downgrade the model-derived “very likely” uncertainty quantification for the range 2.6°C–4.8°C to “likely.” Thus, while there is a >90% probability that a state-of the-art model run will result in a temperature change between 2.6°C and 4.8°C, there is only a >66% chance—in the authors’ judgment—that the real climate system will produce this outcome. So the IPCC judges that there is a up to an additional 24% probability that the actual climate outcome (for the RCP8.5 scenario) will be outside the central range of the models and that we will experience something entirely different than what the models suggest. This is a clear indication that climate scientists do not believe the raw outputs of model ensembles to represent real-world probability distributions.

The reassignment of probabilities described above is based on an informal process of expert judgment: the discussions and iterations between authors during the long writing and review stages of the report. We commend the forthrightness with which the IPCC acknowledges the role of expert judgment in their assessment process. At the same time, we recognize some drawbacks to the (relatively unstructured) way in which they go about it (details below). The main conclusion we draw from the example is that the IPCC’s own results and methods contain the assertion that there is significant uncertainty about climate projections, and that this uncertainty is best quantified by appeal to expert opinion.

This conclusion should not be limited to global mean temperature. Indeed, global mean temperature is the variable in which there is (rightly) the most confidence about climate models’ ability to represent relevant processes.Footnote ⁸ So for the local-scale variables that matter to adaptation planners, we would expect to see a probability downgrade of at least the same magnitude as the one given for global mean temperature. In effect the 24% represents an imprecise lower bound to our uncertainty about other variables.Footnote ⁹ The conclusion we draw from this is that the case for quantifying uncertainty through expert judgment is even stronger when it comes to local-scale variables, and that uncertainty in local climate projections should be assessed through a suitably structured process of expert elicitation. This conclusion stands in stark contrast with current practice, where model results are often interpreted directly as a probability distribution of future real-world climate outcomes (this happens, e.g., in UKCP09).

4. Lateral Knowledge Transfer

The climate services sector faces a situation in which decisions call for scientific input, but there is severe uncertainty about the modeling on which scientists can rest their advice. Climate change adaptation, however, is not the only field that satisfies this description. Indeed, the situation is comparable in many policy-relevant areas of science and engineering, for example, toxicology, public health, and nuclear safety. To illustrate what has become a standard approach to the problem, we now have a look at a concrete case in environmental regulation. Our claim is that the example will illustrate important lessons for climate services.

Zoning plans for sites that use or manufacture large volumes of toxic substances typically include a buffer zone in which no housing development can occur. How large should the buffer zone be? A political process can determine the acceptable level of risk for residents of neighboring developments, but what distance limits the risk to this level? The Ministry of Housing, Physical Planning and Environment of the Netherlands sponsored a study to help answer this question for a handful of common industrial chemicals, including chlorine, ammonia, and hydrogen chloride (Goossens et al. Reference Goossens, Cooke, Woudenberg and Der Torn1998).

The key uncertainties concerned quantities for which there are no direct empirical observations: concentrations and exposure times that are fatal for humans. The standard approach to estimating those quantities models human toxicology using data from animal studies, treating the laboratory animal as an analog model for humans, and extrapolating concentrations and exposure times by adjusting for body weight, lung capacity, and so on. This modeling exercise delivers answers, but how much can they be trusted? Humans are not 70 kg rodents, and different species may react through different physiological pathways altogether. Moreover, extrapolation results are sensitive to debatable scaling assumptions. And of course, there are no data against which to test the resulting predictions (if there were, these would have already resolved the issue).

Furthermore, a sole reliance on animal studies fails to take advantage of other relevant sources of knowledge. For many chemicals, there are well-developed, qualitative toxicological models that describe metabolic processes, functional changes in the impacted organs, and clinical expression of these organ-function disturbances. There may also be quantitative chemical and biological understanding of their “kinetics” within the body: rates of absorption, distribution, metabolism, and elimination. This knowledge is relevant to estimating the required concentrations and exposure times and to judging the reliability of the extrapolations from animal studies, but there is no established quantitative method of feeding everything into a hopper and turning the crank for an all-things-considered best estimate.

To integrate the available evidence into actionable findings, the experimenters recruited 27 toxicologists from industry, academia, and government (about five experts per chemical). The experts reviewed the available research and provided, through a carefully structured interview, their all-things-considered subjective judgments for the required concentrations and exposure times. The interview process, or elicitation, draws on a large body of research on the formatting and sequencing of questions to reduce overconfidence and minimize the effects of cognitive biases. In this particular case, the experimenters also developed a set of seed questions (with known answers) to evaluate expert competence, and then pooled judgments mathematically, giving experts with higher marks on the seed questions greater weight, yielding a single best-estimate dose-response curve for each chemical. From this, regulatory authorities could derive the buffer zone depth that limits the risk of lethal exposure to the safety standard previously set.

The study just described is an example of structured expert elicitation (SEE), a family of methods designed to incorporate expert knowledge into risk management and strategic and policy decisions. SEE has spread from early applications in defense planning and aerospace engineering to wider use in the nuclear sector (Cooke Reference Cooke2013) and in support of regulatory and other policy decisions (Aspinall Reference Aspinall2010; Morgan Reference Morgan2014), especially environmental regulation, food safety, public health, and disaster management. There are a variety of methods; some include deliberation between experts, others (like the above) elicit judgments separately and then aggregate the results, and yet others report a full range of opinion, disagreements and all (for wide-ranging reviews of applications and discussions of the method, see Cooke and Goossens Reference Cooke and Goossens2000; Goossens et al. Reference Goossens, Cooke, Hale and Rodić-Wiersma2008; Martini and Boumans Reference Martini and Boumans2014).

5. Changing Paradigm

In section 3 we argued that there are severe uncertainties in regional climate projection, and that these uncertainties are best assessed via expert judgment. Section 4 looked at an analogous case of environmental planning where SEE is used to assess uncertainties and synthesize different sources of knowledge into actionable results. Putting these two strands of argument together leads to our core suggestion: that all local climate change projections intended to support decision-making should be produced through a process of SEE.

To be clear, SEE does not replace modeling exercises on this approach. Rather, the suggestion is to quit regarding downscaled GCM outputs as the only (or the privileged) source of information and to use model outputs alongside other sources of information. Experts have much to draw on, including physical understanding of large-scale processes such as the greenhouse effect, energy conservation principles (sometimes violated by model behavior), historical data sets, projections based on simple models and statistical methods, and reconstructions of climate change in the distant past. Knowledge about model development is also essential, including understanding of the effects of grid-scale resolution, of including or excluding particular processes (or including them only through ad hoc parameterization), of mathematical idealizations of conceptual relationships, and of the numerical approximations required to process the mathematical idealizations. Simulations of the recent past have also been statistically compared to corresponding observations and plausible reasons for inaccuracies debated. The relative strengths and weaknesses of alternative model formulations are continually debated as well. All of this knowledge and understanding (and more) can inform expert judgments about local-scale climate projections.

There are many variations in SEE methodology and a large literature on its theory and application. If one accepts our proposal, many questions remain about how best to implement SEE for climate change projections, including how to pick the experts, what constitutes a reasonable quorum of relevant experts, and how best to incentivize participation and address conflicts of interest. Other questions concern the mathematical formalism used to encode an expert’s judgments (e.g., precise vs. imprecise probability) and whether the individual judgments are aggregated (Genest and Zidek Reference Genest and Zidek1986) or presented as a group. Cooke (Reference Cooke1991, Reference Cooke, Steenbergen, van Gelder, Miraglia and Vrouwenvelder2014) and Aspinall (Reference Aspinall2010), for example, promote a version of the former, while Morgan (Reference Morgan2014) generally prefers not to aggregate. We believe that no one method is right or wrong in general. The challenge is finding the right tool for the job, and this is a task for future research.

Irrespective of the specifics, there are a few general issues we wish to address. One knee-jerk reaction to our proposal may be that results produced with SEE are too subjective to inform evidence-based policy. This is wrong. All climate modeling exercises involve subjective decisions that can significantly influence the outcome, from the selection of models to include in an ensemble, to the range of parameter perturbations and initial conditions explored in a sensitivity analysis, to the method of downscaling (not to mention the decisions made in constructing a GCM in the first place). By asking experts to assess and communicate the consequences of these decisions as best as they can, considering all available information, the proposed approach is more transparent about the subjectivity, but it is not more subjective. The alternative to SEE is not objective calculation; the alternatives are unstructured expert judgment (as in the IPCC report) or unstructured nonexpert judgment (when model-world results are communicated to policymakers ill-equipped to assess their real-world importance). Reliable objective calculation of probabilities based on nonlinear model output has been argued elsewhere to be infeasible (Frigg et al. Reference Frigg, Bradley, Du and Smith2014). SEE does not guarantee trustworthiness, but it can compensate for (some of) the (known) inadequacies of climate models. Echoing a notorious remark of Churchill’s about democracy, we submit that SEE is the worst form of science advising, except for all those other forms that have been tried from time to time.

Others seem to have reached a similar conclusion and called for the IPCC assessment process to incorporate SEE (Reilly et al. Reference Reilly, Stone, Forest, Webster, Jacoby and Prinn2001; Oppenheimer et al. Reference Oppenheimer, O’Neill, Webster and Agrawala2007; Shapiro et al. Reference Shapiro2010; Yohe and Oppenheimer Reference Yohe and Oppenheimer2011). IPCC author teams have so far made their collective judgments through an informal procedure that does little to address cognitive biases and adverse group dynamics such as deference to high-status participants. And despite the recent emphasis on “traceable accounts,” the expert judgment process is still largely opaque to outsiders. Standard SEE practice requires publishing the interview protocols, raw elicitation results (anonymized), and any procedures for facilitated deliberation and/or mathematical aggregation along with the final results. We support the incorporation of SEE into the IPCC process and recommend that the same ought to be done in the case of local climate projections.

Even those who agree in principle with our conclusion might worry that SEE will offer no new insights in the case of local climate change projections because the outcome is in effect a foregone conclusion. We know, the objection continues, that models underestimate uncertainty and so SEE will simply downgrade all probability distributions, the only question being by how much. The worry is misguided. First, even where the need to downgrade is clear from the start, the degree is not, and the degree matters. Second, this is not the only thing that can happen. In some contexts (e.g., weather forecasting) models can overestimate uncertainty and experts will sharpen the probability distribution. In other cases uncertainties are asymmetrical and can lead to a skewing rather than a widening of the distribution, and in yet other cases the verdict could be that the distribution is systematically biased and needs to be “moved.” Moreover, SEE need not enter the process only at the end of resource-intensive modeling projects, but should also inform their aims and methodology earlier on. This could constrain the choice of variables, timescales, and spatial scales for which it is worth trying to provide decision-relevant model information. Undoubtedly, there are some such limits beyond which projections are simply not plausible; arguably, many climate services products already exceed those limits. If experts judge large swaths of model output useless or misinformative for decision-making, something has gone wrong at an earlier stage.

6. Conclusion

For the purpose of local climate change decision support, direct model outputs can be misleading. The IPCC’s published expert judgment about modeled global mean temperature projections implies that probabilities derived directly from model output on smaller-scale variables are inappropriate inputs to decision-making. Better models and faster computers cannot by themselves make the problem go away, and provision of local climate change projections should instead be reconceptualized as a problem of eliciting expert judgment based on a number of sources including but not limited to model output. Many open questions remain about how best to implement structured expert elicitation for local climate projections.