1. Introduction

In sciences dealing with complex systems, it is common to encounter a range of different models representing the same system. Such models might disagree deeply over the structural relations in the system or in shallower ways over the values of parameters or initial conditions. Since it is often impossible to decide between these models using available evidence, scientists work with whole collections—or “ensembles”—of models. Prominent examples are the CMIP5 ensemble of global climate models and ensembles of hurricane models for the North Atlantic. In some cases, model ensembles indicate disagreements among scientists; in other cases, they reflect agreed latitude in model construction. In either case the ensemble represents (at least partially) scientific uncertainty about the target system.

How should policy makers use model ensembles in making decisions, and how should these decisions reflect the scientific uncertainty associated with them? Mainstream decision methods such as expected utility theory assume that all relevant uncertainty is captured by a single probability measure and so do not (without supplementation at least) provide an adequate answer to this question. Recent decades have seen the development of numerous decision rules for situations in which decision makers face what is known as “ambiguity,” when precise probabilistic estimates of all decision-relevant quantities are unavailable (for a survey, see Kriegler Reference Kriegler, Gert de Cooman and Zaffalon2007; Gilboa and Marinacci Reference Gilboa, Marinacci, Acemoglu, Arellano and Dekel2013; Heal and Milner Reference Heal and Milner2014). There is also a nascent decision-theoretic literature on model uncertainty (see esp. Marinacci Reference Marinacci2015).

This article aims to advance this discussion by reworking a recently developed decision theory called the confidence approach (Hill Reference Hill2013; Bradley Reference Bradley2017) to tackle inputs from model ensembles. The aim is to demonstrate its fruitfulness; in particular, the benefits of its structured approach to managing ensemble uncertainty. We illustrate the approach by applying it to a scenario involving hurricane models used for insurance pricing. The lessons from this case are applicable to many policy-making scenarios. The framing of our article is crucial: we work from the decision maker’s perspective. They are typically nonexperts. The challenge is to design a normatively appealing decision procedure for their use, which is nevertheless sensitive to the state of the relevant scientific knowledge.

In section 2 we introduce hurricane modeling and our insurance case study. Section 3 argues that current practice—using a weighted average of the hurricane models’ outputs—is problematic, and it would be desirable to have an alternative approach. We introduce such an approach in section 4 and apply it in a simple form in section 5. Section 6 considers various ways of constructing the main ingredient in our approach: a nested family of sets of probabilities. Section 7 concludes with a program for developing the approach.

2. Hurricane Insurance Decision-Making

Our case study is drawn from a research collaboration with scientists working for a large insurance company. The insurer is based (and regulated) in the United Kingdom but operates worldwide. As part of its US property insurance, the company offers cover for damage resulting from hurricanes. This practice relies on estimating the probability of the insured-against events (i.e., destructive hurricanes) and the damage they cause. It is often not economically efficient for insurers to invest in the expertise and capabilities required to do this, and so they buy predictive models from commercial modeling companies. These companies employ teams of environmental scientists, statisticians, and programmers to construct simulation models to determine the probability of hurricane “landfalls” along the US Atlantic coast.Footnote ¹

The modeling firms face a problem: there is significant uncertainty in hurricane modeling, derived in part from disagreements about the underlying science. The result is that there are multiple models representing the same system. The FCHLPM (2007) carried out an assessment of the modeling industry using an ensemble of 972 models (Guin Reference Guin2010). Risk Management Solutions (RMS), a leading modeling firm, uses an ensemble of 13 models to generate the medium-term rate (MTR), their preferred prediction of hurricane landfall frequency (Sabbatelli and Waters Reference Sabbatelli and Waters2015).

Any company selling models to insurers must decide how to navigate this landscape. Which model(s) should they build as part of their offering? Offering more than one model better represents the landscape, but presenting insurers with a collection of models creates a further problem for them: How does one decide when faced with not one model probability but 13 or 972? The most common solution when working with ensembles is to average the outputs from each model, and one task of this article is to lay bare the limitations of this process. To add specificity to the problem, and show how it arises in an important real-world application, we now give a brief overview of the RMS model ensemble. We chose RMS because they are a leading hurricane modeling firm and because they are open about their use of an ensemble of models.

A catastrophe model for insurance works in four stages, covering (1) the hazard, in this case a hurricane; (2) the physical damage it creates, which requires modeling the vulnerability of buildings and infrastructure to wind, water, and so on; (3) calculating insurer exposure, by looking at insurance policy terms; and (4) financial modeling of the insured losses that result. We will consider only the first component. Shome et al. (Reference Shome, Rahnama, Jewson, Wilson and Michel2018, 37) provide a classification of RMS hurricane models for the North Atlantic, with model names reflecting the sometimes competing choices made in the modeling process. Briefly reviewing these types effectively conveys the diversity of the models in this ensemble.Footnote ²

• • “Direct” models use historic hurricane landfall counts as input and make a landfall prediction.

• • “Indirect” models use storm formation data from the Atlantic basin to make a prediction of hurricane activity in the basin and then convert that prediction into a landfall prediction using the estimated proportion of basin storms that finally make landfall along the US coastline (Jewson et al. Reference Jewson, Bellone, Laepple, Nzerem, Shree, Lonfat, Shay, Penzer, Coughlin, Elsner and Jagger2009, 12).

• • “Indo-Pacific” models include the impact of sea surface temperatures (SSTs) in the Indian and Pacific oceans on hurricane formation through their effect on wind shear in the Atlantic basin.

• • “Shift” models identify periods of higher or lower than average hurricane activity or SSTs in the historic data. This is due to the Atlantic multidecadal oscillation (AMO), and probabilities of transitions from high- to low-activity periods are estimated using historic data on tree ring sizes, a method due to Enfield and Cid-Serrano (Reference Enfield and Cid-Serrano2006) (Jewson et al. Reference Jewson, Bellone, Laepple, Nzerem, Shree, Lonfat, Shay, Penzer, Coughlin, Elsner and Jagger2009, 14).

• • “Active baseline” models, a category mutually exclusive with “shift,” reflect an alternate hypothesis on the AMO: the low-activity period in the 1970s and 1980s was due to SST cooling induced by high atmospheric aerosol content, primarily volcanic aerosol (Booth et al. Reference Booth, Dunstone, Halloran, Andrews and Bellouin2012). If correct, SSTs will not revert to a cool phase in the future, and one should not apply a probability of shifting back to a low-activity hurricane generation phase. These “active baseline” models therefore do not include the Enfield and Cid-Serrano probabilities in their forecasts and subsequently forecast higher landfall rates than the shift models (Sabbatelli Reference Sabbatelli2017).

The ensemble is built up by taking combinations of the above methods. It starts with two models: direct and indirect. By adding models with Indo-Pacific SSTs, we get to four. We then add shift and active baseline variants of all four—leading to 12 models. The thirteenth is a long-term rate (LTR) model, included for comparison. RMS’s LTR is a statistical model based on historical landfall and basin storm data, and it models hurricane frequency as constant in time (Shome et al. Reference Shome, Rahnama, Jewson, Wilson and Michel2018, 33). RMS’s certification as a modeler for the American market (by the FCHLPM) is granted on the basis of their LTR model, and so, although RMS advertises the MTR ensemble average as providing their state-of-the-art view of hurricane risk, the LTR is often used as a reference view.

This list shows that the models included in the ensemble are not merely variants of the same model (obtained, possibly, by varying parameter values). The models fall into groups that are genuinely different and in some cases mutually incompatible. How should an insurance company use this ensemble to determine the price of its policies?

3. Averaging and Its Limitations

The problem that scientists face is how to extract the information contained in the ensemble and make it available to users. This is a thorny issue because it is far from clear how to interpret the (often conflicting) outputs from different models. A popular method is to calculate a weighted average of all model outputs and use this average for decision-making (Clemen [Reference Clemen1989] and Armstrong [Reference Armstrong2001] provide reviews from economics and management).Footnote ³ This has the advantage of allowing standard decision methods, which require a single probability as an input, to be applied.

Averaging works in some cases, and where it works one should use it. However, it does not work in all cases, and hurricane insurance is such a case (Philp et al. Reference Philp, Tom Sabbatelli, Wilson, Collins and Walsh2019). The most significant problem from our perspective is that such decision methods discard useful information about the state of scientific uncertainty. As Morgan (Reference Morgan2014) points out, averaging focuses attention on the mean projection only. However, the spread of results is itself important information. First, when we are interested in extreme events like major hurricanes, then we are explicitly concerned with the shape of the distribution and not just the mean. Second, the spread tells us something about the state of our knowledge about a question. To the degree that there is spread, it reflects scientific uncertainty about the system and our lack of precision in modeling its relevant features. This by itself is valuable information that the decision maker might want to use.

Although averaging does not preclude communication of the spread, in the expected utility paradigm it is unclear how this information is to be used, since the expected utility of an action or policy depends only on the decision maker’s probability for relevant contingencies (in addition of course to the utility of outcomes). Evidently the same probability can be obtained by averaging over very different sets of candidate probability functions. Consequently, expected utility maximization precludes making decisions in a way that is sensitive to the state of scientific understanding as expressed by the spread in the ensemble projections. But in decisions with high stakes it is reasonable to seek to calibrate one’s choices to the level of uncertainty contained in the scientific projections that one is drawing on (see Bradley Reference Bradley2017; Hill Reference Hill2019). The procedure we develop in sections 4 and 5 overcomes this limitation by making structured use of ensemble spread.

Additional problems undermine decision-maker confidence in averaging procedures. Weights for averaging are typically constructed by scoring models on skill. This often involves hindcasting: reproducing a piece of the historical record (which the model has not “seen” before). This method faces the problem that the historical data set used to score these models is small, as large hurricanes are infrequent. HURDAT2, the standard database for hurricanes hitting the Atlantic coast of the United States, is moderate in size, with ∼300 storms to date and only one-third of those counting as “major hurricanes.” If we split the data set by region the numbers drop precipitously.Footnote ⁴ As Shome et al. (Reference Shome, Rahnama, Jewson, Wilson and Michel2018) point out, actuaries judge that there are insufficient data to form a reliable statistical model to predict future events. The data are also insufficient to meet regulatory requirements. The national regulator in the United Kingdom requires insurance companies to design their portfolio in a way that they go bust at most once in 200 years. Even on a generous reading there are at most 120 years of usable hurricane frequency data. But 120 years of data do not provide a reliable understanding of the tail events and the shape of the distribution at those longer return periods. Shome et al. (Reference Shome, Rahnama, Jewson, Wilson and Michel2018) cite this paucity of data as a reason for using quasi-physical simulation models, whereby modelers create “statistical storms” to expand and “fill in” the data set. This process, however, relies on the (scant) historical evidence, and so it cannot remove the problem of restricted evidence.

A further problem concerns the choice of a scoring rule (SR).Footnote ⁵ The problem is that the weights are set by the rule, and so the average value is sensitive to this choice. But the range of SRs on offer is so diverse that almost any reasonable answer could be selected by one of them (Stainforth, Allen, et al. Reference Stainforth, Allen, Tredger and Smith2007, 2155). The Australian Bureau of Meteorology maintains a website on probabilistic forecast verification techniques, which covers more than 50 SRs, visualization techniques, and analytical approaches to measure the success of probabilistic forecasts. The categories are nonexclusive, and for a given problem there may be multiple appropriate rules with different features (Australian Bureau of Meteorology 2017).Footnote ⁶ So, experts often disagree over which rules to use when, and individual experts may endorse more than one rule as appropriate for a given situation. The debate over these rules covers both technical matters (e.g., which is better suited to rare event predictions) and values (about what counts as a “good” prediction). Decision makers ought ideally to select a rule embodying their values (i.e., that corresponds to what they regard as important), and yet the technical complexity of the subject means that decision makers are often not in a position to participate meaningfully in a choice of rule. Crucially, the problem of choosing a SR is very similar to the original problem of choosing/formulating an answer from the range present in the model ensemble. Any method of choosing a SR requires deciding between disagreeing experts, and so we may well ask why we do not simply apply these same considerations to the “first-order” problem of model disagreement.Footnote ⁷

We conclude that averages are a reliable guide to action only when uncertainty is small (and known to be so), enough data are available for meaningful scoring, and different SRs produce similar results. There may be situations that satisfy these requirements, but hurricane modeling is not one of them. Averaging is therefore not an optimal procedure to make decisions on hurricane insurance. Practitioners feel similarly: insurers have expressed some of these concerns to us, and in practice they will “factor in” their dissatisfaction by, for example, multiplying the average event probabilities by some $\alpha >1$. They have, however, no principled way of determining the value of α, which is typically set in an ad hoc manner by managers removed from the detail of the modeling. The nature of these problems is such that they are unlikely to be resolved by tweaks to the aggregation methodology; a completely new approach is needed.

4. The Confidence Approach

In this section we outline the theoretical basis for our alternate approach to using results from the model ensemble, and in the next we apply this approach to a simple insurance problem. This work applies a relatively new theory of decision-making under ambiguity, which we call the confidence approach, developed by Brian Hill (Reference Hill2013, Reference Hill2016).

The confidence approach makes use of “imprecise probabilities,” sets of probability functions that generate sets of values (typically, intervals) for each event, as a way of capturing features of agents’ uncertainty—specifically the empirical ambiguity (Bradley [Reference Bradley and Zalta2019] presents an overview). Imprecise probabilities support decision rules that take such sets as inputs, as opposed to the single probability function required by the classical decision theory of Savage. There are many such rules on offer, but they all face similar challenges.

First, how does one determine the relevant set of probability functions or, equivalently, the extent of the uncertainty the decision maker faces? The question has significant implications for applications of these decision rules. When the set is large, many of them lead to levels of caution that could be regarded as excessive (e.g., such that much of today’s hurricane insurance business would not be written). But when it is small, it may not capture all uncertainty relevant to the decision maker. Despite this, imprecise decision theories often fail to separate the uncertainty agents face from their attitude to it, for instance, by taking the set of probabilities representing the agent’s uncertainty to be the one with respect to which they maximize the quantity that the decision theory in question takes to be significant, for example, minimum expected utility (Gilboa and Marinacci Reference Gilboa, Marinacci, Acemoglu, Arellano and Dekel2013; Hill Reference Hill2019).

Second, these theories, like expected utility theory, do not allow for the differing importance of various decisions to affect how a decision maker responds to uncertainty (Hill Reference Hill2013), in part because of the aforementioned failure to separate the uncertainty from the response to it. But intuitively, how much it matters that we lack scientific certainty on some question, and how cautious we want to be in making choices as a result, should depend on how much is at stake for us in these choices. The confidence approach mitigates against these concerns, while drawing on the benefits of using imprecise probabilities.

We start with a high-level description of the approach, using a trivial but intuitive example; along the way, we note how policy decisions will differ. Here is the trivial case: suppose you are deciding whether to place a bet on your favorite contestant, Baga Chipz, winning a drag competition. To bet you pay £50 up front. If she wins you are paid back your £50 and receive another £50; if she does not win you lose your £50. So, this bet has a positive expected monetary return whenever the probability of her winning is strictly greater than 0.5. We will show how the confidence approach determines whether this is a fair or advantageous bet.

First, we represent your beliefs with a family of nested sets of probabilities. Each set represents a claim that you accept about the relevant probability, while the nesting captures the logical relationship between these claims. In our example, these claims could range from the very imprecise (indeed trivial) claim that “the probability of Baga Chipz winning is between 0 and 1” to the very precise “the probability of Baga Chipz winning is 0.42.” Figure 1b shows such a nested family schematically.

Figure 1.

Confidence approach: a, set of point estimates of probability; b, nested family of sets; c, confidence levels; d, stakes and cautiousness select a level.

Next, we consider your confidence in each of these claims. Confidence should reflect the “weight of evidence” supporting a claim—a term, coined by Peirce (Reference Peirce1878) and popularized by Keynes (Reference Keynes1921, 78), describing the property of evidence that makes us more sure of our probabilistic judgment, even when the judgment itself may remain constant. “Confidence” is thus a (second-order) attitude toward a (first-order) claim, reflecting an evaluation of the state of knowledge underpinning it. It has the following logical structure: one cannot be more confident in more precise claims. So, you cannot be more confident that the probability of Baga’s win is 0.42 than you are that it is in the interval [0.3, 0.5]. This is a simple consequence of the former’s inclusion in the latter.

In principle we could have any number of sets in the nested family. For practical purposes, it is useful to coarse grain down to “levels of confidence” (e.g., low, medium, high). This involves making a judgment about which members of the family of nested sets are equivalent in terms of coarse-grained confidence and grouping these together as in figure 1c. The idea is that there is no decision-relevant difference between the grouped sets. As narrower sets exclude more possibilities, we therefore have a pragmatic motivation to work with only the most precise interval in each level—in figure 1c this corresponds to the bottom interval in each level. In our example, this means restricting your consideration to three probability intervals, say 0.42, [0.3, 0.5], and [0.2, 0.6] reflecting claims you endorse with low, medium, and high confidence.Footnote ⁸ Every claim wider than 0.42 but narrower than [0.3, 0.5] is considered confidence equivalent to 0.42 (i.e., low confidence), and so forth. Put another way: if you saw decision-relevant differences between the intermediate intervals, you would further fine grain.

How we coarse grain will partly be a matter of convention, but it is motivated by an important consideration: connecting the relative ranking of a particular decision’s family of sets to a background standard of confidence. An ordinal ranking cannot say anything about how much confidence we have in any claim; it can only tell us how that claim is related to other claims. If the outcome of my bet is that I will be shot if I lose, I want to be very confident about my probability estimate; “very” reflecting the absolute importance of the decision and not just that I want more confidence in it than other estimates of drag queen odds. A decision maker can do this by developing a sense of what counts as “enough evidence to warrant high confidence” and applying that standard across decision problems through the labels applied in this coarse-graining step. If there is poor evidence supporting all claims in the family, perhaps the top coarse-grained level should only be considered “medium confidence.”

Coarse graining to levels pegged to such a background standard of evidence allows our notion of “confidence” to decouple from the situation-specific information in front of the agent. This allows us to make decisions in a way that reflects their importance, relative to other decisions we make. To accomplish this systematically, the confidence approach adds two more features to standard decision theory.

The first is the “stakes” of the decision: the agent’s assessment of how important it is. The key feature of stakes is that it partially orders decisions (i.e., that we can compare two decisions and rank them in terms of importance). How this is done can vary, but for formal simplicity, we think of stakes as a number on a 0–1 scale.Footnote ⁹ The term “stakes” is chosen to imply that it should be a feature of the potential outcomes, as in our betting example where you stand to lose (or win) £50. There is a wide range of potential functions of these outcomes that could measure your stakes; Hill (Reference Hill2016) discusses their differences. For simplicity let us take the stakes to be a function purely of the worst possible outcome—losing £50. Assessing the relative importance of a decision in which you stand to lose £50 involves reflecting on other decisions you make, the value of £50 to you, and so forth. For the moment let us simply assume you regard this as a moderately important decision and assign $s=0.5$.

Second, agents have as a feature of their psychology a function called “cautiousness,” which determines how much confidence they require to decide, given the stakes. Cautiousness thus takes the stakes as input and outputs the coarse-grained level of confidence required to make the decision; figure 1d illustrates, with cautiousness denoted D. A different degree of caution, colloquially speaking, is represented by a different cautiousness function. “More cautiousness” means that more of the 0–1 stakes range is mapped to a high confidence level. In our simple example, the question to ask is, “How much confidence do you need in order to make moderately important decisions, with stakes around 0.5?”

Cautiousness represents an attitude to ambiguity on the part of the decision maker. It is therefore subjective and will need to be elicited.Footnote ¹⁰ Let us suppose that after such an elicitation we determine that you require medium confidence for decisions of moderate importance—so that, for you, $D\left(0.5\right)=\mathrm{medium}$. You can now select a probability interval: the narrowest interval in the level of confidence that your cautiousness demands, given the stakes of the decision. In the case of your drag queen bet, that interval is [0.3, 0.5].

In a policy-making context, we may wish to do more than elicit the attitude of the individual decision maker and look to public and political debate to settle what level is appropriate. In our insurance setting, it will be natural for this to represent the firm’s ambiguity attitude, a policy on the conditions under which it is allowable to sell insurance.

We now reconnect with imprecise decision theory. The probability interval selected by the confidence procedure can be used with any imprecise decision rule. For the sake of specificity, we will illustrate with one popular rule: maxmin expected utility (MMEU). This rule says that act f is preferred to act g if and only if the minimum expected utility of f, with respect to the set of probability functions, is greater than the minimum expected utility of g with respect to the same set (Gilboa and Schmeidler Reference Gilboa and Schmeidler1989). That is, it recommends choosing cautiously, by acting to guarantee the best outcome if things turn out to be the worst way they could be, from your perspective.

One benefit of the confidence approach is that we have two “levers” of ambiguity attitude: the cautiousness function and the decision rule. Although MMEU is highly ambiguity averse, this aversion can be attenuated by the choice of cautiousness function—specifically, by choosing a function that recommends moderate levels of confidence for a wide range of stakes. (The opposite choice could boost MMEU’s ambiguity aversion.) Decision makers who are not completely ambiguity averse can thus still use MMEU, for instance, because it is a very simple rule to implement. We adopt it for precisely this reason.

Applying MMEU leads to bad news for your bet on Baga Chipz. Things turn out worst if the probability is at the low end of your medium-confidence interval, where you expect to lose money on this bet because the probability is below the priced value of 0.5. You therefore should not place the bet. In a single simple decision like this, the confidence approach may seem at once complex and permissive. In the next section, this heavy machinery will show its value.

5. Confidence, Models, and Insurance

We now apply the confidence approach to our case: making an insurance pricing decision, using input from an ensemble of scientific models. Our example simplifies some details of actual insurance pricing by considering a very simple portfolio with only one contract. This does not influence the philosophical points we wish to make about the treatment of outputs from a model ensemble, and the approach can be applied to more complex portfolios.

5.2. The Scientific Input and Aggregation

Consider a simplified scenario that captures the salient features of the RMS ensemble from section 3. Our scientific modelers construct 10 models, m ₁, …, m ₁₀, which encode different views about, for example, how hurricanes move across the Atlantic and how the factors influencing their generation will turn out in 2021. As the details will not matter here, we will not describe how these models work except that they generate p(E) and that one of them—m ₈—was built for a different region but works for Florida. Table 1 shows 10 numbers that we will use as our model outputs. The “standard” approach would be to score these models on their predictive skill, as described in section 3. Suppose that we have done this, using a popular SR R. The normalized scores and outputs for p(E) are shown in table 1. Using these normalized scores, we can calculate the ensemble average: $p^A\left(E\right)=0.0072$. This is what is standardly used for decision-making, especially in cases in which it has higher “skill,” according to rule R, than even the best model.

Table 1.

Model Outputs for Toy Example

	m 1	m 2	m 3	m 4	m 5	m 6	m 7	m 8	m 9	m 10
p(E)	.0070	.0071	.0068	.0074	.0076	.0061	.0083	.0086	.0091	.0092
Weight (%)	23.7	20.7	15.8	11.6	11.5	7.3	3.2	3.0	1.7	1.6

Let us consider how the standard approach would price the contract. Using p^A(E), the expected damage is ${⟨d⟩}^A=p^A\left(E\right)d\left(E\right)=0.0072\times \mathrm{-100,000}=-\$720$. The required holdings are $H=\$\mathrm{100,000}$. If we take the cost of capital to be $y=0.05$, then we have minimum price ${\pi }^A>\mathrm{yH}-{⟨d⟩}^A=\$\mathrm{5,720}$.

As we noted above, skeptical underwriters often introduce an inflationary factor for safety. This may take the form of crudely doubling the aggregate probability p^A(E). Going through the calculations with that probability, we get the “safety” price ${\pi }^S>\$\mathrm{6,440}$, which is much higher than the “technical” price π^A. This is clearly ad hoc, but decision makers reported making such moves to us, in the absence of a structured method for dealing with ambiguity.

5.3. Pricing with Confidence

To apply the confidence approach we must formulate claims about p(E). Here is a simple method of doing so; we discuss more elaborate alternatives in the next section. We will assume that SR R has reliably identified the best model, m ₁, and build intervals around it. Our “lowest,” most specific claim is that $p\left(E\right)=0.007$. We form wider intervals by including predictions in order: $I_1=0.007$, $I_2=\left[\mathrm{0.007,\; 0.0071}\right]$, …, $I_{10}=\left[\mathrm{0.0061,\; 0.0092}\right]$.

We now coarse grain and form levels. As our insurer has made no decisions of this sort before, it does not have an ongoing assessment of evidential quality and so constructs the confidence levels using only the model outputs. For simplicity we can suppose that these 10 models represent all the major relevant scientific views. As a first pass, the insurer could decide to coarse grain by model support: using up to 4 models will yield low confidence, 5–7 medium, and 8–10 high. However, in consultation with the modelers the insurer notes that this would result in the narrowest interval in the high level being I ₈. But the modelers doubt m ₈, as it was built for a different region. This is a scientific reason to avoid having a decision depend on it, and so the insurer revises the coarse graining so that low involves models 1–4; medium, 5–8; and high, 9 and 10:Footnote ¹¹

$\begin{array}{c}ℒ=\left\{0.007,\left[\mathrm{0.007,\; 0.0071}\right],\left[\mathrm{0.0068,\; 0.0071}\right],\left[\mathrm{0.0068,\; 0.0074}\right]\right\}\\ ℳ=\left\{\left[\mathrm{0.0068,\; 0.0076}\right],\left[\mathrm{0.0061,\; 0.0076}\right],\left[\mathrm{0.0061,\; 0.0083}\right],\left[\mathrm{0.0061,\; 0.0086}\right]\right\}\\ \mathscr{H}=\left\{\left[\mathrm{0.0061,\; 0.0091}\right],\left[\mathrm{0.0061,\; 0.0092}\right]\right\}\mathrm{.}\end{array}$

The agent now regards the intervals within each level as being of equivalent confidence and so will make decisions using the narrowest interval available at a level. Figures 1a–1c illustrate the process just described.

How can we characterize the stakes facing this insurer? This contract will constitute its whole business, and so the risk of ruin is high. Still, no one’s life is at stake, and there is no impact on anything outside of the realm of this decision (no other business that might be taken down). So, the insurer concludes that its stakes are moderately high, $s=0.75$. Next, we describe the insurer’s ambiguity attitude (which corresponds to cautiousness). As insurance of natural catastrophes involves significant ambiguity, it seems reasonable to assume that this insurer is not overly ambiguity averse. Here is one cautiousness function that exhibits only moderate ambiguity aversion: stakes below 0.6 require low confidence, 0.6–0.9 require medium, and above 0.9, high. Applying this to the example outlined above, we see that the decision maker resolves to use level ℳ and its narrowest interval $I_5=\left[0.0068,0.0076\right]$.

We can now apply our chosen decision rule. In insurance, higher probabilities represent worse payoffs for the insurer, and so MMEU selects the highest probability in the range: $p^C\left(E\right)=0.0076$. We therefore have the following expected damages ${⟨d⟩}^C=0.0076\times \mathrm{-100,000}=-\$760$. The holdings are exactly as before. We therefore get ${\pi }^C>\mathrm{yH}-{⟨d⟩}^C=\$\mathrm{5,760}$.

So, the confidence model recommends pricing the same contract (at least) 0.7% higher than the averaging approach. Crucially, π^C is 10.5% below the “safety” price π^S. This is a very large difference when pricing large insurance contracts—representing money lost to undue, and unjustified, caution. The difference between π^C and π^A, although smaller, is still significant, but note also that its size is an artifact of our toy example: selling only one contract imposes exceedingly high capital costs. If we sold 20, and spread the capital cost evenly among them so that $h=H/20$, the prices would be

${\pi }_{20}^A>\mathrm{yh}-{⟨d⟩}^A=\$970,{\pi }_{20}^S>\$\mathrm{1,690},{\pi }_{20}^C>\$\mathrm{1,010},$

in which case the confidence price is 4.1% higher than the aggregate price and 40% below the “safety” price. Our structured approach to uncertainty classifies a number of sales as imprudent that would go ahead under the aggregate price but far fewer than are excluded by the ad hoc safety price. This shows that “rule of thumb” uncertainty management (i.e., making ad hoc adjustments to the average probability) is not only baseless; it is not cost effective. Note also that the stakes and cautiousness functions, while subjective, are stable attitudes of the decision maker that persist across decisions. The confidence approach therefore ensures consistency across sets of decisions in a manner that ad hoc uncertainty management cannot.

Let us pause here to consider a potential objection: while the ad hoc safety price is obviously unjustified, in cases in which the average has greater “skill” than the best model, the decision maker should simply rely on the average. In reply we say: decision-making using the average is overly reliant on the scoring process, which we noted in section 3 suffers from a paucity of data and a degree of arbitrariness in the choice of a SR. The confidence approach mitigates these worries by introducing a flexible degree of robustness. As we discuss below, there are different ways one might construct the nested sets, and it may well be reasonable to center them on an average. But the key to the confidence approach is that, as the stakes increase, one uses wider intervals around the center, thereby guarding against concerns about how that center was identified. So even when the average has greater “skill” than even the best individual model, one should prefer the confidence approach to using simply the average.

6. Methods for Constructing Nested Sets

The simple implementation of the confidence approach to model outputs described above is by no means the only option. Our view is that there is no “one size fits all” method for the construction of nested sets, given the diversity of target systems and modeling endeavors. Instead the set-construction method will depend on the specifics of the ensemble. In this section we make a start on a “toolbox” for model-based decisions: outlining several potential set-construction methods and identifying what each requires of the ensemble and when it is likely to perform well.

In section 5 we constructed our intervals by starting with the best model, m ₁, and including the next best (according to rule R) each time. But we could also describe what we did as starting with m ₁ and including the next closest model each time, with respect to the Euclidean distance between outputs. In the toy example these procedures generate the same result, and we did not specify which we were following at the time. But in general, we may not have a reliable rule (sec. 3), and these two orderings may diverge. We now outline a decision tree for how to construct a nested hierarchy in the general case.

The first question is: Can you identify a model output or outputs to act as a center for the nesting? (We consider different ways you might successfully accomplish this in sec. 6.1) If yes, then: Do you also have a reliable ordering of model outputs? If yes, then we recommend forming the nesting in line with this ordering (sec. 6.1.1). But note that in section 3 we outlined various problems with scoring models in the hurricane case—at this point in the decision tree, our case likely yields a no. In this case, we recommend forming the nesting by including models in distance order (sec. 6.1.2).

If you cannot identify a center model (sec. 6.2), then we ask: Can you defend the use of one of a suite of statistical methods that construct a center (and a nesting to go with it)? In section 6.2.1 we consider centering around the average using either a partial ordering or distance, and in section 6.2.2 we discuss a method that uses central intervals of a Gaussian distribution to define confidence levels. If you cannot do either of these, then you are in our worst-case scenario and must use only the widest envelope of your model outputs (sec. 6.2.3).

7. Conclusion

The standard approach to working with model ensembles is beset with problems. Aggregation relies on a nonunique predictive test and SR, whose choice is difficult to motivate to decision makers. It requires significant data, which may not be available. Crucially, it misrepresents the state of scientific knowledge to decision makers by producing a single value for p(E), without reflecting the underlying uncertainty. This is compounded by decision makers not knowing what to do with uncertainty information, were it to be given to them.

In the confidence approach we are as explicit as possible about uncertainty at every stage. Decision makers are presented with a variety of options: different sets of probabilities, each with an attached cost to their use in the form of the confidence it can support. One can always demand more specificity, but it is clear what is given up when doing so. There is a natural, and we think valuable, link between the importance of the decision, the confidence that importance demands, and the formulation of decision input.

In our insurance case study the benefits are marked. Current practice tries to build the “missing” uncertainty back in, in a costly and ad hoc manner. The confidence approach, however, allows decision makers to respond to uncertainty in a principled but flexible manner. In practical terms this would be done by replacing an opaque “technical” process (aggregating model outputs) with a structured process of value elicitation, in order to formulate the stakes and cautiousness functions.

The major research question facing this approach is how to construct the hierarchy of nested sets. In this article we have begun a partial taxonomy of methods for set construction and the conditions under which they are applicable. Careful work is required to determine where specific cases lie, and there are surely additional methods not covered here. One obvious candidate, much discussed in the climate literature, is expert elicitation, which could also be used to construct the confidence levels.

The approach outlined here is not restricted to insurance or hurricane modeling. In principle, the approach can be expanded to cover any decision support using a model ensemble—including nonprobabilistic outputs. Doing so would better reflect uncertainty and strike a balance between cautious decision-making in the face of uncertainty and avoiding complete decision paralysis.