11.1 Introduction
Scientific evidence is fundamental to solving a suite of real-world issues and research is crucial in informing solutions to pressing issues such as climate change, food security, evolved resistance and land management (Thomas et al., Reference Thomas, Cameron and Green2004; Godfray et al., Reference Godfray, Beddington and Crute2010; Hicks et al., Reference Hicks, Comont and Coutts2018; Watson et al., Reference Watson, Evans and Venter2018). This evidence takes a range of forms, including the results of small- and large-scale experiments (Firbank et al., Reference Firbank, Heard and Woiwod2003), meta-analyses (Johnson & Curtis, Reference Johnson and Curtis2001; Batáry et al., Reference Batáry, Báldi and Kleijn2011), systematic reviews (Pullin & Stewart, Reference Pullin and Stewart2006) and predictive models (Taylor & Hastings, Reference Taylor and Hastings2004; Stratonovitch et al., Reference Stratonovitch, Storkey and Semenov2012). Decision-makers need to be able to choose between options using the best evidence available (Sutherland & Freckleton, Reference Sutherland and Freckleton2012).
Unfortunately, ecological systems are enormously variable at just about every scale that we study them (Holling, Reference Holling1973). This variability has numerous sources and, collectively, they contribute to what may be known as ‘uncertainty’. In recognising the role of uncertainty, it is important to recognise that this may arise both as an intrinsic property of the system as well as a nuisance through inadequate data or observation. In terms of intrinsic sources, for example, spatial variability results from variations in conditions from place to place (Tilman & Karieva, Reference Tilman and Karieva1997), while temporal variability similarly results from variations in systems through time (Huston, Reference Huston1994). On the other hand, the measurements of the system may contain inaccuracies. For instance, observational variance is a consequence of our inability to perfectly measure systems, instead relying on sampling in order to build up a picture of the dynamical properties of the system (Dennis et al., Reference Dennis, Ponciano and Lele2006; Freckleton et al., Reference Freckleton, Watkinson and Green2006).
Addressing all types of variability and stochasticity is important in making decisions, and we need to recognise the different sources and how they contribute to uncertainty. Consider a simple example: imagine that we are attempting to implement a conservation measure to protect an organism and that a management intervention, I, may be an effective conservation action if implemented, and this yields a benefit, b. However, there is a cost, c, to implementing the action. If we know that the action is certain to work, there is a simple calculation: all other things being equal then, assuming they are measured in the same units, if b > c then it would be worth performing I. If this is not true, then I is not a favourable approach.
However, because variability is pervasive, the situation in conservation management is rarely so simple. We might not be certain that I is always effective and instead suppose that we know that I is effective with probability p; p could have multiple interpretations depending on context. For example, in a spatially variable system, I might be effective in a fraction p of sites, but not in others: p thus measures the spatial variance in outcomes. Alternatively, the evidence for I being an effective strategy might be mixed, and therefore p could measure some aspect of our belief that I works.
When such uncertainty exists, the condition for a manager choosing to apply I becomes pb > c. Note that typically c should be known reasonably accurately as this will be costed in terms of the resources required to enact I. The benefit is now weighted by the uncertainty in efficacy of I. In terms of making correct management decisions, this simple condition suggests a number of interesting observations. First, as uncertainty increases (i.e. p gets smaller) the likelihood of employing I decreases. If p measures spatial or temporal variability in outcomes, then this is sensible because if I is less likely to work, so a manager should be less inclined to choose it. On the other hand, if p measures a lack of knowledge of the effectiveness of I, then the inequality suggests conservatism: do not take action unless it is known that I is effective with a high probability (p > c/b). If p is measuring such uncertainty then the recommended action has nothing to do with the actual effectiveness of I. Being conservative thus results from ignorance.
A second significant behaviour occurs when both p and c are low: the likelihood of I working is believed to be small but the cost is also small. In this case, employing I may still be favoured by a manager if the benefit is very large and one might describe this as superstitious behaviour (i.e. doing something in the face of little evidence that it will work because the benefit is high and the cost is low). A large number of interventions possibly fall into this category.
Overall, this illustrative example demonstrates that the amount of uncertainty can contribute a great deal to the overall management outcome. In both of the hypothetical situations outlined above, the management applied, and consequent outcome, is suboptimal because it leads to biased impressions of the costs and benefits. Characterising uncertainty is thus vital.
11.2 Recognising types of uncertainty
The source of uncertainty is important and authors have proposed various approaches to classifying uncertainty in management. Regan et al. (Reference Regan, Colyvan and Burgman2002) point out that many of the sources of variability leading to uncertainty described above may be termed epistemic (i.e. uncertainty in the system itself and its measurement). They also highlight a second source of uncertainty, namely linguistic uncertainty. This results from uncertainty in the language used to describe actions or systems, as well as resulting from the conveyance of information. As an example, in the UK there was a programme for government-hired shooters to exterminate ruddy ducks (Oxyura jamaicensis). During the cull, coot (Fulica atra), black-necked grebe (Podiceps nigricollis), common pochard (Aythya ferina) and common scoter (Melanitta nigra) individuals were also shot (Henderson, Reference Henderson2009). This resulted in part from inadequate communication with shooters (Henderson, Reference Henderson2009), who were not ornithologists and failed to distinguish between species. Consequently, there is a possibility of confusion, with procedures subsequently being developed to ensure that confusion is minimised. Although such uncertainty is undoubtedly important, I will concentrate on epistemic uncertainty sensu Regan et al. (Reference Regan, Colyvan and Burgman2002), although some of the points made below could equally apply to a more inclusive definition.
Broadly speaking, it is useful to distinguish intrinsic uncertainty (analogous to the variance in model parameters in an ecological or statistical model) from knowledge uncertainty (by analogy with the measurement error or lack of data in a model). The reason for making the distinction between these two types of uncertainty is important: one is a property of the system itself, while the other is caused by a lack of understanding or data. The two are interactive, and this is perhaps the greatest challenge to making robust predictions in management. If the management outcomes are uncertain both in terms of intrinsic variability and knowledge then they will be largely unpredictable. In this circumstance, it is necessary to question the recommendations given, as well as to consider whether the approach to prediction is the correct one. Another option is to consider models that use an alternative more stable formulation (Taylor & Hastings, Reference Taylor and Hastings2004; Freckleton et al., Reference Freckleton, Sutherland and Watkinson2011).
11.3 Science versus practice: different perspectives on uncertainty
Scientists and practitioners have different perspectives, even if they are working on the same problem. The question of how to resolve this difference is a thorny one (Bradshaw & Borchers, Reference Bradshaw and Borchers2000; Sutherland & Freckleton, Reference Sutherland and Freckleton2012) and there is a pervasive perception of a science–policy gap (Bertuol-Garcia et al., Reference Bertuol-Garcia, Morsello and El-Hani2018). Bradshaw and Borchers (Reference Bradshaw and Borchers2000) highlighted a series of ways in which the perspectives of science and practice may be misaligned. Of these there are two in which uncertainty plays a particularly important role.
11.3.1 Probabilistic, qualified evidence
In the introductory example above, the evidence for the effectiveness of a management intervention was measured as a probability. In terms of providing evidence, this is a routine way in which a scientist would express their recommendation. However, for implementing management, this can be problematic. For instance, telling a manager that there is a 70% chance that the intervention will work is only partly addressing the question of the manager, namely should they undertake the action or not? How is a manager to know whether their particular circumstances are likely to lead to them being in the 70% of cases in which the intervention works or in the 30% in which it fails?
In this context, the meaning of probabilities conveyed by scientists may not always be fully clear. Consider an everyday example. We might be told by a weather forecaster that there is a 50% chance of rain today. However, the meaning of that probability is not typically explained. Here are four interpretations.
(i) It will either rain everywhere or nowhere: it could be one or other of these outcomes, for example, because it is not possible to predict the precise location of a weather system.
(ii) It will rain for 50% of the time during the forecast period: for example, there are patchy rain clouds that are continually moving.
(iii) It will rain in 50% of places: for example, there are rain clouds cover 50% of the area that do not move.
(iv) The forecaster is unable to tell us whether it will rain or not and is telling you to flip a coin.
The technical interpretation of a probability in a weather forecast is that this probability represents the fraction of times a given outcome (e.g. raining within a defined set of areas) occurs in a set of stochastic realisations. This definition, interestingly, can incorporate all four of the above interpretations. Nevertheless, the probability quoted is a form of knowledge uncertainty that has a very specific meaning: it is a measure of model uncertainty/variance.
This highlights a second aspect of scientific evidence that is problematic from the perspective of management, namely that scientific evidence is usually qualified. The statement ‘there is a 50% chance of rain’ from a scientific perspective should also be qualified by the statement ‘across a set of simulations, given the assumption that the model is correct’. If the model is wrong then the prediction could be greatly different.
The task of a manager is to convert such evidence into action (i.e. the binary outcome of whether to act or not). As noted in the introduction, the decision then involves costs and benefits, defined in the widest sense and including values. To continue the hypothetical example, carrying an umbrella is low cost and high benefit, so a 50% chance of rain would render this a good strategy. On the other hand, a manager who is spraying a pesticide requires good conditions, and a 50% chance of rain would potentially carry an unacceptable risk that this costly action (in terms of fuel, time and chemicals) would fail.
11.3.2 General versus situational outcomes
The aim of science is typically to find answers that are as general and robust as possible. A scientist faced with evaluating the effectiveness of a management intervention will attempt to find whether there is evidence of its effectiveness, on average, and then probably focus on understanding the mechanisms that drive it. In contrast, a manager is faced with the task of managing a given site over a defined time period. There is a potential conflict between these perspectives, as the scientific perspective typically averages over variation arising from site-specific variations, whereas this is precisely the variation that a manager is focused on. For a scientist, the local variation at a specific site is essentially nuisance variance.
Although perhaps something of a caricature, there is undoubtedly a real problem in addressing these differences in perspectives. The situation is complicated by the difference in success measures for scientists and managers: scientists prove success by presenting results that are of interest to a wide range of others and that do not focus on specific instances (e.g. in scientific papers); managers measure success based on the state of their site. This difference in perspectives is reflected in the contrasting ways that scientists and managers treat uncertainty. From the science perspective the variation around the mean is a quantity that is to be minimised where possible; in contrast, a manager needs to know where their site sits with respect to this variation, and whether local circumstances render the overall average outcome pattern inapplicable.
11.4 Addressing uncertainty
In general, it is important that uncertainty is recognised and tackled to avoid common ‘traps’ (Millner-Gulland & Shea, Reference Millner-Gulland and Shea2017). These traps are varied, but include ignoring or not accounting for uncertainty, as well as focusing on irrelevant uncertainties and not clearly stating the objectives in framing problems (Millner-Gulland & Shea, Reference Millner-Gulland and Shea2017). Here I review three case studies, showing that there is a line of argument that ignores uncertainty and another that embraces it. In each case the value of conclusions, both for the scientist and the practitioner, require that uncertainty is fully evaluated.
11.4.1 Ignoring uncertainty should not be an option
One of the most important causes of uncertainty is lack of information. This is particularly an issue when information is lacking on rare and difficult-to-observe species, meaning that clade-wide conservation assessments are potentially compromised. The International Union for Conservation of Nature (IUCN) is an important organisation that collates data on the conservation status of species from a wide range of taxa into a set of threat states (Mace & Lande, Reference Mace and Lande1991). This extensive and important exercise informs conservation strategies in a range of contexts (Rodrigues et al., Reference Rodrigues, Pilgrim and Lamoreux2006). The basis for the assessment is a five-point scale of threat status for wild extant species. Species are classified as Least Concern (LC), Near Threatened (NT), Vulnerable (VU), Endangered (EN) or Critically Endangered (CE). Extinct in the Wild and Extinct are categories of extinction beyond these five points, representing species loss.
The amount of data required to apply these criteria varies between taxa. In some cases the amount of information required is quite low. For example, the Nechisar nightjar (Camprimulgus solala) is classified as VU despite being known from only a single wing and a single sighting. On the other hand, for some groups (e.g. mammals and amphibians) the data requirements for the assignment of conservation status are more exacting. Those species for which sufficient information is not available are assigned a status termed Data Deficient (DD). The number of DD mammal species is a considerable fraction of the group (483 of 4186 species; i.e. >10%) of mammals studied by Jetz and Freckleton (Reference Jetz and Freckleton2015).
Denoting species as DD is, effectively, a way of dealing with uncertainty. It is essentially the same as ignoring missing data in an analysis. This way of dealing with data uncertainty is, however, fraught with pitfalls, and a large literature exists on dealing with missing data and associated uncertainty (Nakagawa & Freckleton, Reference Nakagawa and Freckleton2008). It is well understood that non-randomness in the pattern of ‘missingness’ can yield highly misleading analyses.
In the case of conservation assessments, the concern with DD mammal species is that the factors that drive data deficiency are closely related to those that determine extinction threat. For instance, if species are difficult to observe it is likely to be because they only occur at low density in remote locations, or population trends are unknown because they are so rare. It is easy to see that this set of criteria could lead to species being ignored from conservation assessments even though they are threatened.
Jetz and Freckleton (Reference Jetz and Freckleton2015) tested this hypothesis by applying a framework for phylo-spatial modelling of IUCN threats, then using this to predict the probability that DD species are threatened. Species that are DD are predicted to have much higher threat probabilities than those that have been assessed already (Figure 11.1). The fraction of threatened mammal species is therefore underestimated by the current system of assessment.
The importance of dealing with uncertainty in conservation assessments. We used models to generate threat probabilities for mammals. (a) These probabilities do an effective job of distinguishing species that are Least Concern (green bars) from those that are Critically Endangered (orange bars); (b) our models were used to predict threat probabilities for species that were Data Deficient (DD) (pink bars) compared to species that were assessed (grey bars) (i.e. to reduce uncertainty in assessment).
Interestingly, the same is not true of birds (Lee & Jetz, Reference Lee and Jetz2011), as a much smaller fraction of them are considered DD because a lower threshold of information is required to assess threat status. Thus, the recent taxonomic explosion that has led to the creation of 1000 new species of birds (del Hoyo et al., Reference del Hoyo, Collar and Christie2014, Reference del Hoyo, Collar and Christie2016) has not resulted in 1000 species being assigned to the DD category.
This example illustrates an important point about uncertainty that is relevant to conservation and management. Ignoring uncertainty by simply excluding cases where data are missing runs the risk of introducing bias and so, in general, should be addressed if at all possible (Millner-Gulland & Shea, Reference Millner-Gulland and Shea2017). In the introduction I noted that the likelihood of implementing an action is low, irrespective of its actual effectiveness, when there is great uncertainty associated with its effectiveness (i.e. the parameter p is low). In this example, data-deficiency data result in no action being taken (p is low because of uncertainty), although the evidence (Figure 11.1) is that the intervention (assigning status of ‘threatened’) is justified with high probability.
11.4.2 Providing more data/evidence
The preceding example highlights that, where possible, additional data should be used to plug gaps in knowledge. One of the ways that scientists tend to qualify conclusions (see Section 11.3.1) is to say that we cannot be confident because more data are required. As argued by Millner-Gulland and Shea (Reference Millner-Gulland and Shea2017), this can prevent effective management-relevant advice being given.
The example from Jetz and Freckleton (Reference Jetz and Freckleton2015) (see also Safi & Pettorelli, Reference Safi and Pettorelli2010; Bland et al., Reference Bland, Orme and Bielby2015) addressed this qualification by extracting as much information as possible out of the existing data using advanced statistical methods. There are a large range of techniques that have been used to infer missing data and it is not possible to review them here, except to point out that suitable methods have been developed (Nakagawa & Freckleton, Reference Nakagawa and Freckleton2008), or that the problem can be dealt with using flexible statistical frameworks, such as Bayesian modelling (Gelman et al., Reference Gelman, Carlin and Stern1995). Another recent application used models to infer the maximal population growth rate of several shark species for which this demographic rate has not been otherwise estimated (e.g. Pardo et al., Reference Pardo, Cooper and Reynolds2018).
In many cases, however, the bottom line is that sufficient data do not exist and there is no option but to collect more. Data are time-consuming and expensive to collect. Engaging in a programme of data collection will delay implementation and use up resources that could be targeted at on-the-ground management. Frequently there will not be resources available for data collection and hence the knowledge gap is never plugged.
On the assumption that more information could be obtained, a key question arises: will collecting more information improve management decisions (Maxwell et al., Reference Maxwell, Rhodes and Runge2015)? Canessa et al. (Reference Canessa, Guillera-Arroita and Lahoz-Monfort2015) highlight a measure called the ‘Value of Information’ (VoI). This measure is the difference in outcome between the expected management action based only on whatever prior information was available, and action taken with new information provided (Yokota & Thompson, Reference Yokota and Thompson2004; Canessa et al., Reference Canessa, Guillera-Arroita and Lahoz-Monfort2015). They provide an example that is typical of many in conservation or land management. Imagine that a species of conservation concern occurs in one location within a protected area. The aim of conservation is to maximise the size of the population in the area over a specified time period. In order to meet this aim, one strategy could be to create a new population. However, imagine further that there is a chance that a disease could be present that would limit the effectiveness of the reintroduction. The VoI in this case reflects the change in estimated effectiveness that would be achieved by testing for the presence of disease before starting the reintroduction programme. Thus, a test might be performed and return a positive or negative result. Given a prior estimate of the prevalence of the disease, the difference between initial and updated estimates can be calculated using Bayesian updating. These differences then measure the VoI provided by conducting testing. This represents the possible improvement in decision-making through the removal of uncertainty.
11.4.3 Addressing uncertainty through benchmarking
A manager might apply a conservation intervention which, if the outcome is positive, leads to a question of whether the intervention should be used again, or even recommended to another manager. Informal communication of outcomes of this sort are not unusual in land management (Henrich, Reference Henrich2001).
From a scientific perspective, this is not an acceptable way of proceeding unless appropriate controls and experimental design are used in the evaluation of the method. Furthermore, the intervention would ideally be evaluated at more than a single site. This reflects, of course, the tension between the situational and general perspectives of practitioners and scientists. There are pitfalls in both views. There is of course, no guarantee that if management appears to work at one site that it is not simply due to natural variation. Figure 11.2a gives an example of this from an agricultural case study. At one site a specific intervention was used and appeared to be successful. However, compared with the outcome on a set of farms that did not use the technique, there is no obviously large effect. On the other hand, if we are too picky about standards of evidence or data then there is a real danger that useful information will be discarded.
Uncertainty and benchmarking in weed control. (a,b) Predicted responses of populations of the weed Alopecurus myosuroides to rotational management. The initial frequency of weeds at each sowing density was the same in each case (dashed blue line). Each grey line represents a matrix generated from a different field following two forms of management. (a) What would have been the density (0 = zero, L = low, M = medium, H = high and VH = very high) of an average field had it been planted with spring barley. This is compared with (b) the predicted response from maintaining winter wheat. The red line in (a) represents a single field that was managed with variable sowing densities. Figures (c–e) compare the observed effect of management with difference sources of background variation to disentangle the uncertainty in management. We generated models for each field: 22 in winter wheat and 12 rotated from winter wheat to spring barley, and their results are presented in rank order. The effect range is the estimate of the random effect for each field, location or rotation.
Developments such as evidence-based conservation promote the collation of evidence on the effectiveness of management (Sutherland, Reference Sutherland2003; Sutherland et al., Reference Sutherland, Pullin and Dolman2004; see also Chapter 4). The idea here is twofold. First, if the same management has been used in different places then, even if individual interventions do not meet the criteria of a randomised trial (as in Figure 11.2a), the collective body of evidence might be useful. Resources such as www.conservationevidence.com allow this work to be synthesised. Second, using systematic review approaches, it is possible to synthesise this information to provide answers to management problems (Pullin & Stewart, Reference Pullin and Stewart2006; see also Chapter 7).
In the example shown in Figure 11.2a, a single manager implemented one management intervention. On its own this is not enough to determine effectiveness. However, if many people implement the same management then it may be possible to use non-intervention cases as a benchmark and compare the difference with those places where interventions were made. For example, Figures 11.2a and 11.2b show the distribution of weed population sizes in fields subject to intervention (Figure 11.2a) compared with those in which no intervention was made (Figure 11.2b). There is an apparent difference in outcome, but clearly with a high degree of variance. Modelling the data (Figure 11.2c–11.2e) reveals that, although there is an effect of the intervention (Figure 11.2e), there is also a high degree of variance resulting from the initial state (Figure 11.2c) or from the variation in population dynamics from field to field (Figure 11.2d). Consequently, the effect of management, although measurable (Figure 11.2e), is relatively small compared with the intrinsic variability of this system. In this example, the results in Figure 11.2c–e confirm the expectation that the specific management intervention should work, but they also confirm anecdotal local reports that the effectiveness of this approach is patchy, and suggest that frequently the positive effects observed may be attributable to other factors (the large negative effect sizes in Figure 11.2c and d).
Benchmarking of this sort could be extremely valuable in aiding management decisions (Freckleton et al., Reference Freckleton, Hicks and Comont2018). Technological advances, such as widespread instrumentation of agricultural machinery, UAS technology (Paneque-Gálvez et al., Reference Paneque-Gálvez, McCall and Napoletano2014; Lambert et al., Reference Lambert, Hicks and Childs2018) and remote sensing (Kerr & Ostrovsky, Reference Kerr and Ostrovsky2003; Turner et al., Reference Turner, Spector and Gardiner2003) offer the possibility of widescale automated data collection at massive scales. When combined with ecological models, such data could provide a hitherto impossible resource for reducing uncertainty in predicting future management outcomes.
Open access