1. Introduction

Risk assessment controversies in biology and other sciences often revolve around disagreements regarding the nature, interpretation, and justification of methods and models used to learn from incomplete and uncertain data. While philosophers of science are ostensibly interested in helping to clarify, if not also to resolve, matters of evidence and inference, they are rarely consulted in practice for this end. Where philosophers are called on to play a role in risk debates, for example, on science panels, their input has largely been focused on the role of ethical and other value judgments in risk policy disputes. As welcome as such participation has been, our position is that issues about values in evidence-based policy call for corresponding attention to methodological issues that enter in collecting, interpreting, communicating, and evaluating the evidence. In Mayo and Hollander (Reference Mayo and Hollander1991), these were dubbed issues of “acceptable evidence” in deliberate contrast to policy questions about “acceptable risk.” That risk assessment judgments intertwine with ethical and value judgments demands a greater methodological understanding that allows for a critical or metascientific scrutiny of the uncertainties, assumptions, and threats of error along the manifold steps in risk analysis. Disagreements that might be attributed to diverging policy and ethical values may actually be the result of divergent assumptions guiding the construction and use of models, and disagreements in the foundations of uncertain knowledge and statistical inference. Although these issues are generally intermingled in debates, the philosopher of science's penchant for laying bare presuppositions of claims and arguments would afford real progress in understanding. If this is correct, then restricting the invitation for philosophical involvement to those wearing a “bioethicist” label overlooks the most constructive ways in which philosophers of science can and should contribute to these debates. If a label is needed, perhaps bioevidentialist would do.

An important advance represented by our co-symposiasts is the recognition that philosophers of science can serve an important role in developing and conducting a metascientific analysis of aspects of risk debates. Even where there is no dispute as to whether a given harm is of concern, they rightly observe, what is often disputed is whether there is evidence of the existence of that harm or hazard, and this in turn may revolve around such choices as which end points to measure and how risks are “framed.” For example, as Thompson (Reference Thompson2006, in this issue) notes, there is a higher estimate of risks of genetically modified (GM) crops if the focus is on the initial stages where uncertainties are high rather than on a later stage after which problematic cases are likely to have been weeded out. Or, again, associating hormesis with mechanisms of natural selection, Elliott (Reference Elliott2006, in this issue) observes, renders it of greater scientific respectability than when associated with homeopathic ideas. Such conceptual analysis can help shed light on the nature of risk debates, but more is required to criticize and help to adjudicate competing risk assessments. The question is: Why stop with conceptual analysis? Why not critique the reliability of the evidence and inferences? The goal of this paper is to give a brief and partial sketch of the kind of metastatistical scrutiny we have in mind. We will apply these ideas to two examples raised by co-symposiasts Thompson and Elliot, GM crops and hormetic effects.

2. Metastatistical Critique of Risk Inference Options

To evaluate how much of a controversy in risk assessment is due to uncertainties in data and how much to conflicting values (social, ethical, economic, religious) requires being able to critically evaluate what the evidence is; and as risk evidence invokes probabilistic and statistical methods, an adequate metascientific scrutiny requires coming to grips with these methods. This does not mean that philosophers of science become statisticians, toxicologists, or the like: that would be both too much and too little. Too much because it would be impractical to become experts in all the arenas involved; too little because there is a great deal of confusion and foundational unclarity among such ‘experts’. A sufficient understanding of the inference methods together with a platform for raising questions about fallacies and pitfalls, we argue, could go a long way toward developing a metascientific (and metastatistical) scrutiny with real bite.

In particular, philosophers of science can serve an important role in developing and conducting an analysis of the various judgments and decisions required to determine if data constitute acceptable evidence of a given risk;Footnote ¹ we might call these risk inference options Footnote ² (e.g., choice of statistical significance levels, dose-response models). Because there is latitude for choice among possible inference options, and each choice influences the chance of obtaining evidence for a given risk (or benefit), much risk controversy revolves around these inference options. Notwithstanding the latitude in choosing inference options, we argue, it is possible to determine how different choices influence a method's ability to detect risks, that is, its risk (or benefit) detecting capacity. This would be the basis for systematically addressing the following questions:

1. 1. How do various methodological choices made in the generating, modeling, and interpreting of data alter a test's risk detection capacity? (For example, Do data-dependent searches alter risk detection ability? If so, how should ‘selection effects’ be taken into account?)

2. 2. What uncertainties and errors have been well ruled out? Which have been overlooked and why? (e.g., extrapolations beyond the lab). Are given policy standards met or flouted?

3. 3. What are the statistical and the substantive assumptions in collecting and modeling the data? How well are they satisfied,Footnote ³ and what are the consequences for the reliability of inference of their being violated in the analysis at hand?

3. How to Find Out the Truth with Metastatistics

In a typical test of risk, we set up a null hypothesis, H ₀, that asserts that there is no increased risk, and an alternative hypothesis, H ₁, that says there is:

1. H ₀: a risk difference δ is 0,

2. H ₁: a risk difference δ is nonzero.

Or H ₁ might be that δ is greater than 0 (one-sided test). A sample of size n is represented by a set of random variables $\boldsymbol{X}=(X_{1},\:\ldots,\: X_{n}) $ , and the data $\boldsymbol{x}_{0}=(x_{1},\:\ldots,\: x_{n}) $ is a realization of X. The experiment might involve studying and reporting the observed difference in risk among those exposed (or ‘treated’) and those unexposed (or ‘untreated’) denoted by d(x₀).Footnote ⁵ One uses the observed d(x₀) to learn about the underlying risk increases that gave rise to the data, as given by the parameter δ in the statistical hypotheses.

A familiar test rule is: Reject H ₀and infer that data x₀ provide evidence of a risk increase if and only if d(x₀) is statistically significant at the α level.Footnote ⁶

When we speak of a test ‘detecting a risk’, we mean ‘it reports a statistically significant result’ (at the chosen significance level α); with an ‘insignificant result’, by contrast, the null hypothesis is not rejected. Although this is often abbreviated as ‘H ₀ is accepted’, it is intended to be understood as data x₀ do not provide evidence against H ₀. (Subtleties between Fisherian significance tests and Neyman-Pearson tests will not alter our points, but see Mayo and Spanos Reference Mayo and Spanos2006 and Mayo and Cox Reference Mayo and Cox2006).

4. How to Tell the Truth (about Insignificant Results) with Metastatistics: GM Crops

Consider the case of GM crops, in particular, plants genetically modified to have pesticidal traits (now called ‘plant incorporated pesticides’), such as genes to cause crops to produce Bt (Bacillus thuringiensis) toxin. A concern is the possible danger to nontarget species such as earthworms or monarch butterflies. Successful EPA petitions to deregulate a Bt crop are based on evidence of acceptable risks to nontargeted species. This evidence in turn is based on finding that the results of lab exposures are not statistically significant in tests of null hypotheses such as:

1. H ₀: Bt crops do not adversely effect untargeted species.

The concern is that failure to reject the null may be due, not to absence of effects, but to the experiment and statistical tests not being stringent or powerful enough to detect them. The metastatistical rule for interpreting insignificant results comes into play. For example, as discussed in Marvier (Reference Marvier2002), an experiment on four replicate batches of earthworms, 10 to a batch, were exposed to soil that included leaves from either transgenic Bt cotton or nontransgenic cotton, and after 2 weeks (too short a time to expect differences in survival rates) the exposed worms gained 29.5% less weight on average than the others. Because this difference is not statistically significant the study concluded that this particular Bt toxin did not impair weight gain in earthworms.

However, due to the low sample size (four replicates), and the large variability among replicates, the test has low capacity to detect adverse weight effects. Given that the number of replicates the EPA requires fails to take into account within sample variation, Marvier reports, very few of the experiments that resulted in statistically insignificant results had a high (90%) probability of detecting even a 50% change (either in survival or weight decrease). Nearly all had little power to detect risks of concern; abbreviate it as δ*.

Although H ₀ ‘passed’ test T, the test it passed was not severe—it is highly probable that H ₀ would pass this test, even if the increased risk is actually as large as δ*. From (SC), a failure to reject H ₀ with test T does not license inferring that the increased risk is less than δ*. What counts as a “risk of concern” reflects policy values, but this critique does not.

The severity criterion also directs us to find specific values of δ that are large enough to be ruled out by dint of the insignificant result. The EPA test had a fairly high probability of detecting a weight change of 56.37% or more; thus the insignificant result may warrant ruling out a 56% decreased weight (between Bt-treated and control worm groups). Generally the reports in the literature provide what is needed for a metastatistical analysis; if not, that alone is grounds for questioning.Footnote ¹⁰

There are lessons both for planning and interpretation. Pre-data: one should specify the effect size of interest (decreased survival, weight, offspring) and calculate the sample size for reasonable power to detect it. Post-data: one should scrutinize the severity attained—based on the actual outcome, variability, and so on. For good discussions, see Marvier (Reference Marvier2002) and Burgman (Reference Burgman2005, Chapter 11), as well as earlier references.

Rules of thumb using confidence intervals are not immune. Appealing to confidence interval (CI) estimates of risk is often thought to avoid misinterpreting insignificant results, but more care is needed. For example, a common rule of thumb is that if both the 0 effect and the risk of concern δ* are included in the interval estimate formed from data x₀, then the results are ‘inconclusive’ (see Burgman Reference Burgman2005, 341). However, the data may provide reasonably severe evidence of the presence of risk δ* (using our criterion); and thus a report of ‘inconclusive’ may not be warranted.

5. A Bioevidentialist Critique of Significant Effects: Hormesis

Hormesis refers to a phenomenon in which a substance that is deleterious at high doses causes a response in the opposite direction at low doses (we can call such low dose reversals ‘improvements' to steer clear from calling them ‘benefits’). Attention to “framing effects” in risk controversies, as usefully delineated by Elliott (Reference Elliott2006, in this issue) for this example, reveals that the way risks are characterized can have a psychological impact in risk controversies. But applying our “bioevidential” scrutiny lets us go much further in waging an effective yet nontechnical critique.

Calabrese (Reference Calabrese2005), a leading proponent of the hormetic hypothesis, has argued that hormesis is a widespread adaptive, stimulatory response. For example, while high doses of dioxin cause increases in tumors, Calabrese cites data showing a suppression of tumors at low dose exposure to dioxin. Here we have a case where rejecting one or more null hypotheses:

1. H ₀: no benefit (or even harms) at low doses,

is the basis for inferring evidence of improvements or decreased risk at low doses. So right away the metastatistical question directed at a positive or statistically significant result (3.3 (ii)) comes to mind: Have they properly controlled type I error probabilities (false positives)? We know from the metastatistical rule for interpreting statistically significant results that if data x₀ are to provide acceptable evidence for the presence of an effect then high severity demands that it not be highly probable to have reported such evidence erroneously. The onus is on the proponents of hormesis to supply convincing evidence that they are not open to misconstruing random effects as genuine.

Before discussing this case we want to emphasize that we are not purporting to decide one way or another about the controversial theory of hormesis—for starters, this short discussion could not do justice to so complex an issue (Mayo and Spanos Reference Mayo and Spanos2007). Our goal is to illustrate how a metastatistical critique can provide standard ways for nonspecialists to raise questions even in dealing with complex evidence-based risks before arguing about what policies might be warranted assuming some risk evidence. Considering this case also helps to illustrate the point raised in Section 3.2: it does not suffice that a low type I error is reported—the actual type I error probability may be a lot higher or it may be uncontrolled altogether, due to certain features of data-dependent selections. Finally, this case would seem to be of interest to philosophers of science both because of the relevance to evidence-based policy and the fact that it is regarded as a possibly revolutionary change in the standard models used in toxicology (Calabrese Reference Calabrese2005).

Evidential warrant for a paradigm change? Although some hormetic effects are apparently uncontroversial, existing use of the linear threshold model in toxicology already allows taking these into account (via U or J shaped models) on a case by case basis. Calabrese and Baldwin (Reference Calabrese and Baldwin2003) want to go much further: they claim to have provided sufficient evidence to actually change the default assumption in toxicology: “These findings challenge the long-standing belief in the primacy of the threshold model in toxicology (and other areas of biology involving dose-response relationships) and provide strong support for the hormetic-like biphasic dose-response model characterized by a low-dose stimulation and a high-dose inhibition” (ibid., 246; emphasis added). As Crump (Reference Crump2001) points out, however, this would demand evidence of a near universal prevalence of hormesis. So the evidential hurdle for the bioevidentialist to consider is whether there is evidence of a sufficiently general hormetic effect. Given the difficulty of detecting the low dose effects of interest, Calabrese, Baldwin, and Holland (Reference Calabrese, Baldwin and Holland1999) decide to obtain their evidence of hormesis through a literature search of $n=10,000$ studies. By putting together those that show apparently hormetic-looking risk assessments they make a case for this “strong support.” But is there acceptable evidence for this?

Among various methodological questions to which these studies give rise, we limit ourselves to a question about the effect of ‘hunting for statistical significance’ (Mayo Reference Mayo1996; Mayo and Kruse Reference Mayo, Kruse, Cornfield and Williamson2001; Mayo and Cox Reference Mayo and Cox2006). Already aware of how type I error rates increase with hunting procedures, our bioevidentialist quickly grasps the gist of Crump's concern: “In order to properly control for the false-positive rate one would need to know how extensive the search was that located the data set. If the data set was the most hormetic looking out of 100 examined, then to conduct a statistical test for hormesis at the standard 0.05 level one should use $p=0.0005$ (the solution to $1-(1-p) ^{100}=0.05$ ) rather than $p=0.05$ ” (ibid., 672). In other words, the researchers would have needed a vastly smaller significance level for each case examined in order for the overall type I error probability to be small. Notice that the task for the bioevidentialist is not to figure out precise significance levels or other error probabilities, it is to point out the kinds of fallacies that must be put to rest. It might next be noted that the data on which they base their inferences are not themselves a random selection from all studies but, rather, are based on a point system they devise, which itself merits scrutiny. On this point system, data are taken as evidence of hormesis simply because a study could have shown evidence of hormesis, whether or not it actually did. (“A data set could achieve a score as high as 6 (high end of the low evidence region for hormesis) even if there was no evidence for hormesis” [Crump Reference Crump2001, 675].)

An effective strategy to demonstrate lack of control of the type I error probability is to apply the test to data deliberately generated to have the null hypothesis true (no hormesis). Such a simulation allows determining the expected distribution of scores from studies in which a hormetic effect is not present (i.e., false-positive rate.) Our bioevidentialist could make use of Crump's report: “Results of this simulation … demonstrates the scoring system does not control the false positive rate (indication that a hormetic effect is falsely identified when none is present). … Using the same scoring system, between 94.9% and 99.7% of the simulated data sets showed some evidence of hormesis (score > 2), even though no hormetic effect was present” (Crump Reference Crump2001, 675).

Unless the results of this simulation are themselves faulty, it appears that Calabrese et al. (Reference Calabrese, Baldwin and Holland1999) have not put the hormesis hypothesis to a stringent or severe test: their data collection and analysis makes it far too easy to produce apparently supporting evidence even where we know the hormetic hypothesis is false. Although philosophers of science would not be expected to run such simulations, using critical information that exists or even asking whether such a challenge could be answered are important first steps.

6. Concluding Comments

We have argued that neither sensitivity to social and ethical values, nor conceptual clarification alone, suffices for the responsible analysis and understanding of today's evidence-based risk assessments and risk debates. Although issues of acceptable evidence are generally intermingled with those of acceptable risk management, the philosopher of science's penchant for laying bare presuppositions of claims and arguments would afford real progress in understanding. If this is correct, then restricting the invitation for philosophical involvement to those wearing a “bioethicist” label precludes the vitally important role philosophers of science may be able to play as bioevidentialists. We hope to encourage a move in that direction.