1. Introduction

Comparative cognition is the interdisciplinary study of the evolution, development, and function of cognitive processes and mechanisms in human and nonhuman animals. A central controversy within the field concerns ways of guarding against bias in the course of interpreting nonhuman animal (henceforth simply “animal”) behavior. This concern about bias is disproportionally aimed at guarding against the overattribution of sophisticated cognition to animals (de Waal Reference de Waal1999; Sober Reference Sober, Daston and Mitman2005; Andrews Reference Andrews, Beauchamp and Frey2011). As a result, the field has tended to prefer explanations of animal behavior that advert to the putatively simplest cognitive mechanisms, processes, or abilities. This practice results in what I call an underattribution bias, or the systematic underascription of complex cognition to animals (Fitzpatrick Reference Fitzpatrick and Lurz2009; Andrews Reference Andrews, Beauchamp and Frey2011). This bias is problematic because no strong reasons have been offered for assuming that the most probable explanation of animal behavior involves the simplest mechanism (Meketa Reference Meketa2014). Moreover, comparative cognition provides no account of what it means for one mechanism to be simpler than another. As a result, a preference for simple cognitive ontologies is unjustified or incoherent, and the underattribution bias must be removed.

In this article, I identify one powerful source of the underattribution bias in experimental comparative cognition, which I argue stems from the choice of the statistical null hypothesis (H ₀). Specifically, I show that the current standards for choosing the H ₀ favor a parsimony-based H ₀, which has the effect of shifting the burden of proof in favor of putatively simple explanations of animal behavior. Having identified this source of the underattribution bias, I propose reformulating current statistical methodology in order to remove it and the underattribution bias it produces. I recommend replacing the present parsimony-based standard for choosing an H ₀ with an empirically and theoretically informed procedure. By revising current statistical methodology in this way, this solution trades in the manageability that comes with opting (a priori) for parsimonious nulls for the greater methodological reliability that a more empirically informed choice of null hypotheses offers. This, in my view, is a favorable trade-off. More generally, my account illustrates how values like parsimony may come to play a central, though implicit, role in statistical scientific methodology.

2. The Source of Bias

The underattribution bias emerges from the dominant statistical analysis method in comparative cognition known as the Neyman–Pearson method (NPM) of hypothesis testing. The NPM is a standard version of a frequentist approach to statistical analysis, that is, one that aims to “provide a reliable decision procedure with controlled long term error rates” (Dienes Reference Dienes2008, 148). Although alternatives such as Bayesian and likelihoodist methods are available, the frequentist NPM is the current orthodoxy in comparative cognition.1Footnote ¹ My solution works within this dominant paradigm, retaining its desirable features while eliminating the source of bias. Put another way, my solution should be viewed as a reformation of methods in comparative cognition, rather than as a revolution.

The standard NPM offers a decision procedure for accepting or rejecting hypotheses by controlling the error rates (long-run relative frequencies) of two types of errors: Type I errors, which reject a true H ₀, and Type II errors, which accept a false H ₀. The Neyman–Pearson tradition assumes that Type I errors are the more serious of the two; however, as I show below, justifications of this assumption have changed over time. Type I error rates are denoted as α and Type II error rates as β. To modify slightly Dienes’s (Reference Dienes2008) formalization of the relationship between error types and their relative frequencies, we may say that

$\begin{array}{c}\alpha =\text{def }P\left(\text{rejecting }{H}_0|H_0\right),\\ \beta =\text{def }P(\text{accepting }{H}_0|\neg H_0).\end{array}$

The NPM provides a means for controlling error rates in a way that minimizes the risk of making both types of errors. However, researchers typically set the risk of Type I errors lower than the risk of Type II errors. The more serious the researcher considers the error, the smaller they will set the value for that error rate. Treating Type I errors as more serious, therefore, means making α very small, while keeping β either large or, in some cases, not controlled at all. This preference for making Type II errors over Type I errors is what Peter Godfrey-Smith (Reference Godfrey-Smith1994) refers to as the error-rate asymmetry. In the next section, I explain how the NPM plays an important role in the underattribution bias.

3. Zeroing In on the Null Hypothesis

Earlier assessments of the underattribution bias identified the error-rate asymmetry as the source of the problem but largely ignored the impact of the H ₀. While they were right to identify this asymmetry as contributing to the underattribution bias, they were wrong to underemphasize the role of the H ₀.

Elliott Sober (Reference Sober, Daston and Mitman2005, Reference Mitman and Lurz2009), for example, argues that there is no reason to prefer making Type II errors over Type I errors in comparative cognition. According to Sober, both types of errors are equally undesirable, since both errors are equally wrong. Sober advocates ridding the field entirely of the error-rate asymmetry, concluding that “the only prophylactic [against risk of error] is empiricism” (Sober Reference Sober, Daston and Mitman2005, 97). Building on Sober’s work, Andrews (Reference Andrews, Beauchamp and Frey2011) attributes the preference for Type I error to an exaggerated worry over the alleged systematic overattribution of sophisticated cognition to animals.Footnote ² She agrees with Sober that the matter is ultimately empirical, but she departs from his conclusion regarding the seriousness of each type of error. She argues that Type II errors are in fact more serious than Type I errors because they restrict future research (Andrews Reference Andrews, Beauchamp and Frey2011). On her view, preferring to make Type II errors means preferring to wrongly conclude that, for example, Portia spiders do not “plan” when taking long detours to attack their prey from a more advantageous position. Once such a judgment has been made, it no longer makes sense to ask further questions about the features of Portia spiders’ planning abilities, such as whether they employ mental maps or are domain specific. As a result, a potentially fruitful research program never gets off the ground. Risking more Type I errors, on the other hand, promotes a further refinement of experimental inquiry. While some of these refinements may terminate in dead ends where the original conclusion turns out to be false (e.g., if Portia spiders do not plan after all), Andrews argues that science must be willing to take such risks.

Despite discussing the biasing effects of the error-rate asymmetry, neither Sober (2001, Reference Sober, Daston and Mitman2005) nor Andrews (Reference Andrews, Beauchamp and Frey2011) questions the fact that the H ₀ is treated as the absence of the mental feature under investigation.Footnote ³ However, as I will now show, the choice of the H ₀ is at least as likely to be a source of bias as the error-rate asymmetry. Once the H ₀ is understood to be a source of bias, a solution to the underattribution problem will become clear.

Consider Andrews’s claim that the error-rate asymmetry is responsible for the underattribution bias. She is right as long as the H ₀ is defined as the absence of the cognitive abilities in question. However, if the H ₀ were defined as the presence of rich cognitive abilities, the result would be the opposite of a bias toward underattribution: the error-rate asymmetry would favor guarding against rejecting rich cognitive hypotheses, and comparative cognition would be biased toward overattribution. That a change to the H ₀ can produce such a dramatic difference illustrates its importance in experimental design and in inference making from experiment. It suggests that an improperly formulated H ₀ generates the underattribution bias.

To illustrate this point more concretely, consider a case where replacing the H ₀ while retaining the error-rate asymmetry would have resulted in a preference for putatively sophisticated cognitive explanations. Foote and Crystal (Reference Foote and Crystal2007) tested rats for their ability to monitor their own uncertainty. Uncertainty monitoring is a form of metacognition, or awareness of one’s own or another’s mental states, and is considered to be a sophisticated cognitive ability. In this experiment, rats were first trained to classify audio tones of different durations into either long or short. Correct answers resulted in six food pellets being given, while incorrect ones resulted in none. Once they reliably classified the tones, the rats were then required to classify three types of tones: (1) clearly short, (2) clearly long, and (3) tones of ambiguous duration. In this “forced-choice” condition, the rats performed poorly on the harder, ambiguous tones, as expected. Next, the rats were tested in an “open-choice” condition, in which they were given the option to decline a test. Declined tests were rewarded with three pellets and the option to move on to more tests. When given the choice to decline tests, the rats opted to decline the ambiguous (“harder”) tests significantly more often than the unambiguous tests, even though declining a test resulted in a smaller food reward than answering correctly. More importantly, the rats’ overall accuracy improved once they were allowed to opt out. Foote and Crystal concluded that the rats were aware of their own uncertainty and hence showed metacognitive abilities.

Let us abstract away from the details of the experimental setup and consider the hypothesis that is being tested. Here, the H ₀ is that rats do not possess metacognition. The hypothesis they wished to demonstrate was that the rats are capable of metacognition. With the error-rate asymmetry in place, the burden of proof falls on the metacognitive hypothesis. In fact, in a follow-up paper on metacognition, Crystal and Foote (Reference Crystal and Foote2009) clearly state that the default (H ₀) hypothesis is and should be that rats lack metacognitive capacities. The reason, they argue, is that the behaviors they observed in the 2007 trials could be explained by allegedly simpler mechanisms, such as stimulus-response learning. As such, metacognitive explanations bear the burden of proof.

But now consider what would happen if the H ₀ in Foote and Crystal’s experiments were richly cognitive explanation of the rats’ behavior, for example, that rats are capable of metacognition. In this case, alternative hypotheses, such as stimulus-response learning, would bear the burden of proof. Were the H ₀ switched in this way, then preferentially guarding against Type I errors at the expense of making more Type II errors would result in an inversion of the underattribution bias. That is, it would produce more instances of accepting the hypothesis that, in this case, rats possess metacognitive abilities when, in fact, rats do not. This example suggests that choice of the H ₀ is at least as important as the error-rate asymmetry: while the asymmetry introduces a bias, the nature of this bias is specified by the content of the H ₀. In fact, the H ₀ may be more important than the error-rate asymmetry because while the asymmetry can only be made more or less pronounced (depending on the values of α and β), the content of the H ₀ can embed any number of problematic assumptions.

Why, then, have animal cognition scientists and philosophers of science assumed that the H ₀ is naturally defined as “nonpresence” or “no effect”? How can we explain the tendency among scientists to tacitly accept that the content of the H ₀ is fixed as the nonpresence of the cognitive property under experimental investigation? An account of the historical origins of and possible justifications for the NPM furnishes an explanation for this tendency. Moreover, this history reveals that nothing about the NPM requires retaining this standard practice of choosing an H ₀ of “nonpresence.”

4. The NPM in Context

According to Godfrey-Smith (Reference Godfrey-Smith1994), the original justification for the NPM was pragmatic rather than epistemic. In essence, the NPM was developed as a systematic means of shaping the behavior of scientists, rather than directing their beliefs. While the original NPM included an accept/reject procedure for experimental hypotheses, this was intended as a behavioral strategy, where “accepting” a hypothesis meant acting as if the hypothesis were true (Godfrey-Smith Reference Godfrey-Smith1994, 280–82). This pragmatism was motivated by Jerzy Neyman’s belief that “when evidence is inconclusive all talk of ‘inferences’ and ‘reaching conclusions’ should be abandoned” (Mayo Reference Mayo1992). On this view, the NPM could not be used to support beliefs in the probable truth of a hypothesis. Instead, in Godfrey-Smith’s words, to take the H ₀ to be ‘true’ means to act as “if the world is in a state such that the action associated with H ₀ is better than the alternative action” (Godfrey-Smith Reference Godfrey-Smith1994, 281). Thus, on the pragmatic justification of the NPM, Type I errors (mistaken rejections of the H ₀) are more practically serious, since a Type I error threatens to leave that feature of the world to which the experiment has access worse off than it otherwise could have been.

This pragmatic justification did not sit well with subsequent scientists and statisticians, such as R. A. Fisher, who wanted a statistical system to provide evidence for the truth or falsity of a hypothesis—something that the original NPM explicitly avoided doing (Anderson et al. Reference Anderson, Burnham and Thompson2000; Gigerenzer Reference Gigerenzer2004; Dienes Reference Dienes2008). The ultimate result was that the pragmatic justification was abandoned and has since largely fallen out of scientific consciousness. However, the method of preferentially controlling for Type I errors was retained, creating a gap between the methodology and its raison d’être.

Uncoupling the methodology from its justification resulted in the proliferation of alternative justifications. One such alternative justification, which Godfrey-Smith labels the semantic justification, most closely resembles the NPM as it is used in comparative cognition research today. Like the pragmatic justification, the semantic justification treats Type I errors as more serious, though it does not provide a reason for doing so. Unlike the pragmatic NPM, the semantic view interprets the accept/reject procedure both behaviorally and epistemically (Godfrey-Smith Reference Godfrey-Smith1994, 287). That is, acceptance of a hypothesis is interpreted as evidence for the truth or likelihood of that hypothesis. This version of the NPM is “semantic” because it specifies the semantic content of the H ₀ by fixing the null as the absence of the feature or effect being experimentally investigated. Godfrey-Smith labels this null the ‘natural null’—call it H _N. On the semantic view, the Type I error is a mistaken rejection of the H _N. Since Type I errors are considered more serious, the semantic view advises erring on the side of concluding that no feature or effect was detected. Godfrey-Smith rightly argues that the H _N thereby shapes experimental outcomes by smuggling Occamist commitments to ontic simplicity into the semantic NPM. That is, when combined with the error-rate asymmetry, the H _N results in a preference for nothing over something. Whether Occamist simplicity is a virtue in science generally is a question with no clear philosophical consensus. What is clear, however, is that the original Occamist justification for preferring simple explanations—namely, that the world is simple—has long been abandoned.Footnote ⁴ Godfrey-Smith writes that psychology uses the semantic NPM almost exclusively, and as we have seen, the same is true of comparative cognition. Thus, the standard statistical methodology in comparative cognition is the product of a series of revised justifications, the latest of which contains an unjustified—though not unjustifiable—preference for guarding against Type I errors and an Occamist H _N.

One lesson to draw from this historical sketch of the NPM is that because statistical methods are a product of intellectual history, the commonsense view that they are objective (in the sense of being value-free) is mistaken. As is the case in comparative cognition, they may come with embedded values such as simplicity. Moreover, as a statistical methodology becomes standard in a given field, its philosophical foundations and the conceptual and normative commitments it entails may gradually fade into the background. As a result, these commitments cease to be subject to direct scrutiny, shaping experiments in ways that researchers may not have endorsed had these commitments been made explicit. Avoiding the misuse of statistics, therefore, requires more than, for example, preventing the gerrymandering of data through cherry-picking statistical techniques—it requires critical analysis of the foundations of the dominant statistical methodologies. By reevaluating the reasons behind choosing a given statistical strategy, these analyses also identify means of reforming statistical orthodoxies adopted by a given field. I now turn to my suggested reform of the NPM, which is aimed at correcting the underattribution bias in comparative cognition.

5. Reforming the NPM: Introducing the “Contextual Null.”

Thus far, I have shown that the underattribution bias stems from the choice of the H ₀ rather than from the error-rate asymmetry (the preference for guarding against Type I errors over Type II errors), which merely acts as a vessel for the bias. As we have seen, the error-rate asymmetry and the choice of the H ₀ are intimately related. However, as suggested by Godfrey-Smith’s analysis, the error-rate asymmetry only becomes a problem when the H ₀ is problematically biased. Thus, the error-rate asymmetry should be retained when the H ₀ is not problematically biased. Minimally, we can say that the H ₀ is unbiased when researchers have strong independent evidence of its veracity. Occamist simplicity does not meet this criterion of evidence, since we have no reason to expect that explanations adverting to simple cognition will be more likely to be true. Since the H _N is a blanket a priori preference for just such simplicity, it is the source of the parsimony-based underattribution bias and must be removed from the semantic NPM.

I recommend replacing the H _N with what I call a “contextual null” (H _C)—one that reflects what is known about the animals under study, or its epistemic “context.” In contrast to the H _N, the H _C is defined against a suite of empirical and theoretical information about the research subjects—information that is relevant for predicting the most probable H ₀. This includes evolutionary, ontogenetic, and ecological considerations; information about species-typical and individual behavioral profiles; neuroanatomical information; and information from related studies and observational data. Evolutionary considerations include, inter alia, the species’ phylogenetic proximity to species about which more is known in order to gauge the likelihood of homologous cognitive structures and abilities. Ontogenetic considerations relevant to constructing the H _C include, for example, possible developmental constraints on the evolution of the relevant cognitive and morphological traits, and the effects of the environment on gene expression. Ecological context refers to information about the subjects’ behavior in their natural habitats, including the nature of their social structure and their hunting and foraging strategies (such as the use of tools). Moreover, these categories may be mutually informing. For example, the role of a neuroanatomical structure can be indirectly inferred from its status as homologous (similar in form or function owing to common ancestry) with respect to another species about whose neuroanatomy and cognitive mechanisms more is known. What matters for present purposes is that when this information is relevant to the experimental question, it ought to inform the choice of the H ₀.

Much of the above information already shapes researchers’ decisions about what to test and on which animals. For example, Foote and Crystal drew on earlier experiments when deciding to test rats for metacognition. They knew that rats had previously exhibited elements of episodic-like (autobiographical event) memory (Babb and Crystal Reference Babb and Jonathan2006). This evidence was relevant because “knowledge about temporal information may play a central role in the organization of behavior … , and consequently, detailed knowledge about when events occur … may have contributed to our experimental evidence for metacognition” (Foote and Crystal Reference Foote and Crystal2007, 552). If researchers have good independent reasons to think that their test subjects will display the feature being tested, this suggests that the information is probability conferring. And, crucially, if this information is probability conferring, then it should inform researchers’ choice of the H ₀. However, as the results of earlier experimental studies may have been affected by the previous choice of a biased H ₀, these studies ought to be scrutinized to ensure that unwarranted metaphysical preferences have not been smuggled in, and that relevant empirical and theoretical information was included.

Evolutionary considerations, too, already influence scientists’ decisions about which species are likely to possess a given cognitive ability. However, they do not affect researchers’ choice of the H ₀. For example, chimpanzees’ close phylogenetic proximity to humans is a frequently cited reason to test them for the presence of human-like abilities, such as tool use and metacognition, but not for changing the H ₀ to reflect this probabilistic judgment.

However, that evolutionary considerations already drive the research projects suggests that scientists consider such information to be probability conferring with respect to the experimental hypothesis. Consider the following example of research that has been guided by both ecological and developmental considerations. Furlong, Boose, and Boysen (Reference Furlong, Boose and Boysen2008) tested chimpanzees for causal understanding by observing their tool use abilities. The research seemed promising because chimpanzees use tools under natural conditions: for example, they dip sticks into ant mounds to catch ants and use leaves to scoop up water. If tool use proved to be flexible—that is, a chimpanzee could generalize from one tool-using situation to another—that would suggest some degree of causal understanding. A previous study on chimpanzee tool use by Daniel Povinelli (Reference Povinelli2000) obtained negative results, concluding that chimpanzees lack the competency for flexible use of implements. Furlong et al. responded by suggesting that Povinelli’s chimpanzees may have been developmentally stunted as a result of being brought up under socially impoverished conditions. Furlong et al. tested chimpanzees with different socialization backgrounds and found that the ability to manipulate tools in a flexible manner—one suggestive of causal understanding—was positively correlated with robust social histories. Furlong et al. clearly saw the utility of using what we know of social ecology and its effects on chimpanzee intellectual development in developing their experiments.

The upshot is this: the H _C ensures that Type I errors will always be more epistemically serious because probability-conferring evidence is built into the H ₀. Sometimes this means of choosing a null will produce an H ₀ positing a simple cognitive ontology. In such cases, a simple ontology will be chosen as the H ₀ not as a result of a blanket Occamist preference for simplicity, but rather as a result of epistemically informed calculations. The current proposal respects the intuition that default hypotheses ought to be those that we have the best reason to adopt—an intuition that already drives research plans, as in the case of Crystal and Foote’s and Furlong et al.’s research programs.

Finally, this proposal requires that it be not only conceptually defensible but also implementable: how can scientists separate the relevant background information in formulating an H _C from all other data? A full answer would require an account of relevance. While such an account is beyond the scope of this article, any future account should contain the following desiderata: an exclusion rule and a stopping heuristic. The former ought to limit the scope of information to what is useful for predicting the most plausible null hypothesis, while the latter should allow the researcher to end her search for predictively useful information prior to exhausting all possible information. The exclusion rule limits the scope of her search to what is relevant for prediction, while the stopping heuristic makes the job manageable. Though less elegant than the simple H _N, the H _C proposal is more likely to arrive at accurate descriptions of animal cognition—which is, of course, the paramount goal of the field.

6. Conclusion

I have argued that earlier assessments of the underattribution bias in comparative cognition have ignored an important feature of the received “semantic” NPM—namely, the null hypothesis. The history of the NPM shows that it is open to interpretation. This makes it a candidate for modification such as removing the ungrounded Occamist H _N and replacing it, as I suggest, with the H _C. This proposed modification respects the intuition that the burden of proof should be on the hypothesis that has the least empirical and theoretical evidence on its side. Thus, my account preserves the risk-controlling structure of the original NPM while eliminating the questionable metaphysics of the semantic NPM and replacing it with a sensitivity to probability-conferring empirical and theoretical information. The reformulated NPM is, thus, more likely to arrive at reliable results, even if doing so requires moving away from the appealing simplicity of the standard NPM.

More generally for the philosophy of science, this article illustrates how metaphysical assumptions, such as preferences for simple ontologies, can be grafted onto the standard statistical practices of an entire discipline. As I have shown, statistical methods are not always value- or theory-neutral data-processing instruments. Therefore, any account of the reliability of experimentation or experimental results ought to include a more careful scrutiny of the theoretical foundations of the statistical methods they employ.