Introduction

Control experiments are a key element of study design in the life sciences.Footnote ¹ As a recent editorial states: “Experimental design is the practice of planning a study to reach specified objectives. … Formal experimental design focuses on the designation of appropriate controls, random assignment of subjects that receive an experimental treatment, replication, and blocking to remove the potential effect of confounding variables” (Nature Method 2017, 929, emphasis added). Scientific articles routinely mention controls,Footnote ² and handbooks and instruction manuals on methods and protocols in the life sciences regularly call for the inclusion or setup of control experiments.Footnote ³ Science journals commonly ask their reviewers to evaluate the appropriateness of controls; indeed this task is regarded as one of the core elements of successful reviewing: “Importantly, reviewers should identify critical weaknesses, suggest experiments to bolster support of the central claims, and comment on the robustness, validity and reliability of data. Feedback on the adequate description of methodology, the appropriateness of experimental design and required controls, the validity of data analyses and statistics, and the quality and reproducibility of data is central in assessing the strength of a manuscript” (Nature Cell Biology 2017, 1005, emphasis added). An experiment that is not controlled in the right way is not acceptable. Moreover, in informal conversations, scientists go even further and ascribe epistemic significance to controls, stating that finding a control is equivalent with “figuring out what the experiment is doing.” Not only does control play justificatory roles for working scientists, but it has a constitutive function as well.

The notion of control is ubiquitous in scientific practice and scientific publications; it is taken for granted and rarely defined.Footnote ⁴ One of the few existing systematic studies is Boring (Reference Boring1954). In this article, psychologist and historian of psychology Edwin Boring distinguished several meanings of control and situated control experiments in the contexts of nineteenth- and twentieth-century science and philosophy of science. Apart from Boring’s contribution, there is not much discussion in philosophy of science and indeed in the sciences themselves about what an appropriate control is, whether it is possible to give a general characterization of a control, or how the appropriateness of controls could be properly evaluated.

In everyday contexts, the term “control” is associated with oversight, management, and constraint. We seek to control riots, passports, pests, or temperatures. In this broad understanding of “control,” experimental control means the check that an experiment is methodologically sound.Footnote ⁵ Here I am interested in experimental controls as particular kinds of checks—different from, say, replications or error statistics or validations of data analyses. In this article, I am taking my cue from Boring’s work, but I will argue that Boring’s explication of “control” is ambiguous, awkwardly straddling conceptual, normative, and practical issues. I propose a revised set of distinctions.Footnote ⁶ Along the way, I also show how the careful study of past scientific terminology can be philosophically fruitful.Footnote ⁷

In a sense, my discussion of control experiments parallels the discussions in philosophy of science about different roles of experiments in scientific practice. Philosophers of experiment have shown that experiments have a life of their own (Reference HackingHacking 1983), that they can play exploratory as well as evaluative roles (Steinle Reference Steinle2002), and indeed that they have many different roles, including articulating an existing theory, measuring quantities that are of physical interest, demonstrating a successful new experimental technique, or correcting previous incorrect or misinterpreted results (Franklin Reference Franklin2016). My discussion highlights different functions for experimental controls in the realization of an experiment: experimental controls may serve as checks, and they may serve as probes.

Boring’s approach does not account for an important dimension of experimental control, namely, its constitutive role in knowledge production (“figuring out what the experiment is doing”). The approach to experimental control that I propose can illuminate this role: I show that experimental controls as probes have a constitutive function in knowledge generation. My account of experimental controls also sheds new light on the conception of exploratory experimentation, and it clarifies what can and what cannot be expected from reviewers of scientific journal articles giving feedback on experimental controls.

Concepts of Control

Boring (Reference Boring1954) advanced two conceptual claims and one historical thesis. The first conceptual claim is that three meanings of “control” must be distinguished in the context of experimentation: “control” in the sense of restraint (keeping conditions constant), “control” in the sense of guided manipulation (causing an independent variable to vary in a specific manner), and “control” in the general sense of check or comparison. The third, general notion of control is the oldest. It stems from the French word contre-rolle and is etymologically tied to keeping a duplicate or a roll of accounts (573).

Boring’s second conceptual claim is that control “in the sense of a check or comparison … appears in all experimentation because a discoverable fact is a difference or a relation, and a discovered datum has significance only as it is related to a frame of reference, to a relatum” (Reference Boring1954, 589). The historical thesis is that our philosophical understanding of experimental control originates in Mill (Reference Mill1843/2006), although Mill himself did not use the term “control” in this context.Footnote ⁸

Boring’s claim that check or comparison appears in all experimentation lends itself to different readings, none of which is entirely convincing. We could read it as an empirical claim about comparative experimentation. In a comparative trial, two entities (as similar as possible) are put in similar experimental situations, but only one receives treatment.Footnote ⁹ In this interpretation of Boring’s, claim it would be incorrect that control appears in “all experimentation.” Perhaps he meant to say that every proper experiment involves (or should involve) some kind of comparative act. Then we would be left with the task of defining what kind of comparison would be a proper experimental control. Or perhaps he meant to advance an interpretation of causality, along the following lines: we understand an experimental outcome (a claim about independent variable I and dependent variable D) against the background of an (implicit) assumption about what would have happened had the variable I not been changed. This claim is as plausible as counterfactual theories of causality are, but it is an interpretation of the meaning of the concept of cause and is not useful as a guide for good experimental practice.

Boring’s historical claim does not help to clarify the issue. As evidence for the view that “the methodological status of control as a check or comparison is to be understood by reference to J. S. Mill’s Method of Difference,” Boring (Reference Boring1954, 589) cites the first edition of the Oxford English Dictionary, Murray (Reference Murray1893). This dictionary defines (the “modern scientific use of”) control as “a standard of comparison used to check the inferences deduced from an experiment by application of the Method of Difference.” In the same passage, the control experiment is defined as “a test experiment with this end in view” (Murray Reference Murray1893, 927). Boring assumes, quite plausibly, that the dictionary definition alludes to one of Mill’s methods of experimental inquiry.Footnote ¹⁰ However, it is important to keep in mind that Mill’s method of difference captures an inference pattern to attain a conclusion, whereas Murray’s definition invokes pragmatic strategies to “check the inferences”: “control” in the sense of carrying out a set of complementary trials.

Mill described the method of difference as the only type of inference by which one can ever “arrive with certainty” at causes (Mill Reference Mill1843/2006, 394). The secure arrival at a cause requires an ideal, the perfectly controlled experimental design, whereby an independent variable is caused to vary in a specific manner (it is manipulated in the first trial but not in the second), and the background is restrained (held constant across both trials).Footnote ¹¹ Needless to say, in a concrete research context, the perfectly controlled experimental design will never be realized, as it is impossible to keep all background conditions constant across different trials and as there is an element of arbitrariness and context dependence in the delineation of the causal field that is relevant for the context in question.Footnote ¹²

It is, however, possible to bring the philosophical conceptualization of experimental control closer to the realm of the working scientist by distinguishing different strategies for attaining experimental control. For this purpose, it is instructive to take a closer look at the scientific sources that served Murray as the sources for his dictionary definition of “control.” Just like the current edition of the Oxford English Dictionary would, the first edition provides illustrative examples for the usage of the term “control,” citing phrases from Charles Darwin’s work on insectivorous plants (Reference Darwin1875), Charles Darwin and Francis Darwin’s work on the movement of plants (Reference Darwin and Darwin1880), and a journal article on immunology (Hankin Reference Hankin1890). Careful reading of these examples reveals that the scientific uses of control are structurally different. I contend that these differences matter epistemically and are relevant to the work of the scientific reviewers today.

Murray’s sample phrase “Four bladders were first tried as a control experiment” is a quote from Darwin (Reference Darwin1875, 4313). The phrase “Radicles without any attached squares, which served as standards of comparison or controls” is a quote from Darwin and Darwin (Reference Darwin and Darwin1880, 162). The phrase “Control mice died of tetanus within 48 hours” is a quote from a scientific article on immunology that was published in Nature (1890, 122; the original source reads “forty-eight hours”). All phrases refer to scientific experiments. All three experimental designs could be understood as real-world approximations of perfectly controlled experiments, but the important thing is that they exemplify different pragmatic strategies of comparison and checking. In the second and third examples, comparisons are being made between an entity that receives a certain treatment and another similar entity in a similar situation that does not. The Darwins compared the natural growth of radicles (primary plant roots) to the growth of radicles of the same plant with little cardboard bits attached. The article published in Nature, which announces a cure for tetanus and diphtheria, describes an inoculation experiment. Six mice received each 2 cc of a tetanus antiserum and withstood the subsequent inoculation with tetanus, while the “control-mice” that had not received the serum “died within forty-eight hours” (Hankin Reference Hankin1890, 122). In both cases, the assumption is that all relevant background conditions remain constant across the trials. This is a tacit assumption; neither the Darwins nor Hankin explicitly addressed the issue of background conditions.

The first example is different. To learn about the process of digestion in insectivorous plants, Darwin (Reference Darwin1875) investigated the “bladders” of certain plants (he was working with Utricularia neglecta, or bladderwort). The bladders are those parts of the carnivorous plant that ingest the prey. Darwin had observed that the absorption of decaying animal matter had generated protoplasm in the processes inside the bladders. He conducted experiments to establish whether the decaying animal substance was essential to the generation of protoplasm or whether the plants could also generate protoplasm by absorbing carbonate and nitrate of ammonium. To find out, he needed to make sure that the bladders he used in his experiments did not contain any rotting animal matter, or, in his words, to ascertain “positively that the bladder, though translucent, does not contain some minute animal in the last stage of decay” (413). These experiments were his control experiments. The best way to proceed was to cut bladders “longitudinally into two; the quadrifids were examined with No. 8 of Hartnack, then irrigated, whilst under the covering glass, with a few drops of the fluid under trial, kept in a damp chamber, and reexamined after stated intervals of time with the same power as before. Four bladders were first tried as a control experiment, in the manner just described, in a solution of one part of gum arabic to 218 of water, and two bladders in a solution of one part of sugar to 437 of water” (413, emphasis added). After these experiments, Darwin proceeded to test a number of bladders with various fluids, including a solution of one part of carbonate of ammonia to 437 of water, a solution of gum (one part to 218 of water), a solution of carbonate of ammonia, a solution of one part of urea to 218 of water, a “putrid and very offensive infusion of raw meat,” and many more (413–15). In this series of trials, the control experiments were carried out before—in preparation of—the actual investigation. Their purpose was to determine the background conditions that might have an impact on the digestive processes within the bladders.

This strategy emerges even more clearly from the following passage in a later section of the book. Here, Darwin discussed the digestive function of Drosera (sundew). The question was whether the secretions of the plant contained a digestive ferment like pepsin (see Darwin Reference Darwin1875, 94–96). In this experiment, Darwin neutralized the acid of the secretion with an alkali and examined whether digestion would stop and would begin again when more acid was added. However, “it was necessary first to try two control experiments namely, whether the addition of minute drops of water of the same size as those of the dissolved alkalies to be used would stop the process of digestion; and, secondly, whether minute drops of weak hydrochloric acid, of the same strength and size as those to be used, would injure the leaves” (94, emphasis added). Again, the control experiment is the preparatory trial that was carried out to gather information about the background conditions for the actual experiment on the digestive function of sundew.

Just a few years after the publication of Murray’s (Reference Murray1893) Dictionary, another dictionary conceptualized the distinction that one can extract from Darwin’s experimental reports. Whitney’s (Reference Whitney1897) Cyclopedia distinguished between “controlling experiment” and “control-experiment.” The explication of “controlling experiment” is as follows: “Controlling experiment, in chem., a corroborating or confirmatory experiment” (1237). An example from Bowditch (Reference Bowditch1867, 149) illustrates the meaning of the term: “For a controlling experiment, the gas may be passed for a short time through the alcoholic ammonia alone.” The phrase is taken from the description of a procedure to detect sulfide of carbon in the gas and is, in fact, itself a quote from a journal article published in 1861. In the experiment, the gas passes through a solution of alcoholic ammonia fortified with a few drops of a saturated solution of lead. The solution takes an orange color, which indicates the presence of sulfide of carbon in the gas. The dictionary also offers a definition of “control-experiment,” which reads: “control-experiment … An experiment made to establish the conditions under which another experiment is made” (Whitney Reference Whitney1897, 1237). This definition is not illustrated with an example.

The two terms “controlling experiment” and “control-experiment” conceptualize two different strategies of experimentation in actual research contexts. In comparative trials, two similar experimental entities are placed in as nearly as possible the same situation and are subjected to as nearly as possible the same treatment except for the intervention of interest (the change of the independent variable).Footnote ¹³ “Controlling experiment” refers to one of these maneuvers. The controlling experiment either does not involve an intervention or it involves an intervention with a known effect. In the example from immunology, the controlling experiment is the experiment on the mice that did not receive antiserum. In Bowditch’s case, the gas is passed through a chemical solution that does not contain an indicator.

The other strategy—termed “control-experiment” in the Cyclopedia—is the strategy of investigating those additional factors that may have an impact on (disrupt, change) the relation between the independent and the dependent variable in an experimental situation. This notion of experimental control is not covered in the account that Boring advanced. The two specific meanings of control that he distinguished both had to do with managing the known variables in an experiment: “control” in the sense of keeping known background conditions constant and “control” in the sense of causing the independent variable to vary in a specific manner. The control-experiment, as the Cyclopedia characterizes it, is preparatory. It is done to investigate the conditions before making the actual experiment and the controlling experiment.

We can now say that one of the examples from Murray’s (Reference Murray1893) Dictionary, the example drawn from Darwin’s book on carnivorous plants, is an example for a “control-experiment,” that is, an experiment to establish the conditions under which another experiment is made. The experiments on infected mice and growing plants are examples for “controlling experiments” that are done to corroborate or confirm the original experiment. In all definitions and examples for scientific strategies of controlled experimentation, it is taken for granted that the experimenters use only known, stable investigative tools and operate in stable, known environments.Footnote ¹⁴ In the next section, we will see that the distinction between “control-experiment” and “controlling experiment” is epistemologically significant.

Functions of Control: Experimental Control as Probe and Test

So far, the nineteenth-century dictionaries have served me to expose two notions of experimental control in addition to the everyday notion of oversight and management and the ideal of the perfectly controlled experimental design (which enables “with certainty” the establishment of causes according to the method of difference). On the one hand, there is the strategy of conducting a controlling experiment for another experiment. In my interpretation of the notion, “controlling experiment” encompasses all three practical meanings of control from Boring’s (Reference Boring1954) study. A controlling experiment is informative in the following circumstances: it subjects an entity that is similar to the entity in the original experiment to as nearly as possible the same treatment as in the original experiment, except for the intervention. The background conditions remain as stable as possible across both trials.

On the other hand, there is the strategy of conducting control-experiments. In my interpretation of the notion, a “control-experiment” establishes the conditions under which another experiment is made.

Using this interpretation emerging from the nineteenth-century dictionary definitions, it is now possible to distinguish different functions of experimental control. “Controlling experiments” function as checks, as attempts at confirmation of the outcomes of the original experiments. “Control-experiments,” by contrast, function as probes.

We could say that the purpose of control-experiments is to build up a confounder repertoire, that is, a repertoire of factors that muddle (disrupt, distort) the relation between independent and dependent variables in an experimental situation.Footnote ¹⁵ The term “confounder repertoire” resonates with Deborah Mayo’s term “error repertoire,” but the underlying idea is not the same. Mayo has argued that experimental inquiries can be understood as “piecemeal learning from errors,” whereby complex experimental problems are broken down into manageable piecemeal questions. Each of these questions can be understood as a question about whether a certain type of error is absent. These types of errors include mistaking artifacts for real effects, mistakes about causal factors, or mistakes about the value of a parameter (Mayo Reference Mayo1996, 17–18). I prefer the notion of confounder to the notion of error because the presence of confounders does not necessarily lead to erroneous results; confounders may confuse, and no result can be obtained. More importantly, Mayo’s “error repertoire” is a transcontextual catalog of “canonical errors”; it includes common types of experimental mistakes. According to Mayo, these types of errors can be found through the synoptic analysis of mistakes made in the past (452). In my account, the confounder repertoire is context specific; it encompasses the kinds of factors that are likely to interfere with an actual experimental inquiry.

An error repertoire as Mayo envisages it is built with the benefit of hindsight. Looking back and drawing on our current knowledge, we can characterize certain past experimental results as mistaken; we can point to the sources of these errors and can say what type of error was committed. The experimenters seeking to build specific confounder repertoires for their experiments do not have the benefit of hindsight. They may not even know where to begin their search.Footnote ¹⁶

Consider again the strategy of conducting control-experiments. I extracted this interpretation of experimental control from a close reading of the Cyclopedia definition (Whitney Reference Whitney1897) quoted above. On yet closer reading, the characterization of “control-experiment” in the original Cyclopedia formulation lends itself to two different interpretations. “Establish[ing] the conditions under which another experiment is made” could mean figuring out what factors may have an impact on the experimental situation (identifying the repertoire of confounders). It could also mean establishing what the specific conditions are in a given situation. Both are important pretrial tasks in the practice of experimentation, but they are not the same. The Cyclopedia does not offer a quotation from a scientific source to illustrate this definition, but if we consider again the example from Darwin’s book on insectivorous plants, the distinction between two probing strategies becomes clearer. Diagnosing and determining the kinds of confounders that must be taken into account are two different tasks. In other words, there are two functions for probing strategies: diagnostic and determinative.

The diagnostic probing task is to figure out where to look for confounders. This step is constitutive for the knowledge that the experiment generates. In Darwin’s case, it is the task of figuring out what factors might possibly affect the production of protoplasm and the digestive processes in carnivorous plants. Because Darwin published at a time when he had already accomplished this task, in his text it does not even appear as a task; the diagnostic steps are not described.

The determinative task for experimental controls is to determine what confounders are indeed present in a given experimental situation. In other words, even if we have our confounder repertoire, we still have to figure out how we could best control for each of the confounders. Once it was clear to Darwin that only empty bladders should be used to study the process of protoplasm generation in plants, he still had to figure out how the bladders could best be cleaned. Control-experiments as determinative probes pose logistic problems, which are not always easy to solve. On the contrary, the logistics of determining confounders may pose considerable, and at times insurmountable, engineering challenges.

My discussion of the twofold probing function of control-experiments—diagnosing and determining—illuminates the notion of exploratory experimentation that goes back to Friedrich Steinle.Footnote ¹⁷ Drawing on examples from the history of electricity, Steinle has characterized a kind of experimentation that is performed in situations where no well-developed theories are available.Footnote ¹⁸

One important characteristic of exploratory experimentation, according to Steinle, is that it is a mode of knowledge generation. According to Steinle, exploratory experimentation generates a specific form of knowledge, namely, new concepts. Steinle also shows that past episodes of exploratory experimentation often encompass a range of experimental strategies, such as parameter variation, simplifying and purifying an experimental arrangement, and finding appropriate concepts to express the empirical rules governing the experimental project.

What exactly does it mean that exploratory experimentation has an epistemic function? After all, Steinle agrees that applying these strategies does not guarantee that new knowledge can be generated. Steinle’s idea is as follows: in hindsight, we understand that in many experimental contexts we now regard as generative of new knowledge, exploration was involved. Charles Dufay’s experiments on attracting and repulsing substances were exploratory in the sense that no theory of electricity guided these experiments. Retrospectively, we can characterize the products of such generative exploration as new concepts—Dufay’s conception of positive and negative electricity in this case (see Steinle Reference Steinle, Schickore and Steinle2006). For Steinle, exploratory experimentation has an epistemic function in the sense that exploration typically contributes to the generation of new concepts, although these strategies of exploratory experimentation are not a recipe for cooking up new ideas and they do not guarantee that the new ideas thus generated will withstand further scrutiny.

Thinking about the structure and function of experimental control reveals that there is a deeper sense in which exploratory experimentation has an epistemic function, and thinking about experimental control can help to pinpoint this function. We have seen that the umbrella term “experimental control” covers different kinds of experimental strategies, some probing, others testing. The strategies lumped together under the umbrella “exploratory experimentation” are also of different kinds. Some of them—simplifying and purifying an experimental arrangement—are determinative. They presuppose that a confounder repertoire for this experimental arrangement is in place. Other strategies—finding a variety of parameters—are constitutive for an exploratory project in that they generate the confounder repertoire for the experiment.

There are no general recipes for finding confounder repertoires. Moreover, and more importantly, our knowledge of relevant confounders will always be limited, and we need to assume that there are unconceived alternatives to the confounders we identified.Footnote ¹⁹ The confounder repertoire enables our experimental learning, yet there is a degree of freedom in it. Mayo mentions in passing the “problem of choosing an appropriate test procedure for a given inquiry,” but her emphasis is on what to do once one has “specified the kind of argument from error one wishes to sustain.” Once this is accomplished, the background knowledge “directs one to appropriate choices of test statistics and corresponding error characteristics” (Mayo Reference Mayo1996, 298–99). My distinction between two probing functions highlights the freedom in making these choices and clarifies what this freedom means for the knowledge gained through experiments. Not every choice is a plausible choice, yet there may be reasonable disagreement about the composition and comprehensiveness of confounder repertoires. Drawing up such a repertoire of confounders plays a constitutive role for our knowledge as it is produced in our experimental practice.

Conclusion

I have proposed an account of experimental control that is relevant to actual scientific practice. My framework comprises the following readings of “experimental control”:

1. I. The ideal of the perfectly controlled experimental design (which enables “with certainty” the establishment of causes according to the method of difference).

2. II. Pragmatic strategies of experimental control

   1. 1. Checks

      1. a) The task is to create and maintain a stable, known experimental environment.

      2. b) The task is to conduct a “controlling experiment” for another experiment.

   2. 2. Probes

      1. a) Determinative: the task is to determine what confounders are indeed present in a given experimental situation (“control-experiment”).

      2. b) Diagnostic: the task is to figure out where to look for confounders.

This framework for thinking about experimental control can do justice to the significance that scientists attach to controls and helps to better understand the tasks of reviewers for scientific journals who are expected to assess the appropriateness of required controls.

What matters for reviewers seeking to give feedback on controls are the pragmatic strategies (II). My framework makes clear that evaluating controls involves quite different tasks and different sets of evaluation criteria. Perhaps different kinds of reviewers are needed to accomplish these tasks and to apply these criteria. Reviewers face several challenges, depending on whether they are assessing experiments functioning as checks (1b) or experiments functioning as probes for confounders (2) and whether they are concerned with the logistic problem of eliminating or accounting for known confounders (2a) and the diagnostic problem of devising confounder repertoires (2b).

Assessing experimental controls as checks requires assessing whether the experimental setting is stable enough to allow for any conclusions to be drawn from it (1a). It also means assessing whether these experiments yielded results that are significantly different from the original experiments (1b). Assessing logistic solutions for the problem of how to account for known confounders means assessing whether particular means can accomplish given ends, whether these means are satisfactory or maybe even constitute improvements in accuracy or precision, speed, or practicality (2a). Assessing the plausibility of confounder repertoires is a fundamentally different task because it raises the question of whether all relevant confounders have been taken into account (2b). The problem is that our knowledge of confounders is always incomplete. Here reviewers face the problem of unconceived alternatives to the proposed confounder repertoires.

It would thus be useful to remind authors and reviewers that controlling an experiment and evaluating controls is as complex and as demanding as making and evaluating the experiments themselves. It would be useful to address the different pragmatic dimensions of experimental control in instructions for authors and reviewers and to ask authors and reviewers for justifications and discussions of the choices of confounder repertoires and controlling experiments.

This account of experimental control explains why scientists refer to experimental controls as extremely important parts of their experiments, noting that finding a control is equivalent with knowing what they are doing in the experiment. My interpretation of control-experiments as diagnostic and determinative probes captures scientists’ intuitions about the fundamental epistemic role of experimental control, the intuition that finding a control is equivalent with figuring out what the experiment is doing.