1. Introduction
Recently, philosophers have argued that experiments do more than test hypotheses. Additional roles have included determining whether scientific instruments are functioning properly (Galison Reference Galison1987) and exploring new phenomena when theories are either absent or in turmoil (Steinle Reference Steinle1997, Reference Steinle2002; see also Hacking Reference Hacking1983 and Radder Reference Radder and Radder2003).
I want to consider an additional amendment to our account of experimental research. I argue that exploratory experimentation—experimentation that is not guided by hypothesis (or theory; I will use these terms interchangeably)—has a broader and more systematic role in scientific inquiry than is commonly recognized. I shall provide both an example of such inquiry and a characterization of how exploratory experimentation becomes more attractive when certain conditions are fulfilled. I focus on the availability of ‘wide’, also known as ‘high-throughput’, instruments (those which allow the simultaneous measure of many features of an experimental system) and suggest that, with wide instrumentation, exploratory experimentation is more productive than it is otherwise.
My account is inspired by recent claims, such as the following, that biological practice has moved from a primarily hypothesis-directed method to one that is increasingly exploratory, coincident with the development of wide instrumentation:
The way we do experiments has changed. No longer do we necessarily form null hypotheses, design experiments to test them and derive answers to challenge them. In the new, high throughput world, we can perform thousands of experiments at once, provide millions of possible answers and then start asking questions. (Elgar Reference Elgar2002, 4)
After setting out traditional approaches to experimental method, I use two experiments to probe the effects of instrumentation on experimental method and then consider the value of exploratory experimentation, suggesting that wide instrumentation can increase the efficiency of an exploratory strategy.
2. Experimental Methods
Philosophers frequently take the scientific method to concern the proper justificatory relation between data and theory. Yet scientific inquiry involves a range of activities each of which plausibly involves following a method—the framing of research programs, the design and execution of experiments, the analysis of data, and the formulation and testing of theories. I will focus on just one of these activities, the design of experiments, and ask to what extent experiments are or should be guided by hypotheses.
Two major approaches to the experimental method have been inspired by Francis Bacon and Karl Popper. What follows are not intended as accurate reconstructions, but only as familiar stock positions.
The Baconian method holds that scientists ought to perform experiments or collect data in a broadly exploratory manner prior to theorizing, that they should investigate the world “without premature reflection or any great subtlety” (Bacon Reference Bacon[1620] 2000, 110). Bacon provides a colorful illustration of this when he describes a utopian community of inquirers with a strict division of labor: One class tries “new experiments, such as themselves think good”; a second collects the results in tables; and then a third constructs theories based on these observations (Bacon Reference Bacon[1627] 2002, 486–487). (I have somewhat simplified Bacon's description.)
The Popperian method rejects this putatively aimless data-gathering, suggesting instead that scientists should experiment only after theories have been constructed and pointed predictions made. Hence, Popper's reversal of Bacon's regimen: “The theoretician puts certain definite questions to the experimenter, and the latter, by his experiments, tries to elicit a decisive answer to these questions, and to no others. All other questions he tries hard to exclude” (Reference Popper[1935] 1999, 89).
There are other ways besides Popper's and Bacon's that theory and experiment can be related,Footnote 1 and I will introduce further distinctions when necessary. For now, let us explore the application of these experimental methodologies with two examples. In the first, we can identify a neo-Popperian method and, in the second, a neo-Baconian one.
3. Research Snapshots: Investigating Gene Expression
3.1. The Northern Blot
In the 1980s Lynna Hereford and a number of collaborators performed experiments aimed at understanding the regulation of histone production in yeast (Osley and Hereford Reference Osley and Hereford1981; Hereford et al. Reference Hereford, Osley, Ludwig and McLaughlin1981, Reference Hereford, Bromley and Osley1982; Osley et al. Reference Osley, Gould, Kim, Kane and Hereford1986). Since DNA content increases over the cell cycle as the cell prepares to divide in two, it is not surprising that the concentration of histones (proteins around which DNA must wind to form chromosomes) likewise increases. Although Hereford and her associates knew that histone levels fluctuated over the cell-cycle, they did not fully understand how this was regulated. They were considering two theories: (1) the transcription and stability of histone mRNA is regulated over the cell cycle, leading to the variable protein output; or (2) histone mRNA is stable over the cell cycle, and variability is explained by variable rates of translation into protein. Over a number of experiments, Hereford and her associates worked to understand the system well enough to distinguish among these alternatives.
Let us focus on a particular experiment (Hereford et al. Reference Hereford, Osley, Ludwig and McLaughlin1981), in which the “fundamental issue” the team faced was “whether the temporal pattern of histone synthesis is a reflection of the periodic transcription of histone genes in the cell cycle” (367). At the time, it was believed that histone concentrations were regulated after the transcription of histone mRNA. In order to evaluate whether there was regulation of transcription as well, the team measured histone mRNA concentrations in yeast cells at different times during the cell cycle. Variation in mRNA levels over the cycle would indicate that transcriptional regulation was occurring.
The experimental procedure was simple: The scientists harvested mRNA every few minutes from a culture of yeast cells that were in the same stage of the cell cycle and analyzed how much histone mRNA was present using a Northern blot. The Northern blot is a technique for measuring the approximate quantity of a particular kind of mRNA present in a sample. First, the sample is separated by size on an electrically charged 2D-gel, and then it is labeled using a colored or radioactive probe that binds to the particular mRNA of interest.
The Northern blots showed that there were dramatic differences in the quantity of histone mRNA over the course of the cell cycle. Scientists concluded that histone protein levels were regulated, at least partly, via fluctuations in histone mRNA levels. They further concluded that mRNA levels were transcriptionally controlled, although it is not possible here to explore the reasoning behind this conclusion. Later experiments persuasively established transcriptional control of histone production (Hereford et al. Reference Hereford, Bromley and Osley1982; Osley et al. Reference Osley, Gould, Kim, Kane and Hereford1986).
This example is important to us as an illustration of one role theory can play in experimental design. Let us focus simply on the role theory played in the choice to monitor histone mRNA levels over the cell cycle. Although it is debatable whether the histone mRNA measurements were taken in order to test a hypothesis, it seems clear enough that the measurements were taken in order to provide evidence that did bear on the truth of candidate theories. In this sense, the experiments were certainly theory-directed. An experiment that is directed by theory need not test theory, but only be planned, designed and performed from the perspective of theories of the object in question (Radder Reference Radder and Radder2003).
There are two levels at which theory directed the investigations. In order to contrast this case with our next one, it is useful to distinguish between them. One level of theoretical direction was provided by a systematic knowledge of molecular biology and classic examples used to illustrate possible molecular mechanisms, such as the lac operon. This knowledge led biologists to conjecture that fluctuating protein levels might be regulated by fluctuating mRNA concentrations. Let us call this the theoretical background. A second kind of theory that guided inquiry—if theory is the correct term here—concerned the behavior of the particular objects being measured, histones. Scientists had theoretical reasons to believe that histone proteins were cell-cycle regulated; they also had empirical evidence of this. Let us call this the local theory.Footnote 2 Coupling local theory with the theoretical background, scientists came to suspect that histone mRNA levels might vary over the cell cycle as well.
Finally, it is worth noting that the use of Northern blots to measure the cell cycle regulation of mRNA levels was common among those using “traditional” instrumentation in the late 1970s and early 1980s (Price et al. Reference Price, Nasmyth and Schuster1991; Breeden Reference Breeden2003). Over the course of two decades of using such theory-directed experimentation, approximately 100 cell-cycle regulated genes were uncovered and the mechanisms of their regulation understood. Let us now consider how cell-cycle regulation experiments are conducted when different instrumentation is available.
3.2. The DNA Microarray
Paul Spellman et al. (Reference Spellman, Sherlock, Zhang, Iyer, Anders, Eisen, Brown, Botstein and Futcher1998) also carried out a series of experiments aimed at understanding the regulation of gene expression during the cell cycle. Despite the common subject matter, there are significant differences between the design of these experiments and the Hereford experiments described above. These differences start with the type of instrumentation being used. By the late 1990s, biologists interested in gene expression were switching from the Northern blot to DNA microarrays. Microarrays can measure the relative expression of all cellular mRNA, rather than the one or two mRNA expression profiles that could be measured using the Northern blot.
Microarrays are glass slides covered with a grid of very small spots. Each spot holds hundreds of copies of a segment of coding DNA produced from a library containing segments of all genes in an organism. The simplest kind of microarray experiment compares the mRNA expression profiles of cells in two conditions. First, mRNA is harvested and changed to the more stable cDNA; the cDNA from the control condition is labeled red and the experimental green. These are added together on a single microarray. The cDNA spreads over all the spots, and the sequences attach only to those spots that contain complementary sequences. If a given spot is more red than green, this indicates that there is more of one kind of mRNA in the control condition than in the experimental.
Both microarrays and Northern blots measure the presence and quantity of mRNA. The difference between them lies in the breadth of measurements that can be taken. While a Northern blot can take one measurement at a time in an experiment that can take more than a day to perform, a microarray can make 25,000 equivalently informative measurements at once. Microarrays are the functional equivalent of thousands of Northern blots, more than could be performed in one scientist's lifetime.
With an understanding of the instrumentation, let us consider the Spellman experiment in particular. It began in a manner reminiscent of the Hereford experiment. Scientists first synchronized the cell cycles stage of a culture of yeast. Samples from the population were periodically harvested and then the levels of all mRNA measured on a DNA microarray. Rather than using probes that would allow them to detect just one kind of mRNA, they detected and measured virtually all mRNA present in the cell.
In total, the scientists collected about 400,000 individual measurements of mRNA levels, and, using a Fourier analysis, determined that about 800 genes exhibited mRNA levels that varied significantly over the cell cycle. This was a dramatic increase from the approximately 100 genes already known to possess this property, such as Hereford's histones.
In general, the Spellman experiment was not theory-directed in the same way that the Hereford experiment was. It did not begin with any particular hypothesis or set of mechanisms that were being explored. Instead, the goal was to survey the cell cycle regulation of all genes in the yeast genome and to create a “comprehensive catalog of yeast genes whose transcript levels vary periodically within the cell cycle” (3273). This information was then to help scientists create a picture of the “logical circuitry of transcriptional control in the cell cycle” (3293).
What the Spellman experiment lacked was the local theory possessed by the Northern-blot experiments. Hereford was driven to investigate histones, in particular, and their regulation, by clear evidence that histone production was regulated over the cell cycle. The Hereford group believed that “only a small number of eukaryotic genes are temporally regulated during the cell cycle” (Osley et al. Reference Osley, Gould, Kim, Kane and Hereford1986, 537). If cell cycle regulation was to be investigated, it was important to do this by looking at known targets of regulation.
Does the lack of the pursuit of a local theory mean that the Spellman experiment was Baconian? Let us stipulate that Baconian experiments are designed independently of theory. According to this standard, the Spellman experiment was clearly not Baconian. Scientists brought to their experiments a large repertoire of cytological and genetic theories, an understanding of the phases of the cell cycle, and the knowledge that cells undergo numerous changes over the cell cycle effected through the variable expression of genes. In other words, they were still directed by the theoretical background that also played a role in experimentation in the Hereford example.
While the Spellman experiment was directed (or constrained) by a theoretical background, there was no particular hypothesis being pursued—there was no local theory. Biologists using microarrays frequently call their experiments exploratory (e.g., Liang et al. Reference Liang, Cowley, Hessner, Lazar, Basile and Pietrusz2005). We can make sense of such claims in terms of the lack of local theory, rather than the lack of a theoretical framework altogether.Footnote 3 A background theory, on the other hand, appears to be crucial to the success of exploratory experiments, even those carried out using wide instrumentation. Background theories, among other things, direct inquirers to the kinds of properties that could possibly have a causal role in their local investigations, even if they do not posit particular causal relationships.
This brings us to ask the positive purpose of the Spellman experiment and others like it. I suggest that we conceive of this family of exploratory experiments as mapping activities. Maps, of course, need not be static or structural in the way a map of the genome or a country might be. Maps can track dynamic interactions such as ocean currents. But no matter whether they are functional or structural, an explorer mapping a territory need not be characterized as testing some hypothesis of the terrain underfoot, nor need she be explicitly directed by a hypothesis. In certain cases, she no doubt is, such as the cartographer on a quest to map the Northwest Passage. But she could also simply try to fill in a gap on an otherwise painted globe. The theoretical background serves to guide the explorer to look for certain classes of objects whose activities are known, as a class, to relate to one another, but it need not direct the explorer to one group of those objects over another.
The Spellman experiment was an attempt to explore—or map—the cell-cycle regulation of all genes in yeast. Rather than a hypothesis directing them to a particular part of the complex web of interaction between different cellular constituents, the exploratory experiment served to find interesting patterns of activity from which scientists could later generate a hypothesis. Along these lines, after having identified the roughly 800 cell cycle regulated genes, the Spellman group examined the promoters on each of these genes, looking for and finding similarities between them that they later hypothesized explained their similar behaviors.
The data that came out of the Spellman experiment was of such enormous quantity that the experimenters hardly began to explore it themselves. Instead of even attempting a full analysis of their results, they posted their data on a website, which was then available for other researchers to mine for insights (http://cellcycle-www.stanford.edu). That data has been subsequently analyzed and it is commonplace for some ‘laboratories’ to dedicate all of their resources to analyzing the data produced by other labs, looking to learn from it and asking questions about it that the collectors never dreamed of. In this sense, microarray instrumentation has led to the kind of division of labor envisioned in Bacon's exploratory program.
4. Evaluating Methods
These questions [that biologists are now asking] are not hypothesis driven but rather discovery based. Cell and molecular biology have been powered by hypothesis-driven research for many years, but with the advent of genomic methods such as microarrays, people are asking different types of questions—‘What if we … ?’ (Campbell and Heyer Reference Campbell and Heyer2003, 120)
The Spellman experiment just discussed is one of many exploratory experiments carried out recently in biological laboratories. A university biologist communicated to me a telling anecdote on this point. He recalled that during a recent meeting one biologist remarked that a proposed experiment sounded quite radical, for it was hypothesis-directed! It is our task to consider the logic behind this shift from hypothesis-directed to exploratory experimentation, and to assess, such as it is possible here, its value.
This presents a challenge. It is easy to doubt the worth of some exploratory experiments; they can seem downright absurd. The physicist George Darwin, a son of Charles, reportedly said that every once in a while one should carry out a crazy experiment, like blowing a trumpet in the tulips each morning. Nothing was likely to happen, but if something did, it would be quite a breakthrough (Hacking Reference Hacking1983).
We can make sense of the recent popularity of exploratory methodology in biology—and the unpopularity of trumpeting in tulips—by considering something that seems, to a philosopher, rather mundane, efficiency.
Traditionally, discussions of the worth of a scientific method have not been about efficiency, but about something far nobler, truth. From an epistemological point of view, experiments should be carried out so as to maximize the truth of our theories. Both Popper and Bacon were largely concerned with truth when recommending experimental methods.
Although questions about truth are important, they are not the only questions. Working scientists, and the societies that support them, do not want just the truth—they want truth now. They are interested in experimental methods that are efficient, methods that waste neither time nor resources with experiments that teach us little or nothing of interest. In other words, rather than being concerned only with validation, scientists are interested in heuristics, a branch of scientific methodology involving tactics that can accelerate the pace of scientific advance (Laudan Reference Laudan1981).
Heuristics are means to ends, and their ability to serve those ends can be evaluated like any other empirical hypothesis. This is obviously attractive for those interested in a naturalized methodology.Footnote 4 Suitably framed, a heuristic focus will allow us to consider the benefits of exploratory and theory-directed experimental methods.
Viewed as heuristics, experimental methods are not necessarily universal. Which method is best can depend both on the goal that it is used to achieve and the context in which it is to be used. Let us consider these in order. All things considered, scientific methods should be serving the aims of inquiry in general, but characterizing this is a task too large to deal with here. We will have to be satisfied to specify a plausible local aim. An attractive goal, at least for those, like biologists, who are investigating complex, causal systems, would be that of finding difference-makers. Relative to one or a set of possible experimental outcomes, difference-makers are causal factors that change the state of the measured outcome. For example, while varying the experimental variable, the scientist records some parameter in the behavior of the two systems. Such an experiment finds difference-makers when it does not have a null result—when there is some difference between control and experimental conditions in the measured outcome. We prefer heuristics that maximize the efficiency of finding such relationships.
A second requirement for evaluating a heuristic is establishing its context. Based on the examples from Section 3, we are interested in the different contexts created by two different classes of instruments: narrow and wide. Most people are familiar with narrow instruments; they include the thermometer, the Northern blot, and the magnetometer. Narrow instruments allow the gathering of either an individual measurement, or a small collection of measurements, per trial. Wide instruments allow scientists to assess many features of an experimental system. This could work in at least two ways. It could function as a parallelization of a narrow instrument: The wide instrument affords scientists several thousand data points per experiment compared with a single data point afforded by a narrow instrument. Alternatively, a wide instrument might allow scientists to make just one measurement at one time, but to do so very quickly, so that it could cycle through many different measurements.
5. Making Sense of Exploratory Experimentation
Pleasing as it would be to evaluate methodological efficiency in a formal way, as computer scientists evaluate their algorithms, this will not be possible here. Instead, let us informally consider how the experimental methods might influence the efficiency of scientific investigations, whether applied in biology or in another field.
An exploratory experiment carried out with narrow instrumentation begins with the selection of a factor to measure and a pair of conditions to sample. Because the scientist is not directed by a local theory, she might probe relationships unknown to her. If she were using a Northern blot to study yeast, she might add alcohol to a yeast culture and measure its influence on the mRNA production of a newly cloned gene. Her chances of finding a connection, and thus a difference-maker, will relate to properties of the system under investigation, such as the connectedness of its elements. Yet with an experiment unguided by hypothesis, null results would be, however depressing, unsurprising.
The scientist with narrow instrumentation conducting theory-directed experimentation would act differently. She would use not curiosity but information about likely causal connections to direct her choice of an experimental intervention and a choice of outcome measure. It seems plausible that such a hypothesis would serve to increase the chance—over a random selection—that the intervention would have some effect on the measured outcome. The hypothesis-directed experimenter would not only find more difference-makers in her initial experiments, but as she integrated information acquired from earlier studies into her knowledge store, she might be in a better position to form hypotheses that increased her rate of discovering difference-makers.
It thus seems plausible that theory-directed experimentation is more efficient than exploratory experiments for those using narrow instrumentation. If true, it might be that the efficiency of theory-directed inquiry, rather than the logic of falsification or confirmation, is the best explanation for the ubiquity of theory-directed experimentation in scientific practice. Along these lines, local theory might play a role like the clue-sheet in a scavenger hunt—although it might technically be possible to find all of the hidden objects through brute scrutiny of the whole domain of the hunt, it is much more efficient to use the information from the clues to find the treasure. Given the enormous domains of investigation in the natural world, this question of efficiency very quickly grows into one of feasibility.
A scientist with access to wide instrumentation is less beholden to such a guide sheet. When a large part of the domain is being monitored, it is likely that any particular experimental intervention will have a measurable effect on some measured outcome. Assuming that the experimentalist had access to the computing necessary to analyze her data, she would learn much more from her experiments than would either of our narrow experimentalists. Furthermore, she would be able to investigate connections that the narrow-experimentalist would not consider asking about for fear of wasting time and yielding a negative result.
Might wide instrumentation be productively used to carry out theory-directed experiments? Since such experimentation has thus far only been used in exploratory investigations, we can only speculate on the answer to this question. It does appear that the nature of such experiments would depend on the state of science in the relevant field. If scientists do not know very much about a system, their hypotheses will be sufficiently general that experimentation is, even when guided by hypothesis, still exploratory in that many possible outcomes are left open by hypothesis. However, as exploratory mapping of a system progresses, scientists might begin refining sophisticated models about how a complex system works. There might come a time when theory-directed experimentation, even with wide instrumentation, would become necessary for any further progress.
Although we originally illustrated the nature of exploratory experimentation with an example from functional genomics, this account of exploratory practices is domain-neutral, depending only on the kinds of instruments available. If we can make sense of the value of exploratory experimentation in terms of the presence of wide instrumentation, we should expect to find it elsewhere in science where wide instrumentation is found. This is just what we come across. Although I cannot give a full account here, let me mention a few fields that are undergoing similar exploratory shifts in methodology.
• Functional imaging instruments, such as the fMRI, have allowed some cognitive scientists to pursue more conjectural exploratory experiments compared to those carried out previously using electrophysiological methods (Sommer and Wichert Reference Sommer and Wichert2003; cf. Kosslyn Reference Kosslyn1999).
• Combinatorial libraries of objects have allowed both pharmaceutical manufacturers and materials scientists to engage in less theory-driven and more exploratory experiments (Case Reference Case2003).
• Instruments recently developed in systems biology besides the DNA microarray, such as those used in proteomics, have aided systems biologists in carrying out exploratory experiments. Such instruments, like the microarray, can make rapid measurements of biological properties of interest (Cambell and Heyer Reference Campbell and Heyer2003, ch. 6).
6. Conclusion
In this paper, I have illustrated a type of exploratory experimentation that has not received much attention since it has only recently begun to be practiced. Bacon's proposal that minions roam the earth collecting data for the overseers to interpret, intriguing though it is, was simply infeasible until the development of computational technology that makes it possible to handle the vast amounts of data culled from wide instrumentation. But at least as important is the existence of background theories which help the explorers pick, from among the infinite possible experiments, those that probe objects possibly relevant to the phenomena under investigation. These are precisely the conditions fulfilled by successful DNA microarray experiments and other exploratory research using wide instrumentation, and these are precisely the conditions that were missing not only in Bacon's time, but also when George Darwin stood among the morning tulips with a trumpet at his lips.
