1. Introduction: The “Replication Crisis”
In the current debate about replicability in psychology, we can distinguish between (1) the question of why not more replication studies are done (e.g., Romero Reference Romero2017) and (2) the question of why a significant portion (more than 60%) of studies, when they are done, fail to replicate (Open Science Collaboration 2015). Debates about these questions have been dominated by two assumptions, namely, first, that it is in general desirable that scientists conduct replication studies that come as close as possible to the original and, second, that the low replication rate can often be attributed to statistical problems with many initial studies, sometimes referred to as “p-hacking” and “data massaging.”
I do not wish to question that close (or “direct”) replications can sometimes be epistemically fruitful, especially with regard to the task of uncovering sloppy statistics. However, I contend that the focus on formal problems in data analyses has come at the expense of questions about the notion of replication as such, as it pertains to psychological experimentation. In this article I address one aspect of this topic by highlighting that psychological experiments (and hence also any attempt to replicate an original experiment) are typically infused with conceptual and material presuppositions and uncertainties, thus making low replication rates not so surprising. I will argue that this fact cannot be remedied by more rigid replication studies, nor does it necessarily constitute a crisis. Rather, it draws our attention to the fact that much psychological research concerns itself with exploration rather than replication.
In section 2 I briefly review some philosophical arguments as to why there can be no exact replications and, hence, why attempts to replicate always involve individuation judgments. Section 3 will address a distinction that is currently being debated in the literature, that is, that between direct and conceptual replication, highlighting limitations of both. Section 4, finally, will argue that a significant part of experimental research in psychology is geared toward exploring the shape of specific phenomena or effects and that the type of experimentation we encounter there is not well described as either direct or conceptual replication.
2. The Replication Crisis and the Ineliminability of Concepts
When scientists and philosophers talk about successfully replicating an experiment, they typically mean that they performed the same experimental operations/interventions. But what does it mean to perform “the same” operations as the ones performed by a previous experiment? I take it to be trivially true that two experiments cannot be identical: at the very least, the time variable will differ. Replication can therefore at best aim for similarity (Shavit and Ellison Reference Shavit and Ellison2017), as is also recognized by some authors in psychology. In this vein, Lynch et al. (Reference Lynch, Bradlow, Huber and Lehmann2015) write that “exact replication is impossible,” arguing that the most advocates of direct replication can aim for is to get “as close as possible,” that is, to conduct an experiment that is similar to the previous one (333). In the literature, such experiments are also referred to as “direct replications” (e.g., Pashler and Harris Reference Pashler and Harris2012).Footnote 1
But the notion of similarity is also notoriously problematic (e.g., Goodman Reference Goodman1955), since any assertion of similarity between A and B has to specify with regard to what they are similar. In the context of experimentation, the relevant kinds of specifications rely on conceptual and material assumptions, many of which are not explicated, about the kinds of factors one is going to treat as relevant to the subject matter (see also Collins Reference Collins1985, chap. 2). Such conceptual decisions will inform what one takes to be the “experimental result” down the line (Feest Reference Feest2016). For example, if I am interested in whether listening to Mozart has a positive effect on children’s IQ, I might design an experiment, which involves a piece by Mozart as the independent variable and the result of a standardized IQ test at a later point. Now if I get an effect, and if I call it a Mozart effect, I am thereby assuming that the piece of music I used was causally responsible qua being a piece by Mozart. Moreover, when I claim that it is an effect on intelligence, I am assuming that the test I used at the end of the experiment in fact measured intelligence. These judgments rely on conceptual assumptions already built into the experiment not only qua choice of independent and dependent variables but also qua judgments of what are relevant features of those variables. In addition, I need material assumptions to the effect that potentially confounding variables have been controlled for.Footnote 2 I take this example to show that effects are typically investigated under a description (i.e., involve concepts).
One obvious rejoinder to this claim might be to say that replication attempts simply imitate what the original experiment did, with no particular commitment to what is being manipulated or measured. But even if direct replications need not explicitly replicate effects under a description, they nonetheless rely on “individuation judgments” (Soler Reference Soler2011). For example, the judgment that experiment 2 is relevantly similar to experiment 1 involves the judgment that experiment 2 does not introduce any confounding factors that were absent in experiment 1. However, such judgments have to rely on some assumptions about what is relevant and what is irrelevant to the experiment, where these assumptions are often unstated auxiliaries. For example, I may (correctly or incorrectly) tacitly assume that temperature in the lab is irrelevant and hence ignore this variable in my replication attempt.
It is important to recognize that the individuation judgments made in experiments have a high degree of epistemic uncertainty. Specifically, I want to highlight what I call the problem of “conceptual scope,” which arises from the question of how the respective independent and dependent variables are described. Take, for example, the above case in which I play a specific piece by Mozart in a major key at a fast pace. A lot hangs on what I take to be the relevant feature of this stimulus: the fact that it is a piece by Mozart, the fact that it is in a major key, the fact that it is fast, and so on. Depending on how I describe the stimulus, I might have different intuitions about possible confounders to pay attention to. For example, if I take the fact that a piece is by Mozart as the relevant feature of the independent variable, I might control for familiarity with Mozart. If I take the relevant feature to be the key, I might control for mood.
Crucially, even though scientists make decisions on the basis of (implicit or explicit) assumptions about conceptual scope, their epistemic situation is typically such that they do not know what is the “correct” scope. This highlights a feature of psychological experiments that is rarely discussed in the literature about the replication crisis, that is, the deep epistemic uncertainty and conceptual openness of much research. This concerns both initial and replication studies.
3. Is the Dichotomy between Direct and Less Direct Replication Pragmatically Useful?
While the above serves as a warning against a naive understanding of direct replication, it might be objected that such replications can nonetheless make unique epistemic contributions. This is indeed claimed by advocates of both direct and less direct (“conceptual”) replication alike. I will now evaluate claims that have aligned the distinction between “direct” and “conceptual” with some relevant distinctions in scientific practice, such as that between the aim of establishing the existence of a phenomenon and that of generalizing from such an existence claim on the one and that between reliability and validity on the other. I will argue that these distinctions are heuristically useful but on closer inspection bring to the fore exactly the epistemological issues just discussed.
3.1. Existence versus Generalizability
Many scientists take it as given that there cannot be two identical experiments but nonetheless argue that there is significant epistemic merit in trying to get close enough (i.e., to conduct direct replications). In turn, the notion of a direct replication is frequently contrasted with that of a conceptual replication. In a nutshell, direct replications essentially try to redo “the same” experiment (or at least something very close), whereas the conceptual replications try to operationalize the same question or concept/effect in a different way. The advantage of direct replication, as viewed by its advocates, is that by being able to redo an experiment faithfully and to create the same effect, one can show that the effect was real: “Exact and very close replications establish the basic existence and stability of a phenomenon by falsifying the (null) hypothesis that observations simply reflect random noise” (LeBel et al. Reference LeBel, Berger, Campbell and Loving2017, 256).
Advocates of conceptual replication do not deny this advantage of close replications but hold that we want more than to establish that a given effect—created under very specific experimental conditions—is real. We want to know whether our findings about it can be generalized: “When the goal is generalization, we argue that ‘imperfect’ conceptual replications that stretch the domain of research may be more useful” (Lynch et al. Reference Lynch, Bradlow, Huber and Lehmann2015, 334). From a strictly Popperian perspective, the idea that nonfalsification of the hypothesis of random error can provide proof of stability and existence is questionable, of course. But even if we abandon Popperian ideology here and take the falsification of H0 (that the initial effect was due to random error) to point to the truth of H1 (that there is a stable effect), the question is how to describe the effect. In other words, when claiming to have confirmed an effect, we have to say what kind of effect it is. And there we face the following dilemma:
a) Either we describe the effect as highly specific to very local experimental circumstances, involving the choice of a specific independent variable, delivered in a specific way, and so on.
b) Or we describe it in slightly broader terms, for example, as a Mozart effect.
Advocates of direct replication might indeed endorse something like a, thereby exhibiting the kind of caution that motivated early operationists, in that no claim is made beyond the confines of a specific experiment. If, however, psychologists endorsed a description such as b, they would immediately run into the question of conceptual scope, that is, the question under what description the independent variable can be said to have caused an effect. I argue that no amount of direct replication can answer this question, and hence, even if direct replication can confirm the existence of an effect, it cannot say what kind of effect. By asserting this, I am not saying that it is never useful to do a direct replication. My claim is merely that it will tell us little about the subject matter. More pointedly, direct replication can (perhaps) provide evidence for the existence of something, but it cannot say existence of what. Zwaan gets at a similar point when he states that replication studies “tell us about the reliability of those findings. They don’t tell us much about their validity” (Reference Zwaan2013).
In a related vein, I argue that direct replication, with its narrow focus on ruling out random error, is epistemically not very productive, because it has nothing to say about systematic error. Systematic error arises if one erroneously attributes an effect to a specific feature of the experiment, when it is in fact due to another feature of the experiment. This can include, but is not limited to, the above-mentioned problem of conceptual scope. Fiedler, Kutzner, and Krueger (Reference Fiedler, Kutzner and Krueger2012) make a similar point when they argue that a narrow focus on falsification (with the aim of avoiding false positives) can be detrimental to the research process. Differently put, by privileging direct replication, we are not in a position to inquire about the kind of effect in question. This question, I argue, is best addressed by paying close attention to the possibility of systematic error and, hence, by doing conceptual work. In other words, experimentally probing into systematic errors of conceptual scope is a valuable and productive part of the research process as it enables scientists to gradually explore what kind of effect (if any) they are looking at.Footnote 3
3.2. Generality
I have argued that (a) scientists typically produce effects under a description and (b) that it can be epistemically productive to probe the scope of the description and to investigate the possibility of systematic error in experiments that draw on such descriptions. It is epistemically productive because it forces scientists to explore the nature and boundaries of the effect they are investigating. With this I have argued against a narrow focus on direct replication, and I have cautioned against overstating the epistemic merits of such replication. But when we are concerned with effects under a description, we are confronted with questions about the adequacy of the description. It is this question that advocates of conceptual replication claim to be able to address when they emphasize that their approach can deliver generality (over mere existence).
We have to distinguish between two notions of generality, namely, (a) what kinds of descriptions can one generalize or infer to within the experiment, and (b) does the effect in question hold outside the lab (see Feest and Steinle Reference Feest, Steinle and Humphreys2016)? These types of generality are also sometimes referred to as internal versus external validity, respectively (Campbell and Stanley Reference Campbell and Stanley1966; Guala Reference Guala and Mäki2012), where the former refers to the quality of inferences within an experiment and the latter refers to the quality of inferences from a lab to the world. The notion of generalizability raises questions about both kinds of validity. My focus here will be on internal validity, that is, with the question of whether the effect generated in an experiment really exists as described by the scientist.Footnote 4
Internal validity can fail to hold because of epistemic uncertainties regarding confounding variables both internal and external to experimental subjects. For example, prior musical training might make a difference to how one responds to Mozart music, but the experimenters may not have taken this into consideration in their design. But internal validity can also fail to hold by virtue of what I have referred to as the problem of conceptual scope (e.g., we may refer to the effect as a Mozart effect when it is in fact a major-key effect). Effectively, when I treat a major-key effect as a Mozart effect, I have misidentified the relevant causal feature of the stimulus. In turn, this means that I will neglect to control for a major/minor key, as I will regard this as irrelevant, which can result in systematic errors. In both cases, scientists can go wrong in their individuation judgment. What is at stake is not whether there is an effect but what kind of effect it is. Now, given that those kinds of problems can occur, we turn to the question of whether “conceptual replication” has an answer. I will now argue that it does not.
To explain this, let me return to the above characterization of conceptual replication, according to which such replication consists in repeating an experiment, using different operationalizations of the same construct. For example, a conceptual replication of an experiment about the Mozart effect might operationalize the concept Mozart effect differently by using a different piece of Mozart music or a different measure of spatial reasoning. But there is a major caveat here: if I want to compare the results of two experiments that operationalized the same construct differently, I already have to presuppose that both operationalizations in fact have the same conceptual scope, that is, that they in fact individuate the same effect. But this would be begging the question, since after all—given the epistemic uncertainty and conceptual openness highlighted above—that is precisely what is at issue. Differently put, experiment 2 might or might not achieve the same result as experiment 1, but the reason for this would be underdetermined by the experimental data. Thus, the problem of conceptual scope prevents us from being able to say whether we have succeeded in our conceptual replication.Footnote 5
Given the uncertainties as to whether one has in fact succeeded in conceptually replicating a given experiment, I am wary of the language of replication here. If anything, I would argue that the method in question should be regarded as a research strategy that is aimed at helping to demarcate and explore the very subject matter under investigation. But as I will argue now, this is perhaps better described as exploration not replication.
4. Putting Replication in Its Proper Place
The conclusion of the previous paragraphs seems pretty bleak: direct replication is either extremely narrow in what it can deliver, or it runs into the joint problems of confounders and conceptual scope. Conceptual replication, however, cannot come to the rescue, because it is either begging the question or runs into similar problems. Should we then throw up our hands and conclude that the crisis of replication is much more severe than we previously thought? This would only follow, however, if replication were in fact as central to experimental practice as it is sometimes taken to be. I claim that it is not. My argument has three parts. The first part holds that exploring (the possibility of) systematic errors is an important part of the investigative process, which is not well described as replication. Second, if we take seriously this process of exploring and delineating the relevant phenomena, we find that there is indeed a great deal of uncertainty in psychological research, but this, in and of itself, does not necessarily constitute a crisis. Finally, while it is fair to say that there is a crisis of confidence in current psychology, it is not well described as a replication crisis.
Let me begin with the first point. I have argued that direct replication (even when it is successful) is of limited value because it can at most rule out random error but fails to address systematic error. But if we appreciate (as I have argued we should) that any kind of replication inevitably involves individuation judgments, it is obvious that there is always a danger of systematic error because there is always a possibility of overlooked confounding variables, often stemming from unstated auxiliaries. One important class of confounders follows from what I have referred to as the problem of conceptual scope, that is, the difficulty of correctly describing both the independent variable responsible for a given effect and the dependent variable.Footnote 6 Epistemically productive experimental work, I claim, therefore needs to focus on systematic errors, specifically those brought about by unstated auxiliary assumptions pertaining to the conceptual scope of the effect in question.
Indeed, if we look at the story of the Mozart effect, we find that this is exactly what happened. This example also nicely illustrates my claim about the conceptual openness and epistemic uncertainty in many areas of experimental psychology. The Mozart effect was first posited by Rauscher, Shaw, and Ky (Reference Rauscher, Shaw and Ky1993). It can now be regarded as largely debunked. While it is true that several people tried (and failed) to replicate the effect (e.g., Newman et al. Reference Newman, Rosenbach, Burns, Latimer, Matocha and Vogt1995; Steele Reference Steele2000), it is important to look at the details here. It is not the case that the effect was simply abandoned for lack of replicability. Rather, when we look at the back and forth between Rauscher and her critics, we find that the discussion turned on the choices and interpretations of independent and dependent variables. In this vein, Newman et al. (Reference Newman, Rosenbach, Burns, Latimer, Matocha and Vogt1995) and Steele (Reference Steele2000) used different dependent variables, prompting Rauscher to concede that her effect was more narrowly confined to the kind of spatial reasoning measured by the Stanford-Binet test. I suggest that we interpret this case as one in which Rauscher was forced to confront (and retract) an unstated auxiliary assumption of her initial study, namely, that the spatial reasoning subtest of the Stanford-Binet (which she had used as her dependent variable) was representative of spatial reasoning more generally. Likewise, her choice of the Mozart’s Sonata for Two Pianos in D major as the independent variable was put under considerable pressure by critics, who suggested that the relevant feature of the independent variable was not that it was a piece by Mozart but that it was upbeat and put subjects in a good mood (Chabris Reference Chabris1999). My point here is that the debates surrounding the Mozart effect are best described as conceptual work, exploring consequences of possible errors that might have arisen from the problem of conceptual scope. At issue, I claim, was not primarily whether Rauscher really found an effect but rather what was the scope of the effect.
I argue that this is a typical case. Rather than, or in addition to, attempting to conduct direct replications of previous experiments, researchers critically probed some hidden assumptions built into the design and interpretation of the initial experiment. My point here is both descriptive and normative. Thus, I argue that this is a productive way to proceed. However, I claim that it is not well described as (direct or conceptual) replication. Rather, what we see here is a case in which scientists explore the empirical contours of a purported effect in the face of a high degree of epistemic uncertainty and conceptual openness, and this is precisely why the case is not well described as employing conceptual replication. The reason for this is quite simple: for a conceptual replication to occur, one needs to already be in possession of some well-formed concepts, such that they can be operationalized in different ways. It also presupposes that in general the domain is well understood, such that operationalizations can be implemented and confounding variables can be controlled. But this completely misses the point that researchers often investigate effects precisely because they do not have a good understanding (and hence concept) of what it is.
Therefore, I argue that while direct replication can only contribute a very small part to the research process, conceptual replication cannot make up for the shortcomings of direct replication. Instead, productive research should (and frequently does) proceed by exploring, and experimentally testing, hypotheses about possible systematic errors in experiments. Such research, I suggest, can contribute to conceptual development by helping to explore and fine-tune the shape and scope of proposed or existing concepts. The fact that this is riddled with problems does not in and of itself constitute a crisis, let alone a replication crisis.
5. Conclusion
The upshot of the above is that when we talk about the importance of replication, we need to be clear on what we mean by replication and why it is so important, precisely.Footnote 7 Looking specifically at the possible role of replication in experimental research in psychology, I have argued that if by replication we mean either “direct” or “conceptual” replication, we need to first of all be clear that direct replications are not nonconceptual. I then turned to some alleged epistemic merits of direct replication, for example, that they can establish the existence of effects or the reliability of procedures that detect effects. I argued that insofar as such replications involve concepts, they run (among other things) into the problem of conceptual scope, that is, the difficulty of determining, on the basis of independent and dependent variables of experiments, what precisely is the scope of the effect one is trying to replicate. I highlighted that this is a real and pernicious problem in experimental research in psychology, because of the high degree of epistemic uncertainty and conceptual openness of many fields of research.
While my emphasis of the conceptual nature of replication may suggest that I would be more favorably inclined toward conceptual replication, I have argued that conceptual replication runs into similar problems, for similar reasons: the very judgment that one has successfully performed a conceptual replication of a previous experiment presupposes what is ultimately the aim of the research, namely, to arrive at a robust understanding of the relevant area of research. Thus, I argue that since conceptual replication presupposes a relatively good grasp of the relevant concepts, it is begging the question, and I suggest instead that researchers (should) engage in a process of specifically investigating possible systematic errors in original studies as a means for developing the relevant concepts. This process is not best described as one of replication, however. Summing up, then, I conclude that replications are less useful and important than is widely assumed—at least in the kind of psychological research I have focused on in this article.
Now, in conclusion let me return to the notion of a crisis in psychology as it is currently discussed in the literature. Obviously, I do not mean to deny that there is a crisis of confidence in (social) psychology (Earp and Trafimow Reference Earp and Trafimow2015) as well as in other areas of study. However, using the analysis provided in this article, I argue that this crisis is not well described as a crisis of replication. Rather, it seems to be to a large degree a crisis that turns on questionable research practices with regard to the use of statistical methods in psychology (see Gelman and Loken Reference Gelman and Loken2014). Let me emphasize that I am not denying that failure to replicate can point toward questionable research practices, especially regarding statistical methods. However, given the conceptual openness characteristic of much psychological research, failure to replicate is also to be expected. Thus, I suggest that a broader focus on the notion of replication provides us with a deeper appreciation of the conceptual dynamics characteristic of experimental practice in psychology.
