Exact? Direct? Conceptual? How we label a replication attempt depends on its similarity to the original study (Zwaan et al.). But “similarity” in whose view and with respect to which parameters? When a replication study reproduces the original finding, the distinction between direct and conceptual is less fraught – any discrepancies between the original and replication procedures provide evidence for generality. But, when a replication result differs from the original, any procedural changes become candidate explanations for the discrepancy; whether or not it counts as “direct” depends on how you interpret those differences in procedures.
Kahneman (Reference Kahneman2014) hoped to eliminate this ambiguity by encouraging researchers conducting replication studies to consult with the original authors about the appropriate procedures. The Registered Replication Reports developed at Perspectives on Psychological Science and now hosted at Advances in Methods and Practices in Psychological Science consult the original authors extensively to verify the accuracy of the replication protocol. ManyLabs projects attempt to do so, as well, as do the new Preregistered Direct Replications in Psychological Science. Although such contacts are a professional courtesy that can and often do improve the precision of a replication protocol, they are not always possible (i.e., the original researcher may be deceased or otherwise unresponsive). Furthermore, in principle they should not be necessary –published articles are the currency of our field, and they should stand alone as a statement of what is needed to reproduce a finding.
Much as preregistration makes clear whether an analysis was planned in advance or exploratory, an original article can make clear whether a subsequent replication is direct or conceptual if it explicitly defines constraints on generality (a COG statement; see Simons et al. [Reference Simons, Shoda and Lindsay2017] for details). Most psychology papers generalize from the tested samples – of participants, stimuli, testing contexts – to broader populations (e.g., all college students, any emotional scene, any testing cubicle). All papers should explicitly define these target populations and indicate which presumed generalizations are derived from empirical evidence, which are theoretical predictions, and which are hunches. A COG statement defines the scope of the original finding, clarifies which aspects of the samples must be reproduced in a direct replication, and identifies claims of generality that need further testing in conceptual replications. It indicates those aspects of the original study its authors believe to be essential, thereby defining the conditions that the authors believe would constitute a direct replication.
Current publishing practices incentivize making the boldest and most general claims possible: A study testing undergraduate psychology students is generalized to all adults; a laboratory effect in a controlled, computer-based task is generalized to real-world decision making; interactions with a trained confederate are generalized to romantic relationships; studies with WEIRD samples are generalized to non-WEIRD populations (Henrich et al. Reference Henrich, Heine and Norenzayan2010). Such broad conclusions are inherently more interesting and provocative, but rarely are justified and often are not justifiable. When constraints on generality are left unstated, authors implicitly encourage readers (including editors and reviewers) to generalize broadly, even if they would explicitly discourage readers from doing so.
Without a COG statement, both original authors and those conducting replications can dispute the meaning of a replication study, assigning the burden of proof to “do it right” to each other. An original author might assume narrow generality for their finding across contexts, so they place the burden of proof on the replicator to ensure that the replication mimics all aspects of the original context. They can dismiss unsuccessful replications as fatally flawed on account of changes in context that, in hindsight, they believe are important. In contrast, a replicator might assume maximal generality in the absence of stated constraints. When an original author posits a change in context as the reason for an unsuccessful replication, the replicating author will expect the original author to demonstrate that it was not a false positive (e.g., by making the effect come and go with changes to a moderator).
Adding a COG statement overcomes the ambiguity of classifying replications as direct or conceptual after the fact because it specifies the target populations for the original claim, allowing anyone to draw from those same populations. It bypasses the need to consult the authors of a publication to identify those constraints, allowing the published finding to serve as the unit of analysis. It also focuses our collective research enterprise on what should be our goal – not just determining whether or not effects exist, but identifying boundary conditions and mechanisms. Direct replications will test the reliability of those procedures thought to be essential, narrowing the purported scope of an original effect or perhaps demonstrating conditions under which it does not occur. Conceptual replications will test whether the proposed constraints on generality are accurate, leading to a more refined understanding of the robustness of the effect. A systematic program of research would evaluate how the size of an effect varies as a function of those constraints; how large is the effect within the narrowest, idiosyncratic confines of the original study procedures and how quickly and in what ways does it change when the restraints are loosened.
Exact? Direct? Conceptual? How we label a replication attempt depends on its similarity to the original study (Zwaan et al.). But “similarity” in whose view and with respect to which parameters? When a replication study reproduces the original finding, the distinction between direct and conceptual is less fraught – any discrepancies between the original and replication procedures provide evidence for generality. But, when a replication result differs from the original, any procedural changes become candidate explanations for the discrepancy; whether or not it counts as “direct” depends on how you interpret those differences in procedures.
Kahneman (Reference Kahneman2014) hoped to eliminate this ambiguity by encouraging researchers conducting replication studies to consult with the original authors about the appropriate procedures. The Registered Replication Reports developed at Perspectives on Psychological Science and now hosted at Advances in Methods and Practices in Psychological Science consult the original authors extensively to verify the accuracy of the replication protocol. ManyLabs projects attempt to do so, as well, as do the new Preregistered Direct Replications in Psychological Science. Although such contacts are a professional courtesy that can and often do improve the precision of a replication protocol, they are not always possible (i.e., the original researcher may be deceased or otherwise unresponsive). Furthermore, in principle they should not be necessary –published articles are the currency of our field, and they should stand alone as a statement of what is needed to reproduce a finding.
Much as preregistration makes clear whether an analysis was planned in advance or exploratory, an original article can make clear whether a subsequent replication is direct or conceptual if it explicitly defines constraints on generality (a COG statement; see Simons et al. [Reference Simons, Shoda and Lindsay2017] for details). Most psychology papers generalize from the tested samples – of participants, stimuli, testing contexts – to broader populations (e.g., all college students, any emotional scene, any testing cubicle). All papers should explicitly define these target populations and indicate which presumed generalizations are derived from empirical evidence, which are theoretical predictions, and which are hunches. A COG statement defines the scope of the original finding, clarifies which aspects of the samples must be reproduced in a direct replication, and identifies claims of generality that need further testing in conceptual replications. It indicates those aspects of the original study its authors believe to be essential, thereby defining the conditions that the authors believe would constitute a direct replication.
Current publishing practices incentivize making the boldest and most general claims possible: A study testing undergraduate psychology students is generalized to all adults; a laboratory effect in a controlled, computer-based task is generalized to real-world decision making; interactions with a trained confederate are generalized to romantic relationships; studies with WEIRD samples are generalized to non-WEIRD populations (Henrich et al. Reference Henrich, Heine and Norenzayan2010). Such broad conclusions are inherently more interesting and provocative, but rarely are justified and often are not justifiable. When constraints on generality are left unstated, authors implicitly encourage readers (including editors and reviewers) to generalize broadly, even if they would explicitly discourage readers from doing so.
Without a COG statement, both original authors and those conducting replications can dispute the meaning of a replication study, assigning the burden of proof to “do it right” to each other. An original author might assume narrow generality for their finding across contexts, so they place the burden of proof on the replicator to ensure that the replication mimics all aspects of the original context. They can dismiss unsuccessful replications as fatally flawed on account of changes in context that, in hindsight, they believe are important. In contrast, a replicator might assume maximal generality in the absence of stated constraints. When an original author posits a change in context as the reason for an unsuccessful replication, the replicating author will expect the original author to demonstrate that it was not a false positive (e.g., by making the effect come and go with changes to a moderator).
Adding a COG statement overcomes the ambiguity of classifying replications as direct or conceptual after the fact because it specifies the target populations for the original claim, allowing anyone to draw from those same populations. It bypasses the need to consult the authors of a publication to identify those constraints, allowing the published finding to serve as the unit of analysis. It also focuses our collective research enterprise on what should be our goal – not just determining whether or not effects exist, but identifying boundary conditions and mechanisms. Direct replications will test the reliability of those procedures thought to be essential, narrowing the purported scope of an original effect or perhaps demonstrating conditions under which it does not occur. Conceptual replications will test whether the proposed constraints on generality are accurate, leading to a more refined understanding of the robustness of the effect. A systematic program of research would evaluate how the size of an effect varies as a function of those constraints; how large is the effect within the narrowest, idiosyncratic confines of the original study procedures and how quickly and in what ways does it change when the restraints are loosened.