No CrossRef data available.
Article contents
Three ways to make replication mainstream
Published online by Cambridge University Press: 27 July 2018
Abstract
Zwaan et al. argue convincingly that replication needs to be more mainstream. Here, I suggest three practices for achieving that goal: Incremental Replications, which are built into each experiment in a series of experiments; Reciprocal Replications, which are reciprocal arrangements of co-replications across labs; and Didactic Replications, which are replications used for training.
Information
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © Cambridge University Press 2018
References
Chawla, D. S. (2017, April 7) Online platform aims to facilitate replication studies. The Scientist. Available at: https://www.the-scientist.com/?articles.view/articleNo/49161/title/Online-Platform-Aims-to-Facilitate-Replication-Studies/.Google Scholar
Foss, D. J. & Blank, M. A. (1980) Identifying the speech codes. Cognitive Psychology 12(1):1–31. Available at: https://doi.org/10.1016/0010-0285(80)90002-X.Google Scholar
Foss, D. J. & Gernsbacher, M. A. (1983) Cracking the dual code: Toward a unitary model of phoneme identification. Journal of Verbal Learning and Verbal Behavior 22:609–32. Available at: https://doi.org/10.1016/S0022-5371(83)90365-1.Google Scholar
Garnham, A., Traxler, M., Oakhill, J. & Gernsbacher, M. A. (1996) The locus of implicit causality effects in comprehension. Journal of Memory and Language 35:517–43. Available at: https://doi.org/10.1006/jmla.1996.0028.Google Scholar
Gernsbacher, M. A. (1989) Mechanisms that improve referential access. Cognition 32:99–156. Available at: https://doi.org/10.1016/0010-0277(89)90001-2.Google Scholar
Gernsbacher, M. A. & Hargreaves, D. J. (1988) Accessing sentence participants: The advantage of first mention. Journal of Memory and Language 27:699–717.Google Scholar
Gernsbacher, M. A., Keysar, B., Robertson, R. R. W. & Werner, N. K. (2001a) The role of suppression and enhancement in understanding metaphors. Journal of Memory and Language 45:433–50. Available at: https://doi.org/10.1006/jmla.2000.2782.Google Scholar
Gernsbacher, M. A., Robertson, R. R. W. & Werner, N. K. (2001b) The costs and benefits of meaning. In: On the consequences of meaning selection: Perspectives on resolving lexical ambiguity, ed. Gorfein, D. S., pp. 119–37. APA. Available at: https://doi.org/10.1037/10459-007.Google Scholar
Grahe, J., Brandt, M., IJzerman, H. & Cohoon, J. (2014, February 28) Replication education. Association for Psychological Science: Observer. Available at: https://www.psychologicalscience.org/observer/replication-education.Google Scholar
Traxler, M. J. & Gernsbacher, M. A. (1992) Improving written communication through minimal feedback. Language and Cognitive Processes 7:1–22. Available at: https://doi.org/10.1080/01690969208409378.Google Scholar
Traxler, M. J. & Gernsbacher, M. A. (1993) Improving written communication through perspective-taking. Language and Cognitive Processes 8:311–34. Available at: https://doi.org/10.1080/01690969308406958.Google Scholar
Zwaan, R. A. (2017, May 8) Concurrent replication. Blog post. Available at: https://rolfzwaan.blogspot.nl/2017/05/concurrent-replication.html.Google Scholar
Zwaan et al. provide convincing arguments for the value of replication – and the need to make replication practices mainstream in psychology. However, due most likely to limits of space rather than limits of vision, Zwaan et al. stop short of providing concrete steps researchers can take to make replication mainstream. Here, I suggest three practices researchers can adopt to better incorporate replication into their labs.
Incremental replications
We might think of replication as a practice that occurs in a separate lab with different researchers. Also the other two replication practices I will discuss can occur that way. But our own studies can also benefit from the verification of replication, within our own lab and within the same studies. An obvious step is to conduct exact replications within a series of experiments (e.g., “Experiment 2: Replication. We tested an additional 120 subjects using the same materials and procedures as we used in Experiment 1,” and “Experiment 4: Replication. We tested an additional 120 subjects using the same materials and procedures as we used in [Experiment 3],” Gernsbacher & Hargreaves Reference Gernsbacher and Hargreaves1988, p. 704 and 706).
More parsimoniously, we can conduct, within the same study, what I am calling incremental replications. For example, in a series of experiments investigating how readers understand pronouns, I probed participants immediately before versus after they read a pronoun in one experiment. In another experiment, I again probed participants immediately after they read a pronoun, but in this second experiment I also probed them after they finished reading the entire sentence (Gernsbacher Reference Gernsbacher1989). In this way, across experiments but within the same study, I tried to incrementally replicate each of the previously tested probe points (see also Garnham et al. Reference Garnham, Traxler, Oakhill and Gernsbacher1996, for a similar approach).
As another example, in a series of priming experiments, we manipulated two types of primes in a first experiment and manipulated again one of those two prime types along with a different prime type in a second experiment (Gernsbacher et al. Reference Gernsbacher, Keysar, Robertson and Werner2001a). In another series, we manipulated three prime types in a first experiment and repeated two of the three prime types across other experiments (Gernsbacher et al. Reference Gernsbacher, Robertson, Werner and Gorfein2001b). These incremental replications in within-subject designs also allowed us to assess the stability of our previous results in slightly different contexts (the value of which Zwaan et al. highlight).
Incremental replication is also valuable in between-subject designs. For example, in a series of between-subject treatment experiments, we repeated the baseline condition in subsequent experiments with other subjects (and juxtaposed with other treatments), which allowed us to assess the baseline condition's stability (Traxler & Gernsbacher Reference Traxler and Gernsbacher1992). In another series of experiments, we repeated the control condition in subsequent experiments, which allowed us to assess its stability (Traxler & Gernsbacher Reference Traxler and Gernsbacher1993).
Reciprocal replications
We might also think of replication as a practice that occurs only after a study has been peer reviewed. However, I would rather receive confirmation (or disconfirmation) of the stability of my results earlier rather than later. Zwaan (Reference Zwaan2017) in material left on the target article's editing floor, describes how this can be done.
A research group formulates a hypothesis that they want to test. At the same time, they desire to have some reassurance about the reliability of the finding they expect to obtain. They decide to team up with another research group. They provide this group with a protocol for the experiment, the program and stimuli to run the experiment, and the code for the statistical analysis of the data. The experiment is preregistered. Both groups then each run the experiment and analyze the data independently. The results of both studies are included in the article, along with a meta-analysis of the results.
Zwaan (Reference Zwaan2017) calls this practice as concurrent replication, and my recommendation goes one step further: Make the process reciprocal. Lab A attempts to replicate Lab B's study, while Lab B is doing the same for Lab A's study. Platforms such as StudySwap (deemed “a Craigslist for researchers” by Nosek in Chawla Reference Chawla2017) and Psychological Science Accelerator are ideal for reciprocal replication. Reciprocal replications should take some of the adversarial sting out of traditional replications.
Didactic replications
Lastly, we can make replication more mainstream by embracing it as a training tool. When I was a first-year doctoral student, in one of my first meetings with my advisor, he walked to his filing cabinet, pulled out a recently published article, and suggested I spend my first semester trying to replicate the results. The fact that this didactic activity occurred nearly 40 years ago might be surprising. More surprising might be the fact that the first author of the study my advisor tasked me to replicate was, indeed, my advisor (Foss & Blank Reference Foss and Blank1980).
As it turned out, the previous study only partially replicated (Foss & Gernsbacher Reference Foss and Gernsbacher1983). Learning how to execute an experiment from a published article was an incredibly valuable training experience. (Most likely this is why beginning cooks are encouraged to follow a recipe precisely, before adding their own flourishes.) Deciphering why the previous study only partially replicated was an even more valuable training experience. I believe I learned more about experimental design, stimulus creation, and the myriad other steps involved in doing good science than I would have learned had I joined an in-process study or tried to generate a new study from scratch.
The didactic value of replication has been advocated by others, most notably Grahe and his “Collaborative Replications and Education Project” (Grahe et al. Reference Grahe, Brandt, IJzerman and Cohoon2014). Along with Incremental Replications, which are replications built into each of a series of experiments to attempt to replicate parts of previous experiments, and Reciprocal Replications, which are reciprocal arrangements of co-replication, Didactic Replications can make replication more mainstream.