In their target article, Zwaan et al. make a spirited and persuasive case for an increased emphasis on replication of psychological studies. Yet I want to challenge their conclusion that “there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science” (Abstract).
The word “experiment” appears 52 times in the target article, and it appears that the authors are mostly concerned with theory-testing experiments (often using college students or recruited Web samples). Direct replications are almost always feasible for such studies. They are not without cost, including the opportunity cost of replicating an old result rather than testing a new result. But the costs are not great, which is perhaps why Zwaan et al. mention only one type of cost – “reputational costs.”
But nonexperimental and/or field methods make up a substantial minority of published psychological research (Cortina et al. Reference Cortina, Aguinis and DeShon2017; Lipsey & Wilson Reference Lipsey and Wilson1993). For such studies, both words in the phrase “direct replication” aspirations are problematic.
The notion of a “direct” replication is problematic in field studies, because standardization is so difficult to achieve – even if it were desirable. Different studies of the same question occur in different locations, in different years, with different implementation details, client caseloads, resources, and political environments. A cautionary tale is provided by a National Institute of Justice effort to replicate an influential experiment on the effect of arrests in domestic assault cases, forcing analysts to search for covariates to account for dramatically disparate results across different cities (see Berk et al. Reference Berk, Campbell, Klap and Western1992). Such situations show that such categories as “construct validity,” “statistical conclusion validity,” and “external validity,” although conceptually distinct, are very fuzzy in practice.
In any case, field studies and archival analyses tend to be much more costly and much less feasible to replicate. Indeed, replication is impossible (or nearly so) for empirical studies based on real-world events, such as homicide rates, gun purchases, presidential elections, terrorist events, and responses to natural disasters, or changes in law. Moreover, maximum sample sizes are often constrained by circumstances beyond the control of the researcher.
Despite all of these problems, field research is indispensable for addressing many important questions. When we cannot replicate such studies, are there other ways of enhancing their statistical reliability and validity? Some areas of physics have had to confront a similar dilemma. Major experiments in particle physics can be conducted at only a few facilities (sometimes only one), at enormous expense. Some questions in cosmology can only be answered at present using a small set of observations (e.g., of supernovae) that, although not rare in the universe, are very costly and difficult to detect.
To make the best use of such data, and to minimize the risk of p-hacking and other forms of biased inference, many physicists routinely perturb their data (e.g., by adding noise and/or bias, scrambling data labels, or “salting” the data with fake events [see Klein & Roodman Reference Klein and Roodman2005]). Blinding methods are increasingly used in forensic science for similar reasons – the conditions under study were historically unique and cannot be replicated to compensate for analyst bias (Robertson & Kesselheim Reference Robertson and Kesselheim2016). Recently, physicist Saul Perlmutter and I have examined such methods and their suitability for psychological science (MacCoun & Perlmutter Reference MacCoun and Perlmutter2015; Reference MacCoun, Perlmutter, Lilienfeld and Waldman2017). Different methods blind different features of the results (e.g., point estimates, effect sizes, statistical significance). More research is needed to examine their suitability and effectiveness for the behavioral sciences. But it seems likely that blind analysis can make results more credible and reduce some of the biases that reduce the replicability of research findings.
In their target article, Zwaan et al. make a spirited and persuasive case for an increased emphasis on replication of psychological studies. Yet I want to challenge their conclusion that “there are no theoretical or statistical obstacles to making direct replication a routine aspect of psychological science” (Abstract).
The word “experiment” appears 52 times in the target article, and it appears that the authors are mostly concerned with theory-testing experiments (often using college students or recruited Web samples). Direct replications are almost always feasible for such studies. They are not without cost, including the opportunity cost of replicating an old result rather than testing a new result. But the costs are not great, which is perhaps why Zwaan et al. mention only one type of cost – “reputational costs.”
But nonexperimental and/or field methods make up a substantial minority of published psychological research (Cortina et al. Reference Cortina, Aguinis and DeShon2017; Lipsey & Wilson Reference Lipsey and Wilson1993). For such studies, both words in the phrase “direct replication” aspirations are problematic.
The notion of a “direct” replication is problematic in field studies, because standardization is so difficult to achieve – even if it were desirable. Different studies of the same question occur in different locations, in different years, with different implementation details, client caseloads, resources, and political environments. A cautionary tale is provided by a National Institute of Justice effort to replicate an influential experiment on the effect of arrests in domestic assault cases, forcing analysts to search for covariates to account for dramatically disparate results across different cities (see Berk et al. Reference Berk, Campbell, Klap and Western1992). Such situations show that such categories as “construct validity,” “statistical conclusion validity,” and “external validity,” although conceptually distinct, are very fuzzy in practice.
In any case, field studies and archival analyses tend to be much more costly and much less feasible to replicate. Indeed, replication is impossible (or nearly so) for empirical studies based on real-world events, such as homicide rates, gun purchases, presidential elections, terrorist events, and responses to natural disasters, or changes in law. Moreover, maximum sample sizes are often constrained by circumstances beyond the control of the researcher.
Despite all of these problems, field research is indispensable for addressing many important questions. When we cannot replicate such studies, are there other ways of enhancing their statistical reliability and validity? Some areas of physics have had to confront a similar dilemma. Major experiments in particle physics can be conducted at only a few facilities (sometimes only one), at enormous expense. Some questions in cosmology can only be answered at present using a small set of observations (e.g., of supernovae) that, although not rare in the universe, are very costly and difficult to detect.
To make the best use of such data, and to minimize the risk of p-hacking and other forms of biased inference, many physicists routinely perturb their data (e.g., by adding noise and/or bias, scrambling data labels, or “salting” the data with fake events [see Klein & Roodman Reference Klein and Roodman2005]). Blinding methods are increasingly used in forensic science for similar reasons – the conditions under study were historically unique and cannot be replicated to compensate for analyst bias (Robertson & Kesselheim Reference Robertson and Kesselheim2016). Recently, physicist Saul Perlmutter and I have examined such methods and their suitability for psychological science (MacCoun & Perlmutter Reference MacCoun and Perlmutter2015; Reference MacCoun, Perlmutter, Lilienfeld and Waldman2017). Different methods blind different features of the results (e.g., point estimates, effect sizes, statistical significance). More research is needed to examine their suitability and effectiveness for the behavioral sciences. But it seems likely that blind analysis can make results more credible and reduce some of the biases that reduce the replicability of research findings.