In an excellent article in the Journal of Politics, Cyrus Samii (Reference Samii2016) demonstrates that our capacity to generalize from results in sample data to a large population is far more limited than is often thought, even when we employ regression methods that are generally regarded as especially well suited to exactly that end (and even assuming valid and reliable measures). Although this specific claim will probably surprise many researchers, his larger point is more in the nature of a reminder of what most scholars already believe: that, to be secure, knowledge must be grounded in the incremental work of numerous scholars rather than in dramatic single studies. This view is such a staple of textbooks and lectures in research methods that it could be regarded as a truism.
Almost as much of a truism, however, is that scholarly activity does not reflect this understanding. Far too seldom do we undertake the vital work of testing whether notable findings in one domain hold up for other actors, in other circumstances, or when measured or analyzed in different ways. There are probably many reasons for this professional shortcoming that JLC has little capacity to affect. But there is one undoubted factor that JLC can help address: journals overvalue novelty, because novel findings and arguments generate citations and provide the material for exciting press releases.
Journals’ emphasis on novelty means that a replication study has almost no chance of being published unless the replication fails and the failure can be presented as persuasive evidence for an alternative theory. This distorts incentives for scholars, with the result that important claims are left in a state of limbo or, worse, widely treated as more definitive than they really are.
To do its part in combating this problem, as of today JLC is opening its pages to most types of replications, successful or unsuccessful. There is one exception: although successful direct replications, employing the same data and models as the original study, are valuable, we lack the journal space to publish them. But we will welcome any study, quantitative or qualitative, that asks whether findings in one domain extend to another or can be replicated using different measures or models. These papers will be evaluated for publication on the basis of (a) the importance of the work being replicated, (b) the clarity and logic of the reasons given for questioning whether the findings of the original study hold in other circumstances or when analyzed in a different way, and (c) the quality of the replication analyses. There will be no requirement that the theoretical framework or findings of the replication study contradict the theoretical framework or findings of the study being replicated.
Naturally, replication studies will typically be briefer than conventional articles and may follow a slightly different structure. However, there will be no difference between these and other articles in how they appear in the journal. We hope scholars will take us up on this invitation to conduct more of these studies that are so crucial to our communal search for knowledge.
