Psychological science has shifted from observational accounts to technology- and statistically-mediated research. Wundt's reaction time experiments describe everyday experience partly because he used simple equipment: a dropped ball-bearing and a telegraph key (Wundt, Reference Wundt1883). Experiments that rely on more complex technologies or analyses require methodological decisions. Because many such decisions are embedded in our research pipelines, our claims are not about directly observed reality. Our data are inextricably connected to the methods used in its collection and analysis. The generalizability (née replicability) crisis stems from the intrinsic trade-off between experimental complexity and generalizability.

For a simple system like Wundt's telegraph key, a small change in input produces a small change in measured output. Such results ought to generalize because we can model noise in the system as Gaussian. But multistep research pipelines are complex systems and a trivial change in inputs can produce huge output changes. The complex pipeline has many hyperparameters and noise in these hyperparameters accumulates with each step of analysis, producing non-Gaussian noise in the output. In complex systems, a more reasonable prior belief is that findings will not generalize.

For example, it is popular, and wrong, to imply that functional magnetic resonance imaging (fMRI) measures brain activity. fMRI measures fluctuations in magnetic fields. These fluctuations are correlated with ionic charge exchange, which in turn is correlated with the deoxygenation of blood, which in turn correlates with brain activity. Because this chain of correlations is so long, it is possible to reject certain claims in the fMRI literature as unrealistically strong (Vul, Harris, Winkielman, & Pashler, Reference Vul, Harris, Winkielman and Pashler2009). Yet this skepticism of overly strong results also depends on a long chain of correlation: the literature establishing the test-retest reliability of fMRI. Estimates of reliability themselves vary widely depending on the analysis pipeline and experimental parameters. This does not mean that fMRI tells us nothing about the brain, it simply means that the kind of things that fMRI tells us about the brain are interpretable only within a rich context of historical results and consolidated knowledge.

Thus, psychological scientists' linguistic claims about “the mind” are actually claims about the models and methods of their own research contexts. As Yarkoni points out, failure to generalize is a predictable consequence of the gap between the linguistic claims of psychological science and quantitative models. To borrow a statistical term, our discipline has “overfit” its theories to its methods. We've become highly specialized evaluators of our chosen domain. But this specialization precludes a deep understanding of competing research practices; when specialists review others' work, they are only qualified to evaluate surface-level linguistic claims. We are more skillful at predicting the tendencies baked into our own methods, subjects, and analyses than we are at predicting the behavior of people and animals. As we gather more diverse data (Henrich, Heine, & Norenzayan, Reference Henrich, Heine and Norenzayan2010), we begin to perceive how much our data mirror our own perspectives.

However, just as psychological science is fundamentally interwoven with its own research methods, public scientific debates are best understood in sociohistorical context. All hypotheses struggle for survival. Successful claims must survive a selection process that balances the claims' fertility and lifespan. As in natural selection (Pianka, Reference Pianka1970), scientific “parents” may pursue different strategies to promote the survival of their research “offspring.” Yarkoni implies that psychological science is pursuing what is essentially an r-strategy: creating simplistic claims that breed quickly but die off under scrutiny. These claims thrive when audiences are sophisticated enough to engage with linguistic claims, but lack the detailed knowledge required to evaluate the quantitative evidence. Yarkoni argues for a K-strategy: constructing bigger, tougher, longer-lived claims. The replicability community, Yarkoni says, wants durable science built on robust methods.

Ironically, by labeling the entire class of methodological concerns a “crisis,” methodologists have invented a broad, qualitative, linguistic claim that is easy to spread. Instead of pedantic discussions of overfit methods, we can point to a lack of rigor with a single phrase. What a brilliant rhetorical move! At a stroke, methodologists can build a broad, interdisciplinary audience with the attention-grabbing, quick-spreading “generalizability crisis,” retain their authority as no-nonsense K-strategists, and create a better climate for metascience.

There is ample precedent in the history of science for leveraging manufactured conflict to spread scientific ideas. Even losers reap rewards. We remember Camillo Golgi in the same breath as Ramon y Cajal, even though the former's model of the synapse was flatly incorrect (Glickstein, Reference Glickstein2006). More commonly, no one is precisely wrong: In the early 2010s, there was a sometimes contentious debate over whether recollective memory could be clearly distinguished from strong familiarity. This debate fizzled out without clear resolution, yet the conflict produced some fruitful new methods (Koen, Aly, Wang, & Yonelinas, Reference Koen, Aly, Wang and Yonelinas2013). Even methodological vagueness itself can be a wellspring for debate. Meta-analyses show that the degree of learning transferred between tasks is closely related to the degree of task similarity (Giovanni Sala, Deniz Aksayli, Semir Tatlidil, Tatsumi, & Gondo, Reference Giovanni, Deniz Aksayli, Semir Tatlidil, Tatsumi, Gondo and Gobet2019). Thus, the “distance” between any pair of tasks could be empirically measured. Yet rather than developing a consensus measure for task similarity, the field actively debates whether a particular finding represents “near” or “far” transfer (Redick et al., Reference Redick, Shipstead, Harrison, Hicks, Fried, Hambrick and Engle2013). Ambiguous measures give investigators more latitude in building their brand of credulity or skepticism.

Perhaps the generalizability crisis was inevitable. The mind is a system more complex than even the most elaborate research pipeline. All evidence for general principles should be met with extreme skepticism. Reliable quantitative tools can trick us into overgeneralizing when we confuse the tool for the phenomenon. Yet those who can invent more reliable tools have a clear pathway to greater authority within the discipline. As psychological science selects for better tools, it draws practitioners increasingly from the ranks of technologists and statisticians. Our discourse must be tailored to this new social reality. Our audience is us. We are eager for stories where hard-nosed, skeptics skewer weak findings. The move to metascience is a logical endpoint for a discipline studying itself.