The argument advanced by Yarkoni (this issue) is, at its core, the problem of provisos, a skeptical dilemma in the philosophy of science posed by Hempel (Reference Hempel1988) and Lange (Reference Lange1993): Claims that require ceteris paribus assumptions (i.e., that hold only “all else being equal”) either overgeneralize, in which case they are false, or they depend on an unbounded set of provisos, in which case they are vacuous. What distinguishes Yarkoni's account from prior formulations is his simultaneous embrace of both horns of the dilemma: Scientists must fully align their verbal and statistical claims, and either abandon generalization or use very large models that favor prediction over explanation.

Building models large enough to match verbal claims might be quite difficult, and the target article says very little on how to go about it. Disturbingly, we are warned that model growth could be unbounded – we should be prepared to add factors “ad infinitum,” with the understanding that “reality is not under any obligation to only manifest sparse causal relationships that researchers find intuitive.” Without qualification, this opens the door to two more skeptical challenges: underdetermination and the problem of unconceived alternatives. If the true causal structure of reality is complex and unbounded, then any empirical finding could be caused by more than one independent mechanism, and all scientific theory would be underdetermined by the available evidence (Quine, Reference Quine1951). Worse, if the nature of causality is inaccessible to human intuition, then even if we cobble together an instrumentally useful theoretical framework, we should expect it to be undermined by alternatives that we could not possibly conceive of (Stanford, Reference Stanford2006).

Following these skeptical implications, it seems to us that the true antagonist of Yarkoni's story is not the unfortunate tendency of social scientists to make sloppy verbal generalizations, but rather the complex causal structure of reality itself. Better alignment between verbal and statistical claims may make the problems of causal complexity easier to see, but it will not do much to solve them. Where Yarkoni asks why social scientists are so slapdash in their verbal treatment of statistics, we would ask instead: Are the institutions of social science competent to manage the challenges posed by the causal complexity of social phenomena?

One reason for optimism is that social scientists do occasionally come to generalizable conclusions without large statistical models or full alignment of verbal and statistical claims. For example, Knetsch and Sinden (Reference Knetsch and Sinden1984) speculated that the endowment effect would generalize broadly, despite using a model that did not permit generalization beyond students at the University of New England trading lottery tickets for $3 in the year 1984. Later research iteratively modified this generalization scope, showing that the endowment effect extends across goods (e.g., Rowe, D'Arge, & Brookshire, Reference Rowe, D'Arge and Brookshire1980; van Dijk & van Knippenberg, Reference van Dijk and van Knippenberg1998), age groups (Harbaugh, Krause, & Vesterlund, Reference Harbaugh, Krause and Vesterlund2001), cultures (Maddux et al., Reference Maddux, Yang, Falk, Adam, Adair, Endo and Heine2010), and species (Lakshminaryanan, Keith Chen, & Santos, Reference Lakshminaryanan, Keith Chen and Santos2008), but is limited by market features (Kahneman, Knetsch, & Thaler, Reference Kahneman, Knetsch and Thaler1990), learning (Apicella, Azevedo, Christakis, & Fowler, Reference Apicella, Azevedo, Christakis and Fowler2014; Coursey, Hovis, & Schulze, Reference Coursey, Hovis and Schulze1987), and expertise (List, Reference List2003). At the risk of overgeneralizing from a single example, this shows that the familiar, messy process of iterative science is in some cases capable of untangling the causal structure of social phenomena. Importantly, this process depends on scientists pragmatically understanding over- or under-generalization as opportunities to contribute. Nuanced research cannot get off the ground if there is nothing to nuance.

But how should scientists identify unmeasured factors? The target article recommends “careful, critical thinking,” which is somewhat vague. In practice, this critical thinking does not take place inside the head of a single scientist, but through dialog within a broad community of scholars with diverse backgrounds and areas of expertise. Accordingly, the diversity of the scientific community may be a better target for intervention than statistical modeling practice. Achieving what Harding (Reference Harding1992) calls “strong objectivity” will require creating variation not only in experimental factors, but also in the social standpoints of the people who select those factors, determining the direction of scientific research.

Diversifying the population of researchers is not the only way institutions can mitigate the problem of unconceived alternatives. For example, a bias against the publication of null results disincentivizes research that could limit the generalization of previous findings. Exactly this problem plagued research on social priming, contributing to the replication crisis in psychology. Similar effects may follow from a preference for funding “low-risk” projects by eminent scientists (Stanford, Reference Stanford2019) and the gatekeeping function of prepublication peer review (Heesen & Bright, Reference Heesen and Bright2020).

One promising approach is to explicitly incentivize revealing unconceived alternatives. Beyond alignment of verbal and statistical claims, authors could be required to anticipate the unmeasured conditions they suspect are necessary and sufficient to realize the observed effects. This would have the upshot of precommitting researchers to accept certain conceptual replications, even when faced with null results. Building on Yarkoni's advocacy of predictive, translational research, institutions could encourage interdisciplinary collaborations for identifying mechanisms of action and limits to generalization, as well as adversarial collaborations between researchers with competing hypotheses. These efforts could be augmented by the maintenance of prediction markets for aggregating the knowledge and expertise of the broader scientific community (e.g., https://socialscienceprediction.org/).

These recommendations assume that the causal complexity of social phenomena is not so extreme as to bring researchers to skeptical grief. The (admittedly partial) successes of the social sciences seem to support this assumption, while the field's failures seem plausibly caused by perverse incentives and institutional mismanagement. We are enthusiastic about some of Yarkoni's recommendations – in particular, designing for variation is a fantastic idea and we have sought to do so in our own research (Sievers, Lee, Haslett, & Wheatley, Reference Sievers, Lee, Haslett and Wheatley2019). But insofar as generalization depends on identifying blind spots and missed opportunities, reforms that focus on the statistical practices and thinking habits of individual scientists will likely fall short of the mark. We must also foster a diverse scholarly community that is incentivized to reveal what those who came before them have missed.