The target article highlights a vital problem in psychology: the inferential gap between statistical models and verbally-expressed psychological theories is too rarely appreciated or respected. However, I am perhaps more optimistic that the core problem can be solved.

My suggestion is, in short, to recognize the distinction in statistics between statistical hypotheses and substantive hypotheses, and to treat them differently from one another (e.g., Hays, Reference Hays1994). In psychology, a statistical hypothesis simply describes the distribution of a trait across a population, such as a behavioral tendency or level of task performance. A substantive hypothesis makes claims about the causal structure responsible for that distribution, such as the function of the cognitive systems operating. Yarkoni's paraphrase of a conservative conclusion about ego depletion could describe a statistical hypothesis: “crossing out the letter e slightly decreases response accuracy on a subsequent Stroop task” (sect. 6.3.1, para. 2). The attending substantive hypothesis might be “psychological processes requiring attention are subject to ego depletion.”

Only the statistical hypothesis is directly tested using statistical methods. The substantive hypothesis must be evaluated separately (though not entirely independently). The result is a two-step process of evaluation (see Dacey, Reference Daceyforthcoming).

Each step in the process implements one of Yarkoni's suggested courses of action. The first step is statistical inference proper, evaluating a statistical hypothesis based on the data by one's preferred statistical method. This step implements Yarkoni's suggestion that we draw more conservative inferences (sect. 6.3.1). The statistical result only bears directly on the limited statistical hypothesis, such as Yarkoni's limited conclusion, quoted above. The second step is the evaluation of the substantive hypothesis, which should implement Yarkoni's suggestion that we embrace qualitative analysis (sect. 6.2). This step requires evaluating how the decision about the statistical hypothesis bears on the substantive hypothesis, taking into account other relevant evidence. This is an inference to the best explanation that is not likely well-modeled by quantitative tools or formal logic (contra concerns about affirming the consequent; section 6.3.6).

Crucially, this means that a single statistical result will usually be very weak evidence for the substantive hypotheses of interest. The mind is complicated and, as Yarkoni highlights, there are many sources of variance, so it is very rare that one substantive hypothesis cannot explain, or at least accommodate, a statistical result. The statistics simply say “here is an effect that our theories should explain.” We decide which candidate explanation is best in the second, qualitative step. This must consider all relevant findings as targets of explanation, not just the most recent: we should resist viewing a single experiment as a stand-alone test of competing substantive hypotheses. I'd even suggest that many experiments be seen as one part of a larger project of characterizing or mapping the capacities involved, not as tests of one substantive hypothesis against another at all. It is a mistake to try to make statistical inference do all of the work of theory choice.

This approach will not fix all of the problems the target article mentions, but it would go a long way towards addressing the core problem. I take this to require changes in the way findings are reported in empirical papers, and perhaps in the way theoretical interpretations are argued for in review papers. However, I don't take it to require any drastic change to the science.