For Yarkoni, the discipline of psychology suffers from at least two problems. First, the operationalized variables that are used in empirical work do not track the underlying structure of their hypotheses. This point has been made before with reference to failures to follow the hypothetico-deductive chain (Harris, Reference Harris1976) and in a recent discussion of flexible versus hard to vary theories (Szollosi & Donkin, Reference Szollosi and Donkin2021). Second, the statistical assumptions made when using operationalized variables are in error.

We find ourselves in broad accord with Yarkoni's first diagnosis. But whilst he expresses agnosticism about psychological constructs, we believe construct formation is a cause of Yarkoni's problems. There is a deep history to be written about the use of constructs, but the case of behaviorism will help to make a point. Watson's original view was that only observable data could be included within an explanation of stimulus–response transitions. However, when mathematical accounts proved untenable, this led to the introduction of unobservable constructs that were derived from observable data in order to generate an account. This was referred to as mediational neo-behaviorism (Moore, Reference Moore2013). This derivation was from data collected in the laboratory; neo-behaviorism did not lead with the construct; it was not something to operationalize. Skinner noted that traditional psychology had a contrary practice, defining terms such as memory using unobservable constructs that were not derived from observable data (Skinner, Reference Skinner1945). He advocated looking to the reinforcement history of those terms within the discipline to understand what work they might be doing for scientists.

Skinner's point is related to Popper's discussion of definitions in science, in which he argued that the practice was to read definitions from left to right as <an x consists of p₁ to p_n properties> (Popper, Reference Popper1945). This Aristotelian tradition introduces a form of essentialism, such that the project of science is to look for the essence of x. Instead, Popper claimed that definitions should instead be a form of shorthand. Once we understand that p₁ to p_n cohere in some way, for example, we can decide to name that kind of coherence x. Both Skinner and Popper committed to a clear-sighted form of empiricism.

It is often forgotten that behaviorism emerged as an antidote to introspection, which permitted verbal speculation about the architecture of internal behavioral causes. It was not that behaviorists denied inner experience, but they understood the scientific perils of trying to operationalize such models. Construct-led psychology necessarily has an introspective quality, and that practice leads to untethered ideas and a somewhat desperate attempt to empirically ground them. For the reasons that Skinner and Popper noted this will fail us scientifically: ideas, and more formally constructs, are best grounded when they emerge from empirical soil. Skinner also noted that the practices of cognitive psychologists were similar to behaviorists, in that they manipulated input variables and measured outputs, and were methodological behaviorists at best. Why not simply note regularities, titrate them, and then develop constructs? These points relate to Yarkoni's endorsement of a form of natural history.

De Regt claims that the unobservable mediational constructs that arose in behaviorism provided theoretical intelligibility, permitting the development of a functional explanatory framework that yielded prediction (de Regt, Reference de Regt2017). De Regt makes this more formal with his Criterion for Understanding Phenomena, which states that a phenomenon is understood if and only if it has an adequate explanation based on an intelligible theory. Furthermore, that theory must “conform to the basic epistemic values of empirical adequacy and internal consistency” (p. 92). Criteria for judging intelligibility include the ability of scientists to derive qualitative judgments about that theory without having to pursue exacting calculations. This package provides the necessary and sufficient conditions for scientific understanding. What we should note in the context of Yarkoni's argument is that here theory is being built in concert with empirical derivation, piece by piece.

Popper also revealed that there is no such thing as theory-free observation. Deciding what to measure is a theoretical choice and Yarkoni is well aware of this, and yet he avoids the discussion of grounding psychology in deeper theory. By this, we explicitly mean seeking some unity with biology, through the adoption of highly corroborated theories such as evolutionary theory, in order to provide a justifiable constraint on construct development. It is justified by the simple fact that behavioral plasticity is a phenotypic trait in the evolutionary framework (Meyers & Bull, Reference Meyers and Bull2002). In the last 30 years, this has been attempted by evolutionary psychology, but that particular exercise has sought to bolt traditional psychological constructs to hypotheses about adaptation, and to use adaptationist considerations to remove hypotheses that are not working (Dickins, Reference Dickins2003). A better use of evolutionary theory would be to adopt the optimality-led practices of behavioral ecology (Parker, Reference Parker, Lucas and Simmons2006) and then look to develop constructs to explain internal causation of behavior, with a clear view of what behavior is for (cf. Curry, Mullins, & Whitehouse, Reference Curry, Mullins and Whitehouse2019). Recent work taking a strong phylogenetic perspective on cognition, and borrowing from ecological psychology, which also had a distrust of construct-led science (Gibson, Reference Gibson1979), is carefully rebuilding the conceptual architecture of cognitive science (Bechtel & Bich, Reference Bechtel and Bich2021). This work is cautious and thoroughly aware of all the assumptions it is making, building toward intelligible theory and understanding.

Yarkoni's statistical points about random and fixed effects are sound and we take his point that a portion of empirical psychology is really qualitative by nature. But we do not see the need to embrace this. Instead, in keeping with our recommendations above, we would advocate a stronger emphasis upon grounding psychology and deriving hypotheses from a biological “bottom up” – at least until workable idealizations of causation can be derived to allow future prediction (Potochnik, Reference Potochnik2020). Doing this would introduce more steps into the derivation of hypotheses. This would include using modeling solutions to test the coherence of hypotheses.