2. Mechanisms, extrapolation, and mechanism individuation

Mechanisms are structured systems composed of interacting components and operations that collectively produce specific phenomena. According to the influential characterization proposed by Machamer et al. (Reference Machamer, Darden and Craver2000), mechanisms are defined by entities and activities arranged in an organized manner, ensuring the regular production of the phenomenon. The identity and boundaries of any given mechanism are not determined independently of the phenomenon; rather, they are always relative to it. For instance, when explaining the circulation of blood, the heart functions as the relevant mechanism precisely because its coordinated parts—such as chambers, valves, and rhythmic muscle contractions—systematically interact to achieve the continuous circulation of blood.Footnote ²

The phenomenon to be explained in (behavioral) policy contexts is typically the behavioral change caused by a specific intervention. The corresponding mechanism consists of the interacting parts and processes responsible for the change, including individuals; their beliefs, preferences, actions, cognitive processes; and the broader social or institutional contexts mediating these interactions. These mechanisms, moreover, possess a characteristic temporal structure: They begin from clearly specified initial conditions (in this context, the policy intervention), proceed through intermediate causal steps (such as shifts in individual beliefs, changes in preferences, shifts in social norms, or alterations in behavioral incentives), and ultimately culminate in the final behavioral outcome.

How can evidence of such mechanisms justify the extrapolation of a result from the study context to the target domain? Proponents of mechanism-based extrapolation have argued that “knowledge of the mechanisms running from cause to effect and of the kinds of things that can interfere with them enhances our ability to reliably decide whether a causal relationship found in one population will or will not obtain in another” (Steel Reference Steel2007, 5). As Joyce and Cartwright (Reference Joyce and Cartwright2020) argue for education policies, one needs to know that the local, social, cultural, and physical structure can afford the necessary causal pathways for the intervention to lead to the outcome in the target domain; one needs to know that the support factors are in place and the derails are absent.

Yet confirming similarity between mechanisms in the source and in the target requires precisely the knowledge we often lack about the target domain, which is precisely Steel’s (2008) “extrapolator’s circle.” Steel (2008) proposes to address this challenge using “comparative process tracing,” which involves breaking down the mechanism into clearly defined, distinct causal steps and compare only those stages in which the source and the target are most likely to differ.

Steel’s comparative process tracing reduces the evidential burden for extrapolation by focusing only on crucial steps in the mechanism. Yet it presupposes that there is a uniquely individuated mechanism to compare across contexts (cf. Joyce and Cartwright’s “necessary causal pathway”), an assumption that does not always hold. For example, consider extrapolating from a behavioral intervention designed to foster habit formation. Habit formation, as recent research highlights (Volpp and Loewenstein Reference Volpp and Loewenstein2020), can emerge through several distinct underlying mechanisms such as reinforcement learning, status quo bias, commitment strategies, or social influence processes. Consequently, there is no reason to assume that an intervention’s measured effect size within a student population would replicate, say, among older adults living in a care-home environment. Justifying the extrapolation of such an intervention demands careful consideration of whether the differences between the populations (in cognitive abilities, motivations, routines, social networks, etc.) might substantially alter the workings of the underlying habit-formation mechanism at work. For instance, learning-based habit formation hinges largely on factors like the immediacy of reinforcement and frequency of behavior repetitions, while commitment-based habit formation often depends on external incentives or explicit commitment devices. Therefore, extrapolating a policy from students to seniors would require knowing precisely through which specific mechanism (learning-based, commitment-based, or otherwise) the intervention operates in each distinct population.

Yet once we distinguish between different mechanistic types, such as “learning-based” versus “commitment-based” habit formation, we can still ask whether further differentiation is warranted. Indeed, under each mechanism type, it is possible to identify multiple more specific causal sequences. Each of these sequences, moreover, could be represented at the cognitive, behavioral, or even neuronal level of description. Yet only some of the possible representations of the mechanism will be helpful in identifying relevant differences between these populations. Imagine, for example, that motivational factors moderate habit formation in the elderly but not amongst students; otherwise, the mechanism in both populations is the same. If motivational factors are absent in the original student-centered model, mechanism-based extrapolation might incorrectly lead us to conclude that the observed effect transfers to the senior population because that mechanism individuation does not allow for a representation of the relevant moderator in the senior population. Thus, the mechanistic model may be accurate for the student population yet inadequately individuated for its extrapolation to the senior population.

To sum up, these difficulties arise because mechanisms as things in the world can be represented by models at varying degrees of abstraction and granularity. Thus, when assessing whether the mechanism in the target is sufficiently similar to the mechanism in the source, we must first address the question of how to represent the mechanism. This in a nutshell is the challenge of mechanism individuation.

3. Addressing the mechanism individuation challenge in behavioral public policy

We have argued that mechanism-based extrapolation needs to address the challenge of mechanism individuation, that is, at what level of abstraction and detail to represent the mechanism. As with any representation, the correct assessment criterion is not accuracy per se but rather adequacy for purpose (Parker Reference Parker2020). Different policy or inferential goals can warrant different criteria of relevance. We now apply this simple idea to describe the mechanism individuation challenge in the context of scaling up and extrapolating claims about the efficacy of a policy. In what follows, we take a mechanistic model to identify relevant mediators—variables directly situated along the causal pathway from intervention X to outcome Y—and relevant moderators—variables external to the direct causal pathway that determine whether, and under which conditions, the causal relationship remains stable, weakens, or disappears. Mechanistic evidence is evidence about those mediators and moderators, and such evidence can be obtained through a variety of methods including RCTs. In other words, what makes it mechanistic evidence is mainly what the evidence is about, namely evidence about mediators and moderators (Marchionni and Reijula Reference Marchionni and Reijula2019).

3.1. Scaling up

Scaling-up means implementing a policy intervention to a substantially larger domain than the original study domain—whether in terms of geographic scale, population size, or budget. Unlike other forms of extrapolation, scaling up typically assumes that the scaled-up domain includes the original one and shares some, if not all, of its features. We show that insofar as the extrapolative inference is to be grounded in mechanistic reasoning, the challenge of mechanism individuation arises here too.

Consider the following example. In a randomized field experiment carried out by Fryer et al. (Reference Fryer, Levitt, List and Samek2020), disadvantaged aged 3- to 4-year-old children participated in two new preschool education programs aimed at improving both their cognitive and noncognitive skills. At the same time, their parents attended biweekly incentivized meetings with their own child’s teacher to learn how to support the child’s learning at home. The hypothesis was that parental investment (P) was an important moderator of the causal effect of schooling (S) on achievement, and that S could affect P in some way. The researchers therefore decided to intervene on both S and P at the same time. More than 600 children participated in the experiment, which ran across two consecutive school years. Both educational programs had large cognitive impacts in the study situation, with treated children testing far above those in the control group.

Researchers then questioned whether the results from their experiment could be scaled up to the national level. The researchers’ original mechanistic model could have supported an affirmative answer: In it, P amplified the effect of S on achievement. However, they worried about other moderators that could come into play at the national level such as limited teacher quality or constrained budgets—factors that were absent from the “best-case” experimental setting where outstanding teachers and a generous budget had been available, and which subsequently were not represented in the original mechanistic model. As they considered scalability, they were unsure whether their initial mechanism individuation was sufficient for the purpose of a scaling-up inference.

To address this concern, the researchers proposed to adopt a “bigger picture” approach. This approach consists in explicitly examining from the outset “what constraints the idea will face at scale, what key factors can impact scaling, and whether the mediation paths and moderators are in place at scale” (List Reference List2024, 496). Rather than measuring effect size under ideal conditions and then face scalability problems, they advocate to run the small-scale experiment in a way that already anticipate possibly relevant large-scale moderators and individuate the mechanism accordingly. A mechanistic model derived exclusively from a best-case design might accurately capture the P-moderator yet fail to represent moderators relevant to scalability like teacher quality and budget constraints. The model, although accurate, would thus provide an inadequate basis for mechanism-based inference to scale.

We can capture this individuation challenge more schematically. Suppose an intervention I is shown in an RCT to affect an outcome O, with additional studies indicating that factor X moderates I’s effect on O by acting on a mediator M:

$$\matrix{ {\rm{X}} \cr \downarrow \cr {{\rm{I}} \to {\rm{M}} \to {\rm{O}}} \cr }$$

Now imagine that I is to be scaled up beyond the experimental sample. Researchers learn that M still mediates I’s effect in the new domain and confirm X is present to a similar extent. They conclude that I will work at scale as well. However, the initial mechanistic model (I → M → O) may be too coarse-grained for extrapolating to a larger scale. In particular, M may decompose into a sequence M₁ → M₂, where X moderates M₁ but another moderator (Y) influences M₂. Although Y did not appear in the original setting (e.g., teacher’s quality), it surfaces at scale. As a result, the intervention might fail under these new conditions. In this case, the mechanism has not changed; rather, the way in which it was identified in the initial study was too coarse grained for scaling-up purposes. Had the researchers accounted for the large-scale context and recognized the relevance of Y from the outset, they could have finer-grained the mechanism as follows:

$$\matrix{ {\!\!\!\!\!{\rm{X}}\ \ \ \ \ \ \ {\rm{Y}}} \cr \!\!\!\!\!{\downarrow }\ \ \ \ \ \ \ {\downarrow } \cr {{\rm{I}} \to {{\rm{M}}_1} \to {{\rm{M}}_2} \to {\rm{O}}} \cr }$$

By representing Y as a potential moderator already during the small-scale original experiment, they would have been better positioned to determine whether the intervention would work at scale.

4. Evidence aggregation

So far, we have discussed extrapolation from a particular study domain to a particular target: scaling up an intervention to a broader domain or transferring it to a new context. Yet a great deal of policy inference proceeds indirectly, namely, by making use of aggregated evidence, mostly in systematic reviews or meta-analyses. As a reaction to the failures of using such evidence for policy, many have argued for consideration of evidence about mechanisms alongside the correlational evidence that is often the only kind of evidence aggregated in such policy-oriented aggregation exercises.Footnote ³ Once it is established that a policy causes some outcome in one or more domains, then one moves to assess whether the mechanism in the new domain is similar enough to the mechanism described in the studies. This proposal is certainly an improvement over standard evidence-based procedures, but it misses an important step, namely the assessment of whether all the evidence is indeed evidence about the same causal relation and what affects its stability across varying contextual conditions.

To illustrate, consider the following example. Delmas et al. (Reference Delmas, Fischlein and Asensio2013) conducted a meta-analysis of information-based energy conservation experiments, analyzing 156 published trials from 1975 to 2012. This review represented the state of knowledge in this domain at the time of publication, and most of the included studies were of sufficient quality to provide evidence for the existence of a causal relationship between information-based interventions and energy saving behavior. One notable result of this meta-analysis is that the effect sizes substantially differ across studies, however. On average, individuals in the experiments reduced their electricity consumption by 7.4 percent. Yet some reviewed studies found much stronger effects—up to 55 percent reductions—others showed no effect, and some even exhibited consumption increases (Delmas et al. Reference Delmas, Fischlein and Asensio2013, 733). The reviewed studies were performed in different countries, with different demographics, at different times. The review also grouped together a wide array of interventions under the label “information based,” including energy saving tips, real-time feedback, audits and consulting, monetary savings information, and even social comparisons. The heterogeneity in effect sizes suggests that certain factors or moderators must have varied across contexts, shaping how people responded to the interventions. Consequently, one cannot simply apply an information-based intervention in a new setting and expect the “average” effect size reported in the meta-analysis to hold.

This poses a problem: This and similar meta-analyses seek to aggregate evidence about the efficacy of (types of) policy interventions. However, if the studied relationships are found not to be stable, how can the aggregated evidence be useful for policy making, and how should it be used? Clearly, we cannot straightforwardly extrapolate from a single aggregate effect when the underlying causal processes vary so much from context to context. A promising answer, we think, is to revise evidence aggregation practices so that interventions are grouped not just by broad intervention categories such as “information-based interventions,” but by the mechanism through which they operate (Grüne-Yanoff et al. Reference Grüne-Yanoff, Marchionni and Nuotio2024). To continue the example from section 2, rather than pooling every habit-formation intervention into one meta-analytic category, researchers could separate interventions driven by reinforcement learning, status quo bias, commitment strategies, or social influence. In that case, if the target domain is believed to be likely to engage a status quo bias mechanism, one would look primarily at the effect sizes from contexts where status quo bias—and any relevant moderators—were indeed in play.

However, this approach faces the mechanism individuation challenge. Consider an aggregator who classifies interventions under a single mediator M instead of individuating it by a more fine-grained chain M₁ → M₂. The same issues we identified earlier in section 3.1 reappear here: If applied to a context where a moderator Y, which was not present in the original studies, influences M₂, the evidence will be misleading. Or aggregators might choose to describe the mediator of their education intervention as “increasing nutritional competences of mothers,” instead of increasing nutritional competences of the decision maker, thus leading to similar extrapolation problems as those discussed in section 3.2.

Now, imagine two habit-formation interventions, both seemingly working through reinforcement learning—a common mediator. But in one study the intervention only works when daily reminders are used. In the other, the effect disappears when peer support is not available. Thus the two studies show sensitivity to different moderators. Should we treat these two interventions as operating through the same type of mechanism because both involve reinforcement learning? Or as instances of two different types because their causal performance depends on different contextual supports? Even if one argues that they share a common mediator, consistent evidence of different moderators should prompt finer-grained individuation for policy inference. Ignoring the role of reminders or peer support could lead to inaccurate generalizations about “reinforcement learning interventions” as a whole.

These problems are exacerbated by the division of epistemic labor inherent in such aggregation practices: Neither those who perform the individual studies, nor those who aggregate the evidence, will likely have information about what domains their evidence might be extrapolated to. Thus, in contrast to the inferences discussed in section 3, it is not possible to correct misconstrued mechanism individuation with the help of reference to the intended target domains.

What can be done, given that we cannot perform a forward-looking comparison to a not yet specified target? Our proposal is to look backward and systematically consider all currently available evidence for possible moderators. In other words, for the purpose of evidence aggregation, evidence should be aggregated for interventions that operate through the same type of mechanism, and the mechanism should be individuated in such a way that all available evidence of moderation can be included. For example, if there is evidence that an intervention’s effect size is sensitive to some moderating factor, then a mechanistic model employed to classify effect sizes of such an intervention should represent this moderator-sensitivity. To continue the schematic example from section 3, this might mean basing the meta-analytical taxonomy on a mechanism model that individuates M₁ → M₂ if there is any evidence for the moderating effect of Y, even if Y is not present in any of the reviewed studies. In this way, even though moderators are “external” to the causal pathway, they help define the boundaries of mechanism types for practical purposes like aggregation. By tailoring mechanism individuation to incorporate all known or theoretically plausibly important moderators, we can ensure that the resulting categories are more robust for subsequent scaling or policy transfer.