2.1. Epistemic Values in EBP

It is important to note that there is perhaps no universally accepted set of epistemic values common to all activities under the EBP heading. More fundamentally, it may be contested whether there is something like a unified EBP paradigm at all. The EBP movement, particularly as it changes over time and in response to various criticisms, is difficult to precisely demarcate as a unified paradigm with distinctive and invariant objectives, methods, underlying epistemic value presuppositions and so forth.Footnote ¹ Moreover, it is also difficult to find clear-cut commitments to particular sets of epistemic values, which means that the reconstruction I offer below will be just that: a reconstruction that is potentially imperfect and incomplete in its rendition of the actual roles of epistemic values in EBP.

That being said, it seems uncontroversial that there is a kernel of epistemic values that are common to a broad variety of activities under the EBP heading. This is because many of these activities are conducted in accordance with standard methodological recommendations offered by key institutions, such as the Campbell Collaboration, the What Works Clearinghouse, CONSORT, GRADE, the Cochrane Collaboration, JPAL and others. Many of these methodological recommendations, in particular so-called quality-of-evidence ranking schemes that rank different kinds of evidence according to quality and credibility, overlap in their emphasis on a kernel of epistemic values. These values are not coextensive with traditional epistemic values in the context of theory choice or appraisal such as those offered by Kuhn (Reference Kuhn1977). Instead, for empirical paradigms such as EBP the core epistemic values of interest concern the estimation of policy effects and include the methodological rigour that should be exercised when using different methods to obtain such estimates; the unbiasedness and precision of these estimates; and the ability to obtain causal conclusions about policy interventions on grounds of such estimates.

More specifically, methodological rigour is broadly understood as an attitude towards focusing on the use of only those methods that are believed to be (most) reliable in producing correct estimates of the quantities one is interested in measuring; the thorough exercise of appropriate precautions to ensure and demonstrate that the assumptions required for the successful use of these methods are valid; as well as a clear preference for methods that require few substantive assumptions to begin with.

Unbiasedness is typically understood in the sense that the effects estimated by using certain methods should identify only the quantities that one is interested in measuring, e.g. the causal effect of an intervention on X on an outcome Y, rather than, for instance, also capturing the effects of other variables on Y.

Precision is the requirement that studies should be adequately powered to detect the effects of interest, e.g. by ensuring that sample sizes are sufficiently large and that error bounds on the estimates of interest (although not always computable without substantive assumptions, see e.g. Deaton Reference Deaton2010) are sufficiently small to minimize uncertainties about the magnitudes and signs of the effects being estimated.

Finally, the ability to obtain causal conclusions about policy effectiveness is the central idea that EBP research should focus on the production of quantities that are actionable for the purposes of policymakers, i.e. quantities, which, if correctly estimated, can figure as a basis for designing successful policy interventions. For that, the estimated effects must be successfully causally attributed to the intervention variables of interest, as otherwise subsequent intervention on such variables may fail to produce the expected or desired effects.

The above values seem central to EBP in the sense that they seem to underpin many aspects of standard EBP methodology, i.e. a set of salient methodological principles that are shared among proponents of the paradigm and are widely circulated in methodological recommendations, guidelines and manuals that advise practitioners on how to conduct and evaluate studies on policy effectiveness. For instance, EBP methodological recommendations specifically focus on certain epistemic targets, i.e. causal conclusions about policy effectiveness that are informative for policy formation. Moreover, EBP methodology is premised on principles concerning the relative desirability of certain kinds of evidence, e.g. by using quality-of-evidence ranking schemes to express a strict preference for experimental and quasi-experimental over purely observational evidence. Finally, EBP methodology emphasizes the relative ability of different methods with respect to generating desirable kinds of evidence; again by focusing on RCTs and quasi-experimental designs as opposed to observational studies. Together, these methodological tenets mediate between epistemic values and method choice in the sense that EBP methodological recommendations, such as those issued by quality-of-evidence ranking schemes, seem to advocate the use of certain methods, notably RCTs, specifically because these methods are considered to best promote the achievement of crucial epistemic values such as those outlined above.

With this brief reconstruction of key EBP epistemic values in place, let me turn to reconstruct the role of value-freedom and -neutrality ideals in EBP.

2.2. Value-Freedom and -Neutrality in EBP

It is not obvious that EBP proponents in general pursue any specific ideal with respect to value-freedom and -neutrality, i.e. concerning whether non-epistemic, moral values may play a role in the conduct of EBP research, and whether the outputs of this research may involve substantive moral value presuppositions. This is particularly difficult to tell as EBP methodological guidelines rarely comment on value-related issues.

Despite the difficulties involved in finding explicit commitments to ideals of value-freedom and -neutrality, it seems plausible to think that the EBP movement rests on a relatively broad axiological presupposition that some division of labour with regard to settling normative issues of what values public policy should promote, and settling factual issues of what are effective interventions to promote these values, is possible. In other words, while it is clear that evidence-based policy invariably involves moral and political values when it comes to specifying what outcomes policies should promote, EBP seems to rest on the assumption that agreement on the desirability of policy outcomes can be separated from the production of evidence speaking for the effectiveness of interventions in realizing these outcomes. On this view, whether or not a policy intervention such as reducing class sizes is effective in increasing student performance is an issue that can (and should) be settled independently of whether it is in fact desirable to increase student performance. The latter is supposed to be a question of values, the former a question of empirical facts pertaining to ‘what works’; and it is the focus on documenting ‘what works’ that EBP is expressly committed to.

This broadly parallels traditional ideals regarding the role of non-epistemic, moral values in economics, where economists have frequently invoked the metaphor of economists as social engineers, who provide factual answers to policy questions independently from and typically after policymakers have settled issues concerning the relative desirability of social outcomes (cf. Hausman and McPherson Reference Hausman and McPherson1996). While EBP proponents may not subscribe to this particular ideal, EBP methodology seems to presuppose at least that some such division of labour is possible in the sense that one can empirically investigate the effectiveness of policy interventions largely independently from questions concerning the desirability of the outcomes that they bring about (see Vedung Reference Vedung2010 for a similar reconstruction). Let me expand on what this suggests for the role of value-freedom and -neutrality in EBP.

First, it seems that EBP involves some ideal of value-freedom in the sense that non-epistemic values are generally not and should not be involved in shaping the conduct and outcomes of EBP research internally. According to such an ideal, while non-epistemic values may be involved in selecting the kinds of policy issues and interventions being studied, the outcome variables of interest, and may act as constraints on whether conducting RCTs is morally permissible, non-epistemic, moral values are generally not and should not be involved in the choice and application of methods once these issues are settled (see Biddle Reference Biddle2013: 124 for a related sketch of such a demarcation). For instance, the choice between RCTs and observational studies, the collection of data for such studies, or the interpretation of estimates obtained from such studies, should not vary with respect to researchers’ (or policymakers’) convictions concerning the desirability of the policies under scrutiny. These internal aspects should be guided by epistemic values alone.

Second, EBP seems to involve some version of value-neutrality in the sense that the outcomes of EBP research are intended to be value-neutral insofar as they should not, and generally do not, issue unconditional normative claims regarding the desirability of social outcomes. At most, if there are normative claims issued in the dissemination of EBP research, these claims take the shape of hypothetical imperatives, i.e. normative claims that are conditional on some substantive value presupposition but do not endorse this value presupposition as such. In order for EBP research to maintain value-neutrality, the adequacy of presuppositions speaking for the desirability of some social outcome must be settled independently from (and perhaps prior to) generating information about the effectiveness of different interventions in producing the outcome. If such independence is achieved, then even if EBP research sometimes issues normative claims, e.g. when researchers suggest that some intervention is preferable to others because it is more (cost-) effective, these claims would still be value-neutral since they remain non-committal on the adequacy of the substantive moral value presuppositions involved, i.e. whether the outcomes of interest are in fact morally desirable. These issues are left to policymakers and otherwise suitably legitimized agents to settle.

With this brief reconstruction of the central epistemic values in EBP as well as the role of value-freedom and -neutrality in place, let me focus on the main issue that I am interested in, which is that standard EBP methods are differentially suitable for the pursuit of different kinds of moral and political values, specifically values that do or do not put emphasis on the distributive consequences of policies. This differential usefulness creates a trade-off relation between epistemic and moral values in EBP. To explain why this is so, let me begin by offering some background on the epistemic challenges involved in producing evidence on the distributive consequences of policy interventions.

3.1. Heterogeneity Information from RCTs

More formally, treatment effect heterogeneity is the systematic variation in the sign and/or magnitude of individual treatment effects among agents who are subject to a given intervention. In a potential outcomes framework (Rubin Reference Rubin1974; Holland Reference Holland1986), given an outcome of interest Y, the individual treatment effect (ITE) for individual i is the difference between her potential outcome Y_i(1) given the treatment and her potential outcome Y_i(0) in the absence of treatment, other things being equal. Since only one of the two values of Y_i can ever be observed, ITEs are in principle unobservable magnitudes (Rubin Reference Rubin1974).

RCTs offer a partial remedy for this inferential dead-end by permitting the estimation of average treatment effects (ATEs) instead of ITEs. This is achieved through balancing the net effects of confounding factors as well as interactive covariates by means of random assignment of subjects to experimental and control conditions, and multiple blinding of trial participants, those administering treatment and those recording and interpreting outcomes. Provided that randomization (and blinding etc.) are successful in that the net effects of confounders and interactive covariates (including interactions among them) are approximately balanced between treatment and control groups, an ideal RCT can help obtain, in expectation, an unbiased estimate of the ATE by taking the difference in means of Y for treated and untreated units, or $\widehat {ATE} = \ \overline {{Y_t}} ( 1 ) - \overline {{Y_c}} ( 0 )$.

This estimate of the sample ATE, however, does not permit inferences about ITEs. At best, and in the absence of any knowledge about interactive covariates as well as heterogeneity in their individual-specific realizations, the ATE estimate can figure as the expectation of the ITE for an individual randomly drawn from the experimental population. But as soon as there is variation between individuals in their realizations of interactive covariates, and consequently variation in ITEs, inferences about ITEs are largely precluded and information on heterogeneity cannot be recovered from $\widehat {ATE}.$

This has significant bearing on the transferability of trial results, i.e. the extent to which the ATE from an experimental population A can be expected to be replicated in some other population B. Two jointly sufficient conditions for the transferability of trial results (in the particular sense adopted here) to some out-of-sample target are first, that the treatment variable plays the same causal role in the production of the outcome in the target as it does in the experimental population, i.e. that the mechanisms in both populations are sufficiently similar with respect to the causal effect to be extrapolated. The second condition is that the distribution of interactive covariates of the treatment effect is the same in both populations (see e.g. Cartwright and Marcellesi Reference Cartwright, Marcellesi, Crangle, de la Sienra and Longino2015 for such conditions).Footnote ³ So the transferability of experimental results to targets hinges not only on sufficient similarity in mechanisms between populations but also on whether there is heterogeneity induced by differences in interactive covariates as well as how these variables are distributed among agents in the populations of interest. This problem has received attention from a variety of econometricians, methodologists, philosophers of science and EBP proponents (e.g. Hotz et al. Reference Hotz, Imbens and Mortimer2005; Duflo et al. Reference Duflo, Glennerster, Kremer, Schultz and Strauss2008; Imbens and Wooldridge Reference Imbens and Wooldridge2009; Bareinboim and Pearl Reference Bareinboim and Pearl2013; Cartwright and Marcellesi Reference Cartwright, Marcellesi, Crangle, de la Sienra and Longino2015).

Treatment effect heterogeneity does not only affect the transferability of trial results. It also creates a second challenge for EBP. The challenge is that in the absence of information on heterogeneity, RCTs are not suitable for informing policy formation processes that are concerned with the distributive consequences of policy (see e.g. Manski Reference Manski2000). More specifically, policymakers may often be interested in knowing not only whether an intervention is effective on average but also in how effective the intervention will be for specific types of agents, how treatment effects are distributed among agents, with respect to which observable pre-treatment characteristics they exhibit heterogeneity, whether heterogeneity obtains in magnitude or also in sign, etc.

Such information is crucial particularly when it is reasonable to suspect that agents may be harmed by an intervention, even though the ATE might be positive. In these scenarios, several pertinent distributive concerns arise, e.g. is it at all permissible to implement a policy that will render some agents worse off? If so, how should we adjudicate between the negative welfare consequences for these agents and the net effectiveness of the intervention? What are the thresholds of proportionality that we should use to decide whether benefits on the part of some outweigh losses on the part of others? Can the policy be targeted so that it predominantly affects those who will most likely benefit from the intervention? And so forth.

As these considerations suggest, policymakers may be interested in pursuing a variety of different distributive values. Yet, in order to pursue these values effectively, in the sense that the available evidence gives them good reasons to believe that an intervention will in fact promote them, policymakers require information on treatment effect heterogeneity, i.e. whether there is heterogeneity at all and how heterogeneous treatment effects are distributed with respect to agents’ observable characteristics. As I have argued above, RCTs cannot provide such information on their own.

This does not mean that EBP methodology is at a complete loss in this regard, however, as one way to address this problem is to perform so-called subgroup analyses. While this is feasible as long as researchers have obtained pre-treatment data on potential interactive covariates of the treatment effect, I argue below that performing such analyses, when judged against the background of standard quality-of-evidence ranking schemes circulated in EBP, comes at the expense of sacrificing several key EBP epistemic values. This creates a tradeoff between several epistemic values central to EBP and the pursuit of moral and political values such as equality and priority for the worst-off.

3.2. Subgroup Analysis as a Remedy for Informing about Heterogeneity

Following Duflo et al. (Reference Duflo, Glennerster, Kremer, Schultz and Strauss2008), subgroup analyses partition experimental populations into subgroups according to observable pre-treatment characteristics such as age, sex, ethnicity, prior education etc. They then typically further partition subgroups into different categories or strata, for instance age groups. Given this stratification, a difference-in-means estimation can be performed on the partitioned data to obtain conditional, subgroup-specific ATEs (CATEs); this helps us tell whether treatment effects differ between subgroups. A somewhat more sophisticated alternative to this are regression-based approaches, where potentially interesting interactive covariates of the treatment effect are modelled as interaction terms with the treatment indicator in a standard regression framework. In doing so, it is possible to obtain information on the significance of interaction effects between observable variables and the treatment indicator, which may be taken to suggest that the variables in question induce heterogeneity in the treatment effect of interest.

Even so, while subgroup analyses provide at least tentative information about heterogeneity, they are also subject to several pertinent methodological concerns. Let me expand on two particularly pressing concerns and explain how they bear on the achievement of EBP epistemic values.

First, the information that CATE- and regression-based approaches can generate is purely correlational in nature, and hence subject to standard concerns about endogeneity and consequent bias. For instance, consider the case where we estimate a significant positive interaction between microfinance access and prior business ownership, suggesting that agents who previously owned a business will benefit more from microfinance access. Such a finding does not permit the straightforward conclusion that prior business ownership is a causally relevant interactive covariate of the treatment effect. This is because the significance of the estimate may be attributable to common-causes, e.g. because business ownership is strongly correlated with business education, and it is business education that is causally relevant for inducing different behavioural responses to microfinance access, but prior business ownership in the absence of (or conditional on) business education may not contribute at all to microfinance treatment effects.Footnote ⁴ In this case, if business education is not included in the regression, our estimates of individual-level heterogeneity with respect to prior business ownership will be upwards biased.

Randomization at the treatment stage does not alleviate this problem because although it ensures, in expectation, that the net effects of prior business ownership and business education on the outcome are distributed equally between treated and untreated units, it leaves the covariance between the two variables untouched. So if we run a regression with an interaction term including only prior business ownership and find that there is a significant interaction with the treatment indicator, this result can be misleading. This is because the variable that truly induces the subgroup differences, business education, will be captured by the error term, and since it is still correlated with prior business ownership, this yields a biased estimate of the interaction between treatment and prior business ownership.

More generally, parameter estimates for interactive treatment effect covariates will invariably remain subject to such concerns unless researchers are prepared to entertain the relatively strong assumption that the subgroup variables of interest are uncorrelated with the error term of the regression (cf. Pearl Reference Pearl2014), i.e. there may be no common causes of the putative subgroup variable of interest and the outcome that are captured by the error term and that could induce the apparent interaction of the subgroup variable with the treatment indicator. However, it is precisely such questionable identification assumptions, which are necessary for unbiased identification in any standard regression context, that EBP proponents are keen to avoid. Indeed, randomization is expressly emphasized in the methodological literature as the key strategy to help avoid making such assumptions.

This means that using subgroup analyses to obtain unbiased estimates of heterogeneous treatment effects and straightforward causal conclusions about the role of interactive covariates that induce them is typically precluded, threatening at least two EBP epistemic values at once.

A second worry about subgroup analyses is with regard to the precision of effect estimates, including concerns about statistical power. In short, the more subgroups one specifies, the higher the probability of obtaining spurious results. For typical significance levels at P < 0.05 even a moderate number of subgroups, strata partitions and corresponding hypothesis tests will render the occurrence of spurious results exceedingly likely. At the very least, suitable statistical corrections are in order to remedy the consequences of multiple testing for the prevalence of false positives. Yet, while recommended by some EBP proponents (e.g. Duflo et al. Reference Duflo, Glennerster, Kremer, Schultz and Strauss2008: 65), this is rarely carried out in practice (Fink et al. Reference Fink, McConnell and Vollmer2014: 47). In short, these concerns about hypothesis testing pose a threat to the envisioned methodological rigour mandated in standard EBP methodological guidelines.

Moreover, to alleviate concerns about insufficient statistical power, sample sizes may need to be expanded for subgroup analyses to be informative at all. For instance, in order to detect a heterogeneity signal of the same magnitude as the sample ATE and with the same precision as the sample ATE estimate, a difference-in-means estimation on just one subgroup partitioned into two strata requires a fourfold expansion of the original sample size (Varadhan and Seeger Reference Varadhan, Seeger, Velentgas, Dreyer, Nourjah, Smith and Torchia2013: 38). Yet, subgroup-specific effects may be smaller than ATEs, particularly in relatively homogeneous trial populations.Footnote ⁵ So adequately powered subgroup analyses may frequently require much greater expansions of sample sizes to permit detection of heterogeneous effects. In short, this suggests that the epistemic value of precision is threatened for the investigation of subgroup-specific effects.

These and other, related concerns limit the extent to which subgroup analyses can inform about treatment effect heterogeneity in a way that lives up to the epistemic standards imposed by EBP methodological guidelines. Standard methodological recommendations (e.g. Varadhan and Seeger Reference Varadhan, Seeger, Velentgas, Dreyer, Nourjah, Smith and Torchia2013) suggest that subgroup findings should at most be considered exploratory in the sense that they may prompt additional investigations such as novel trials on subgroups of interest, but are insufficient to warrant definitive conclusions about heterogeneity by themselves. However, while conducting novel RCTs on subgroups appears to be a viable strategy for addressing some of the above concerns, this requires prior identification of the relevant subgroups. Unfortunately, we are rarely in the epistemically fortunate position to know which individuals are most likely to incur welfare losses in advance, since that depends on knowing what the causally relevant interactive covariates of the treatment effect are, how they affect the outcomes of interest as well as which agents exhibit beneficial or harmful realizations of these variables. So information on heterogeneity, possibly obtained from subgroup analyses, is still required even if we are willing to conduct subsequent RCTs on particular subgroups to obtain unbiased estimates of subgroup-specific effects.

The extant EBP literature has only recently started to address treatment effect heterogeneity issues. Yet, even though there are several recent social policy and development studies that perform at least tentative and exploratory heterogeneity analyses, they frequently fail to address one or more of the concerns outlined above (see e.g. Fink et al. Reference Fink, McConnell and Vollmer2014) or tend to focus on heterogeneity that obtains between estimates obtained in different trials, which is a related but conceptually distinct issue from the within-trial and between-subject heterogeneity that I consider here. Specifically, between-trial heterogeneity may not only occur as a result of individual-level differences in treatment effects, but also as a result of trial-specific differences such as differences in the interventions being tested or the quality of treatment implementation, as well as differences in the methods used to estimate their effects. Investigations of such between-trial heterogeneity are mostly focused on determining whether heterogeneity is random or systematic, attributing heterogeneity to differences between trials, and deciding whether trial results, despite such differences, may still be aggregated in systematic reviews of effectiveness evidence. Such studies are hence not typically concerned with exploring individual-level heterogeneity in treatment effects and even less so with the distribution of treatment effects within populations.

Let me expand on how these epistemic challenges for informing about heterogeneity create a trade-off between epistemic and moral values in EBP and how this trade-off challenges both value-freedom and neutrality in EBP.

4.1. Value-freedom

First, if the value-free ideal underlying EBP is understood as saying that non-epistemic values are generally not and should not be involved in shaping the conduct and outcomes of EBP research internally, e.g. by influencing the collection of data as well as the choice between different methods, estimators, model specifications and so forth, then the desirability of this ideal is challenged. Without suitable changes to EBP methodology, pursuing values that depend on the distribution of treatment effects on grounds of EBP evidence is presently inhibited. If this situation should be remedied, then this requires changes to EBP methodology that enable and facilitate the production and use of evidence on treatment effect heterogeneity. However, and this is the crucial point, these changes would be effected by moral values, since it is the desired ability to effectively pursue moral values on grounds of EBP evidence that motivates the requisite changes to methodology. To the extent that such changes to methodology are justifiable and justified, this suggests that value-freedom in EBP is not a desirable ideal, even at key methodological stages such as issuing widely disseminated recommendations for method choice, data collection, model specification, estimation and interpretation.

It is important to emphasize that this may be a transient state of affairs only, because as soon as requisite changes to standard methodological recommendations are implemented, and the evidence that is produced in accordance with revised recommendations becomes more informative about distributive consequences of policy interventions, then this may obviate further changes to methodology that are driven by non-epistemic, moral values. So value freedom in EBP may only be transiently undesirable. This helps to push back against the concern that the changes to EBP methodology suggested above are just the first step in permitting arbitrary influences of moral values on the internal stages of EBP research.

To further push back on such concerns, it is important to emphasize that not all changes to EBP methodology will promote the goal of increasing the extent to which EBP evidence is informative for the pursuit of widely shared moral and political values such as equality and priority for the worst off. This goal is sufficiently specific to rule out changes to EBP methodology that are motivated by other, potentially idiosyncratic values, e.g. researchers’ personal convictions about the desirability of policy interventions, or profit-maximization motives by trial sponsors who prefer methods that are more likely to produce effect estimates that are in accordance with their financial interests. Such value-influences would not be compatible with the kinds of changes to EBP methodology that facilitate the pursuit of a broader set of moral and political values on grounds of EBP evidence. So value-freedom in EBP may still remain an important ideal, but there are reasons to think that it is presently undesirable, as the pursuit of an important class of values on grounds of EBP evidence is inhibited until EBP methodological tenets undergo suitable revisions.

The role for values outlined here hence differs, and extends considerably beyond extant contributions such as Heather Douglas's (Reference Douglas2009), who argues that moral values play important and ineliminable roles in handling the uncertainties that are involved in using evidence for policy purposes; and that such values may play legitimate indirect roles in the assessment and use of uncertain evidence. The arguments provided here suggest that there are cases where the role of non-epistemic moral values may be even more extensive than previously considered as they may also, legitimately, play direct roles in governing the production of evidence, e.g. when a particular moral value figures as a reason in itself (cf. Douglas Reference Douglas2009: 96) to recommend a particular method over another, or to use a method in a certain way that helps promote the pursuit of the moral value in question, even if this proceeds on pain of sacrificing other, epistemic values.

Two important qualifications need to be added here. First, it is clear that there can be cases in which policymakers can offer precise characterizations of the questions they are interested in answering, and with a view towards the particular values that they are interested in pursuing. In these cases, when the questions to be addressed are fixed, it seems that even if values were involved in shaping the questions that are pursued by EBP researchers, they would not meddle with the outputs of that research. This seems a clear-cut case where values play only an external role.Footnote ¹³

My concerns apply to a different type of case, however: cases where evidence is used ‘off the shelf’. This relates to an important goal of EBP, i.e. to build libraries of evidence that policymakers and practitioners can consult when addressing certain types of policy problems, or implementing certain kinds of interventions. In these cases, evidence is produced before the aims of the particular use-case are determined, including any values that are to be pursued. Here, it seems that non-epistemic values may play direct, and legitimate roles at internal stages. They can act as reasons in themselves, explicitly mentioned in methodological recommendations, and on par with other, epistemic values, for choosing certain methods over others, even if this comes at an epistemic price, e.g. less precise estimates, or increases in the risk of bias.

Of course, this does not mean that any sort of wishful thinking is going on, as for instance when certain methods are chosen on the grounds that they are more likely to produce particular kinds of answers. But that is precisely how my case departs from Douglas's concern about wishful thinking cases, i.e. cases where values systematically and illegitimately meddle with our research results. In contrast, in the present case it seems that even though values play a direct role at internal stages, such as method choice and model specification, this is not necessarily illegitimate. The reason is that the choice of method may have important non-epistemic ramifications concerning the kinds of policies that are likely to be implemented on grounds of EBP evidence, and these ramifications may, legitimately, be anticipated when governing the production of evidence.

This points to a second important qualification, which is that I am not suggesting that moral values need to act as reasons in themselves at internal stages of particular studies. My concerns are rather with the methodological recommendations that are circulated in EBP, and that shape the conduct of individual studies. Hence, we might say that value-freedom in EBP is transiently undesirable, not at the level of individual studies, but rather at the methodological level of issuing recommendations for the production and use of different kinds of evidence. In analogy to study-level internal roles for values in determining what method to use, which model specification to use, how to collect data, and how to interpret results, I am concerned with internal roles for values at the level of general methodological recommendations pertaining to these issues. Hence, my claim is that value freedom is, at least transiently, undesirable at the level of widely disseminated methodological recommendations, although perhaps not at the level of individual studies.

Let me expand on related concerns about value-neutrality in EBP.

4.2. Value-neutrality

Recall that for the purposes of this paper I understand value-neutrality in EBP as the idea that EBP research outputs should not be premised on substantive moral value judgements about the desirability of social outcomes or the interventions that bring about these outcomes. So in disseminating the results of policy evaluations EBP researchers may issue at most conditionally normative recommendations, i.e. recommendations that are conditional on substantive value judgements but do not endorse these judgements as such. For instance, when a policy is considered most (cost-) effective because it best promotes an outcome such as household endowment of the rural poor, nutritional health in children etc. it may be explicitly recommended on grounds of its effectiveness. However, whether or not the outcomes that the intervention best promotes are in fact morally desirable should be settled independently, and EBP policy recommendations should remain neutral with respect to such questions. This issue is for policymakers and other suitably legitimized agents to settle.

As the previous discussion suggests, this type of value-neutrality is undermined by how the trade-off between the informativeness of evidence and central EBP epistemic values is settled in practice. More specifically, adherence to widely disseminated methodological recommendations and the epistemic values underlying them means that inferences about policy effectiveness remain limited to average effectiveness assessments and hence do not encompass information about the distributive consequences of interventions. So the standard way of operationalizing what it means for a programme to be effective brackets concerns about distributive issues (see Biller-Andorno et al. Reference Biller-Andorno, Lie and Meulen2002 for similar concerns about evidence-based medicine). As it stands, an effective programme is considered a good programme to the extent that the outcome of interest tracks a relevant moral or societal good. However, even if this good is uncontroversial in itself, effectiveness as standardly construed in EBP still only means effectiveness on average, not some effectiveness for everyone, or sufficient effectiveness for the worst-off, or equal effectiveness for all policy subjects. It is clear that effectiveness in one sense does not imply effectiveness in others, so policies that are effective on average can have distributive consequences that are undesirable relative to a wide variety of values, and hence would not be considered effective relative to these values.

To achieve value-neutrality in the envisioned sense it is hence not enough to ensure that the desirability of the outcomes of interest as such has been settled, or can be settled independently. It is also necessary to maintain neutrality with respect to the ways in which these outcomes may be realized. A given change in aggregate outcomes can usually be achieved in various ways, each of which may have dramatically different distributive consequences for target populations, and some of these may be intrinsically more or less desirable than others. If policymakers who wish to use EBP evidence care about differences between agents, and about absolute and relative changes in outcome distributions, then effectiveness as standardly construed in EBP is not informative about the moral permissibility or desirability of the policies under scrutiny and might be misleading about what effective programmes are ultimately able to achieve, relative to the specific moral and political values that policymakers pursue.

So, at least presently, it seems that the dissemination of EBP research is premised on the implicit presupposition that the relevant magnitude for deciding which policy to implement, is effectiveness in terms of average effects. This fails to be value-neutral in the envisioned sense because it presupposes that average effectiveness is the magnitude of interest to policymakers, rather than delegating the question of whether it is to policymakers and other suitably legitimized agents to settle. In a nutshell, in order to maintain a traditional ideal of value-neutrality, additional value presuppositions such as the above must at least be made explicit for EBP policy recommendations to remain value-neutral in the envisioned sense.

Let me close with some general remarks on the role of non-epistemic values in EBP going forward.

4.3. EBP and Values – Where Next?

The previous discussion raises important questions about what role non-epistemic moral and political values should play in EBP, and consequently how potential tradeoffs between epistemic and non-epistemic values should be settled at the level of widely circulated methodological recommendations. Should EBP proponents give up on their commitment to central EBP epistemic values, and if so, which epistemic values? Or should they bite the bullet and concede that standard EBP evidence fails to be informative for the pursuit of a wide range of moral and political values?

It seems unclear whether there is a single, univocal answer to how this trade-off should be settled. It seems plausible to think that the precise nature of this trade-off will depend on concrete contextual details pertaining to the kinds of questions that users of evidence seek to address, the nature of the policy settings that these questions pertain to, and the nature of the methods that are available for addressing these questions; specifically their relative ability to provide certain kinds of information and the extent to which they promote certain kinds of epistemic values. These aspects can vary importantly between cases.

What I want to offer here is hence not a set of definitive answers, but rather some general suggestions for moving forward. Some obvious suggestions include that the problems outlined above should be explicitly recognized in EBP methodological guidelines, including an explicit discussion of the differential usefulness of evidence for the pursuit of different purposes and values, as well as recognition of the fact that evidence concerning heterogeneous treatment effects is crucially important for the pursuit of a wide range of values and purposes. With the importance of such evidence more clearly emphasized, it would also seem desirable if more efforts were exerted in the way of offering up to date advice on recently proposed methods such as machine learning approaches for detecting heterogeneity (e.g. Athey and Imbens Reference Athey and Imbens2016, Reference Athey and Imbens2017) including recommendations for how these methods should be used, and how the evidence supplied by them can be integrated to yield more comprehensive assessments of the distributive consequences of policy interventions. Moreover, it seems sensible to suggest that EBP methodological recommendations should offer explicit guidelines for how authors of primary EBP research and meta-analyses should comment on potential limitations of their research. For instance, one may recommend that authors expand more explicitly on the deliberations underlying their choice of specific methods, including, if applicable, a commentary on why they chose not to use certain methods and how and why they may have chosen to sacrifice informativeness of the evidence in the pursuit of adherence to certain epistemic values. This is similar to what Heather Douglas (Reference Douglas2009) and many others (e.g. Elliott Reference Elliott2017) have been calling for, i.e. more transparency on the part of researchers about choices that may invariably involve non-epistemic considerations.

My second, and more general suggestion is that a general framework is needed to help facilitate deliberation about the role of values in EBP, i.e. a framework that can help structure and elucidate different ways in which epistemic and non-epistemic values may play a role in EBP methodological recommendations going forward.

In standard EBP methodological recommendations, the production of different kinds of evidence is presently recommended on epistemic grounds, i.e. evidence-ranking schemes rank different types of evidence according to their ability to promote standard EBP epistemic values. As the previous discussion makes clear, however, even evidence that ticks all the boxes on the epistemic desiderata underlying such rankings will neither necessarily, nor typically, be useful for the pursuit of all important purposes. This suggests that there is not only an epistemic dimension to recommending the production and use of certain kinds of evidence, but also a non-epistemic dimension concerning what kinds of evidence are useful for the pursuit of different values and purposes.

Recognizing the importance of both dimensions suggests that it may be useful to adopt a framework that integrates both epistemic and non-epistemic considerations relevant for governing the production and use of evidence for EBP. This is not an entirely new idea, of course. There have been several proposals in the philosophy of science literature to apply so-called coupled epistemic-ethical frameworks (e.g. Tuana Reference Tuana2013) to various policy-relevant activities in the special sciences, e.g. in climate science (Tuana et al. Reference Tuana, Sriver, Svoboda, Olson, Irvine, Haqq-Misra and Keller2012) and evidence-based medicine (e.g. Katikireddi and Valles Reference Katikireddi and Valles2015). While I am not suggesting that these frameworks can be straightforwardly applied to the present case, it seems that integrating epistemic and ethical concerns might be useful for investigating, structuring and eventually settling the above tensions between epistemic and non-epistemic values in EBP.

There are various ways in which this can be fleshed out in more detail. For instance, one way could be to say that in addition to ranking evidence according to its quality on epistemic grounds, there should be complementary attempts to rank different kinds of evidence according to their ability to individually, or jointly with other kinds of evidence, provide comprehensive accounts of policy effects, including details on adverse effects on subgroups as well as changes in the outcome distributions brought about by policy interventions.

These two dimensions together may allow us to systematically explore the particular balance that different kinds of evidence, and different combinations of such evidence, strike concerning the extent to which they promote epistemic and non-epistemic desiderata respectively. Some methods might do well on epistemic desiderata but may be extremely limited in their scope of sensible application; other methods might strike a more even balance. Analysing such trade-offs systematically, across a range of different methods and different types of policy scenarios, may help us get a better grasp of the joint epistemic and non-epistemic consequences of issuing preferences for certain kinds of methods over others. Finally, a joint epistemic-ethical analysis may also help us to distinguish between different kinds of trade-offs that vary in severity as well as cases where trade-offs may perhaps not obtain at all.

Naturally, considering both epistemic and non-epistemic dimensions in recommending the production of particular kinds of evidence will raise the important question of how these dimensions should be weighed against each other. Any particular weighting of these dimensions will reflect some way in which a trade-off between epistemic and non-epistemic, moral values is settled. As I have suggested above, my aim is not to recommend specific weightings, but merely to highlight that it seems important to develop a framework that facilitates deliberation about such weightings by making choices pertaining to them explicit. So similar to other calls in the values in science literature to make the role of non-epistemic, moral values as explicit as possible, my suggestion here is that adopting an epistemic-ethical approach can mark an important first step in facilitating joint deliberation among methodologists, researchers, users of EBP evidence and relevant stakeholders about how trade-offs between quality of evidence on epistemic grounds and the usefulness of evidence for the pursuit of different values should be settled.