We thank the responders for their thoughtful and quite constructive reactions to our manuscript. Their insights and comments are incredibly useful since they span broad disciplines that have each lent insights into the scalability of public policies and the importance of the scale-up effect. In particular, we believe that their complementing thoughts can aid the special issue readers to synthesize the received work. In this note, we provide a brief rejoinder to highlight some of the insights of the commentators.
Before walking through a quick response to each author, we wish to step back and summarize what we are trying to accomplish with this series of papers on the science of using science. As we note in our introduction (Al-Ubaydli et al., Reference Al-Ubaydli, Lee, List and Suskind2020):
Our study is motivated by the goal of digging into the economics of scaling. To date, this literature has largely been devoid of economics, focusing instead on the tools of implementation experts. Yet, a natural progression of the field revolves around the query: How can we combine economics with the experimental method to inform policy at scale? We view this query as the most important question facing evidence-based policymaking today. Indeed, the chain connecting initial research discovery to the ultimate policy enacted has as its most susceptible link an understanding of the science of scaling.
The ‘chain’ that we mention in the passage is elaborated on in footnote 2 of the article, where we state that we view the chain as being composed of three major links: (1) funding basic research; (2) providing the knowledge creation market with the optimal incentives for researchers to design, implement and report scientific results; and (3) developing a system whereby policymakers have the appropriate incentives to adopt effective policies and, once adopted, develop strategies to implement those policies with rigorous evaluation methods to ensure continual improvement.
At this point, the astute reader might stop and pause to wonder why so many of the pundits focused on the third link in our statement (see Ho et al., Reference Ho, Leong and Yeung2020; John, Reference John2020; Mani, Reference Mani2020; Supplee & Kane, Reference Supplee and Kane2020). In this context, the focus for some reviewers seemed to be on what we did not model rather than what we did. In fact, one set of commentators (Supplee and Kane) even questioned how we could possibly derive insights from a model that did not include this third link: “The core economic model does not currently acknowledge the broader context of policy decision-making and therefore may fail to achieve its objectives.”
It is instructive at this point to step back and consider why abstraction is useful and sometimes necessary when modeling in the sciences. One place to start is to go back to the 1980s, when John's father would always get lost driving from their home in Madison, WI, to Orlando, FL, for their family vacation. Invariably, somewhere around Georgia the family would get lost. The car would pull over and John's father would walk inside the gas station and walk out with a local map. At this point, having recently learned something about the earth, John might have commented, “Dad, your core model of the earth does not currently acknowledge the broader context of our great planet; therefore, it may fail to achieve its objectives … the earth is a sphere after all, not flat.” To which his father might have replied, “Son, ever since Galileo, scientists have made useful abstractions to make their points. Just because one part of the model is an abstraction, and clearly not correct, it doesn't mean that the model will fail to deliver useful insights.” He would then promptly unfold his flat map and it would deliver exactly what was needed – it would direct the family back to the proper route.
This is by no means debating the point of whether the third link is of great importance. Indeed, we view this entire research/policy agenda as a ‘weakest link problem’, in that all three links of the chain need to be strong for success, for if one link is weak, then the entire system does not achieve its promise. That said, we also do not believe that any one paper, be it theoretical or empirical, can take on all three links usefully.
For example, List and others have entire research agendas pertaining to the first link: ways to increase charitable giving for science and other public goods (e.g., see List, Reference List2011, and the citations therein). Likewise, very able scholars, some in this special issue, have focused their entire careers on the third link, spanning political science, political economy, law, psychology and sociology, to name just a few. There is no doubt that the third link is important to the problem at hand. We chose in this work to focus on the middle link: on increasing our understanding of how to provide the knowledge creation market with the optimal incentives for researchers to design, implement and report scientific results (Al-Ubaydli et al., Reference Al-Ubaydli, List and Suskind2019).
In this spirit, the idea is to set aside the first and third links and attempt to make a contribution to the knowledge creation market. Under this view, the thought experiment is that if every element in the first and third links is set at its optimal level, would the voltage problem be solved? The answer is unequivocally no – there would still be a voltage drop because of features inherent in the second link. And the size and nature of that voltage drop can be understood through a model. We provide one such model in our work, motivated by economic incentives.
In future work, we plan to add economic and behavioral insights to study the third link, because clearly economics has a lot to say in this case, and judging by revealed preferences, our pundits agree! We leave that research for another occasion, and now turn to a brief response to each author.
Harrison
Harrison (Reference Harrison2020) makes two key points. The first one rests on the critical issue of whether the policy being considered is worth scaling at all. He argues that randomized controlled trials “avoid wanting to make any structural claims about why things work or that they work to improve the welfare of individuals.” Relatedly, his second point emphasizes the danger in just examining the average treatment effect as opposed to its distribution because averages can mask important welfare impacts at the individual level.
We agree that before even addressing the different threats to scalability, a logical preliminary step is to make sure that the policy is desirable in the first place. Harrison implicitly demands that we understand the entire distribution of treatment effects, and indeed we agree.
That was the genesis behind Proposal #5 in our study:
Proposal #5: Leverage multi-site trials to learn about the variation in program impacts across both populational and situational dimensions. In other words, before scaling, understand the program effects across subsets of the general population and characteristics of the situation to understand who should receive the program, where/how it should be implemented and whether it passes a benefit/cost test.
In fact, that sentiment is why we included wish #9 in our ‘Dozen Things Experimental Economists Should Do (More of)’ (Czibor et al., Reference Czibor, Jimenez-Gomez and List2019), a companion study to this one that calls for more designs that provide the full joint distribution of treatment effects.
We are happy to learn that Harrison agrees with us, and his piece represents a great call for estimating the entire joint distribution of treatment effects. Our Proposals #5 and #9 essentially recognize the importance of measuring the heterogeneity of treatment effects, not only across population characteristics as pointed out by Harrison (Reference Harrison2020), but also across situational characteristics (Proposal #9).
Ho, Leong and Yeung
Expanding and complementing our framework, Ho et al. (Reference Ho, Leong and Yeung2020) make six excellent suggestions from the implementation and policy sciences field.
Suggestions #1, #2 and #3 encapsulate the philosophy behind many of our proposals (e.g., Proposals #5, #8 and #9, as well as many elements within the checklist). In fact, the demand for the theoretical underpinning of intervention effectiveness should go beyond population-specific issues and include situation-specific counterparts.
We embrace Suggestion #4, as it is line with our Proposal #7 and the checklist items emphasizing the importance of collaborating with policymakers and implementers to define the appropriate policy-relevant environment.
Ho et al.'s (Reference Ho, Leong and Yeung2020) Suggestion #5 aligns perfectly with our checklist (e.g., see the section for researchers within the checklist) that pushes for greater transparency, particularly regarding the information that concerns the characteristics of the population and the situation of the experimental study. Finally, Suggestion #6 echoes in part comments made by John (Reference John2020), Mani (Reference Mani2020) and Supplee and Kane (Reference Supplee and Kane2020), so we will not expand upon this here other than to point the reader to our thoughts laid out above in the introduction to this rejoinder.
John
Leveraging insights from the political science literature, John (Reference John2020) delves deeper into our footnote 9, noting, “Bureaucrats too are political animals whose interests are tied to those of the politicians who usually command them and who have an interest in the preservation of their bureaus and in promoting their own careers.” We acknowledge that capturing this political dynamic was beyond the scope of our work. Many of the excellent issues raised were pointed out by Ho et al. (Reference Ho, Leong and Yeung2020), Mani (Reference Mani2020) and Supplee and Kane (Reference Supplee and Kane2020).
Our reasoning for our modeling choices are laid out above in the introduction to this rejoinder, but it is important to note that once the policymakers have identified the policy or set of interventions to implement (i.e., a policy that has survived the political dynamic), the proposed framework still applies to minimize the scale-up effect. Ideally, we can probably agree that the political system in place chooses the policy that maximizes net benefits at the societal level (including the possibility of assigning different weights to different individuals). In reality, that is not the case, but even with this constraint, the scientific path for actionable evidence is not affected.
We welcome the attention drawn to the literature that “focuses on the importance of the ease of translation of ideas and recommendations,” and we agree that there is a role for “skilled advocates and entrepreneurs,” who can better communicate not only to policymakers, but also to the recipient constituents on the value of existing scientific evidence.
We also agree that some of the proposals in Al-Ubaydli et al. (Reference Al-Ubaydli, Lee, List and Suskind2020), such as increasing replications, can increase the costs of knowledge generation. However, if we broaden the dimension of costs to include those used to scale-up policies that later show more disappointing benefit/cost profiles than anticipated, the additional costs in replications can be well justified.
Mani
Mani (Reference Mani2020) provides a wonderful exposition in applying the scale-up effect framework from the perspective of developing countries and she points out two key issues to be considered: the “limited state capacity” and the “politics of project selection.”
We agree on the role that state capacity constraints potentially have on the scale-up effect, whether these are limitations on the infrastructure or on information processing and learning by policymakers. Although Al-Ubaydli et al. (Reference Al-Ubaydli, Lee, List and Suskind2020) do not mention these constraints explicitly, they are captured within the representativeness of the situation. Our framework suggests one avenue to address these constraints: we need to incorporate them when evaluating potential interventions because the calculated benefit/cost ratio needs to reflect them in order to minimize the scale-up effect. When these bottlenecks are not identified a priori, we believe it is the critical role of the collaboration between researchers, policymakers and practitioners to attempt to pinpoint potential constraints that would only become apparent at scale. The second avenue would be to look for ways to reduce or eliminate these constraints, certainly a gargantuan task.
We also agree on the importance of the politics in the project selection process, and many of the issues raised by Mani (Reference Mani2020) reflect those raised by Ho et al. (Reference Ho, Leong and Yeung2020), John (Reference John2020) and Supplee and Kane (Reference Supplee and Kane2020), so we do not add further comments here.
McConnell
As we worked on the science of scaling, we recognized early on how much the issues at stake resonated with experts across diverse disciplines and, more importantly, how much each field had to contribute to the scale-up issue for the betterment of society. With beautiful exposition, McConnell (Reference McConnell2020) brings to our attention three important notions from the behavioral psychology literature that deepen and broaden the treatment given in our work: in his own words, (1) perspectives on the importance and challenges of studying and controlling our own behavior; (2) approaches to determining the social value of intervention outcomes and the procedures for achieving them; and (3) recommendations for deploying meaningful, common measures across phases of research. We fully embrace them.
McConnell (Reference McConnell2020) wisely starts by focusing on the set of our 12 proposal that one might argue are the ones facing the most challenge to be implemented. This reminds us of a joke that goes like this: A physicist, an engineer and an economist are stranded in the desert. They are hungry. Suddenly, they find a can of corn. They want to open it, but how? The physicist says: “Let's start a fire and place the can inside the flames. It will explode and then we will all be able to eat.” “Are you crazy?” says the engineer. “All the corn will burn and scatter, and we'll have nothing. We should use a metal wire, attach it to a base, push it and crack the can open.” “Both of you are wrong!” states the economist. “Where the hell do we find a metal wire in the desert?! The solution is simple: assume we have a can opener …”Footnote 1
We acknowledge that maybe our proposals are naive and accept the difficulties in overcoming the existing incentives structure, and that we, ourselves, are active and integral part of it. Nonetheless, our analysis points to it as one of the main challenges that we need to face and overcome.
The last two notions from behavioral psychology emphasize how critical – and difficult –coordination among stakeholders is, which is necessary for the science of scaling to have a meaningful impact on society. McConnell's emphasis echoes, in part, the importance of ensuring that interventions considered for being scaled up do indeed improve the welfare of individuals, but it is broader in the sense that what he calls “social validity” includes stakeholders like caretakers of recipients, program managers in charge of the implementation, policymakers in charge of the regulation and even the community at large, beyond the direct recipients of the interventions themselves. In other words, it is this tight coordination that better aligns incentives among the stakeholders to give us the best chance of scaling a policy that leads to meaningful impact. This need for coordination arises, on a different dimension, as the need to focus on the common measures of both intermediate and final outcomes, but most critically on measuring what matters. We cannot overemphasize how much we agree: we face a lot of challenges in this effort, so let's remove all unnecessary inefficiencies that slow down scientific progress due to speaking different languages!
Raudenbush
We see our contribution as mainly methodological in nature: we turn the paradigm upside down from one that supports evidence-based policies into one that focuses on policy-based evidence generation, or more precisely, better achieve the former. We emphasize that the scale-up effect is at the core of this paradigm. Our main objective is to foster further advances where this framework is applied to narrower policy areas of importance, an example being the 2019–2020 Griffin Applied Economics Incubator at the University of Chicago with a focus on Advancing the Science of Scaling in early childhood education.
Raudenbush (Reference Raudenbush2020) provides a wonderful and thoughtful contribution in this direction to reduce educational inequality. He summarizes how the science of scaling can contribute toward this policy goal of paramount importance to our society. In our view, Raudenbush (Reference Raudenbush2020) raises two important issues: (1) the fundamental role of theory for an effective policy that can meaningfully change outcomes of interest; and (2) the need to broaden the focus from single interventions into a portfolio of interventions that can be implemented within the existing infrastructure and understanding of current practitioners.
We agree with both of these points. Although the exposition of the scale-up model was done in the context of a particular intervention, we appreciate that, for a particular policy, it is likely to take the form of a combination of interventions. Raudenbush's description of an effective school model is a perfect example of such a case. Because of this, the development of a theoretical framework plays such an important role in achieving an effective policy at scale that is implementable in a variety of situations and populations. This is in line with our Proposal #8, although from a broader perspective, as noted earlier.
Finally, Raudenbush makes an insightful comment regarding our Proposal #1, although we believe by α he means what in our notation we denote as π, the post-study probability (PSP). We agree with his main point. We do not necessarily have a firm stance on π = 0.95, except that it is a very conservative criterion. As he correctly notes, if the next best alternative choice is worse, a lower PSP can be used for policy decisions.
Soman and Hossain
Soman and Hossain's (Reference Soman and Hossain2020) thoughtful remarks make two key points: (1) they diverge from our framework by noting that governments and researchers are not independent agents in the knowledge creation market, so there is a need for collaborations between the two in evidence generation; and (2) they expand the concept of ‘voltage drop’ to include the possibility of the intervention actually backfiring when scaled up.
We agree on the second point, as it is possible for the scale-up effect to be so severe that the benefit component becomes negative. Furthermore, we believe that the proposed scale-up model can be used to identify the mechanisms underlying this extreme effect. We welcome this extension.
Regarding the first point, although for obvious reasons our framework models them as distinct agents with differing incentives, we do not deny the importance of the collaboration between them. In fact, this collaboration is tacit in our Proposals #7, #8 and #12 and explicit in our checklist when we emphasize the importance of defining the policy-relevant environment (i.e., in terms of population and situation).
As a summary, Soman and Hossain (Reference Soman and Hossain2020) conclude with 6 proposal that have connections with our 12 proposals and checklist: Proposals A and E to our Proposals #7, #8, #12 and checklist (as mentioned above); Proposal B to our Proposal #11; Proposal C to our checklist (i.e., see the section for researchers within the checklist); and Proposal D to our Proposals #5 and #9. Finally, we note that we agree with the philosophy behind Proposal F, particularly the costs that directly relate to the research process per se (i.e., on our checklist, we refer to the infrastructure to facilitate the execution of field experiments with replication purposes). However, costs representative of the environment where the policy will be scaled up should not be ‘artificially’ reduced.
Supplee and Kane
Supplee and Kane (Reference Supplee and Kane2020) note limitations and challenges to our proposals and make five recommendations moving forward. Interestingly, we not only agree with those proposals, but we also find them to be in line with our own.
(1) Expand research partnership: This is in line with our Proposal #7 and our emphasis in the checklist on the importance of partnering not only with policymakers, but also with practitioners and implementers in order to fully understand the policy-relevant environment, including the constraints raised by Supplee and Kane (Reference Supplee and Kane2020).
(2) Raise the standard of evidence incrementally through systematic reviews: This recommendation essentially captures the essence of the effort in the science of scaling and in part captured by our Proposals #1 and #12. In line with Raudenbush (Reference Raudenbush2020), Supplee and Kane (Reference Supplee and Kane2020) note the restrictiveness of Proposal #1 and its recommendation for a PSP of 0.95. We accept this and point out that this threshold is simply a conservative suggestion. As the wording of the rest of Proposal #1 suggests, the idea behind PSP is to provide a summary statistic of the scientific evidence available behind the intervention. We understand that policy decisions entail a balancing act between benefits and costs, and in certain circumstances it might not be societally optimal to wait until the PSP reaches 0.95. Our point is that a measure like the PSP has to be a component in this decision. Within context, we agree that the incremental approach can be appropriate.
(3) and (4) Prioritize real-world studies and Broaden the factors that researchers consider when designing studies for policymaking: Field experiments and our Proposals #5, #9 and #11 do indeed support these recommendations by Supplee and Kane (Reference Supplee and Kane2020).
(5) Leverage new methods to improve existing models: We fully agree with this recommendation. Although the framework is developed in the context of field experiments, as noted in footnote 3, we believe the same approach can be used for other approaches to data generation or the use of naturally occurring date. More broadly, given the resource constraints that we face, we need to focus on the optimal way of sourcing the policy-based evidence that policymakers rely on to make decisions.
