Al-Ubaydli et al. provide an insightful and directly actionable analysis of ways that social-behavioral research may exert more influence over the development and implementation of public policy. At the core of their paper is a sophisticated understanding of the “scale-up effect,” or the extent to which positive experimental effects replicate as an intervention is implemented more broadly. They offer 12 proposals based on earlier analyses (Al-Ubaydli et al., Reference Al-Ubaydli, List and Suskind2017, Reference Al-Ubaydli, List and Suskind2019) that apply careful economic and behavioral-economic principles to the vexing problem of finding ways to make better use of rational, scientific methods to effectively address societal problems in ways that we can have confidence will improve the lives of individuals and communities.
Somewhat unique to Al-Ubaydli and colleagues’ (Reference Al-Ubaydli, List and Suskind2017, Reference Al-Ubaydli, List and Suskind2019, this issue) approach is the application of economic principles, models and analyses to issues of scaling up. This approach brings clarity and a sharp, technical focus, and offers to non-economists (like me) a fresh perspective and deeper insights into analyses of what we know and what we need to discover.
Al-Ubaydli and colleagues summarize this point well:
We view this focus [on research as an essential input] as important, but also as a shortcoming of the extant literature when it comes to informing policy. In this spirit, what has been lacking is a scientific understanding of whether, and to what extent, we should use the experimental insights for policy purposes. In particular, scholars doing the basic science rarely confront the question of whether the research results scale to larger markets or different settings. And what is the science behind those expectations? (emphasis in original)
Without question, the conjoined fields of behavioral economics and field experiments, paired with investigators’ clear intentions to use science for human betterment, have propelled us forward (Levitt & List, Reference Levitt and List2009; Thaler, Reference Thaler2016; Duflo & Banerjee, Reference Duflo and Banerjee2017). Also, perhaps without question, complementary approaches, analyses and perspectives may contribute further to this effort. One complementary approach, to be explored here, is that of behavioral psychology.
More than in name only, behavioral economics and behavioral psychology share core features that may support complementary analyses. At its roots conceptually (Skinner, Reference Skinner1953) and analytically (Sidman, Reference Sidman1960), behavioral psychology (also referred to as behavior analysis) focuses on the replicable and experimental analysis of causal relations between environmental events and simple to complex forms of individual and group behavior. In the 1960s, this scientific approach was extended and explicitly applied to the challenges of understanding and improving the lives of individuals and communities (cf., Baer et al., Reference Baer, Wolf and Risley1968). In intervening years, this amalgam of the scientific analysis of behavior with the design and dissemination of evidence-based practices has contributed directly to a robust and maturing science and to a growing array of interventions that address myriad concerns and conditions.
A full analysis of the overlap and reciprocal contributions between the analyses offered by Al-Ubaydli and colleagues and the broad field of basic and applied behavior analysis is well beyond the scope of this brief response. However, at least three notions from the intellectual and practical literature in behavioral psychology may be of use here. In particular, it might be fruitful to consider: (1) perspectives on the importance and challenges of studying and controlling our own behavior; (2) approaches to determining the social value of intervention outcomes and the procedures for achieving them; and (3) recommendations for deploying meaningful, common measures across phases of research.
When scientists are the subjects
Among their 12 proposals, Al-Ubaydli and colleagues offer recommendations that cover a variety of somewhat separable topics, including research design (e.g., Proposals #2, #3, #5, #9 and #11), data analysis and interpretation (e.g., Proposals #1 and #10), scientific dissemination (e.g., Proposals #2, #3 and #4) and practical implementation (e.g., Proposals #7, #8 and #12). One could argue, however, that each of these proposals requires, at its base, changes in the behavior of one or more actors. The key to the success of each proposal is an adequate analysis of how to facilitate eventual change in the behavior of these actors. But what science will lead to the needed changes and what variables will sustain these changes in actors’ behavior?
B.F. Skinner (Reference Skinner1975) addressed a similar challenge in a commentary aptly titled ‘The Steep and Thorny Way to a Science of Human Behavior’. In that commentary, Skinner (Reference Skinner1975, p. 42) wondered:
Why has it been so difficult to be scientific about human behavior? Why have methods that have been so prodigiously successful almost everywhere else failed so ignominiously in this one field? Is it because human behavior presents unusual obstacles to a science?
Skinner goes on to note a number of possible challenges. He variously refers to these as digressions or diversions (adding “the word diversion serves me well by suggesting not only digression but dalliance,” p. 42). The foremost challenge is overcoming our feelings about our own behaviors, as we continue to believe that we can assess effects of the environment on our own actions through introspection and to attribute causal status to these epiphenomenal features of behavior change. Rather than attending to truly causal variables occurring outside of the ‘self’, too often we attribute behavioral control to individuals’ capacities and talents. Skinner argues that this is fundamentally a flawed analysis that prevents attention to important causal mechanisms that may lead to desired effects:
In the long run, aggrandizement of the individual jeopardizes the future of our species and cultures. In effect, it infringes the so-called rights of billions of people still to be born, in whose interests only the weakest of sanctions are now maintained. (Skinner, Reference Skinner1975, p. 48)
Skinner (Reference Skinner1975) offers a different approach, one based on a clear and detailed understanding of environmental effects on simple and complex behavior. Skinner sees the promise of this different approach and its early traction and application in a variety of scientific disciplines. He concludes his commentary with this:
Much remains to be done, and it will be done more rapidly when the role of the environment takes its proper place in competition with the apparent evidences of an inner life. As Diderot put it, nearly 200 years ago, “Unfortunately it is easier and shorter to consult oneself than it is to consult nature. Thus the reason is inclined to dwell within itself.” But the problems we face are not to be found in men and women but in the world in which they live, especially in those social environments we call cultures. It is an important and promising shift in emphasis because, unlike the remote fastness of the so-called human spirit, the environment is within reach and we are learning how to change it. (Skinner, Reference Skinner1975, p. 49, emphasis added)
Skinner's (Reference Skinner1975) commentary provides at least two notions worth considering as we assess ways that experiments may play a greater role in public policy. First, in the work of both Al-Ubaydli and colleagues and Skinner, there is a strong commitment to the notion that principles of science and the scientific method can be marshaled for the betterment of humankind. The idea that our behavior can be analyzed and controlled in this way is not a widely held notion among the lay public or, perhaps, our scientific colleagues. However, Al-Ubaydli and colleagues’ examples provide evidence of a growing commitment as well as success in addressing wider sweeps of both the physical and behavioral sciences. In short, we are now conducting science on policies and practices that are worth scaling up. To advocate for expanded use of this scientific approach, we need clear demonstrations of broad benefits and a recognition of the power of this approach and the adoption of these practices by scientists and policymakers.
Second, Skinner's (Reference Skinner1975) analysis clearly suggests the importance of arranging environmental contingencies that will support Al-Ubaydli and colleagues’ proposals. Indeed, we may need to reform many scholarly contingencies that serve to constrain or limit these proposed practices. For example, thoughtful analyses of individual and group decision-making have begun to address the need for changes to things like journal publication decisions, promotion and tenure decisions, grant-making and policy development (Biglan & Embry, Reference Biglan and Embry2013; Wilson et al., Reference Wilson, Hayes, Biglan and Embry2014). The path forward likely requires clear-eyed analyses like those offered by Al-Ubaydli and colleagues, continued empirical analysis of these factors and their contributions to positive change and purposeful actions – richly supported by colleagues and communities – to bring these ideas to broad-scale application. In this way, we can change the environments that in turn change us.
Designing for scale by focusing on desired procedures and outcomes
Most of Al-Ubaydli and colleagues’ proposals have direct consequences for the design and conduct of policy-relevant research, but Proposal #11 has perhaps the greatest bearing on the design of individual studies and subsequent programs of research:
Researchers should backward induct when setting up their original research plans to ensure accurate and swift transference of programs to scale.
In part, this requires focusing on outcomes that individuals, communities and policymakers value and producing these outcomes in ways that various stakeholders find preferable. Applied behavior analysis has grappled with this issue since its inception. Researchers and practitioners discovered that powerful behavior change procedures could produce outcomes that failed to fully meet socially important goals (e.g., Winett & Winkler, Reference Winett and Winkler1972) or used procedures to achieve goals that failed to meet universal or local social standards (e.g., Sidman, Reference Sidman1993). From this arose attention to social validity, or assessment of the social significance of goals of intervention, the social appropriateness of procedures used to meet these goals and the social importance of the effects achieved (Wolf, Reference Wolf1978).
Although applied behavior analysis has maintained a clear and primary focus on objective analysis of environment–behavior relations, as Wolf (Reference Wolf1978) eloquently argued, intervention developers often need ‘subjective’ information from stakeholders to evaluate the contextual fit or appropriateness of any intervention. These stakeholders include direct recipients of interventions, caretakers of these recipients, program managers conducting interventions, policymakers regulating or requiring these interventions, and oftentimes the community at large. As noted earlier, the goal of work being discussed here is to find ways to make better use of science to address problems of the social condition in ways that we have confidence will improve the lives of individuals and communities. In addition to scientific methods, then, our work must include ways to determine what constitutes ‘improvement’.
Since the 1970s, attention to social validity in the many branches of behavioral psychology has deepened and broadened to include: (1) attention to the preferences and requirements of ‘consumers’, or the intended beneficiaries of any intervention (Schwartz, Reference Schwartz1991); (2) relations between interventionists’ preferences or the rated acceptability of interventions and the fidelity with which they implement that intervention (e.g., Peterson & McConnell, Reference Peterson and McConnell1996; Allinder & Oats, Reference Allinder and Oats1997); (3) design principles for developing or selecting interventions (e.g., State et al., Reference State, Harrison, Kern and Lewis2017); and (4) using social validity measures throughout the iterative development of interventions to facilitate implementation (Goldstein, 2016). Three aspects of social validity – the social significance of goals, the social appropriateness of procedures and the social importance of effects – serve as foundational elements for the emerging discipline of implementation science (Fixsen et al., Reference Fixsen, Naoom, Blase, Friedman and Wallace2005; Cook & Odom, Reference Cook and Odom2013; Metz & Miller, Reference Metz, Miller, Carta and Miller2018) that may also bear on the current discussion.
It is likely that design that systematically includes customer preferences (‘voice of the customer’) or that employs design thinking (Both, Reference Both2010; Boy & Narkevicius, Reference Boy, Narkevicius, Aiguier, Boulanger, Krob and Marchal2014; Harte et al., Reference Harte, Glynn, Rodríguez-Molinero, Baker, Scharf, Quinlan and ÓLaighin2017) is more common in business settings than in academic programs of research and development. However, there seems to be good reason to consider carefully what policymakers and practitioners want, and what they are willing and able to do to produce these outcomes, as a necessary feature of intervention development, evaluation and dissemination. This should smooth the process of scaling up, where articulation of goals and priorities and allocation of needed resources (at the individual, group and organizational levels) become instrumental (Duda et al., Reference Duda, Fixsen, Blase, Peisner-Feinberg and Buysse2013).
Measuring what matters
Across several of Al-Ubaydli and colleagues’ proposals, one can see an argument that alignment and coordination of measurement is necessary across all stages of the research, development and broad-scale implementation of science-based policy initiatives. The importance of this alignment is clear in proposals for the replication and confirmation of experimental effects (Proposals #1–3), systematic variation of intervention components and their consequences for program impacts (Proposal #5) and attention to continuous empirical refinement of intervention as it is introduced at scale (Proposal #12). The alignment and coordination suggested by these proposals will increase the confidence in intervention impacts in a wider range of practical situations and will likely contribute to more complete replication of both treatment procedures and their assessed effects.
One of the primary inferences required by policymakers when evaluating a series of experimental studies is the degree to which intervention will produce a desired outcome.Footnote 1 Consistency in measurement across early-, middle- and late-stage experimental implementations makes such inferences stronger.
The logic of this is straightforward: with more instances where intervention inputs or conditions vary but the method of outcome assessment is comparable, the more directly we can infer effects of intervention inputs (even when outcomes vary across experiments). For example, assume researchers have access to two different measures of school engagement. Further assume that these measures appear similar on face, and that past research has demonstrated a modest but reliable correlation of their scores. Now consider two scenarios, one in which two independent research teams demonstrate reliable effects of variations in a positive behavior support program on school engagement and both use the same measure of outcome, and a second where each team demonstrates important change but with different outcome measures. Whereas either scenario likely increases our confidence in the intervention and its variations tested, the first scenario's common outcome assessment across multiple experiments allows for lower-inference analysis (and, thus, greater confidence) when considering strong candidates for scaling up.
Early in the development of applied behavior analysis, Bijou et al. (Reference Bijou, Peterson and Ault1968) described benefits of common or tightly aligned measures of outcomes across phases of research and development. Bijou and colleagues’ (Reference Bijou, Peterson and Ault1968) analysis relates directly to the broader case of planning a program of research that leads to broadly scalable – and effective – interventions. Using common measures across phases of research facilitates inferences of generalizability, improves the basis for moderator-like analyses and produces low-inference bases for intervention application and continuous improvement at scale.
General outcome measures of child development and academic achievement (Fuchs & Deno, Reference Fuchs and Deno1991) provide excellent examples of this alignment. Measures of reading (Deno et al., Reference Deno, Mirkin and Chiang1982), mathematics (Lee & Lembke, Reference Lee and Lembke2016) and written expression (Deno et al., Reference Deno, Mirkin and Chiang1982; McMaster et al., Reference McMaster, Du, Yeo, Deno, Parker and Ellis2011) were constructed and evaluated in specialized research settings and have been used extensively as originally developed to both evaluate school-based interventions and support teachers’ and administrators’ practice (Mellard et al., Reference Mellard, Frey and Woods2012; Tusing & Breikjern, Reference Tusing and Breikjern2017; Whitley, Reference Whitley2019). In this way, alignment of methods and interpretation is strengthened across a wide range of experimental investigations and practical applications.
Considerations regarding social validity, discussed earlier, bear directly on this discussion as well. If researchers select measures for assessing intervention outcomes that can be used across both settings and initiatives and that have demonstrated relation to stakeholders’ preferences, researchers and practitioners can be more certain that interventions are both reliable and produce improvements at all levels of research, development and implementation. The notion, then, is this: to the extent possible, investigators’ efforts both to design from the beginning interventions that address practical requirements for policy-relevant research (Al-Ubaydli and colleagues’ Proposal #11) and to promote replication with diminished voltage effects (Proposals #1–3) will be strengthened by selecting measures that can be used with fidelity in both specialized research and typical practice settings and that represent socially important outcomes for the recipients of intervention.
Conclusion
One other possible point of contact between Al-Ubaydli and colleagues’ analysis and that of behavioral psychology is a deep reliance on low-inference, empirical analyses across multiple studies. Behavioral psychology and applied behavior analysis not only have rested on commitment to objective and empirical analysis, but also have leaned toward an inductive, parsimonious description of cause-and-effect relations to build theory and evidence-based practices (Skinner, Reference Skinner1950). Similarly, Al-Ubaydli and colleagues rely on the accrual and ongoing refinement of information from individual studies, constantly comparing current to prior empirical observations in order to note change. In these ways, both approaches yield increasingly complex, but always trustworthy, descriptions of current knowledge and, as a result, some guidance for what needs to be done next in both research and practical application.
The scientific community is increasingly directing its attention and great capacity to its role in fostering development and promoting a society of individuals who thrive and contribute to their communities, today and for generations to come (Wilson et al., Reference Wilson, Hayes, Biglan and Embry2014; Darcy Mahoney et al., Reference Darcy Mahoney, McConnell, Larson, Becklenberg and Stapel-Wax2019). This is a high calling, but also one with enormous challenges. Success, or even modest progress, will likely require creative and insightful analysis, humility and self-reflection and deliberate and sustained effort. This will require shared effort across multiple scientific disciplines, but also invites individual disciplines to bring their unique perspectives, resources, tools and knowledge to bear on matters of social importance.
We are fortunate to have the analysis and proposals offered by Al-Ubaydli et al. in this issue and in their earlier work (Al-Ubaydli et al., Reference Al-Ubaydli, List and Suskind2017, Reference Al-Ubaydli, List and Suskind2019). Scientists, policymakers and others will benefit from attention to their work and from the ways this work continues to inform and iterate. We also will benefit if we notice ways in which this analysis aligns with, is informed by and in turn informs recommendations arising from other disciplines and perspectives. There is much to do, but we are off to a good start.
Acknowledgments
The author thanks Dr. Howard Goldstein for extensive comments on an earlier draft.
Financial support
The research reported here was supported by the Institute of Education Sciences, US Department of Education, through Grant R305A160034 to the University of Minnesota.
Disclaimer
The opinions expressed are those of the author and do not represent views of the Institute or the US Department of Education.
