The 1996 Personal Responsibility and Work Opportunity Reconciliation Act marked the culmination of a long political fight to reform public assistance for low-income families in the US. The bill replaced Aid to Families with Dependent Children (AFDC), a longstanding entitlement program, with temporary and conditional cash assistance designed to incentivize poor mothers to join the workforce. The success of this conservative policy agenda is commonly attributed to the Republican takeover of Congress during the mid-1990s, but its initial ascendance benefited from the support of new bodies of policy research with newfound influence in the policymaking process (Bane and Ellwood Reference Bane and Ellwood1996; Haskins Reference Haskins2007; Nathan Reference Nathan2000; Weaver Reference Weaver2000). By importing more scientific methods from academia into the field of policy research, experts produced convincing evidence that the rising cost of AFDC could be curbed by moving welfare recipients into the labor market. For the first time, policy evaluation researchers successfully conducted a series of randomized experiments to test a major social policy reform initiative in the context of existing public service agencies, and these experiments found small but statistically significant improvements in welfare recipient earnings and program cost savings. The findings caught the attention of policymakers, contributing to bipartisan support for mandating employment programs as a condition for welfare receipt, and Congress, subsequently, institutionalized the experimental method as the new standard for evaluating welfare reform proposals (Baum Reference Baum1991; Greenberg and Wiseman Reference Greenberg, Wiseman, Manski and Garfinkel1992; Haskins Reference Haskins1991; Nathan Reference Nathan2000; Szanton Reference Szanton1991).
This early wave of social policy experiments marked a shift in the methodological norms and expectations surrounding policy evaluation research and the use of experimental design. Before the 1970s, evaluation researchers expressed concern that the classic experimental method, well established in laboratory research and clinical trials of new medical interventions, could be faithfully implemented in the context of complex public service bureaucracies. Although large-scale randomized trials of proposed Negative Income Tax programs had generated enthusiasm for “social experiments” as a policy evaluation tool, these early field experiments were costly, and they were developed from scratch by academic researchers to test carefully defined hypotheses (Heckman Reference Heckman, Manski and Garfinkel1992). A more applied version of the social experiment that tested reforms within existing service delivery systems only gained legitimacy after the Manpower Development Research Corporation (MDRC) – a non-profit research evaluation contractor – completed a series of experiments demonstrating the positive impact of welfare-to-work employment programs during the 1980s (Gueron and Pauley Reference Gueron and Pauly1991; Gueron Reference Gueron2003). The use of randomized experiments in public policy became more frequent during this period, helping set the stage for the subsequent ascendance of randomized trials across multiple domains of policy research (Banerjee and Duflo Reference Banerjee and Duflo2017; Connoly et al. Reference Connolly, Keenan and Urbanska2018; Greenberg et al. Reference Greenberg, Shroder and Onstott1999).
This article asks how randomized experiments first became methodologically viable as a general tool for evaluating new social policy reforms, and how this evolving expert consensus impacted the reform process. Like any method, field experiments using random assignment are more well-suited to studying some interventions and less suited to others. Social programs involving cash transfers, like the negative income tax proposals of the 1970s, are easily amenable to an experimental research design. Researchers can control the delivery of discrete interventions involving specific cash amounts and eligibility rules across treatment and control groups. In the context of the late twentieth century US welfare reform debates, however, policymakers considered more complex interventions at the nexus of public assistance programs and employment services. The flagship welfare-to-work employment services program for AFDC recipients, the Work Incentives Program (WIN), raised a host of pressing questions about program implementation and service delivery strategies. WIN encouraged AFDC beneficiaries to move off welfare and into regular employment through support services and placement in various kinds of employment programs, ranging from job search activities to vocational training and work experience programs. The effective implementation of WIN required coordination across welfare and employment agencies, monitoring clients throughout the process, and consistently enforcing program guidelines. Beginning in the 1970s, a series of WIN demonstration programs – local pilot projects conducted for the purpose of research – invited researchers to evaluate competing approaches to delivering the WIN program model.
I argue, first, that MDRC overcame methodological challenges in the evaluation of welfare-to-work demonstration programs by reframing the object and objective of its experiments. During the early 1980s, MDRC began designing experiments to evaluate the general WIN program model, which mandated that welfare recipients with school-aged children seek employment, rather than evaluating specific programmatic reforms or administrative improvements intended to make WIN more effective. By redesigning and reinterpreting its policy experiments to estimate the holistic impact of the WIN service delivery system, MDRC avoided evaluating more flexible or discretionary components of these systems that were less amenable to random assignment. I show how this reframing benefited from political trends in welfare reform and research funding and obscured substantive trends in the design and implementation of WIN demonstration programs that might have suggested alternative lessons about how to improve AFDC. Limitations aside, MDRC’s new WIN experiments in the early 1980s produced findings that were both statistically significant and politically relevant for legislators in Washington DC (Baum Reference Baum1991; Greenberg and Wiseman Reference Greenberg, Wiseman, Manski and Garfinkel1992; Haskins Reference Haskins1991).
Second, I argue that MDRC’s decisions about how to frame welfare-to-work programs channeled the welfare reform debate and helped consolidate a larger trend in contemporary US social policy. The contemporary turn toward a more restrictive and punitive welfare state, perhaps best exemplified by the repeal and replacement of AFDC in 1996, has been characterized by a trend away from the frontline expertise of social workers and case managers in favor of more routinized public service systems and market-based interventions premised on individual responsibility (Brodkin Reference Brodkin1986, Reference Brodkin2006; Watkins-Hayes Reference Watkins-Hayes2009; Berman Reference Berman2022). In the case of welfare reform, this trend was already underway before the 1980s, but the institutionalization of randomized experiments in the evaluation of welfare reform proposals circumscribed the kind of reform strategies that states pursued and that experts legitimized. Not only did the immediate impact of MDRC’s initial findings increase bipartisan support for conditioning welfare receipt on job-related activities, but its new brand of policy experiments also helped delineate and focus the welfare reform debate around discrete, rule-based policy interventions, precluding investment in the organizational capacity of public service agencies.
This sociological history of welfare reform is rooted in Michel Foucault’s broader critique of neoliberal government (Foucault Reference Foucault2007, Reference Foucault2008). Foucault’s account of the contemporary evolution of liberal governance goes beyond the rise of ideologies supporting free markets and state withdrawal. His work considers the implicit logic of neoliberal government embedded in the practical and intellectual devices of contemporary rule (Rose Reference Rose1993). In contrast to the disciplinary technologies of mid-twentieth century welfare state liberalism, the innovations of neoliberal government rely on techniques for enumeration, calculation, and monitoring that enable indirect forms of government through market logics and individual responsibility (Barry et al. Reference Barry, Osborne and Rose1996; Burchell et al. Reference Burchell, Gordon and Miller1991; Feeley and Simon Reference Feeley and Simon1992). Neoliberal government, thus, elevates formal quantitative experts like economists and accountants over the more substantive and specialized expertise of social workers, clinicians, and educators.
Exemplifying this trend, the expansion of the US social state since the 1960s prompted a parallel expansion of economics-trained policy analysts within and outside government as government agencies increasingly turned to cost-benefit analyses and causal estimates of program impacts to dictate policy decisions (Berman Reference Berman2022; Breslau Reference Breslau1998; O’Connor Reference O’Connor2009). These new methods for rigorous policy analysis were not simply imported wholesale from academic economics as even the theories and methods of microeconomics research evolve in response to political demand and changes in the field of policy institutions (Breslau Reference Breslau1998; de Souza Leão and Eyal Reference de Souza Leão and Eyal2019; Griffen Reference Griffen2023). This article Interrogates the contemporary history of US welfare reform by probing its recursive relationship with a novel methodological approach to social program evaluation.
This analysis is based on a case study of MDRC and its pathbreaking research during the 1970s and 1980s. MDRC is central to this history not merely because it was one of the few organizations that specialized in program evaluation research using random assignment. Other research firms conducting policy experiments during this period – primarily, RAND, Abt Associates, and Mathematica Policy Research – tested original interventions that directly served individuals without involving existing public service bureaucracies. MDRC, in contrast, was tailored to manage more complex collaborations with local agencies. While earlier policy experiments produced theoretically interesting research findings, MDRC’s pioneering approach to staging experiments in the context of welfare and employment public service delivery systems had direct political influence on national policy debates during the 1980s. Rather than cover the broader scope of randomized trials in contemporary public policy during this early period, this article focuses on one crucial and easily overlooked methodological innovation that helped make the experimental method more widely applicable and politically relevant.
Alongside a review of policy evaluation methodology literature pertaining to early social experiments, this article draws on a range of MDRC research reports, monographs on the organization’s history, first-hand accounts of MDRC leadership, and secondary sources to examine major organizational decisions and key events that shaped the trajectory of its work. MDRC provided archived hard copies of research reports that have not been digitized or widely distributed. I track changes in the design and interpretation of MDRC’s policy experiments over time, and I trace the institutional, discursive, and political conditions that supported or compelled these changes.
After situating my research in existing approaches to the study of policy expertise, this article proceeds by providing background on the methodological challenges facing randomized experiments in social policy research before the 1980s. The subsequent section details the creation and early evolution of MDRC, an organization designed to orchestrate complex, collaborative policy experiments. Moving to the 1980s, I review trends in welfare reform and government research funding that reshaped the market for social policy evaluation research. The article, then, turns to an analysis of MDRC’s influential wave of WIN evaluations to show how MDRC adapted the design and logic of its experiments to bypass methodological challenges that it could no longer attempt to micromanage. Finally, this article offers an account of the political implications of MDRC’s new method, and it concludes with a discussion of the relevance of this analysis for the contemporary politics of social policy.
Studying the rise of formal policy expertise
The relationship between politics and knowledge remains an important area of inquiry for social scientists interested in understanding policy change. A longstanding body of research on the role of ideas in policymaking examines how policymakers use expert knowledge to guide their decisions, how political institutions create openings for specific bodies of expertise to gain influence, and how established bodies of knowledge shape policy paradigms that structure political decision-making (Hall Reference Hall1993; Heclo Reference Heclo1974; Stone Reference Stone2002; Weir and Skocpol Reference Weir and Skocpol1983). Contemporary approaches drawing on the social study of science and technology further complicate this dynamic relation between research and practice by interrogating the “co-production” of knowledge and society through processes of scientific knowledge production that are embedded in state institutions and social movements (Callon and Rabeharisoa Reference Callon and Rabeharisoa2003; Epstein Reference Epstein2009; Eyal Reference Eyal2013; Jasanoff Reference Jasanoff2004). In the context of social policymaking, policy analysts achieve scientific objectivity and political neutrality through their increasing reliance on research methods and substantive knowledge borrowed from academic economists, but the disciplinary knowledge of economists is not completely autonomous from the state. The scholarly theories and methods adopted by policy experts are often forged in dynamic relation to changing policy institutions, leveraging state-supported bureaucracies and data systems to construct measurable and predictable objects of policy intervention (Breslau Reference Breslau1998; de Souza Leão and Eyal Reference de Souza Leão and Eyal2019; Griffen Reference Griffen2023).
Policy research began to rely heavily on the scientific rigor of economistic methods during the 1960s. In the wake of the sudden expansion of the US social state initiated by the Lyndon B. Johnson Administration, government agencies and a growing industry for poverty policy research hired academically trained economists and institutionalized formal methods for causal analysis within the design, implementation, and evaluation of social programs (Aaron Reference Aaron2010; Berman Reference Berman2022; Haveman Reference Haveman1997; O’Connor Reference O’Connor2009). Early critics of this trend emphasize the inherently conservative implications of imposing sometimes unattainable standards for establishing scientific certainty as benchmarks for justifying new programs (Aaron Reference Aaron2010; Lindblom and Cohen Reference Lindblom and Cohen1979; Rossi Reference Rossi1987; Wilensky Reference Wilensky1997). Traditionally, social policy initiatives might be driven by broad social reform agendas like the War on Poverty. New programs would include constant, informal processes for evaluation and adaptation in response to research, staff experience, and community demand. The formal analysis of program impacts reduces policy objectives to easily quantified outcome metrics like individual earnings and net program costs rather than broader structural objectives like the empowerment of marginalized constituencies (Aaron Reference Aaron2010; Berman Reference Berman2022; O’Connor Reference O’Connor2009; Wildavsky Reference Wildavsky1979). Programs serving impoverished and socially marginalized groups, particularly public assistance programs and targeted employment services, lack a politically influential base of support in the US and are, thus, more reliant on technocratic justification rooted in formal policy analysis (Fourcade Reference Fourcade2009: 113; Wilensky Reference Wilensky1997).
Among the methodological techniques for the causal analysis of program impacts, the experimental method was lauded for its potential for rigor and precision even when the scope and practicality of its application was in question. In the absence of a randomly assigned comparison group, economists developed statistical models to control for complicating factors and estimate the impact of social programs, but these analyses raised esoteric methodological debates and often failed to produce statistically significant results (Aaron Reference Aaron2010; Breslau Reference Breslau1998). Repeated efforts to extend the use of randomized trials in social policy evaluation research did not yield much politically impactful research before MDRC’s successful welfare reform experiments in the 1980s. Although scholarly critics continued to question the generalizability and appropriate interpretation of MDRC’s research (Brodkin and Kaufmann Reference Brodkin and Kaufman2000; Epstein Reference Epstein1997; Oliker Reference Oliker1994; Peck Reference Peck1998b, Reference Peck2001), and although these experiments did not match the rigor of laboratory experiments or more carefully controlled field trials (Heckman Reference Heckman, Manski and Garfinkel1992; Heckman and Smith Reference Heckman and Smith1995), they were, nevertheless, embraced by policy experts and institutionalized as the standard method for evaluating new welfare reform proposals (Baum Reference Baum1991; Gueron Reference Gueron, Banerjee and Duflo2017; Haskins Reference Haskins1991). The number of social policy experiments conducted in the US steadily increased during this period with a total of ninety-six studies completed during the 1980s relative to only forty-one during the 1970s (Greenberg et al. Reference Greenberg, Shroder and Onstott1999).
The growing influence of policy experiments in the evaluation of welfare and employment programs reinforced the authority of formal policy evaluation research and elevated a new crop of specialized research organizations, led by MDRC, over the econometric analyses of specialized economists. George Breslau (Reference Breslau1998) attributes the triumph of these research contractors and their experimental method to their ability to combine scientific rigor with applied research that produces easily interpreted, politically usable results. This account is accurate but begs the question, how did research firms like MDRC successfully design experiments that were simultaneously rigorous and applied? The problems facing policy experiments during the 1970s revolved around the difficulty of implementing random assignment and controlling the delivery of policy interventions in the real-world context of complex public service bureaucracies (Morehouse Reference Morehouse1972; Riecken and Boruch Reference Riecken and Boruch1978; Rossi Reference Rossi1987; Weiss and Bucuvalas Reference Weiss and Bucuvalas1980). The rise of policy experiments did not merely entail the uptake of an existing methodology already suited to the task. Accounting for this rise requires examining the entrepreneurial efforts of MDRC as it developed an innovative research design amidst a changing political and institutional context for evaluation research.
The more recent spread of randomized controlled trials in a very different context, the evaluation of international development interventions, illustrates how the rigorous methods of economics research respond to opportunities generated by changing policy institutions. Gil Eyal and Luciana de Souza Leão (Reference de Souza Leão and Eyal2019) attribute the lauded introduction of randomized trials in the field of international development to the growing influence of nongovernmental organizations and private foundations. They argue that the experimental method operated as a “hinge” (Abbot Reference Abbott2005) – a mechanism constructed to serve the distinct interests of academic economists and development organizations, thus linking these actors across fields. More generally, Zach Griffen’s work (Reference Griffen2023) illustrates the embeddedness of subfields of the economics discipline in policy institutions, tracing how the discipline evolved through its involvement with the development of new social programs since the 1950s. The social scientific concepts and methodological tools that state actors rely on to establish their mechanical objectivity are themselves forged through diffuse networks with state agencies and political movements.
In the context of US social policy, this recursive relationship between scientific methods and political change is potentially more dynamic when mediated by private policy research organizations. Compared to the more autonomous scientific expertise of university economists, MDRC prioritized producing politically relevant research, and it relied on funding from government and major foundations. It, nonetheless, depended on the scientific rigor and neutrality of its evaluation research and, specifically, of its randomized policy experiments. This article shows how MDRC established the viability and rigor of its policy experiments even as it relinquished autonomy in the design of these experiments, giving up not just control over the objectives of its research but, crucially, influence over the design and implementation of the demonstration programs that it studied. Sacrificing a degree of methodological precision and potentially valuable political insight, its newly designed policy experiments established a hinge connecting the scientific pursuit of valid and meaningful research findings with the political pursuit of employment-oriented welfare reform. MDRC, thus, proved that the experimental method could be widely applied as the standard for evaluating welfare reform initiatives, and, I argue, this evaluative standard helped to channel the trajectory of welfare reform toward a more restrictive and punitive framework for public assistance.
If policymakers are now accustomed to “thinking like an economist” (Berman Reference Berman2022), then we might better grasp the orientation and limits of contemporary policy paradigms by asking how economists, and economics-informed policy experts, established diffuse expert networks in new policy arenas and helped to construct legible objects for government intervention (Eyal and Buchholz Reference Eyal and Buchholz2010). This approach to analyzing the neoliberal welfare state goes beyond ideological preoccupations with disciplining the poor and the punitive political rhetoric that villainized welfare recipients. It focuses, instead, on the practical and intellectual devices that gave form and function to a political agenda that initially stalled in its efforts to retrench the welfare state and to force welfare beneficiaries to work for their benefits (Pierson Reference Pierson1994). Although the 1996 legislation that transformed US public assistance policy was driven by the growing influence of a longstanding right-wing political agenda, it also depended upon bipartisan support and decades of trial-and-error as local bureaucracies developed policy institutions that could produce reliable and measurable effects in line with multiple political objectives. The purportedly politically neutral and scientifically objective work of evaluation research played an indirect yet instrumental role in forming the institutional means and logic of contemporary poverty policy.
Initial challenges facing social policy experiments
MDRC was established in 1974 to facilitate the task of implementing randomized experiments in the evaluation of social programs. Previous experiments involving federal government demonstration programs ran into complex methodological challenges. In the 1960s, the new Office of Economic Opportunity waged a War on Poverty, in part, by developing local Community Action Programs designed to be responsive to local conditions and test out multiple interventions. Nixon Administration officials, eager to reign in these ambitious progressive programs, introduced more formal evaluation standards that favored the use of randomized trials (Breslau Reference Breslau1998; O’Connor Reference O’Connor2009). Around the same time, the use of the experimental method gained traction in policy evaluation research thanks to a small number of more costly field experiments beginning with the New Jersey Negative Income Tax experiment. The latter experiments were high-profile demonstrations of the rigor and precision of the experimental method, but they involved large-scale field experiments developed from scratch by economists to test academic hypotheses (Heckman Reference Heckman, Manski and Garfinkel1992). The attempt to use randomized trials to evaluate locally-run, multifaceted Community Action Programs required much less expense but presented an entirely distinct set of methodological challenges and failed to replicate the elegance and rigor of the Negative Income Tax experiments (Morehouse Reference Morehouse1972; Rossi Reference Rossi and Weiss1972).
Social experiments involving Community Action Programs succumbed to various challenges on the ground, and these failures sparked a broader methodological debate about the limits of randomization in the evaluation of demonstration programs operating within ordinary public service bureaucracies (Morehouse, Reference Morehouse1972; Rossi, Reference Rossi and Weiss1972; Schulberg and Baker Reference Schulberg and Baker1968; Weiss and Rein Reference Weiss and Rein1970). Social programs often rely on organizational flexibility to tailor services to local clientele, which complicates the work of formal evaluation researchers, who seek to isolate specific and replicable interventions for causal analysis (Morehouse Reference Morehouse1972). Program operators sometimes adjust normal operations to local conditions, including subtle changes to organizational culture and routines that might diverge from the prescribed experimental treatment (Weiss and Rein Reference Weiss and Rein1970: 103–105). An expert panel for the Social Science Research Committee published a volume on policy experiments that cautioned researchers to recognize the tension between operating a public service delivery system, with its moving parts and multiple variables, and implementing controlled experiments, which seek to clearly define and stabilize experimental programs (Riecken and Boruch Reference Riecken and Boruch1974). Many leading methodologists expressed these concerns but continued to promote the potential of “social experiments.”
The methodological challenges raised in this early literature can be summarized in three categories. The most commonly cited issue was the coordination challenge, in which researchers struggled to secure the cooperation of unsympathetic or protective program staff. This problem was not simply a matter of stubborn resistance from local bureaucrats fearful of outside scrutiny. Controlled experiments require intrusive monitoring and intervention in program operations, and the process of random assignment can interfere with preferred recruitment and selection strategies. Second, researchers faced a standardization challenge – the problem of defining and ensuring the delivery of an explicit and consistently executed treatment and control state. Once study participants are randomly assigned, researchers must ensure that program delivery does not deviate from the prescribed treatment and that controls are not exposed to the treatment. Given the complexities involved in the delivery of a public service program, this standardization work is more an art than a science, requiring researchers to frame the core elements of the treatment and relegate certain other processes to a black box. Finally, researchers balanced these challenges with the logistical challenge of designing an experiment within resource and time constraints that offers a reasonable chance of rejecting the null hypothesis. If the study sample is too small or diluted through attrition, or if the research process impedes treatment delivery or otherwise fails to substantially distinguish the treatment group’s experience from that of the control group, then the study will be unlikely to detect statistically significant impacts regardless of the treatment’s potential merits.
Three main policy research contractors – MDRC, Mathematica Policy Research, and Abt Associates – specialized in social policy evaluation using random assignment, and MDRC, in particular, was created for the singular purpose of helping manage the collaboration between evaluation researchers and policy demonstration programs. MDRC was tasked with overseeing and helping develop local policy demonstrations to ensure that they would be compatible with the implementation of random assignment and the successful completion of a formal experiment. In the context of welfare-to-work reforms, this careful coordination involved specifying and monitoring the experience of an experimental treatment group receiving the new version of AFDC as well as the experience of the control group, which was entitled to benefit from the status quo AFDC program as well as normal employment-related services generally available to this population. The careful collaboration of researchers and program staff was necessary to ensure that these distinct program experiences were faithfully maintained in the context of the experiment (Nathan Reference Nathan2000). MDRC, thus, managed competing pressures to control the design and implementation of the demonstration programs that it studied while conceding to the priorities of the policymakers and program staff with whom it collaborated.
MDRC’s early years: supported work and the WIN Research Laboratory Project
The Ford Foundation created MDRC in 1974 with the expectation that it would leverage its prestigious board and academically trained staff to both manage the implementation of demonstration programs and participate in the design of experimental research. Ford’s Vice President for Domestic Affairs, Mitchell Sviridoff, had embarked on an ambitious national demonstration project to test out a new employment program called Supported Work, and he pursued a well-organized, top-down evaluation strategy to draw reliable evidence from the pilot project. Before coming to Ford, Sviridoff had run a major community action-style antipoverty program and grown frustrated with the failure of such programs to produce reliable research findings that could be easily replicated beyond the initial demonstration site. He sought to replace the decentralized, community-driven innovation process with a more systematic, executive-centered approach. To this end, Ford established MDRC as an “intermediary” between the sponsors of demonstration projects like the Ford Foundation, program evaluators, and the local public service agencies that implement these projects on the ground (Breslau Reference Breslau1998; O’Connor Reference O’Connor2009: 232–37). In its National Supported Work Demonstration, MDRC redesigned the program initially developed by the Vera Institute in New York, partnering with private program providers in cities across the country to implement the nationwide version. The firm subcontracted out the experimental research component, approving the initial research design and overseeing its implementation (Brecher Reference Brecher1978).
MDRC was carefully designed to handle the various political, scientific, and managerial components of its first project. Its board of directors included established experts and government officials, led by former undersecretary for the Department of Health, Education, and Welfare Richard P. Nathan. It was staffed with academically trained experts who had administrative experience (Brecher Reference Brecher1978). For the Supported Work study, MDRC recruited Mathematica Policy Research, the organization that conducted the Negative Income Tax experiments, to design and implement the multi-site randomized experiment with MDRC’s supervision and administrative support. While Mathematica was initially hesitant to take on the ambitious evaluation study, MDRC was able to alleviate concerns that the experiment would succumb to logistical challenges by ensuring a sample size large enough to feasibly reject the null hypothesis (Ball Reference Ball1977). MDRC secured this large sample by managing a sprawling demonstration program, standardizing operations across more than a dozen cities. This entailed overseeing program development, recruitment, and implementation while monitoring the random assignment process and ensuring consistency across sites. Local agencies delivering the Supported Work program often relied on partnerships with public agencies in the recruitment process, and when these recruitment efforts created conflicts and potential legal problems, MDRC coordinated with federal government sponsors of the project who could step in to resolve these local issues (Ball Reference Ball1977). This managerial feat illustrated the research contractor’s unique capacity to develop and manage a successful social experiment from start to finish.
The National Supported Work Demonstration was a methodological success, but it failed to resonate with the priorities of policymakers, despite compelling findings and some high-profile media attention (Auletta Reference Auletta1999). The intensive program was shown to increase income from employment for at least one of its targeted subgroups – unemployed mothers on welfare. While the Vera Institute’s original Supported Work program targeted hard-to-reach populations like formerly incarcerated adults and people with substance abuse problems, MDRC added AFDC recipients to help attract funding from federal agencies operating public assistance programs (Brecher Reference Brecher1978). Political momentum for programs promoting employment among welfare recipients was growing, but politicians were preoccupied with lowering the cost of existing programs rather than constructing innovative and expensive interventions like Supported Work (Gueron Reference Gueron2003; Nathan Reference Nathan2000). Moving forward, MDRC would turn its attention to programmatic reforms initiated by government agencies and policymakers, confronting more daunting methodological challenges as it progressively relinquished control over the programs that it studied.
MDRC’s next major project brought it into closer cooperation with government agencies at the federal, state, and local level. The WIN Research Laboratory Project pioneered a new collaborative approach to demonstration research involving the Work Incentive Program (WIN) for AFDC recipients. The WIN program was enacted in 1967 to support and incentivize AFDC beneficiaries – primarily low-income mothers – in the transition from public assistance to full-time employment. Under an earlier AFDC statute, states could request that the federal government waive specific program components to permit isolated demonstration projects testing proposed changes to AFDC and challenges facing the WIN program, in particular, prompted a wave of AFDC demonstration programs in the 1970s (Williams Reference Williams1994). Across the country, WIN implementation was notoriously poor due to limited resources and lack of compliance among frontline staff, and state WIN demonstration programs attempted to improve implementation by modifying the delivery system and specific program components (Burbridge and Nightingale Reference Burbridge and Nightingale1989; Williams Reference Williams1994). In an effort to draw national lessons from this wave of local WIN demonstration programs, the national WIN program director, Merwin Hans, decided to conduct a multi-state evaluation study during the late 1970s. Hans was intrigued by the potential rigor and reliability of social experiments, and he solicited MDRC to develop and manage the national WIN Research Laboratory Project (Gueron Reference Gueron2003; Leiman and Seessel Reference Leiman and Seessel1982).
MDRC reluctantly took on the project and was quickly forced to cede control of the development of individual demonstration programs to local program directors in order to secure their cooperation (Leiman and Seessel Reference Leiman and Seessel1982). The project included four sites testing four distinct interventions, two of which succumbed to conflict and disruption and were never completed despite MDRC’s efforts (Greenberg and Shroder Reference Greenberg and Shroder1997; Leiman and Seessel Reference Leiman and Seessel1982). Of the two completed experiments – conducted in Louisville and Denver, respectively – one of them produced statistically significant findings that seemed more consequential for their methodological implications than any immediate policy implications. The successful Louisville WIN Laboratory study tested a streamlined version of the WIN program, which normally required welfare recipients with school-aged children to participate in one of several possible employment programs. The demonstration program replaced the normal individualized placement process with the prescribed assignment of most WIN clients to a single job search program that aids participants in finding immediate work. The demonstration program also reduced the amount of red tape in the placement process, cutting down the normal wait time between registration and placement in the job search class (Goldman Reference Goldman1981). The experiment found small but statistically significant impacts on individual earnings and welfare program costs.
In contrast to this streamlined version of WIN, the Denver experiment tested a flexible and intensive WIN program that posed more apparent methodological challenges. The demonstration program targeted AFDC recipients with children under the age of five – a group that was not required to register for WIN – and it struggled to recruit a large sample size (Slaughter and Leiman Reference Slaughter and Leiman1981: x). It was designed to provide more intensive counseling and a more extensive set of services to its clients. The experiment found limited statistically significant evidence of effectiveness, arguably because the study’s small sample provided little opportunity to detect the effects of often subtle improvements to the normal WIN program (Greenberg and Shroder Reference Greenberg and Shroder1997: 42; Slaughter and Leiman Reference Slaughter and Leiman1981). Because the Denver demonstration attempted to evaluate a voluntary program featuring some difficult-to-standardize reforms, MDRC’s experiment succumbed to logistical and standardization challenges.
The Louisville experiment was able to avoid these more obvious challenges because it targeted welfare recipients who were required to register for WIN, randomly assigning them at the start of the program. Its experimental program stressed simple bureaucratic changes over difficult-to-standardize case management procedures. Instead of being referred to one of several employment services, welfare recipients in the Louisville experimental group were all initially referred to a single job search program, and their referral and registration for the job search program involved a simplified bureaucratic process that successfully reduced the wait time between their referral and the program start date. These changes were relatively easy to standardize, and this study design generated a relatively large sample size.
Even the Louisville program, however, involved unspoken aspects of the service delivery process that complicated the research design. Its positive impact on employment earnings was presumably due to the increased share of welfare recipients in the experimental group who actually participated in WIN-mandated employment services – primarily, the job search program – but the reasons for this increased participation were not clear. Welfare recipients with school-aged children were generally required to register for WIN and attend their assigned employment service programs, but, across the country, WIN programs often struggled to enforce this mandate and rarely sanctioned welfare recipients who did not comply (Burbridge and Nightingale Reference Burbridge and Nightingale1989). In the case of the Louisville demonstration, program operators decided not to enforce the participation mandate at all. These welfare recipients were required to enroll in WIN and nominally mandated to participate in assigned employment activities, but they were not actually threatened with welfare benefit reductions if they did not comply. High participation rates in the job search program were, nonetheless, influenced by the “considerable pressure” on caseworkers to actively insist on clients’ participation (Greenberg and Shroder Reference Greenberg and Shroder1997: 33). The specific language or strategies used by these caseworkers to effectively persuade WIN registrants to attend the job search program were critical to the success of the demonstration, but these strategies were not explicitly articulated in the design of the program. The features of this program that effectively promoted participation in employment services were arguably contingent on the added attention and motivation surrounding the experimental situation (Goldman Reference Goldman1981: 78; Greenberg and Shroder Reference Greenberg and Shroder1997: 39).
The statistically significant positive impacts of the Louisville program demonstrated that a WIN-style program could be effective and suggested that randomized experiments could accurately measure program impacts, but the ambiguities surrounding the implementation of this demonstration cast doubt on both the rigor of the experiment and the specific policy implications of its findings. Within a few years, this demonstration would be grouped with a subsequent wave of MDRC WIN experiments as examples of both the rigor of this experimental method and, ironically, the effectiveness of the WIN participation mandate. Moving forward, MDRC would adopt a simplified framing of its WIN experiments that deemphasized the particular programmatic components of each service delivery system in favor of the study of the WIN program model as a whole – a model characterized by the participation mandate. This shift became increasingly possible and convenient amidst a series of changes in the market for social program evaluation research and the spread of politically driven WIN demonstration programs.
The evolving market for policy evaluation research
A new market for welfare-to-work policy demonstration research developed in the 1980s due to a provision of the 1981 Omnibus Budget Reconciliation Act (OBRA) that expanded the opportunity and incentives for states to test alternatives to the existing AFDC model. Although the federal government had previously granted states AFDC waivers to pilot programmatic changes, this statute encouraged states to test more ambitious reforms to the WIN program, including the placement of welfare recipients in “work experience programs,” a work-for-benefits model of welfare reform that Reagan had previously enacted, with limited success, as Governor of California (Gueron and Nathan Reference Gueron and Nathan1985; Rich Reference Rich1981). Many states took advantage of this OBRA provision, though most of their demonstration programs prioritized much cheaper job search programs. States opted to test relatively cheap and simple versions of WIN that did not substantially diverge from earlier WIN demonstrations (Greenberg and Wiseman Reference Greenberg, Wiseman, Manski and Garfinkel1992). Many of the new WIN demonstration programs resembled the Louisville WIN Laboratory program. States did take advantage of aspects of the OBRA statute that allowed for significant investment and restructuring of the administrative systems of WIN. For instance, one important OBRA provision permitted states to consolidate WIN within a single agency, thus bypassing the sometimes-challenging coordination between local welfare and employment agencies (Burbridge and Nightingale Reference Burbridge and Nightingale1989).
The Reagan Administration also cut funding for independent social policy research, including large research grants for policy demonstration projects (O’Connor Reference O’Connor2009; Rich Reference Rich2005), compelling MDRC to pursue contracts with states conducting their own WIN demonstration projects. Shifting to the evaluation of existing, government-driven demonstration programs marked a departure from the Ford Foundation’s original vision of MDRC as a mediator that helped design and manage local demonstration programs on behalf of a national-level funding agency. According to Barbara Blum, who was president of MDRC during the early 1980s, the shift implied a loss of autonomy that could jeopardize the rigor of its experiments and divert the firm’s research away from basic, apolitical questions. MDRC leadership, nonetheless, conceded that this transition was a difficult but necessary step toward becoming a leading expert voice in high-level policy discussions (Blum and Blank Reference Blum and Blank1990; Gueron Reference Gueron2003; Gueron Reference Gueron, Banerjee and Duflo2017).
The decision to pursue this new market for welfare evaluation research involved serious internal deliberation about MDRC’s mission, capacity, and relationship to the political process. Given the apparent purpose of the OBRA statute to support a conservative reform agenda, MDRC staff and board members internally expressed concern that evaluating the new state demonstrations would contradict its mission to study programs with the potential “to help disadvantaged members of society” (Gueron and Rolston Reference Gueron and Rolston2013: 99). Judith Gueron, who succeeded Blum as MDRC president, recounts that the organization held a series of meetings before agreeing to accept its new role as “an explicitly nonpartisan institution that believed in subjecting conservative as well as liberal approaches to the same rigorous evaluation standard” (Gueron and Rolston Reference Gueron and Rolston2013: 100). MDRC proceeded to leverage its experience conducting policy experiments, alongside matching funds from the Ford Foundation, to win a large share of the post-1981 state welfare evaluation contracts, conducting nine WIN policy experiments during the mid-1980s under its multi-state Demonstration of State Work/Welfare Initiatives (Greenberg and Wiseman Reference Greenberg, Wiseman, Manski and Garfinkel1992; Gueron and Pauley Reference Gueron and Pauly1991; Gueron and Rolston Reference Gueron and Rolston2013: 120–22).
Settling the science of the social experiment
MDRC’s Demonstration of State Work/Welfare Initiatives largely focused on broad political questions about the cost effectiveness of WIN and the impact of requiring welfare recipients to find employment, marking a shift away from previous WIN demonstration research that evaluated more specific reforms to the design and implementation of WIN. Media coverage of MDRC’s findings reflected this new focus. News reports interpreted MDRC’s experiments as evaluating the mandatory participation rule, which required a large share of welfare beneficiaries to attend assigned employment programs involving job search, job training, education, or work experience. The media declared that “Workfare Works” and trumpeted the efficacy of “requiring able-bodied welfare recipients to work (or prepare for work) in exchange for their grants” (Raspberry Reference Raspberry1986; also Christian Science Monitor 1991; Otten Reference Otten1986; Taylor Reference Taylor1992). Judith Gueron, who served as the director of the State Work/Welfare Initiatives project before becoming president of MDRC in 1986, was repeatedly cited by the press in support of this claim that mandatory WIN programs could simultaneously lower program costs and improve participants’ overall income.
This public-facing interpretation of MDRC’s research corresponded to a subtle shift in MDRC’s more technical framing of its experiments and the programs that they evaluated. As illustrated in Table 1, seven of the nine experiments comprising its Demonstration of State Work/Welfare Initiatives resembled the design of MDRC’s Louisville WIN Research Laboratory study insofar as they randomly assigned study participants during the beginning stages of the demonstration program and evaluated the impact of the program taken as a whole rather than isolating a single component or segment of the service delivery system for evaluation. The other two experiments in the series focused specifically on the impact of a work experience program on WIN clients who volunteered for this service, but these studies showed little consistent impacts and lacked the political influence of the core seven (Auspos et al. Reference Auspos, Cave and Long1988; Freedman et al. Reference Freedman, Bryant and Cave1988). In MDRC’s new terminology, the seven experiments studied “mandatory broad-coverage programs,” assessing the overall impact of WIN programs on the total population of AFDC recipients required to enroll (Friedlander and Gueron Reference Friedlander, Gueron, Manski and Garfinkel1992; Gueron and Pauly Reference Gueron and Pauly1991).
MDRC’s WIN experiments, 1977–1985 *
Table 1. Long description
The table presents data on multiple WIN experiments conducted in various states from 1977 to 1985. It includes columns for study site and project, evaluation start date, participation rules, results, programmatic features, and control state. The table has 12 rows and 6 columns. Row 1: Louisville, WIN Research Laboratory Project (Goldman 1981), 1978, Mandatory, Increased employment and earnings; reduced welfare receipt and payment, Placement in job search, Normal WIN program. Row 2: Denver, WIN Research Laboratory Project (Slaughter and Leiman 1981), 1977, Voluntary, Limited impact on welfare payments, Recruitment; Enhanced WIN services, 1) Not recruited; 2) Normal WIN services. Row 3: Maine, State Work/Welfare Initiatives (Auspos, Cave, and Long 1988), 1983, Voluntary, Increased employment earnings without reduced payments, Job training, Normal WIN program. Row 4: New Jersey, State Work/Welfare Initiatives (Freedman, Bryant, and Cave 1988), 1984, Voluntary, Short-term impact on earnings and welfare payments (faded over time), Job training, Normal WIN program. Row 5: Baltimore, State Work/Welfare Initiatives (Friedlander et al. 1985a), 1982, Mandatory, Increased employment and earnings, Choice of Job search, Unpaid work experience, education, Training, On-the-job training, Normal WIN program. Row 6: Arkansas, State Work/Welfare Initiatives (Friedlander et al. 1985b), 1983, Mandatory, Increased employment and earnings; reduced welfare receipt and payment, Sequence of Job search, Unpaid work experience, No WIN requirements. Row 7: West Virginia, State Work/Welfare Initiatives (Friedlander et al. 1986b), 1983, Mandatory, Reduced welfare receipt and payments, Placement in Unpaid work experience, Normal WIN program. Row 8: Cook County, State Work/Welfare Initiatives (Friedlander et al. 1987), 1985, Mandatory, Small reduction in welfare receipt and payments, 1) Placement in Job search; 2) Sequence of job search, Unpaid work experience, No WIN requirements. Row 9: San Diego I, State Work/Welfare Initiatives (Goldman et al. 1986), 1982, Mandatory, Increased employment and earnings; reduced welfare payment, 1) Placement in Job search; 2) Sequence of Job search, Unpaid work experience, Normal WIN program. Row 10: San Diego SWIM, State Work/Welfare Initiatives (Hamilton and Friedlander 1989), 1985, Mandatory, Increased employment and earnings; reduced welfare receipt and payment, Sequence of Job search, Unpaid work experience, Education and training, No WIN requirements. Row 11: Virginia, State Work/Welfare Initiatives (Riccio et al. 1986), 1983, Mandatory, Increased employment and earnings; reduced welfare payments, Sequence of Job search, Unpaid work experience, Education or training, No WIN requirements.
* Information in this chart is drawn from the above-cited research reports and secondary literature by MDRC researchers that synthesizes these research projects (Friedlander et al. Reference Friedlander, Goldman, Gueron and Long1986a; Friedlander and Gueron Reference Friedlander, Gueron, Manski and Garfinkel1992; Gueron and Pauly Reference Gueron and Pauly1991). In some sites, MDRC conducted multiple randomized trials or assigned the study sample to multiple treatment groups.
A closer look at these welfare experiments reveals that they departed from the precedent for and the statutory purpose of WIN demonstration research by effectively abandoning the effort to evaluate the specific programmatic components that distinguished each demonstration program. For the first time, MDRC designed experiments to test the WIN model in general, assessing the relative merits of different versions of WIN only as a secondary consideration based on the non-experimental comparison of the results of each experiment. WIN demonstration projects were legally permitted for the purpose of testing “administrative innovations to improve the service delivery of the program or small projects extending social services,” and before 1981 WIN demonstrations fulfilled this stated purpose (Williams Reference Williams1994: 14; see also Rogers-Dillon Reference Rogers-Dillon2004). Even the Louisville experiment, which resembled subsequent WIN studies, was intended to evaluate the prioritization of a new job search program alongside a simplified process for placing clients in the program.
MDRC’s decision to frame the object of these experiments in terms of a holistic program model was justifiable because the model was premised, at least nominally, on the WIN mandatory participation rule. AFDC recipients with school-aged children were mandated to enroll in WIN and participate in assigned employment-related activities or risk facing reductions to their welfare benefits. The experience of program participants from the time they were randomly assigned to the treatment group and throughout their experience with the service delivery system was characterized by their exposure to the participation mandate, which incentivized WIN clients to comply with program rules. The impact of any specific component of the program would be difficult to disentangle from the threat of welfare benefits reductions, which could even compel some beneficiaries to find employment on their own (Gueron and Pauly Reference Gueron and Pauly1991: 22, 23). Rather than attempt to separate this incentive from other features of the program experience, MDRC designed experiments that evaluated the impact of the service delivery system as a whole, relegating component features like case management procedures or particular employment services to the black box of program implementation.
This new framing is evident in the way these experiments defined the experience of the control group. Instead of parsing the effectiveness of each WIN demonstration program in relation to the status quo version of WIN that it replaced, MDRC’s post-1981 experiments compared these demonstration programs with an imagined counterfactual state in which WIN did not exist. In most cases, states enacted these demonstrations throughout an entire local jurisdiction or state, and special arrangements were made to provide a distinct experience to a designated control group. In four of the seven experiments, researchers explicitly denied controls access to WIN (Gueron and Pauly Reference Gueron and Pauly1991: 96). Although these welfare recipients were legally required to enroll in WIN, staff left the mandate unenforced or only partially enforced, and they denied controls access to the major employment service programs that made up the core of the demonstration program, offering limited support in finding other resources. For instance, in the San Diego Saturation Work Initiative Model (SWIM) demonstration, “members of the control group were not assigned to SWIM (or WIN) activities but could, on their own initiative, enroll in other community programs” (Hamilton and Friedlander Reference Hamilton and Friedlander1989: 12).
In two of the three other cases, controls were nominally registered in the normal WIN program, but staff considered this existing version of WIN to be poorly implemented and largely defunct. In the San Diego I experiment, controls were “eligible for services from a WIN Program in which activities had been severely curtailed because of national funding cuts” (Goldman et al. Reference Goldman, Friedlander and Long1986: 9). In the West Virginia Community Work Experience Program (CWEP) demonstration, controls “could receive the limited WIN services but … very few members of either group took part in any non-CWEP WIN activities” (Friedlander et al. Reference Friedlander, Erickson, Hamilton and Knox1986b: 53). The experiment involving the Baltimore Options program was an outlier in that it targeted a smaller subsample of the city’s AFDC population and assigned the control group to a version of WIN intended to approximate the status quo Maryland WIN program (Friedlander et al. Reference Friedlander, Hoerz, Long and Quint1985a).
This new approach to WIN evaluation research did not come without costs. In contrast to this new approach, MDRC’s WIN Research Laboratory Project collaborated closely with program directors and local staff to evaluate specific strategies for improving upon the poorly implemented WIN program. The Louisville experiment randomly assigned staff to a special unit serving the control group in order to preserve the status quo WIN experience and compare it with the demonstration program (Goldman Reference Goldman1981: 18). After 1981, MDRC’s experiments largely abandoned this effort to understand the mechanisms that make WIN effective. Designing experiments to test “mandatory broad-coverage programs” as holistic interventions rather than improvements to an existing model does not merely sacrifice potentially valuable substantive insights into why WIN works; it also raises complex questions about the WIN model itself. MDRC’s emphasis on the participation mandate presumed the centrality of the mandate to program effectiveness and risked overlooking other potentially critical features of the WIN model.
Crucially, framing these seven WIN programs as mandatory was accurate on paper, but the demonstrations varied in the extent to which they emphasized and enforced the mandatory participation requirement (Gueron and Pauly Reference Gueron and Pauly1991: 82). On one end of the spectrum, the San Diego SWIM demonstration placed special emphasis on enforcing the mandate and reported high sanction rates, meaning that clients who were not compliant with WIN requirements were often penalized with benefits reductions (Hamilton and Friedlander Reference Hamilton and Friedlander1989). The Baltimore Options program, in contrast, achieved participation rates comparable to other demonstrations despite rarely sanctioning those who did not attend their assigned activities, a practice that was consistent with the program’s philosophy (Friedlander et al. Reference Friedlander, Hoerz, Long and Quint1985a: 101). Other studies left some degree of ambiguity surrounding the importance of the participation mandate for the program’s ability to place clients into employment services and help them find work. The first San Diego program evaluated by MDRC did not explicitly avoid enforcing the mandate, but MDRC reported that, similar to the Baltimore program, it encouraged participation through persistent monitoring and follow up and “most problems were resolved without imposing a sanction” (Goldman et al. Reference Goldman, Friedlander and Long1986: 13). Even in the case of San Diego SWIM, high participation rates could be attributed either to the real threat of benefits reductions or, alternatively, to the extensive resources it invested in case management staff, who, in addition to enforcing the mandate, followed up regularly with clients and helped arrange support services that facilitated their attendance in assigned activities (Hamilton and Friedlander Reference Hamilton and Friedlander1989: 13).
The assertion that some welfare recipients were compelled by the threat of benefits reductions is plausible, but this nominal threat was present in the status quo WIN program, and non-experimental evidence from similar welfare-to-work programs during this period suggests that some new programs achieved success by deemphasizing the mandate in favor of other, less punitive case management strategies. One independent study of MDRC’s WIN demonstrations found that program success depended on effective organizational arrangements rather than any specific policy or rule incentivizing participation. These organizational arrangements included an ideological commitment to client support, which was inconsistent with threatening and enforcing benefits reductions (Hasenfeld and Weaver Reference Hasenfeld and Weaver1996). Another study similarly attributed the impact of new WIN programs to effective case management systems, including appropriate staffing, manageable caseloads, and improved procedures for selecting, counseling, and monitoring WIN clients (Doolittle and Riccio Reference Doolittle, Riccio, Manski and Garfinkel1992). Leading social policy expert Mary Jo Bane contended that these welfare-to-work programs, whether they emphasized mandatory or voluntary services, all relied on strong organizational cultures and effective case management practices to impact welfare recipients’ earnings in the labor market (Bane Reference Bane1989). Many of the post-OBRA 1981 WIN demonstrations took advantage of a new provision permitting the consolidation of welfare and employment agencies in charge of WIN (Burbridge and Nightingale Reference Burbridge and Nightingale1989), but MDRC’s experiments were not designed to evaluate the impact of organizational restructuring.
MDRC’s new methodological approach arguably sacrificed critical insight into the processes that made welfare-to-work programs effective, but it successfully bypassed vexing methodological challenges that would have likely undermined more ambitious experiments. By evaluating the impact of the mandatory WIN program as a whole, MDRC avoided the challenge of standardizing more technical reforms to the service delivery system, and it avoided the parallel problem of standardizing the experience of the control group. Defining and delivering the status quo version of WIN presented obvious ambiguities, and, as MDRC board chairman Richard P. Nathan explained, experiments were vulnerable to the manipulation of the control state by program staff who might arbitrarily deny services to controls or struggle to place them in services because local service providers fill their openings with members of the treatment group (Nathan Reference Nathan2000: 72, 73; also Heckman et al. Reference Heckman, Hohmann, Smith and Khoo2000). In the context of MDRC’s State Work/Welfare Initiatives, the appropriate handling of the control groups was still ambiguous, but because they effectively evaluated WIN demonstration programs in relation to the absence of WIN, program staff in doubt of the proper protocol for serving controls could generally err on the side of denying support and leaving rules unenforced.
Where demonstration programs did not emphasize the mandatory participation rule or otherwise diverged from a relatively uniform and routinized service delivery process, methodological concerns about the standardization of the experimental intervention were more difficult to overlook. In their overview of the post-OBRA 1981 WIN demonstrations, David Greenberg and Michael Wiseman mention the Baltimore and Virginia demonstrations as featuring unobserved procedures for assigning clients to different employment services (Reference Greenberg, Wiseman, Manski and Garfinkel1992: 36). The Baltimore Options program, in particular, refrained from enforcing the mandate and emphasized, instead, an individualized, client-driven placement model premised on caseworker discretion. Such a program presents standardization challenges similar to the Louisville WIN Research Laboratory study, for which program impacts depended upon the discretion of staff under unique pressure to encourage participation widely while targeting services to employable clients (Goldman Reference Goldman1981: 78; Greenberg and Shroder Reference Greenberg and Shroder1997: 39). MDRC bypassed the challenge of standardizing these discretionary case management procedures in favor of evaluating a simpler, holistic policy intervention.
MDRC made the implicit decision to reframe its experiments in part because the political context of welfare reform and evaluation research during the 1980s presented unique constraints and opportunities that compelled this decision. According to MDRC leadership, the firm reluctantly acquiesced to conducting post-hoc evaluations of state-led demonstrations in service of the political objectives of conservative state policymakers (Blum and Blank Reference Blum and Blank1990; Gueron Reference Gueron, Banerjee and Duflo2017; Gueron and Ralston Reference Gueron and Rolston2013). The decision to evaluate these new demonstrations, however, did not necessarily prevent MDRC from tailoring each experiment to test distinct administrative reforms for improving the delivery of WIN. The context of MDRC’s new demonstration research influenced its framing of these experiments, first, by foreclosing methodologically feasible strategies for evaluating more nuanced programmatic reforms. Had MDRC designed seven distinct experiments to test the impact of various service delivery mechanisms, then the methodological validity of these experiments would depend on the closer involvement of researchers with program staff to standardize each specific treatment as well as the status quo experience of the control groups. MDRC’s contractual agreements with these states, however, did not afford the research firm the same authority to help design and manage demonstration programs that it had exercised during the 1970s.
Second, the new political context provided the rhetoric and audience for discussing WIN demonstrations in terms of the mandatory participation rule. State policymakers embraced WIN demonstrations not as a research tool for improving the implementation of an existing program, but instead as a political tool to demonstrate their commitment to moving AFDC recipients off welfare. MDRC’s framing of these pilot programs in its research reports and media coverage supported this narrative. This framing was further facilitated by the relative uniformity of most of the post-1981 WIN demonstrations that MDRC studied. The tendency across states to test streamlined versions of WIN in which most WIN clients were initially funneled into a mandatory job search program lent plausibility to MDRC’s interpretation of these programs as multiple instances of a single, cohesive program model. The demonstrations, however, did not fundamentally differ from pre-1981 WIN reform initiatives. They still involved strategies to improve the implementation and operations of WIN that went far beyond simply enforcing a rule or incentive attached to welfare payments.
MDRC’s decisions about how to design and interpret its welfare experiments were, thus, not overdetermined by the demands of their political sponsors. The organization pursued its foundational purpose to legitimize and institutionalize randomized policy experiments within a new political context that presented unique opportunities and constraints. Their new approach had the benefit of bypassing potential methodological hurdles and rendering their results both credible and politically relevant. This approach also came with the cost of obscuring potentially important features of the demonstration programs that it studied and, thus, undermining competing reform strategies based on the investment in skilled caseworkers and other organizational improvements to local public service agencies. This perspectival shift in expert discourse directly influenced the national welfare reform debate during the late 1980s, and the resulting institutional shift in the procedure for conducting welfare demonstration research had an arguably more important, if less direct, influence on the trajectory of welfare reform leading up to the landmark national welfare reform legislation of 1996.
The implications of MDRC’s experiments for welfare reform
MDRC’s evaluation research was immediately taken up by national politicians and informed the 1988 Family Support Act (FSA), which established a successor program to WIN and encouraged states to continue administering demonstration projects to test further innovations. The experimental findings directly influenced Congressional decision-making, helping to create a new bipartisan consensus around the efficacy of mandatory employment programs for public assistance recipients (Baum Reference Baum1991; Haskins Reference Haskins1991; Szanton Reference Szanton1991). The celebrated use of randomized policy experiments was institutionalized in an FSA requirement that new demonstration projects include provisions for an evaluation study involving random assignment.
Beyond explicit instances of the application of research findings by policymakers, this institutionalization of randomized trials in welfare demonstration research shaped the course of the welfare reform debate in more subtle ways. First, demonstration programs that managed to avoid randomized trials were cast in doubt. In particular, the most high-profile liberal alternative to mandatory welfare-to-work programs, the Massachusetts Employment and Training (ET) program, provided employment services to recipients of the state’s public assistance program. Massachusetts decided to forego an evaluation involving random assignment because the ET program was completely voluntary and program directors feared that implementing random assignment would interfere with the recruitment process (Nightingale Reference Nightingale1991). Denying a large share of volunteers access to the program in order to construct a control group would potentially interfere with the program’s ethos of inclusivity and its ability to recruit the most motivated participants. In general, voluntary programs face ethical concerns and difficulty maintaining the quality of the applicant pool in the context of random assignment (Heckman and Smith Reference Heckman and Smith1995).
Although the state’s statistical analysis of program outcomes concluded that the ET program had a positive impact on welfare recipients’ earnings (Nightingale Reference Nightingale1991), a competing analysis funded by the Urban Institute argued that these outcomes reflected positive changes in local labor markets and did not offer clear evidence of program effectiveness (O’Neill Reference O’Neill1990). In contrast to the credibility of a randomized experiment, Massachusetts’ evaluation of program impacts was susceptible to expert critique and immediately cast in doubt in the face of competing evidence (Peck Reference Peck1998a). Even if a program like ET had justifiable reason to avoid a randomized experiment, its credibility inevitably suffered as a result of this choice.
Second, the new requirement to submit welfare reform proposals to randomized trials limited the kinds of demonstration programs that states could enact. Even before the 1988 FSA, the federal government adopted a preference for demonstration research involving randomized experiments, and this informal requirement immediately operated as a constraint on state innovation. The interagency board established to review state requests for AFDC waivers in 1987 insisted that states incorporate plans for an experimental or quasi-experimental evaluation in their demonstration project proposals (Fishman and Weinberg Reference Fishman, Weinberg, Manski and Garfinkel1992). According to Michael Fishman and Daniel Weinberg’s analysis of the waiver process during this period, the board received twenty-six state applications for waivers and granted sixteen (Reference Fishman, Weinberg, Manski and Garfinkel1992). Of the ten that were not accepted, five “encountered difficulty with the Board’s evaluation criteria” (125). The authors note that “the design of certain demonstrations precludes the use of an experimental design,” especially in cases where “outreach efforts are an integral part of the demonstration and control group members cannot be excluded” (123). The federal government’s insistence that demonstration projects involve randomized experiments placed a constraint on the kinds of programs that states would pilot, a constraint that would be likely to undermine voluntary and responsive program models like the Massachusetts ET program.
During the years leading up to the major 1996 welfare reform legislation, state demonstration programs played an outsized role in the national policy debate. A growing wave of welfare demonstration projects after 1988 accelerated in 1992 as welfare reform became a cornerstone issue for both political parties. The corresponding proliferation of evaluation studies and experimental research findings, however, played a surprisingly small role during this period (Rogers-Dillon Reference Rogers-Dillon2004; Weaver Reference Weaver2000). Robin Rogers-Dillon (Reference Rogers-Dillon2004) argues that state-initiated welfare demonstration programs, despite their nominal research purpose, increasingly functioned as shadow institutions, enabling states to undo the institutional legacy of the existing AFDC program, restructure local welfare agencies in line with their reform agendas, and demonstrate the workability of new programs to encourage other states to adopt them (also Brodkin and Kaufman Reference Brodkin and Kaufman2000). Although AFDC demonstration programs during the 1960s and 1970s tested technical changes to the implementation of existing policy, states began leveraging waivers in the 1980s to enact their new reform initiatives, and in the 1990s these politicized demonstrations became both more common and more ambitious (Rogers-Dillon Reference Rogers-Dillon2004: 35; Williams Reference Williams1994). The proliferation of demonstration projects during the lead-up to the 1996 welfare reform legislation added momentum to a growing rightward shift in the national debate as states across the country enacted new versions of AFDC that attached various restrictive rules and conditions to public assistance.
To the extent that the federal government enforced the requirement that state demonstrations include plans for experimental evaluation research, more liberal alternatives to these restrictive versions of AFDC would have been difficult to implement. In many respects, the federal government process for granting AFDC waivers that permit state demonstrations was increasingly lax in years prior to 1996, but an interview study of top state officials examining their involvement with AFDC demonstration projects in the 1990s reveals that the waiver process remained a barrier for state reformers during this period (Greenberg et al. Reference Greenberg, Mandell and Onstott2000). State bureaucrats looked to MDRC’s demonstration research and, to a lesser extent, evidence from other AFDC demonstrations to learn how to design successful proposals that would be approved by the federal waiver process. As one state official explained, “We’ve discussed (these experiments) not as models of intervention, but as strategies of working the waiver process. You hear about what state X got away with doing their waivers” (quoted in Greenberg et al. Reference Greenberg, Mandell and Onstott2000: 379). The way MDRC framed its evaluation research, thus, provided a precedent that subsequent states would reference when designing their new demonstrations, and the pressure to accommodate this version of the experimental method likely impeded efforts to develop alternative models for welfare reform that did not lend themselves to random assignment.
Discussion
MDRC was created to manage the unwieldy challenge of orchestrating randomized field experiments to evaluate real-world social programs implemented within complex public service bureaucracies. It established the methodological viability and political relevance of this novel approach to policy evaluation during the 1980s, at the very moment that it relinquished control over the design and implementation of the programs that it studied. MDRC’s influential policy experiments relied on a simplified interpretation of technical and multifaceted reform projects that abandoned potentially important policy questions in favor of a politically resonant framing of the interventions. The methodological problems that it could not solve directly were relegated to a black box as the experts shifted their attention from administrative practices to a rule-based incentive. This article illustrates how the history of this form of expertise is inseparable from the interventions that it helped to articulate (Eyal and Buchholz Reference Eyal and Buchholz2010), both because the science of the social experiment is contingent on the means and logic of the interventions it tests and because these interventions are rendered legible through the credible scientific methods tailored to test them.
The political limitations and biases of randomized trials in the evaluation of social programs are well documented across a variety of policy arenas. All research methodologies have their limitations, and the reliance on randomized field experiments – whether to test a simple resource transfer rule or a complex public health or education initiative – tends to present practical problems involving the coordination and standardization of program delivery as well as theoretical problems for generating knowledge about causal mechanisms that can be applied beyond the experimental case (Connoly et al. Reference Connolly, Keenan and Urbanska2018; Deaton Reference Deaton2010; Kabeer Reference Kabeer2020; Marchal et al. Reference Marchal, Westhorp, Wong, Van Belle, Greenhalgh, Kegels and Pawson2013). These limitations, however similar across cases, vary considerably depending on the circumstances, means, and objectives of the experimental design. The success of the method depends on the successful negotiation of these methodological problems, overcoming some while rendering others less visible or important. This negotiation involves both the satisfactory resolution of certain technical methodological issues, accomplishing the scientific legitimacy of the research, and the satisfactory accommodation of political objectives, accomplishing the general applicability of the method and the usability of its results.
In short, the ascendance of a new form of policy evaluation expertise depends on the construction of a “hinge” (Abbott Reference Abbott2005; Eyal and de Souza Leão Reference de Souza Leão and Eyal2019) linking the accomplishment of scientific objectives with the pursuit of political objectives. The construction of the hinge linking economists’ randomized controlled trials with international development interventions relied on a new institutional field of support for development aid that had moved away from long-term, ambitious social programs and was receptive to small, short-term interventions that presented fewer methodological challenges. In the case of US welfare reform, in contrast, MDRC adapted its research design and methodological norms to accommodate social programs that were not fundamentally different from earlier welfare demonstration programs that presented thorny methodological challenges. In this case, the relation between research and reform is especially dynamic due to the specialized role of MDRC as a non-profit research contractor, operating in an “interstitial field” between the economics discipline and public policymaking (Griffen and Panofsky Reference Griffen, Panofsky, Eyal and Medvetz2023; Medvetz Reference Medvetz2012). Not beholden to the same degree of scientific autonomy as academic economists, actors in the private policy research industry must adapt to more immediate changes in the political demand for its research, and they are afforded more flexibility for methodological innovation in the service of applied social research.
Probing the processes by which specialized policy experts negotiate the relationship between science and politics sheds light on the political significance of technical advancements in expert knowledge production. Elizabeth Popp Berman (Reference Berman2022) shows that the economistic logic that predominates in US public policy today has been embraced not merely by conservative policy advocates with complementary ideological concerns but, more importantly, by liberal-leaning politicians and experts who are genuinely seduced by a technocratic logic rooted in economics research methods. The scope of what is knowable and provable using the tools of formal policy analysis thus sets the bipartisan boundaries of viable policy options in mainstream US politics. The history of MDRC’s welfare experiments illustrates that the symbiosis between right-wing political movements and the technocratic methods of the liberal social state was not accidental. When Republicans struggled to implement their vision of a more restrictive and conditional public assistance policy, and when poverty policy experts struggled to produce reliable evidence supporting politically viable policy solutions, the two movements came together to forge analytical tools that would legitimize and guide institutional change at the level of local welfare and employment bureaucracies. Liberal-leaning policy experts thus developed tailored research methods to help define workable social programs that could realize conservative policy objectives. The institutionalization of new evaluation research facilitated administrative buy-in and bipartisan support for restrictive welfare reform, in part by rendering competing reform strategies illegible.
Understanding the history of social policy evaluation methods is crucial to our analysis of the logic of the neoliberal welfare state. The concrete devices of rule that characterize neoliberal versions of poverty governance entail data systems and analytical tools for studying population dynamics and measuring the effects of state intervention (Foucault Reference Foucault2007; Rose Reference Rose1993). In general, the reliance on formal methods for the causal analysis of program impacts has immediate political implications for social reform insofar as it isolates policy interventions from complementary policy efforts and broader social objectives, favoring narrow solutions that swim upstream against the current of structural economic inequality (Aaron Reference Aaron2010; O’Connor Reference O’Connor2009). Advancing this general line of critique requires considering even arcane and technical advancements in specific methods for formal policy analysis because these purportedly neutral scientific innovations respond to and help consolidate particular political visions about the role of the state and the nature of social problems. MDRC designed its seminal series of welfare experiments to place multiple programmatic reform efforts under the umbrella of a single program model based on conditional public assistance, and the elevation of a new evidentiary standard based on this precedent precluded subsequent reform initiatives that attempted similar programmatic changes apart from this overarching model. If we are to recover a vision of public goods based on skilled and well-resourced local staff that are responsive to community needs and empowered to confront systemic problems, then we must critically reflect on the available tools for policy analysis and be willing to challenge scientific claims to rigor that rely on a narrowly circumscribed vision of possible political change.
Open access
