The impact of the policy apparatus on evidence

Emerging from debates in evidence-based medicine, evidence-based policy making became a dominant frame for policy formulation after the election of the New Labour government in 1997. This was part of a broader ‘modernising agenda’ for government (Cabinet Office, 1999; Mulgan, Reference Mulgan2005). Cameron and colleagues note (Reference Cameron, Salisbury, Lart, Stewart, Peckham, Calnan, Purdy and Thorp2011: 430) that at the heart of this development was a ‘belief that policy making needed to move away from the subjective use of anecdote to a more objective, dispassionate review of available evidence’. In recent times a pragmatic turn (Pearson, Reference Pearson2010) has emerged in discussions around EBPM. The sense of optimism that accompanied the first UK New Labour government's assertions that it would be guided more by ‘evidence’ of ‘what works’ than ‘dogma’ has dissipated (Nutley et al., Reference Nutley, Walter and Davies2007; Monaghan, Reference Monaghan2011), not least as the complexity of the evidence and policy connection became more apparent (Monaghan, Reference Monaghan2010). The UK Conservative-Liberal Democrat Coalition government (2010-2015) made little pretence of basing policies upon evidence. This was to some extent justified by the prevalence of the narrative of ‘austerity’ as a game-changing context for the development of policies (see Taylor-Gooby, Reference Taylor‐Gooby2012; Reference Taylor‐Gooby2016).

Although austerity still shapes the social policy landscape, EBPM has retained some currency within Government. A solid portfolio of guidance from the UK central government on the process of ‘evidence informed policy’ (Bochel and Duncan, Reference Bochel and Duncan2007) has been produced. HM Treasury's Magenta Book (HM Treasury, 2011) provides guidance to policy makers and analysts for the design and management of evaluations of government projects, policies, programmes and the delivery (2011: 7). The updated version ‘shifts emphasis away from the “analyst's manual” of the previous edition, to a broader guidance document aimed at both analysts and policy makers at all levels of government, both central and local’ (2011: 5). The Magenta Book is a companion to HM Treasury's Green Book (HM Treasury, 2003), which constitutes binding guidance for departments and government agencies with regard to the economic appraisal of the development of policies and programmes. Appraisal and evaluation (and in particular cost-benefit and cost-effectiveness analyses – p. 4) is a key stage of the ‘ROAMEF’ cycle (Rationale, Objectives, Appraisal, Monitoring, Evaluation, Feedback) within Westminster policymaking and intended to represent the UK policymaking cycle, although in recent years it has become less central to this process.

Despite these attempts to provide guidance for policy makers and government analysts (economists, social researchers, statisticians and operational researchers) regarding the harnessing and utilisation of evidence, the publications have arguably added to, rather than alleviated, this void on the grounds that the guidance is, at best, incomplete and highly abstract. For instance, the Cabinet Office (2008) framework for international policy comparisons omits the crucial step of ‘how to transfer/translate policies’ between contexts and at no stage does the Magenta Book consider how policymakers can assimilate knowledge from such evaluations. Additionally, very little attention is given to the basic reporting of evidence to Ministers except to draw, very briefly, upon Vaughan and Buss’ (Reference Vaughan and Buss1998) tips on reporting social science research to busy policy makers.

A key limitation of the EBPM literature concerns, therefore, the relative paucity of data that explains how evidence is used by those charged with the task of using it. In one of the most enduring studies of the evidence and policy relationship, Weiss (Reference Weiss1979) highlights how the relationship between evidence and policy can be seen in instrumental, in conceptual or in symbolic ways. The ‘instrumental’ version, which assumes that scientific knowledge has a direct, linear impact on policy making, has now largely been discredited. The ‘conceptual’ understanding suggests that knowledge offers opportunities for policy makers to change the way they conceive of policy issues and so in this version evidence filters into decisions over time in an ‘enlightenment’ capacity. The ‘symbolic’ version is where evidence is harnessed by policy makers to justify pre-existing policy stances rather than acting as a guide for action.

This account, although useful, remains abstract. There is, however, an emerging body of research that has attempted to consider some of the main mechanisms by which research is filtered into and out of the policy process. A common thread of the more recent accounts of evidence utilisation is that evidence selection in policy making is an unequal competition as the framing of the policy in question often determines the kinds of evidence that are deemed permissible. Liebling et al. (Reference Liebling, Maruna and McAra2017: 11) note how:

Similarly, in their comparative review of diverging tobacco control policies in the UK and Japan, Cairney and Yamazaki (Reference Cairney and Yamazaki2017) note that:

For Cairney and Yamazaki, the sensitive nature of the policy itself impacts on the kinds of evidence that are deemed viable for policy development in the UK and Japan. Stevens (Reference Stevens2007a; Reference Stevens2011) has suggested that to understand how evidence is translated into policy one must consider a number of filtration processes that render evidence selection an unequal competition. Stevens (Reference Stevens2007a) notes how, in contentious areas of policy development, there are often significant socially structural, as well as politically tactical, barriers to evidence use. Citing the examples of immigration and drug use, Stevens notes how the direction of policy in the UK in both these areas over recent times has been towards restricting these activities. This is because those who stand to benefit from ‘liberalising’ the legislation (drug users and would-be immigrants or asylum seekers) are often among the least powerful members of society. They are also among the groups frequently scapegoated and stigmatized for social problems, particularly crime, making political reform less likely (see Ford and Lymperopoulou (Reference Ford and Lymperopolou2016) for a discussion of immigration and crime and Seddon (Reference Seddon2005) and Stevens (Reference Stevens2007b) for the links between drugs and crime).

Elsewhere research (Ingold and Monaghan, Reference Ingold and Monaghan2016), has suggested that several, inter-related, factors broadly shape the process of translating evidence into policy. Firstly, that the substantive nature of policy issue or problem determines the kind and extent of evidence used. Secondly, agenda setting and policy framing influence the search for permissible evidence (Head, Reference Head2008). Thirdly, specific filtration processes (Stevens, Reference Stevens2007a; Reference Stevens2011) are always at play in the relationship between evidence and policy. Here, the kinds of evidence that are deemed useful to policy makers tend to survive, whereas critical evidence falls by the wayside. Fourthly, following lessons from institutionalism, where political institutional arrangements greatly affect policy processes and outcomes (Parkhurst, Reference Parkhurst2016: 9) the apparatus for policy design and implementation can have a direct bearing on the kinds of evidence used and omitted. Fifthly, policy personnel in the form of analysts or policy officials play a decisive role in relation to each aspect.

Despite this emerging literature, there are, as indicated, few studies that have directly analysed the routes by which evidence finds its way into policy from the point of view of those who facilitate or are responsible for this. One consequence is that evidence hierarchies are often used as an explanatory tool to explain the preponderance of the use of certain kinds of evidence in policy over others (Nutley et al., Reference Nutley, Powell and Davies2013). Our research sought to move beyond such accounts, in order to (i) shed light on the dimensions that impact on evidence utilisation; (ii) better understand how evidence is employed in the policy process; and (iii) analyse views of evidence utilisation from a user perspective. Although we would argue that the five factors identified above are inter-related, for current purposes in this paper we investigate to what extent (i) policy itself is a determining feature of evidence selection and utilisation, specifically within (dominant) policy frames; and (ii) policy shapes the kinds of evidence that are deemed permissible and silences other, potentially valuable, contributions. We were particularly interested in how the organisational features of the DWP at the time of our research mediated the framing of the evidence and how this was influenced by the broader policy landscape and times (Zeitgeist). Before turning to the findings from our research, we give below a brief account of the (very politicised) policy context in which our research was conducted.

Research context

The Department for Work and Pensions (DWP) is the UK's largest administrative department and is responsible for pensions, social security benefits and child maintenance policy. It oversees employment policy and delivers employment services both through contracted provision and through Jobcentre Plus. Three workshops were conducted with 75 DWP policymakers and analysts in three of the department's key locations around the UK. Workshop participants were officials and analysts (social researchers, statisticians and economists), ranging from middle-ranking to senior officials and from relatively new entrants to officials of long service. Participation in the workshops was voluntary through self-selection but, from their everyday roles, all participants had an interest in how evidence and policy interconnect. The DWP was a compelling location for this case study given that between 2004 and 2010 it oversaw an unprecedented growth in commissioned evaluations (Legrand, Reference Legrand2012: 330). However, Legrand's assertion that DWP's growth in research outputs demonstrated an ‘extraordinary sea change in their use of research evidence in policy development and evaluation’ assumes that this evidence was actually used in policymaking. During the research period, the DWP was engaged in a wider agenda of experimentation with new approaches across government, including ‘horizon scanning’, which looks across a spectrum of evidence sources to consider how they could shape future policy developments. These initiatives represented a continuation of the machinery of government changes led by the UK Cabinet Office but were also aligned with political imperatives under austerity, such as large-scale expenditure cuts to central government departments and drives for greater efficiency. Consequently, the workshops took place during a time of important organisational change and significant policy change for the DWP.

The most important aspect of the latter was the policy development of, and preparation for, the introduction of Universal Credit (UC). The central aim of UC is to ‘radically restructure’ the existing benefits and tax systems by merging six benefits and tax credits into one single working-age benefit, in order to improve incentives to work (DWP, 2010). UC was introduced in April 2013 in pathfinder areas (pathfinder rather than pilot given that definite national rollout of the policy was already decided), with final completion across the UK expected by 2017. These expectations have not been met. The reform has attracted much media coverage and criticism, with repeated claims that the project had floundered and had been ‘scaled back’ (Wintour, Reference Wintour2013) and more recent revelations about the reality of income loss and the winners and losers of the policy (Savage, Reference Savage2018; BBC, 2016). There have been concerns around the financial difficulties for households resulting from the change from weekly to monthly payments, financial support for childcare and problems with online accessibility. Other concerns include UC's implications for women, particularly the individualisation of benefit conditionality without individualised payment incentives for second earners, who are more likely to be women (Gingerbread, 2013; Millar and Bennett, 2016;). The National Audit Office has published critical assessments of the implementation of UC (2013; 2014) and the government's own Infrastructure and Projects Authority (IPA) recommended a ‘firebreak’ in implementation in early January 2018 in order to consider whether the rollout plan remained appropriate. The Work and Pensions Select Committee (2018) recently praised the DWP for bringing the project ‘back from the brink of complete failure’ but highlighted concerns regarding ongoing challenges (2018: 14) relating to this ambitious policy reform.

UC is the result of repeated political attempts to address the difficult task of simplifying the UK's complex social security and taxation systems. Early in the first governing period of New Labour, incremental change was favoured over more large-scale overhaul, leading to the introduction of Tax Credits and changes to the benefit rules concerning spouses/partners of benefit claimants (Joint Claims for Jobseekers Allowance for couples and the New Deal for Partners). Further changes to social security entitlement followed in the Welfare Reform Act 2009, increasing conditionality for a wider range of individuals. Central actors in the next phase of reform were the Centre for Social Justice (CSJ) and Iain Duncan Smith who was, at the time of our research, the Secretary of State for Work and Pensions. The CSJ first published its UC blueprint for the report Dynamic Benefits (CSJ, 2009), focusing on (dis-)incentives to work in the existing social security system, in particular high marginal tax rates. Government plans for UC were then set out in the Green Paper 21^st Century Welfare and, finally, in the White Paper Universal Credit: Welfare that works and enshrined in the Coalition Government's Welfare Reform Act 2012. These documents drew on a relatively limited range of evidence and, notably, not upon findings from DWP-commissioned evaluations, including those relating to the New Deal for Partners or New Deal for Lone Parents in terms of the real-world complexities of moving into work and changes to circumstances beyond technocratic or rational economic dimensions.

At the time of the workshops, DWP was still finalising aspects of UC and discussion of this important policy change and its affiliated evidence-base pervaded each of the workshops. Workshop discussions related to the overall policy development of UC, its implementation with HMRC (responsible for the management and collection of tax and payment of tax credits) and with local authorities (who administrate Housing Benefit, another payment merged into UC), as well as in relation to DWP-commissioned evaluations of the policy. Drawing on research by one of this paper's authors in three countries in collaboration with the DWP (Ingold and Etherington, Reference Ingold and Etherington2013), discussion in the first workshop largely related to consideration of UC in relation to couples and their decision-making in relation to work and benefits and ‘In Work Conditionality’, now referred to as ‘In Work Progression’ (IWP). IWP is a critical dimension of UC relating to its central aim to incentivise any workFootnote ¹ by extending benefit conditionality to those in work but still in receipt of benefits. IWP is without precedent internationally, with a very limited evidence base (the exception being DWP's Employment Retention and Advancement demonstration project focused on improving job retention and progression (Hendra et al., Reference Hendra, Ray, Vegeris, Hevenstone and Hudson2011)). Consequently, there has been a series of DWP-commissioned IWP pilots to garner what could be termed ‘policy-based evidence’. Additionally, the first DWP-commissioned UC evaluation report was published in 2011, focused on public perceptions of welfare reform and UC (Rotik and Perry, Reference Rotik and Perry2011). This was followed by the first pathfinder evaluation (2013) and estimations of UC's employment impacts based on a limited claimant group in 2 per cent of areas; qualitative and cognitive research including ‘laboratory’ experiments utilising random assignment (DWP, 2017) (underscoring the dominant role of ‘nudge’ and the Behavioural Insights Team in policy post-2010); and evaluations of mixed method ‘test and learn approaches’ in relation to families. Within this policy context, the workshops aimed to tease out policy makers’ and analysts’ understandings of the evidence and policy connection.

Our research was co-produced in consultation with the DWP and resulted in three workshops, the first workshop comprising 10 participants, the second around 25 participants and the final workshop almost 40 participants. The workshops ranged in length from one to two hours. Although there were internal variants, there were constants in the subject matter discussed. In each scenario, we presented conceptual models based on the EBPM and policy transfer/translation literatures as vignettes to stimulate discussion. The key research question posed was: ‘How is evidence used, or translated, in the context of contemporary policy making in the DWP?’ All workshops were transcribed verbatim and data were analysed thematically. The findings are presented thematically in the following three sub-sections: (i) Evidence hierarchy?; (ii) Capability; and (iii) Political feasibility.

Findings