Researching the effectiveness of WtW policies

One extensive and expanding strand of research, which has engaged most directly with the question of ‘what works’ in terms of bringing vulnerable unemployed individuals into work, is that of quantitative studies of the effects of specific programmes and policy measures on labour market participation (see Danneris, Reference Danneris2016, for a mapping review). One of the dominating tendencies is research with a preference for randomised control trials (RCT), or as close to this as one can possibly get (meta evaluations, quasi experiments, quantitative statistical tests) (e.g. Graversen and van Ours, Reference Graversen and van Ours2009; Maibom et al., Reference Maibom, Rosholm and Svarer2016; Rehwald et al., Reference Rehwald, Rosholm and Svarer2017), and, on the back of those, meta-reviews that seek to synthesise findings across multiple individual studies (e.g. Crisp and Fletcher, Reference Crisp and Fletcher2008; Card et al., Reference Card, Kluve and Weber2015). Several of these studies, which are believed to represent the most sound and reliable knowledge in the field (Raffass, Reference Raffass2017), find contradictory or only slightly positive effects of WtW policies in terms of encouraging unemployed individuals into work (e.g. Smedslund et al., Reference Smedslund, Dalsbo, Hagen, Johme, Rud and Steiro2006; Danneris, Reference Danneris2016; Raffass, Reference Raffass2017). One example is an IZA (Institute of Labor Economics) policy article by Martin (Reference Martin2015), who reviewed fourteen studies using cross-country econometric analysis to investigate the impact of WtW policies on employment and unemployment rates across OECD countries. Martin concludes that spending on WtW policies does to some extend reduce unemployment or long-term unemployment (Martin, Reference Martin2015: 11). However, Martin (Reference Martin2015: 26) also notes that there are examples of both successes and failures in terms of activating benefit recipients, and that these vary as a consequence of the design and implementation of activation regimes, which are again determined by the different starting points, institutions and culture between the countries. Martin’s study illustrates the challenges of measuring the effects of social interventions. Thus, several authors have criticised the methodological approaches of this strand of studies, which draw on a linear logic of causality when approaching the question of what works (Koivisto, Reference Koivisto2007; Andersen et al., Reference Andersen, Caswell and Larsen2017). Such methodologies treat both the intervention (WtW policies) and the problem (unemployment) to which it is applied as standardisable, constant and measurable in isolated variables (Shortall, Reference Shortall2012; Newman, Reference Newman2017). As we will demonstrate throughout this article, this is not reflective of how employment services are delivered to unemployed individuals, nor of how interventions ‘work’ in complex situationsFootnote ² .

A few exceptions to the quantitative evaluations have included a broader range of research in their reviews of ‘what works’ (e.g. Griggs and Evans, Reference Griggs and Evans2010; Brady and Cook, Reference Brady and Cook2015). These studies illustrate how the linear logic of cause and effect is too simplistic when dealing with social matters such as unemployment. For instance, Campbell et al. (Reference Campbell, Thomson, Fenton and Gibson2016) review nineteen qualitative studies in their investigation of lone parents, health, wellbeing and employment policies. They conclude that participation in mandatory WtW interventions have limited success in terms of improving lone parents’ health, wellbeing and economic circumstances, but also make visible how several contextually mediating factors such as the unavailability of suitable employment, childcare and social support may counter the potential positive results of employment interventions for this specific group of unemployed individuals (Campbell et al., Reference Campbell, Thomson, Fenton and Gibson2016). In an individual study, Bredgaard and Hansen (Reference Bredgaard and Hansen2012) use a realistic evaluation framework to analyse what works in three activation sites. They find that a complex interplay between individual moderators and mechanisms affects the young unemployed. Thus, they conclude that different types of programmes and activities work differently depending on the individual client and his or her competencies and self-efficacy, etc.

Overall, then, the research that has most directly engaged with the question of ‘what works’ offers few clear answers to inform policy makers in their quest to ameliorate unemployment, even more so for people with challenges in addition to being unemployed (Danneris, Reference Danneris2016)Footnote ³ .

Researching the front-line of WtW bureaucracies

A separate strand of research, that of street-level organisations within WtW, helps to fill the gaps between a policy decision to apply a certain measure or intervention and what (and what does not) happen in terms of bringing unemployed individuals closer to the labour market. While growing, the extent of this research is modest compared to its aforementioned quantitative counterpart, and qualitative methods are used more often (see van Berkel, Reference van Berkel, van Berkel, Caswell, Kupka and Larsen2017 for a mapping review in the European context). Building on the work of Lipsky (Reference Lipsky2010), this research approaches the implementation of policy not as the application of standardised interventions that can be expected to ‘work’ in certain and uniformed ways, but rather as processes of translation and transformation through which policy is made to work in local practices at street-level (Brodkin, Reference Brodkin2011; Brodkin and Marston, Reference Brodkin and Marston2013). A recurring theme is the role of managerial reforms that have introduced New Public Management (NPM), performance management and similar trends in the design and delivery of employment and social policies (e.g. Diefenbach, Reference Diefenbach2009; Heinrich and Marschke, Reference Heinrich and Marschke2010; Brodkin, Reference Brodkin2011; van Berkel and Knies, Reference van Berkel and Knies2016). For instance, Soss et al. (Reference Soss, Fording and Schram2011: 209) demonstrate how performance measurements, when combined with other organisational conditions, encourage frontline professionals to sanction clients. Similarly, Møller et al. (Reference Møller, Andersen and Iversen2016) found that performance management promotes a focus on short-term results, such as quick, temporary jobs, rather than more sustainable solutions to the problem of unemployment. Other authors have examined the role of frontline workers’ attitudes and competencies for how they enact employment policies (Malmberg-Heimonen and Vuori, Reference Malmberg-Heimonen and Vuori2005; Behnke et al., Reference Behnke, Frölich and Lochner2010; van Berkel and van der Aa, Reference van Berkel and van der Aa2012; Malmberg-Heimonen, Reference Malmberg-Heimonen2015), adding yet another dimension to the complexity of making policy ‘work’. Such influences on how policy is delivered to the unemployed individual muddies the picture of ‘what works’, as policy measures and programmes are essentially transformed through their very delivery. Street-level literature, nevertheless, demonstrates that WtW policies have a great (often negative) impact on the lives and experiences of the unemployed, even though the long-term effects in terms of labour market participation, as described previously, may be limited (e.g. Dubois, Reference Dubois2010; Møller and Stone, Reference Møller and Stone2013; Carter and Whitworth, Reference Carter and Whitworth2015; Nothdurfter, Reference Nothdurfter2016). A small sub-strand of studies has examined how employment policies are experienced from the perspective of the unemployed individuals they aim to reach (e.g. McDonald and Marston, Reference McDonald and Marston2008; Danneris, Reference Danneris2016; Danneris and Nielsen, Reference Danneris and Nielsen2018; McIntosh and Wright, Reference McIntosh and Wright2018). The purpose of giving a voice to vulnerable people is commonly found in this strand, extracting unique knowledge from lived experiences and, to some extent, contributing to qualified practices in the delivery of services by illuminating how the unemployed are affected by such policies in complex ways far beyond their participation in the labour market. The street-level literature does not engage with the question of ‘what works’, and is often preoccupied by the perverse effects that arise when policies do not work. Nevertheless, it offers important insights into how policies work in both intended and unintended ways at the street-level and, as such, holds important potential for policy makers as they consider how to address the problem of unemployment.

Researching the communicative practices between front-line professionals and the unemployed

That the lack of the conclusive effects of WtW policies is not (merely) a matter of failed implementation or of a management regime gone wrong is exemplified by the growing literature, which examines what happens in concrete meetings between professionals and unemployed individuals; that is, how policies are brought about in personal interactions (e.g. Eskelinen and Olesen, Reference Eskelinen and Olesen2010; Caswell et al., Reference Caswell, Eskelinen and Olesen2011; Dall and Sarangi, Reference Dall and Sarangi2018; Dall, Reference Dall2018, Matarese and Caswell, Reference Matarese and Caswell2018). Using discourse and conversation analysis, this literature shows an interest in the variations and nuances of individual cases, and offers important insights into the ‘working’ of policies at the micro level. For instance, Mäkitalo and Säljö (Reference Mäkitalo and Säljö2002a, Reference Mäkitalo and Säljö2002b), Mäkitalo (Reference Mäkitalo, Hall, Juhila, Matarese and van Nijnatten2014) and Müller and Wolf (Reference Müller and Wolf2015) demonstrate how institutional (and policy-derived) categories are constituted in and constitutive of client work in public employment services. Another important point to come out of this research is the active and strategic role of both professionals (e.g. Drew et al., Reference Drew, Toerien, Irvine and Sainsbury2010; Irvine et al., Reference Irvine, Sainsbury, Drew and Toerien2010; Toerien et al., Reference Toerien, Sainsbury, Drew and Irvine2015; Hansen and Natland, Reference Hansen and Natland2017) and the unemployed individuals (e.g. Solberg, Reference Solberg2011a, Reference Solberg2011b, Reference Solberg2014; Danneris and Dall, Reference Danneris and Dall2017) as co-producers of how policy measures are implemented. These micro-level studies demonstrate the contingent and negotiated nature of how policy is executed in practice. While such interactions are understood to be happening contingently and continuously between specific people conversing in particular moments in time, general tendencies (in interactions) can often be identified that offer tools for reflection and improved practices for frontline workers (Hall and White, Reference Hall and White2005; Kirkwood et al., Reference Kirkwood, Jennings, Laurier, Cree and Whyte2014). These studies demonstrate how interventions, even if they are described and organised in highly structured and standardised ways, are enacted in concrete, contingent and negotiated interactions between professionals, unemployed individuals and the contexts they are part of. However, as the focus is, naturally, on the verbal interactions of the (human) actors, this strand of research offers limited insight into the many ways that practices are shaped outside of such interpersonal interactions (e.g. through policy discourse and management strategies, etc.). Furthermore, discursive studies offer little insight into the more general consequences of certain policies, by focusing on the detailed ways in which they are made to work and as a result of certain interactions.

In this short journey through the literature, we are presented with a great variety of studies in terms of applied methods, approaches and findings. In each of these strands of research, both positive and critical approaches can be identified – and it is clear that they contribute to the development of the field in very different ways. Important and valuable knowledge exists in all strands of research, but looking at them individually does not help us to further our attempt to improve WtW policies or to move more people off benefits and into work in sustainable ways. Unfortunately, the field is, to this day, characterised by limited dialogue between the different strands of research, and often the most scathing critique of the knowledge produced in one strand of research comes from researchers in the other strands.

The aim of understanding ‘what works’ – in nuanced ways

The study of what works in terms of ameliorating unemployment for the vulnerable unemployed is an important topic and task for researchers engaged in conducting applied and/or applicable research in this specific field. As such, we are aligned with the aim of the wave of research looking to inform and qualify both employment and social policies targeting the challenges of long-term unemployment. However, we believe that there are significant limitations in how the question of ‘what works’ has previously been approached, some of which has actually worked against this aim. As we have previously outlined, the strand of research looking to ‘evidence base’ social and employment policies (and practices) has been oriented towards the identification of causal links between a certain measure and the movement of unemployed individuals into work. However, as has been thoroughly demonstrated by research on street-level organisations, the matter of labour market participation is not the only significant (potential) effect of employment policies. As these additional ‘effects’ can be highly detrimental to the individual, as well as counterproductive in terms of supporting labour market participation (Caswell et al., Reference Caswell, Larsen, Sieling-Monas, Bengtsson, Frederiksen and Larsen2015), their inclusion is imperative when we seek to understand ‘what works’. For instance, Griggs and Evans (Reference Griggs and Evans2010) find that the use of sanctions works in terms of increasing the rate of job entry for clients on unemployment benefits on a short-term basis, but in the longer term – and for welfare benefit recipients – the effects are inconclusive or negative; further, Caswell et al. (Reference Caswell, Larsen, Sieling-Monas, Bengtsson, Frederiksen and Larsen2015) demonstrate how decreasing benefits and increased sanctions lead to a decline in both job search activity and physical, mental and social wellbeing for the vulnerable unemployed. While it is ultimately a political decision whether the effects of labour market participation outweigh the negative consequences in terms of poverty and loss of individual wellbeing, the role of researchers must be to seek a comprehensive understanding of a given phenomenon and, perhaps most importantly, when political powers seek to further their understanding and agenda through substantial and administrative policies (e.g. Schram et al., Reference Schram, Flyvbjerg and Landman2013). Thus, we argue that there is a potential to strengthen research on ‘what works’ by including unintended and spill- over effects, as well as the other consequences of policies that do not directly pertain to labour market participation.

Moving towards a relational causality

Our point is that the effect or consequence of policies pertaining to other dimensions than that of labour market participation is not only a normative one, but is also closely linked to the ontological position from which we approach the issue of unemployment.

Research examining whether a given measure ‘works’ often does so based on the assumption of a linear causality; such a causality operates on a mechanistic ontology that supposes that an intervention is a necessary and sufficient element to produce the sought-after effect (Koivisto, Reference Koivisto2007: 529). However, such successional processes of causality are the exception, and not the norm, in the social domain (Biesta, Reference Biesta2010: 497). An intervention in the employment services is not the operation whose effect was first examined in a more or less controlled environment in evidence-based medicine. Thus, even a fairly ‘simple’ intervention, such as that of sanctioning clients that do not attend a meeting they are obligated to be at, gains its effect relative to the relations the client is part of; in interactions between the client and the frontline worker (Have the clients understood what they have done wrong, so as to act differently next time?), between the client and their everyday life (Does the client have access to childcare, so as to be able to attend every appointment going forward?) and between the frontline worker and the managerial obligations and incentives s/he is working under. In the words of Porpora (Reference Porpora and Dépeltheau2018: 419), ‘Instead of causality always following some homogenous if-then form […] causality is thick, taking a multitude of different forms’. Relational causality is why a given policy measure will always work in more than one ‘dimension’ (e.g. that of labour market participation) and why there will always be other effects than those expected (and measured). As described by Biesta (Reference Biesta2010: 497), ‘[i]n the social domain interventions do not generate effects in a mechanistic or deterministic way, but through processes that – structurally, not pragmatically – are open so that the connections between intervention and effect are non-linear and, at most, probalistic’. Rather than a linear causality, then, it is necessary to understand the processes of change within the social domain of unemployment in terms of a relational causality, by which the causal mechanisms are to be found in relational processes that are continuously performed, produced and reproduced during daily activities and practices (Olesen and Eskelinen, Reference Olesen and Eskelinen2011).

We do see some recognition of this in the research focused on finding ‘effects’. For instance, Martin (Reference Martin2015) first finds a positive effect within macroeconomic employment policies before going on to nuance his findings according to the significant differences in the activation regimes studied, while Bredgaard and Hansen (Reference Bredgaard and Hansen2012) find, at the local level, that the effects of the studied interventions depend on local and individual matters. These more nuanced approaches to effect studies, from our perspective, illustrate how a narrow focus on the linear effects of a given policy measure does not provide a sufficient or full picture of the consequences of such policies. Furthermore, they do not offer much input into the directions that policy makers might look towards for more qualified services. In order to improve applicability, the seemingly proven effects (or lack thereof) must be considered in relation to broader understandings of both the effects themselves and of the way in which interventions work in certain contexts.

A relational understanding of how policies ‘work’

The concept of relational causality calls attention to the ways in which policies work in social practices. As demonstrated by the literature on street-level organisations, policy is not simply applied (in linear ways) at the frontline of public employment services, but goes through a series of ongoing translations and transformations, including managerial and organisational, as well as individual and communicational.

Our proposal of the concept of relational causality builds on the ontological ideas of relational sociologyFootnote ⁴ (Dépelteau, Reference Dépelteau and Depéltheau2018). Depélteau (Reference Dépelteau and Depéltheau2018: 10) outlines a ‘relational turn’ in sociology that refers to ‘the attempt to perceive, define, study and so on social phenomena as fluid social processes rather than solid, determining social substances’. This is in opposition to approaches that study effects as linear causal mechanisms between essentially separate entities (e.g. that of a certain policy and its effects on labour market participation). Rather, any social phenomenon is constituted through and within dynamic relations between a matrix of both human and non-human actors; thus, the ‘effects’ (or other phenomena) emerge and evolve through ongoing relational processes.

Following this ontology, a policy or a specific (policy decided) intervention within the unemployment services may consist of a ‘script’ outlining which actors are to perform what kind of processes and activities in order to achieve a given goal. Yet the actors perform and produce the method in actual and concrete activities in practice; accordingly, the effects of a given practice ‘cannot be explained by the work method [alone], because it is constituted and co-produced in the interactive work activities and processes and it is thus an effect too’ (Koivisto, Reference Koivisto2007: 533, our addition). This does not mean, however, that we cannot gain any knowledge on ‘what works’ in social policy efforts, but that we must take into account the ways in which a given intervention is made to work in practice. We are not arguing for or against any certain methodologies here – on the contrary: in order to capture the many ways in which policy measures gain their effects in and through contextual practices, we need a similarly diverse research that engages with the question from different perspectives. Interventions and services are not universally true and therefore universally applicable (Koivisto, Reference Koivisto2007). They are constituted by and function in socio-material networks (Latour, Reference Latour2005), which means that the a-contextual approach of much of the existing literature on ‘what works’ misses a crucial part of what ‘working’ means in social settings. We have seen this thoroughly demonstrated by both the research in street-level organisations illustrating the role of organisation, management and professional roles, and the micro-discursive research into meetings between frontline workers and unemployed individuals, illustrating the contingent and negotiated character of making policy ‘work’. Unfortunately, such research only engages with the question of what works to a limited extent.