5.1 Introduction
Public administrators around the world are enthusiastically seeking to ‘harness the power of data’, reflected in the growing use of algorithmic decision-making (ADM) systems within governmental organisations, eager to replicate their successful application in commercial contexts.Footnote 1 Many of these networked digital applications include those developed through the application of machine learning (ML) techniques to score, sort and rank individuals to inform how public services and functions are administered, including those which purport to evaluate an individual’s predicted ‘risk’.Footnote 2 A combination of technological advances and socio-economic factors have fuelled digital transformation projects within public administration (in a tendency I have identified elsewhere as the ‘New Public Analytics’).Footnote 3 This chapter focuses on how a particular set of promises, assumptions, beliefs and rationalities (collectively referred to as ‘logics’) prevalent in the commercial sphere are being translated and applied to public sector environments in ways that have fuelled the take-up of these ML-based citizen scoring tools in government.Footnote 4 Although the use of so-called big data analytics (now more commonly referred to as ‘Artificial Intelligence’ or AI) in government for the purposes of categorisation, assessment and prediction at both individual and population level are relatively recent,Footnote 5 the practice of individual risk assessment through actuarial methods has been employed by public authorities for several decades.Footnote 6 By the early 1990s, important state functions conventionally undertaken in highly industrialised states, particularly decisions about whether to release inmates serving jail sentences or in pre-trial custody, were already being discussed and characterised in terms of ‘risk management’ at the same time as the English language publication of Ulrich Beck’s ‘risk society’ thesis in 1992.Footnote 7
The claimed purpose of these citizen scoring tools, both then and now, is to ‘optimise’ the allocation of public sector resources. Yet the current wave of ML-based citizen scoring tools and systems are touted as superior to the actuarial techniques that preceded them, claiming to offering better, more accurate and ‘objective’ or ‘evidence-based’ predictions, based on the success of ML applications in commerce.Footnote 8 My analysis is structured around three celebrated promises invoked to justify the enthusiastic take-up of ML-driven prediction models, notably: (a) more accurate prediction, (b) the generation of ‘actionable insight’, and (c) the value (and superiority) of ‘early intervention’. In so doing, I address three questions.
First, whether ML-based scoring, which has proved successful in the private sector, can be expected to generate ‘more accurate’ predictions to inform decisions made by public authorities about individuals, in comparison with actuarial methods that have long been used by public authorities to assess an individual’s ‘risk’. Second, in what way has the promise of ‘actionable insight’ that fuelled adoption of AI in the commercial sector been invoked in public sector contexts? Third, to what extent have assumptions about the value of ‘early intervention’ to prevent or avoid a future outcome associated with a specific individual contributed to the enthusiastic embrace of citizen scoring, and how can those claims be reconciled with foundational constitutional principles, particularly basic respect for the freedom, dignity and agency of the individual?
I will argue that these three central promises, which have fuelled ML’s enthusiastic adoption by commercial firms, are being unthinkingly adopted by public administrators (at the national, European and global level), failing to recognise that the role of the state is fundamentally different from, and antithetical to, the role of private actors (including for-profit commercial organisations) within liberal democratic states. While the latter are free to pursue private gain provided that they abide by law, public authorities are duty-bound to serve the interests of the public in accordance with, and constrained by their legislative mandate and subject to the safeguards and constraints imposed by constitutional principles, including respect for human rights and doctrines of administrative law. This unthinking and inappropriate translation of ML’s promises from commercial applications into public decision-making tools and systems results in a proliferation of opportunities for injustice and the abuse of public authority in ways that interfere with the rights of individuals while restricting opportunities for affected individuals to understand and contest these decisions. Yet because it is the most vulnerable members of society, for example those convicted of criminal offences, who are parsed and made subject to ‘early interventions’ by public authorities, the use of ML scoring has not attracted significant public resistance. The unjust administration of rights-intrusive interventions and, in turn, the allocation of public benefits and burdens, are being silently yet effectively redistributed through the adoption of ML tools without adequate public oversight, democratic debate or public consent. I demonstrate how this political foreclosure happens, not so much within the space of mathematical measurement and computational calculation, but in the unthinking and misplaced translation of ML’s promises from commercial contexts into the realm of public administration.
5.2 The Promise of More Accurate Predictions
The promise of more accurate prediction, compared with conventional statistical tools, continues to fuel the enthusiastic embrace of ML models across many and varied domains. For commercial organisations whose success depends on attracting and retaining individual customers online as repeat customers, or digital platforms for which greater user ‘engagement’ is directly proportional to the advertising revenue generated,Footnote 9 the promise of accurate predictions concerning an individual’s tastes, preferences and behaviours has been seductive.Footnote 10 Like their retail marketing counterparts, public officials are also keen to accurately predict the future behaviour of individuals, yet their interests and motives are rather different. Since the late 1920s, public officials have sought to predict the ‘riskiness’ of individuals, particularly in the US criminal justice system. Although initially risk was understood as ‘risk of absconding’, it now tends to be understood primarily in terms of ‘predicted dangerousness’.Footnote 11 While so-called ‘first generation’ risk assessment relied upon unstructured clinical assessment, these approaches were succeeded by ‘second generation’ prediction tools during the 1990s when actuarial tools commonly used in the insurance industry were taken up by US criminal justice officials, particularly in seeking to manage the crisis in prison overcrowding resulting from President Nixon’s ‘tough on crime’ policies.Footnote 12 While these prediction tools relied on ‘static’ risk factors (i.e., factors that cannot be changed) drawn from empirical research, ‘third generation’ risk assessment tools later entered into use, incorporating ‘dynamic’ risk factors (i.e., factors that can be changed either slowly or relatively quickly over time, such as marital status, employment and education level) which advocates argued would increase their efficiency by allowing public officials to target and monitor their risk-reduction efforts.Footnote 13 The most recent ‘fourth generation’ risk assessment tools build on third generation tools, but also include case management plans, seeking to match each individual’s unique profile to an evidence-based ‘treatment plan’ by combining risk management efforts with individual case planning.Footnote 14
Scholarly interest in individual risk prediction tools by public authorities are enjoying a revival with the development of tools based on ML prediction models.Footnote 15 The capacity to accurately predict a person’s future behaviour which features in the commercial rhetoric surrounding ML’s touted economic value is increasingly evident in public sector contexts, particularly for public administrators already engaging in individual risk assessment practices to inform decisions. Although the promise of accurate prediction is understandably beguiling, public administrators have largely accepted at face value the underlying assumptions and conditions upon which this promise is based. In the following discussion, I argue that this promise, and the conditions upon which its fulfilment depends, do not withstand critical scrutiny.
For public authorities, ML’s promise of more accurate predictions about an individual’s future behaviour holds out the possibility of overcoming the ‘individualised justice objection’, one of the most sustained and significant criticisms of actuarial tools that rely on conventional statistical techniques.Footnote 16 In particular, the use of actuarial predictions which underpin the practice of ‘risk profiling’ to inform the treatment of persons, particularly in criminal justice contexts, has been persistently criticised on the basis that it fails to attend to the relevant person’s specific characteristics, context and circumstances, thereby departing from the principle of individualised justice which posits that each person has a basic right to be treated as a moral agent with dignity and respect.Footnote 17 Instead, actuarial assessments treat individuals on the basis of characteristics shared with other members of a group.Footnote 18 The use of ML techniques applied to very large datasets, however, is often portrayed as overcoming this limitation. As Cukier and Mayer-Schönberger, authors of the popular science book that first brought ‘big data’ into public consciousness, claim:
using big data we hope to identify specific individuals rather than groups; this liberates us from profiling’s shortcoming of making every predicted suspect a case of guilt by association. In a big data world, somebody with an Arabic name, who has paid in cash for a one way ticket in first class may no longer be subject to secondary screening at an airport if other data specific to him make it very unlikely that he’s a terrorist. With big data we can escape the straightjacket of group identities, and replace them with much more granular predictions for each individual.Footnote 19
This statement rests on a belief that ‘more granular predictions for each individual’ overcomes the individualised justice objection levelled at individual risk assessments produced through conventional statistical techniques while reflecting prevailing commercial rhetoric about the promise of ‘more accurate’ predictions of individual behaviour. Yet this belief is mistaken. Although ‘accuracy’ in predictions concerning an individual’s future behaviour are possible through the use of some types of ML models under certain conditions and depending upon the specific behaviours that it purports to predict, the kinds of ML individual risk-prediction tools that public authorities have begun to embrace continue to be based on the projection onto that individual of the profile of a group into which a given individual is allocated. In other words, ML techniques, as they have been deployed in the public sector to ‘score’ a person’s ‘risk’, continue to rely on the same logic of group projection that underpins actuarial statistical methods they are often claimed to supersede. As Imrey and Dawid explain in relation to ‘recidivism risk’ predictions, unless one observes data on recidivism specifically from a new offender (D), then statistical inference about D’s individual risk must be based on assumptions linking D to a population or process from which recidivist outcome data are available to serve as a source of inference that can inform predictions about D’s individual recidivism risk.Footnote 20 They continue:
Much confusion about individual risk assessment is … of semantic origin … But by placing ownership of risk with the offender on whom it is projected rather than the reference group from which it was derived, such phrasing conflates the concepts of individualized risk, derived from an external group and projected onto the subject [i.e. statistically assessable risk arising within the external group projected onto the individual], and latent individual risk intrinsic to the subject herself. This invites confusion between statistically assessable variability in the ARAI [actuarial risk assessment instrument] production process and variability of conceptual, latent quantities, intrinsic to individuals, for which relevant data are unavailable and hence statistical assessment inapplicable.Footnote 21
Furthermore, when confronted with claims that ML prediction models offer greater prediction accuracy, we must attend carefully to the phenomenon purportedly predicted and over what time horizon. This is particularly important when our concern is with predicting a person’s behaviour, given the enormous variation of real-world human behaviours of interest. The more specific, proximate and directly observable through automated means the candidate behaviour is, the greater the level of attainable prediction accuracy. For example, those engaging in online advertising will typically seek to maximise the number of user-clicks on the web-content links automatically served up to individual users. Hence the kind of human behaviour that these advertisers seek to predict concerns whether a person will ‘click’ on a website link. Impressively accurate predictions are possible now that users’ click-stream data is routinely automatically and continuously collected and parsed via ML models applied across a population of users alongside real-time, automated feedback on whether each prediction about a user’s click through behaviours was successful or not.Footnote 22
In contrast, consider the challenge faced by child protection authorities in predicting whether a particular child is ‘at risk’ of abuse or gross neglect seeking to use algorithmic prediction tools to allocate preventive family support services or to make child protection screening decisions. The interpretive vagueness that surrounds the notion of child abuse, combined with the dearth of reliable data on actual incidents of child abuse and neglect, means child protection prediction algorithms are more often trained on so-called ‘proxy’ data which may be only weakly related (if at all) to abuse incidents across a population.Footnote 23 This generates what Hildebrandt refers to as the ‘reality gap’ due to an ML function failing to correctly define the real-life events that the data supposedly represent. She attributes this gap to the ‘low hanging fruit’ problem, referring to circumstances in which the availability of big data may appear to enable reliable inferences, whereas in fact the data is not relevant or incomplete.Footnote 24 These errors can also be understood in more conventional statistical parlance, understood in terms of the failure to adopt a representative sampling frame for the purposes of statistical assessment. For example, Kendall notes that child abuse prediction algorithms are often trained on administrative data on reported abuse. But much child abuse and neglect are never reported, and many families that are reported may be subject to both surveillance and personal biases meaning the sample data generated by child protection systems is not representative of the prevalence of child abuse across a population. This, in turn, profoundly undermines the ability of those predictive tools to accurately identify those at risk of abuse, while lending weight to claims that racial and class-based discrimination is built into them.Footnote 25
The ‘reality gap’ is compounded when relevant factors and outcomes are omitted from the training data. In particular, data upon which individual risk prediction algorithms are trained cannot capture ‘counterfactual’ outcomes. For example, if a child was taken into care, then the intervention will preclude the production of data on whether that child would have been neglected and abused had that child had not been taken into care. Conversely, if an abused child is not taken into care and that abuse remains unreported, then the reported data will indicate that the prediction recommending against taking the child into care was correct even though it was not.Footnote 26 Prediction accuracy is eroded further due to the omission of other types of salient information which is not included in the training data, such as changes in the level of informal social support available, which is a key factor in child abuse development.Footnote 27
These difficulties remind us that the ‘quality’ of an ML-based prediction tool is not only a product of the quality, size and richness of the data upon which it is trained, but also of the ‘interpretive work’ associated with the collection and processing of data in the service of useful knowledge production.Footnote 28 To be regarded as data, a data point must relate accurately, directly, reliably and consistently to some phenomena that it purports to represent. If data collected for administrative purposes is the source of ‘ground truth’ data upon which an ML individual risk prediction model is trained and developed, great care is required in making assumptions and assertions about what the resulting predictions are taken to represent, especially when those predictions inform rights-critical decisions about how individuals are treated, particularly by public authorities.Footnote 29 Yet the celebratory rhetoric associated with AI has resulted in a tendency to downplay or largely ignore these inconvenient realities.
Furthermore, care is needed when making generalised claims about all ML techniques, given their wide variety and the different ways in that prediction models are configured within socio-technical systems. As König and Krafft observe, it is striking that algorithmic risk assessments are framed in public debates as ‘Artificial Intelligence’ in a manner that has been partly decoupled from actual practice.Footnote 30 Even well-informed scholarly accounts may refer to ‘AI’ when discussing individual risk assessment tools of the kind that purport to predict an individual’s ‘recidivism risk’, failing to recognise that they closely resemble actuarial prediction models produced by conventional statistical approaches and bear little resemblance to the automated dataveillance systems now routinely employed by digital platforms to predict, generate and distribute online content that fuelled the meteoric rise of the Digital Tech Titans who now dominate the platform economy.Footnote 31
Instead, the ML prediction models used by public authorities to generate individual risk predictions are typically static models trained on historic administrative data and do not provide automated feedback about whether a specific prediction was successful or not.Footnote 32 Yet there remains a problematic (albeit understandable) tendency, even amongst well-regarded scholars, to make generalised claims about ML techniques, failing to recognise that capabilities and criticisms that apply to some kinds of ML techniques and prediction models cannot be appropriately generalised and applied to others.Footnote 33 Online prediction tools seek to predict very specific, narrowly defined, directly observable individual behaviour that is reliably and consistently monitored and measured (as indicated by user ‘clicks’) that are dynamically, automatically and continuously updated from feedback on whether the automated predictions were successful or not. For these algorithmic systems, it may be legitimate to assume that past data patterns are a reliable predictor of future behaviour because we have both high-quality longitudinal data (i.e., granular data over time in relation to a specific person) and lateral data (i.e., it extends widely across a population) to enable individual predictions about a very specific, narrowly defined and proximate future behaviour (whether a user will ‘click’ on a link).Footnote 34 In contrast, the individual risk prediction models used, for example, to assess ‘recidivism risk’ by US criminal justice officials, or by child protection agencies, bear greater similarities to conventional actuarial prediction models and for which their real-world prediction accuracy is unknown.Footnote 35
These concerns underscore the need for caution when interpreting the meaning, reliability, quality and significance of ML outputs, particularly claims of ‘prediction accuracy’. For data scientists, ‘accurate prediction’ is conventionally defined in narrow, technical terms based on the model’s performance in accurately predicting outcomes from a portion of data that formed part of the original dataset from which the training data was taken and specifically set aside for the purposes of assessing prediction accuracy (the ‘test set’). Accordingly, the resulting accuracy metrics are not necessarily indicative of accurate prediction in real-world settings: they pertain exclusively to the original dataset used to train, validate and evaluate the model. In this respect, Tobias Blanke rightly argues that ‘high accuracy figures for decision-making algorithms should never be enough to reassure us that these algorithms are correct and make wise pre-emptive decisions’.Footnote 36 Moreover, an analysis by König and Krafft of individual risk assessment tools used by US states in pre-trial settings reveals that the predictor outcomes themselves vary considerably across tools. They found that although these tools always claim to predict ‘failure to appear’, some claim to predict ‘new criminal activity’ while others offer predictions of ‘being rearrested’, while still others blend substantively different concepts offering a composite measure of both ‘failure to appear’ and ‘new crime/arrest’ to produce a single numerical score.Footnote 37 While these scores might assist in the ranking of candidates relative to each other, whether the score itself bears any consistent and reliable relation to social reality is seriously questionable.
It may be tempting to respond that these problems are fundamentally reducible to the lack of high-quality data pertaining to individuals of the kind that Imrey and Dawid claim preclude the prediction of what they call an individual’s ‘intrinsic risk’. Their account might imply that if relevant recidivism data about that individual were available, the individual’s actual ‘intrinsic’ risk of recidivism could be statistically inferred. Yet this belief is also mistaken, based on a fantasy which computer scientist and philosopher Reuben Binns refers to as ‘Laplace’s omniscient demon’.Footnote 38 Binns points out that even patterns in mouse movements, social graphs, recent purchases and so forth are often framed and taken as equivalent to an omniscient demon’s knowledge of the position and forces of nature. He remarks that:
Human behaviours like repaying a loan are fundamentally unlike the process of drawing coloured marbles from an urn or flipping a coin. As Ian Hacking charted, probability emerged in the 17th century as a new way of knowing, and statistical regularities became elevated to a status analogous to laws of nature (Hacking 2006). But patterns at population level are not explanations of any single individual’s behaviour. From the perspective of Laplace’s omniscient demon, no individual person is 60% likely to default on a loan or commit a crime; they either will or they won’t, in the fullness of time, depending on the precise configurations of the position of matter and the operation of natural laws.
But absent such omniscience, individual behaviours are indeterminate, and only predictable at the population level. It is in this space of indeterminacy where we act in ways we personally identify with, and where we attribute both to ourselves and to others freedom, agency and intentionality. Regardless of how we understand the notion of free will at a metaphysical level, attributions of agency persist in the face of population-level statistical regularities. This is one reason why people may object to the use of statistical prediction to make decisions about individuals. Even if every other person with a given set of features acted in a certain way, the nth person sharing those same features might act otherwise.Footnote 39
In other words, as Mireille Hildebrandt has emphasised, the human self is ‘fundamentally incomputable’. No algorithm can be trained on future data: the temporality that grounds us in our humanity and the natality that it entails confronts ML with the fundamental uncertainty of the real world.Footnote 40
While achieving perfect accuracy in predicting human behaviour will forever remain an impossible dream, it is alive and well in the minds of those peddling data-driven ‘solutions’ to cash-strapped public authorities. Although the latest risk-prediction models are, like their earlier counterparts, aimed at facilitating more efficient public sector resource allocation, their novelty lies largely in the normalisation of this objective combined with a flawed belief that ML algorithms and the data deluge accompanying the digital revolution enables the routine and accurate prediction of individual behaviour of all kinds, with scant critical reflection.Footnote 41 Both then and now, the harnessing of statistical tools in the service of more efficient public resource allocation has often been a pragmatic response to crises such as managing prison overcrowding, or easing the fiscal crisis facing contemporary public services.Footnote 42 It is precisely this ‘more for less’ challenge that elevates the importance and attraction of the promise of ‘actionable insight’ for public authorities, to which I now turn.
5.3 The Promise of ‘Actionable Insight’
Directly tethered to the promise of more accurate predictions is the promise of ‘actionable insight’, based on a belief that if an organisation can predict the likely behaviour of a user, they can take pre-emptive action to shape that behaviour to serve organisational interests. For digital platforms, predictive analytic systems that track the online behaviour of individuals across a population are now commonly used to sift, sort and score individuals, enabling the automated provision of targeted content.Footnote 43 Such consumer-facing digital services are essentially a ‘barter’ model in which users agree to continuous monitoring of their online behaviour and the collection of digital traces (including meta-data) thereby generated, in return for services.Footnote 44 Shoshana Zuboff and others have emphasised how reliance on these post-industrial forms of capitalist production, which Zuboff dubs ‘surveillance capitalism’,Footnote 45 requires the continuous and pervasive surveillance of individuals, channelling and controlling flows of personal information while converting them (via the use of AI technologies) into flows of profit, in ways that are obscured from users.Footnote 46
These contemporary forms of digital capitalism have spawned a new modality of production which I have referred to elsewhere as ‘mass predictive personalisation’. Like late nineteenth-century industrial modes of production, mass predictive personalisation has the capacity to operate at scale (and thus on a ‘mass’ basis), but differs from industrial production in significant ways, three of which are directly rooted in the promise of ‘actionable insight’.Footnote 47 First, rather than generating identical units of production, services are personalised to each user’s predicted tastes, interests, preferences, lifestyle and behaviours. Second, service provision operates, by default, on a predictive basis due to the application of algorithmic profiling to infer and predict each user’s service preferences with the aim of anticipating user needs and ‘pushing’ appropriate personalised services. Third, services are continually and automatically reconfigured based on feedback gleaned from monitoring the recipient’s responses without requiring active and intentional feedback concerning the user’s interest in the service provided.
5.3.1 What Does ML-Based Actionable Insight for Public Administrators Consist Of?
For digital platforms, the promise of actionable insight rests on the ability to pre-empt and personalise services offered to users, given that the size of a platform’s user-base is typically directly proportional to its market valuation. But for public administrators, actionable insight from risk prediction tools promises to optimise the allocation of public resources that have been severely constrained since the financial crisis of 2008.Footnote 48 Yet in developing ML risk prediction, the process of model development and the model itself became detached and decoupled from the substantive interventions intended to flow from predictions thereby generated. Instead, the rationale for ML-derived ‘actionable insight’ proceeds on the assumption that more efficient resource distribution will inexorably flow from specifically targeting high-risk candidates, whatever substantive intervention is deployed. It is here that the mundane rubric of process optimisation is at work via a logic of ‘reverse reasoning’. This reasoning begins with the status quo by taking an existing substantive intervention already in use by public officials (X) to address a given social problem then works backwards by narrowing and reframing the key task as one of more precise ‘targeting’ to inform the allocation and distribution of that substantive intervention. The core logic takes the following form:
For any existing substantive intervention X and a pre-specified set of constraints (Y1, Y2 … Yn) how should X be allocated and distributed in order to optimise objective Z?
Examples of this kind of ‘reverse reasoning’, paving the way for the configuration of ML predictions models for organisational use, might include the following:
Given that we already incarcerate convicted criminal offenders but we have a limited number of prison places, how can we optimally allocate prison places to ensure that the most dangerous offenders are incarcerated?
Given that social workers may take children identified as ‘vulnerable’ and ‘at risk’ of abuse and/or negligence into state custody and care without parental consent and the availability of public resources to provide this care is limited, how can we optimally identify and take into care the most vulnerable children?
Given that the police receive more complaints and reports of threats of domestic violence than they can investigate in any given time frame, in which some complainants are likely to be more ‘vulnerable’ than others to future violence, how can we sort and rank these complaints so that the most serious are prioritised, thereby optimising the time and resources devoted by police to these complaints?
This reverse reasoning takes place at the beginning of the development process when technical developers engage in the tasks of problem-framing, abstraction and goal identification necessary to construct a computational model to inform a real-world task. This often entails making several simplifying assumptions which enable computer scientists to create a mathematical model of a real-world task capable of taking input data and algorithmically converting it into outputs to inform that task. As Andrew Selbst and his colleagues explain:
Decisions about what constitutes an input and what constitutes a desirable output are the ‘abstraction’ choices made as part of the design of the resulting system, and they represent the boundary between the interior of the system and the exterior …
For the designer of the system, the processes that lead to the production of the inputs, as well as the processes that follow after the production are not to be considered part of the problem space and are disregarded.Footnote 49
In this way, a wicked problem – one that is impossible to solve perfectly because of its complexity – is reframed and recast as a tractable computational problem for which a ML prediction model can be developed and then rendered in the form of a user-friendly software tool for use by front-line officials.Footnote 50
5.3.2 A Critique of ML-Produced Actionable Insight for Public Administrators
This reverse reasoning and the reductionist problem-framing that it relies upon is closely connected to what I have referred to elsewhere as a process of ‘contextual detachment’ that occurs during the process of rendering a social problem computable.Footnote 51 This entails hiving-off the substantive intervention actually administered based on the output produced by an ML prediction tool, from the process of developing a computational model that will generate a numerical output or score. This logic of ‘decoupling’ is evident in the findings of Cardiff Data Justice Lab researchers who found, based on interviews with developers of public sector data analytics tools for UK local authorities, that developers had little awareness of what happens as a result of the outputs produced from the prediction tools that they were commissioned to build.Footnote 52 This decoupling is problematic for several interrelated reasons.
First, reverse reasoning recasts social problems that necessarily implicate multiple values and interests (and which therefore require normative evaluation) as largely ‘technical’ decisions about how to utilise existing data to accurately identify high-risk targets and to distinguish them from their medium and low-risk counterparts. Rather than asking, for example, how children at risk of abuse and neglect are best protected and supported, the focus is reoriented towards seeking to identify which children should be coercively taken into state custody. It entails reframing the relevant question as a technical targeting problem so that the relevant question becomes whether the prediction model produces ‘better’ scores than those produced by human officials. In other words, the take-up of ML to inform operational decisions prioritises the ‘who to target’ question while treating the ‘how’, ‘what’ and ‘why’ questions as irrelevant. Questions about what policy objectives should be pursued in this domain, how those objectives should be appropriately prioritised and traded off, and what ‘substantive intervention’ should be undertaken in pursuit of them, are partitioned off and excluded from discussion, foreclosing more open-ended reflection and creative and critical debate about the foundational real-world problem that the prediction model is intended to help ‘solve’.
Second, by decoupling and detaching the substantive intervention that is intended to follow the production of a risk prediction, the underlying reasoning reflects the ‘seeing is solving’ fallacy associated with the promise of ML-produced ‘actionable insight’. This is precisely the trap into which Chicago police fell into creating a ‘Strategic Subject List’ (SSL) in 2012. The SSL assigned risk ratings to tens of thousands of Chicago residents based on who was most likely to be caught up in violence (either as a victim or criminal). It was an algorithmic system made accessible to police on patrol which scored people based on myriad factors, including their criminal histories, particularly weapons or drug offences, and whether they had been shot or affiliated with a street gang, with a higher score indicating a greater risk of being involved in violence. Yet the system was quietly abandoned in 2020 following a report by the Inspector General and an independent report by the Rand corporation which found that it had failed to provide a practical way to reduce violence for several reasons,Footnote 53 including its reliance on arrest data in producing risk scores irrespective of whether an individual had been convicted of any crime and the fact that a ‘level of public fear’ about the program overshadowed its efforts to reduce violence. The Chicago experience shows that the investment of significant public resources to develop and implement citizen scoring systems may do more harm than good. Yet, beguiled by the promise of ML-based citizen scoring systems, public administrators may be attracted to software ‘solutions’ while failing to attend to the difficult substantive questions concerning how best to respond to wicked social problems in light of the functions and collective values they are required to serve, and the individual rights and group interests those responses implicate.
Third, despite the invocation of the rubric of ‘process optimisation’ to encourage public administrators to adopt ML-based software systems (including data-driven citizen scoring systems), portraying them as mundane tools to facilitate administrative efficiency, these system are in practice typically configured to prioritise the so-called ‘sigma’ family of values related to economy and parsimony over the ‘theta’ family of values associated with honesty, trust, mutuality and fairness.Footnote 54 The real-world consequences of these systems, including the devastating hardship created for thousands of poor and working-class people in the state of Indiana, are vividly illustrated by Virginia Eubank’s analysis investigations into the impact of the adoption of an automated welfare administration system.Footnote 55 The new system abandoned its former reliance on human case-workers who developed one-on-one relationships with individuals and families following their cases through to completion, replacing it with a centralised data repository accessed via a ‘ticket’ system in which call-centre workers were presented with discrete ‘tasks’ in which no single worker had oversight of a case from beginning to end, and cases were no longer handled in the county where the applicant lived. Her account reveals how myriad system errors and failures were routinely and automatically classed as the individual claimant’s ‘failure to cooperate’, setting in train processes that would result in the termination of denial of benefits, with devastating consequences for the blameless individual. Yet, in the course of determining a lawsuit for breach of contract against IBM in developing and implementing the system, Judge Dreyer found that there had been no breach of contract:
The goals of the project were consistent throughout the automation experiment: maximize efficiency and eliminate fraud by shifting to a task-based system and severing caseworker-to-client bonds. They were clearly reflected in contract metrics: response time in the call centers was a key performance indicator; determination accuracy was not. Efficiency and savings were built into the contract; transparency and due process were not.
According to Dreyer, the problem with Indiana’s automation project was not that the private contractor had failed to deliver, but that the state and its private partners refused to anticipate or address the system’s human costs: IBM had no contractual obligation to measure its impact on poor and working-class citizens.Footnote 56 Although the automated system in question did not employ ML, it powerfully illustrates how the quest for automation-driven efficiency can abandon holistic responses to a person’s real human need into a series of discrete, modularised tasks for automated processing in which values associated with the treatment of individuals with dignity and respect are side-lined and subordinated to the public authority’s quest for transaction efficiency.Footnote 57 Particularly objectionable is the routine configuration of default settings in which the responsibility for any irregularity or error is placed on the affected individuals who then face an uphill battle to persuade a human official to investigate and take steps to ensure it is corrected while their access to basic services or benefits is threatened or withdrawn (aka ‘the computer says no’). In this way, relations between state and citizen are significantly albeit surreptitiously transformed in which the latter are increasingly dehumanised and excluded from active participation in decision-processes that profoundly affect their lives and, in some cases, threaten their basic survival.
The embrace of citizen scoring tools rooted in the promise of actionable insight by public officials is prone to overlook the value-laden nature of technical tools and systems, contrary to Langdon Winner’s persuasive demonstration that ‘artefacts have politics’.Footnote 58 There is a common tendency in both popular and technical scholarship to assume that automated computational systems are fully substitutable for their analogue predecessors which relied more extensively on human labour and judgement. This has facilitated the proliferation of celebratory claims about the capacity of artificially intelligent machines to ‘outperform’ humans. Yet these claims recast tasks historically performed by humans in narrow and circumscribed terms. Although machines may outperform humans with respect to one specific task in which a circumscribed range of values associated with technical accuracy, precision and speed are emphasised, these values may tend in practice to subordinate other important but overlooked and often tacit functions and values associated with existing organisational processes. As I have noted elsewhere, while a story podcast may generate a perfectly timed, uninterrupted narration of a children’s story enhanced with appropriately curated background music and sound effects, hitting the ‘play’ button on my smartphone story-time app to my ten-year-old daughter is a woefully impoverished substitute for the nightly ritual of reading to her from a book while she sits on my lap as she winds down before lights-out at bedtime.Footnote 59
The process of reverse reasoning that has accompanied the take-up of ML-based citizen scoring tools for public administration has enabled questions about the legitimacy of the substantive intervention – coercive incarceration, or taking children into custody, for example – as a response to complex social problems to be side-stepped. By way of abductive logic, the developer begins with the substantive intervention, the constraining factors that limit the availability of that intervention and asks whether available historic data pertaining to those interventions can serve as ground truth to create an ML prediction model configured to optimise a specific goal expressed in numerical terms. The focus of the inquiry then becomes centred on the task of accurately identifying and scoring individuals to identify who to target, rather than justifying the substantive intervention imposed upon the target. This logic is problematic due to its tendency to foster policies of ‘responsibilisation’, in which the locus of responsibility for a social problem is seen to lie with the flagged individual rather than a product of systemic societal forces and failures. This reflects questionable assumptions that result in a tendency to locate responsibility for a person’s behaviour and beliefs in the unfettered agency of the individual.Footnote 60 The hollowness of this logic is apparent when we consider the use of ML prediction tools to identify vulnerable children at risk of parental neglect and who can clearly not be held responsible for their own plight.
This logic is problematic for two additional and related reasons. First, the embrace of citizen scoring systems may foster an unjustified focus on the ‘targeting and ranking of individuals’, while losing sight of the substantive interventions that are intended to flow from the assignment of such a rating, and which may seriously undercut the public authority’s underlying policy goals. This is not to suggest that targeting decisions do not demand explanation and justification: they clearly do, particularly when machine predictions about individuals are viewed as ‘scientific’ and objective, aided by the numerical score delivered via a simple and attractive interface, overlooking the fundamentally incomputable nature of human agency. Rather, attempts to explain why an ML prediction model has flagged a specific individual as high risk is only one of the questions which public authorities within democratic societies are obliged to answer.Footnote 61 Due consideration must also be given to the intervention which this designation of ‘high-risk’ entails, namely: ‘is it acceptable to single this particular individual out for this specific kind of substantive treatment?’ The level of acceptable tolerance of error, and the kinds of error we are willing to tolerate, should be radically different when our concern is with predicting whether an arrested person is considered ‘dangerous’ and should therefore be retained in custody pending trial, compared to predictions about which advertisement may be of greatest interest to a given person.
Second, this logic fosters what I call the ‘status quo swerve’: by simply pointing to the substantive interventions employed in the past directed towards individuals identified as ‘risky’, this is assumed to be sufficient justification for their imposition at scale using automated prediction tools. In so doing, it unthinkingly reinforces and entrenches the status quo and fails to recognise that the automation of a function previously undertaken by a human official may change its very nature while side-stepping and thereby tending to close down discussions about alternative substantive goals, value priorities and substantive policies and interventions.Footnote 62 It is the absence of normative reflection and debate about the appropriate substantive response of public authorities and evaluation that I reflect on when examining the promise of ‘early intervention’.
I directly encountered the ‘reverse’ reasoning described above when participating in policy-level discussions about the use of algorithmic decision-making in public administration at an expert and stakeholder workshop convened in London in mid-2019. During that workshop, a police officer presented his research, which involved working with a data scientist to develop and build a prototype ‘domestic violence predictor’ tool to inform policing decisions. The police officer then invited the audience to ask questions.
‘What are police to do once the model predictively identifies those complainants considered most vulnerable?’ I asked.
‘There is a well-established protocol prescribing what should be done’ he replied, without offering me any explanation of what that the prescribed protocol entailed.
The officer then proceeded to inform me, in a withering tone, that the only question that needed to be asked was simply:
‘Does the machine make better predictions than human officers about whether an individual is “at risk” of domestic violence?’
It was plain that for this officer, questions pertaining to the substantive interventions that should be offered to support and protect members of the public who are at risk of domestic violence was not a question of concern nor one worth asking. From this officer’s perspective, wider questions served as distraction from what really matters: making sure that model ‘correctly’ identifies the most vulnerable complainants.
5.3.3 ML Promise No. 3: The Superiority of Preventive Interventions
The preceding discussion reveals how promises of more accurate prediction and the production of ‘actionable insight’ that have fuelled the take-up of ML prediction tools in online retail, media and marketing are being uncritically imported into the public administration, including rights-critical contexts such as the provision of social welfare benefits, care services and administration of the criminal justice system.Footnote 63 These tools are being deployed on the basis that individual ‘risk predictions’ are both valuable and useful to those who undertake governmental functions. Rhetoric surrounding the ‘actionable insight’ that is believed to flow from ML-generated risk predictions are considered especially valuable because they enable ‘early intervention’, which is regarded as presumptively superior to a ‘wait and see’ approach to intervention. On this logic, any person assessed by an algorithmic prediction model as ‘high risk’ is thereby regarded by the public authority an appropriate target for various forms of early intervention to prevent the threatened ‘risk’ from ripening into concrete harm or wrongdoing. This attitude was evident in the responses of commercial developers of citizen scoring systems interviewed by Cardiff Data Justice Lab researchers. Although interviewees often could not identify or explain how the data systems led to concrete measures, developers proceeded on the understanding that these systems were not intended to be ‘punitive’ but instead to enable ‘early intervention’.Footnote 64 In other words, although the developers recognised that citizen scoring systems might be problematic if used to inform the administration of punitive measures, they believed that non-punitive early interventions administered on the basis of ML-generated risk scores attached to individuals were not.
While state punishment for the commission of a crime necessarily stands in need of moral and legal justification and must be subject to safeguards against the abuse of the state’s coercive authority, the assumption that other forms of ‘early intervention’ do not is mistaken and deeply troubling. It reflects a basic failure to recognise that the mission and raison d’etre of public authorities are entirely different from that of commercial organisations. In particular, the functions and decision-making powers of public authorities are rooted in the constitutional authority of the state from which their legitimate and lawful authority flows, conferring, informing and constraining the authority which public administrators may exercise. Yet the entire thrust of public sector digital transformation initiatives since the mainstreaming of the internet has failed to grasp this crucial distinction between public and private bodies and their respective missions and duties. As Vishanth Weerakkody and Paul Waller observe in their Parliamentary evidence, the practices of ‘digital government’ apply the language and models taken from commerce with no adaptation to the governmental context, without any reference to what governments and public bodies actually do and how they do it: ‘current nostrums are the wrong way round. If I were to join a bank as a technology adviser, would it expect me to teach bankers about technology, or would it expect me to learn the business of banking and give insight into how to do it better with technology, using my deep knowledge? The latter of course.’Footnote 65
This observation takes us to the heart of the logics focused upon in this chapter, illuminating the deeply problematic way in which digital transformation in government has proceeded, including the embrace of ML-powered citizen scoring systems. The authority of public officials to take action, including coercive action, is premised on the authority of the state which, within liberal democratic political communities, is ultimately rooted in the consent of its citizens. Democratic government is ‘of the people, by the people, and for the people’,Footnote 66 and not, as is the case for commercial organisations, for the pursuit of private shareholder gain. For this reason, public authorities are subject to much more onerous legal duties, rooted in constitutional principle and administrative law, that circumscribe and limit the extent to which that authority can be lawfully exercised.
Interventions administered by public authorities, whether in the form of punishment or unilaterally imposed ‘early interventions’ are not presumptively benign or unproblematic, particularly those targeted at specific individuals. In contrast, services offered by private providers are rooted in the economic freedom enjoyed by all persons to offer services to customers provided they abide by law and their actions are not backed by the coercive authority of the state. Although various ‘early interventions’ unilaterally imposed by public authorities do not satisfy legal definitions of punishment, they may nevertheless be coercive, unwelcome, intrusive and rights-violating. Many examples can be found in contemporary practice, ranging from stop and search powers by the police to preventive measures employed by states under the rubric of counter-terrorism prevention, such as asset-freezing orders and travel bans.Footnote 67
While legal challenges against various measures alleging that they entail unjustified human rights violations have been met with a mixed reception from courts, the intrusive and often life-limiting impact that they have for targeted individuals cannot be denied. Within democratic states, the use and administration of these ‘pre-emptive’ interventions are circumscribed by constitutional principles, including respect for human rights and the rule of law.Footnote 68 For example, Noorda argues that just as the imposition of state punishment is subject to limits, constraints on liberty through preventive measures that undermine the autonomy of an affected individual should also be subject to expansive checks to ensure that they are lawful and legitimate, including legal guidelines that set appropriate limits on their use.Footnote 69 Similarly, Ashworth and Zedner persuasively demonstrate that although preventive measures that are not formally characterised as punitive have long been employed within the criminal justice system, they nevertheless stand in need of moral and legal justification and their use should be subject to a principled framework of safeguards to protect individuals from injustice and/or the abuse of the state’s power. While the appeal of the adage ‘better safe than sorry’ in justifying preventive interventions is difficult to deny, liberal democratic societies whose political and constitutional arrangements are rooted in respect for dignity and freedom of the individual as a moral and political agent must be at least equally mindful of the adage ‘let sleeping dogs lie’. It is a fundamental tenet of the rule of law that people are held responsible for what they do, not for what they might or might not do.Footnote 70 It is for this reason that preventive interventions taken by public authorities stand in need of clear justification and must be constrained by constitutional principle. In short, the power of public administrators to adopt ‘early interventions’ to forestall threatened risks is not unlimited, however well-intentioned or efficiency-enhancing the motives underpinning them. The mere fact that these interventions are not formally punitive does not imply that ‘anything goes’.
Although it is beyond the scope of this chapter to identify a set of principles that should limit and constrain the use of preventive interventions to avert a threatened risk, the conceptual framework offered by the ‘preventive justice’ model sketched by Ashworth and Zedner offers a valuable starting point for reflection.Footnote 71 They develop a set of values capable of generating both justifications for preventive measures and limiting principles. These values recognise that while the state has a legitimate duty to prevent harm to protect the security of its citizens, this duty must be tempered and constrained by its duty to respect individuals as autonomous beings, which means, amongst other things, respect for rights, the maintenance of rule of law values, the avoidance of arbitrariness and other kindred principles. Although Ashworth and Zedner confine their attention to preventive measures that entail coercive deprivations of liberty, my approach would seek to build on and refine their basic framework. Thus, to evaluate whether any given intervention can be pre-emptively and unilaterally imposed by the state, consideration must be given to both the degree of epistemic confidence we have in our judgements about the risk that the candidate in question poses and the severity of the proposed intervention, particularly its impact on the rights and interests of that targeted individual.Footnote 72 In other words, my suggestion is that attempts to justify preventive interventions by the state should proceed through a holistic analysis in which the confidence we have in our assessment of the likelihood and severity of the relevant ‘risk’ will directly affect the legitimacy of the proposed intervention so that the lower our epistemic confidence, the weaker the case for more intrusive measures.Footnote 73
The importance of a holistic analysis also points to the need to attend to three further matters: (a) the lack of concrete, systematic evidence attesting to the nature and magnitude of real-world benefits of data-driven citizen scoring systems, (b) the real human costs of such systems, particularly for those about whom scores are generated, and (c) whether the allocation of benefits and burdens that results from these systems are fair, just and democratically endorsed. Recent experience demonstrates that it is the poor, vulnerable and historically marginalised groups and communities who are being forced to bear a disproportionate share of the burden with little (if any) evidence of concrete benefits. In this respect, Lina Dencik and her collaborators rightly draw attention to the lack of attention given to the harms associated with the use of data-driven citizen scoring systems, and the paucity of evidence to establish that these systems lead in practice to more efficient resource allocation. She observes:
engaging with unanticipated harms that may arise from using such models is less salient amongst developers and managers working to implement data systems. In contrast … civil society groups … highlighted concerns about the ways in which a data lens particularly targets those on the margins … As one interviewee noted, ‘it’s not something that the bulk of the population will ever encounter. It’s something you only encounter when you are part of a risk group, a risk population’ (Netpol) …
Without comprehensive evaluation of how these new data arrangements are, or are not, affecting action, engagement and resources, these claims [of more efficient resource allocation] remain unproven. The argument that these systems make it easier for frontline staff to access and share information and assess risk is made with little, if any, evidence provided about how this affects resource allocation or actions taken … Moreover, councils pointed to a lack of resources in pursuing any comprehensive evaluations or impact assessments of transformations in practices and provision with the implementation of new data systems.Footnote 74
These research findings demonstrate that the take-up of citizen scoring systems is rooted in reliance upon promises rather than proof of their real-world benefits, suggesting that precious public resources are being squandered on systems that are unable to deliver on those promises. Although the selective distribution of ‘early interventions’ by public authorities based on individual risk scores may be well-intentioned and understandable, given the relentless pressure of austerity, these systems are fuelling the administration of public policies practices in ways that are presented as unproblematic forms of business process optimisation yet reflect value-laden and contestable political judgements about the appropriate treatment of individuals, groups and the distribution of public resources. Concerns expressed by civil society groups about the ‘unanticipated harms’ produced by these systems, help direct attention towards the unthinking beliefs that underpin these systems, including unexamined faith in the presumptive superiority of early interventions that target individuals, failing to reckon with the way in which those interventions are experienced as adverse by those targeted, while adhering to political commitments that are at odds with the basic foundations of democratic governance in constitutional democracies. It is the rule of law that underpins the entire edifice of democracy, in which individuals are held responsible for their actions and not for their thoughts, beliefs, inclinations or propensities, and certainly not those inferred on the basis of ML predictions parsed on administrative data routinely collected for bureaucratic organisational purposes. There are, of course, circumstances in which the state may justifiably and may even be duty-bound to engage in early intervention before any concrete harm or wrongdoing has transpired, particularly in circumstances where there is a clear and imminent threat of violence to a specific individual, as in the cases of alleged negligent failure by specific public authorities to intervene to protect a known and vulnerable individual against threatened harm.Footnote 75
It is striking that the most enthusiastic advocates of public sector scoring systems speak in the language of commercial imperatives infused by a mindset which regards the role of the state as essentially transactional and thus largely concerned with the purchase and delivery of consumer goods to customers.Footnote 76 This is not to deny the value and importance of operational efficiency, but an unquestioned assumption that early intervention in the lives of individuals based on ML-produced citizen scores requires no special justification is seriously troubling. It produces an undue and excessive focus on the decision about ‘who to target’ rather than the substantive question of ‘what should we do’ to address the particular social problem of concern. As Emily Keddell has observed in her critique of the use of ML-based prediction tools by social workers: ‘The root of the problem is that asking, ‘how can we identify the families who will go on to become abusive’ is fundamentally the wrong question to ask. A better question is “how can we more effectively address the risk factors that contribute to abusive behaviour”.’Footnote 77
Similarly, Featherstone, Morris and White demonstrate how the rhetoric of ‘early intervention’ became central to the ‘child protection paradigm’ which took hold in the policy and practice of British social workers in the mid to late 1990s in which risk became a dominating trope. According to this paradigm, the term ‘child protection’ rather than ‘child welfare’ or ‘supporting families’ is used, the focus being on the assessment of risk to children by family and care-givers, and prioritising procedures and risk averse practices while family support becomes secondary to the primary role of child protection.Footnote 78 This, they argue, has resulted in the replacement of the moral imperatives of mutual obligation that sustained political support for welfare states by a rhetoric of governance that prioritises the lessening of risk, rather than the meeting of need. In calling for the reimagination and reinvigoration of family support policies and practices that focus on family need underpinned by an ethics of care for vulnerable families rather than the highly individualised risk focus that now prevails, Featherstone and collaborators bring normative values and substantive politics back into the frame. This contrasts sharply with the contemporary fascination with ML techniques that reframes the challenge faced by social workers as that of accurate targeting to inform the administration of ‘early intervention’ while side-stepping the need to evaluate the legitimacy and legality of that intervention or the larger social policies from which those interventions emanate. It is through reliance on these logics that the ML prediction models serve to ‘foreclose politics’, invoking Louise Amoore’s terminology in her critique of ML systems that rely on deep learning techniques to inform decisions about public security.Footnote 79 My analysis demonstrates how this political foreclosure happens, not so much within the space of mathematical measurement and computational calculation, but in the unthinking and misplaced translation of ML’s promises from commercial contexts into the realm of public administration. This serves in practice to reinforce and expand public policies that dehumanise individuals, treating them as risk objects to be sorted, scored and singled out for unwelcome targeted intervention, as public administrators become more reliant on ML-generated predictions falsely believed to be ‘scientific’ yet are of highly dubious validity that magnify the production of injustice. As a result, very significant substantive policy changes are in effect taking place with little public awareness, let along robust discussion and debate, flying under the radar even as they violate fundamental rights and side-step the rule of law.
5.4 Conclusion
In this chapter, I have shown how logics that fuelled the successful adoption of ML techniques within commerce are being inappropriately and unthinkingly translated into public sector contexts in pursuit of operational efficiencies in the form of citizen scoring systems. Not only are these systems being embraced by public authorities without any clear, demonstrable evidence that they deliver on their promises, let alone enhance public value, but they are producing policy change by stealth while generating injustice at scale in ways that are difficult to see and often practically impossible to contest. The deeply problematic manner in which the promises of more accurate prediction, of actionable insight, and of early intervention are being relied upon to support the take-up of citizen scoring systems by public administrators is ultimately rooted in a failure to recognise that the role and authority of the state is entirely different from, and antithetical to, the role and freedom enjoyed by private persons (including for-profit organisations) within liberal democratic states operating within a capitalist economy. While the latter are free to pursue private gain provided that they abide by law, public authorities are duty-bound to serve the interests of the public in accordance with, and constrained by, their legislative mandate and subject to the safeguards and constraints imposed by constitutional principles, including respect for human rights and doctrines of administrative law.
The unreflective translation of ML’s promise from commercial contexts into public administration has proceeded on the basis of assumptions, beliefs and forms of reasoning that are portrayed as straightforward and self-evident. They are not. By framing the difficult and often fraught judgements that must be made in the course of carrying out public functions, such as child protection and criminal sentencing, as dreary matters of procedure and process that can be improved by embracing ML systems which offer more precise, accurate targeting and evaluation of individual risk objects, the embrace of citizen scoring systems conceals contestable political viewpoints. Although some ML prediction models may appear to offer dazzling capabilities, reflected in the fanfare accompanying the roll-out of large language models such as ChatGPT, research demonstrates that ML predictions about the life outcomes of individual persons are extremely poor.Footnote 80 Accordingly, the unilateral imposition by public authorities of ‘early interventions’ based on citizen scoring systems informed by ML prediction models, particularly when they are adverse and/or unwanted by those singled out for attention, are not only damaging, dehumanising and rights-violating but anti-democratic in a more general sense. They can undermine the rule of law, while excluding those scored and thence adversely affected from opportunities for meaningful participation or contestation, while reinforcing and deepening existing inequalities between groups that have been historically marginalised and those which have not.
In other words, the unthinking and inappropriate reliance on the promises of ML predictions in the form of citizen scoring systems to inform ‘early intervention’ decisions by public authorities offers a concrete site in which ML systems foreclose politics by side-stepping debate and deliberation about how collectively the state should act (on behalf of the community). This reliance also conceals the political transformation of relations between state and citizen in ways that are adverse to the latter, and to which the general public has not consented. It is striking that early interventions justified in the name of the precautionary principle which accompanied the transatlantic genetically modified organism (GMO) debates attracted intense political controversy given the contestable and speculative normative choices they entail, while those administered against individuals on the basis of ML-based citizen scoring systems attract so little public attention or political concern. These systems are redistributing power, benefits and burdens in ways that are hidden in plain sight, parading under the mundane rubric of business process optimisation, and underpinned by misguided promises of more accurate and precise risk assessment in the quest to enhance the efficiency of public sector operations. These systems are being used to sort, score and unilaterally intervene in the lives of the most poor and vulnerable in society, who lack political clout and resources to contest this sorting, let alone mobilise support to resist and oppose it, flouting constitutional principle while side-stepping the requirement to justify interventions or subject them to public discussion and debate.
Open access