4.1 Introduction

In 2017, the Australian Department of Immigration and Border Protection commenced procurement of a new technology platform for processing immigration and VISA decisions.Footnote ¹ Resistance quickly emerged around the risks of automating critical decisions affecting individual’s rights.Footnote ² Responding to those questions, the Department Secretary Michael Pezullo clarified that:

In a theological flourish, this was described by Pezullo as the ‘Golden Rule’, effectively expressing law’s implicit belief that only humans can be intentional moral agents.Footnote ⁴ In the language of technology governance, Pezullo was specifying the protective operation of a ‘human in the loop’ – a concept that has increased in popularity as a regulatory solution for accountability problems accentuated by automation.

In government decision-making contexts, automation typically takes the form of software systems that process specific elements of decisions or guide decision-makers through decision processes. As they have proliferated, these systems have provoked anxieties about inconsistency with rule of law principles. A substantial literature has diagnosed the ways automation challenges notions of legality on multiple registers. Some question the effects of automation on transparency, equality, and accountability;Footnote ⁵ others dig deeper into the relationship between automation and ideas in legal philosophy.Footnote ⁶ These analyses describe how digital technologies challenge the very media of our governing systems,Footnote ⁷ and undermine critical philosophical precepts such as the undecidable horizons of justice.Footnote ⁸ Frequently, these diagnoses conclude with an of appeal to re-humanise automated decisions in order to preserve their alignment with rule of law prerogatives. Beyond automated administrative decision-making, similar appeals emerge wherever automated systems operate. Across these domains, the ‘human in the loop’ has become a central pillar of humanising efforts and an ascendant regulatory orthodoxy designed to ensure the primacy of human values over mindless technocracy. This chapter argues the human in the loop does more than simply ameliorate technology’s disruptive potential. Through analysis of cases invoking human in the loop in administrative contexts, this chapter argues that, in the name of recuperating rule of law values, a human in the loop is used as a versatile legal technology that enables problematic distributions of accountability and inhibits a richer legal conceptualisation of automated decision systems.

4.2 What’s Wrong with a Human in the Loop?

Although laws demanding human oversight for automated decisions have existed for some time,Footnote ⁹ calls for a human in the loop have recently intensified as both ethical requirement and regulatory prescription.Footnote ¹⁰ This is not surprising considering its intuitive appeal as a regulatory strategy. If automated systems and software are opaque by design, the human renders decisions transparent. If automated decisions risk subordinating humans to technocracy, the human provides the requisite dignity of human consideration. If automated systems discriminate on account of the way data encodes social bias, the human corrects errors and ensures decision quality. If automation produces non-reflective forms of mindless governance, the human grounds the legitimacy and authority of legal commands.Footnote ¹¹ If software dissolves accountability into the procurement, design and engineering of a decision system, the human provides a coherent locus of responsibility. If automation includes emergent and generative qualities, the human garners trust, and normalises regulatory strategies premised on risk management.Footnote ¹²

These intuitions mean the human in the loop has become a sort of Talisman of the appeal to re-humanise – a symbolic regulatory apparatus.Footnote ¹³ At the same time, however, the human in the loop’s capacity to satisfy those demands has been questioned on a number of registers. There are empirical challenges to the human in the loop challenging its capacity to improve decisions,Footnote ¹⁴ satisfy dignitarian demands of human rights instruments,Footnote ¹⁵ or facilitate meaningful transparency.Footnote ¹⁶ Ben Green and Yiling Chen, for instance, shows that human oversight of data-driven judicial decisions does little to mitigate discrimination.Footnote ¹⁷ De-Arteaga, Fogliato and Chouldechova offer more mixed results, identifying a limited human capacity to intervene against malfunctioning algorithmic risk prediction tools in the child welfare context.Footnote ¹⁸ But the impact of the human in the loop on decision quality is, at best, uneven. This is not to suggest that technical systems make better decisions without human oversight. Humans are, of course, capable of improving technologically inflected decision processes. Scholars like Jennifer Raso detail how humans actively manipulate decision technologies, even beyond their intended input parameters, to achieve better outcomes by clinical standards.Footnote ¹⁹ But there is immeasurable complexity in how human interaction with different sorts of data-driven and algorithmic systems, influenced by behavioural biases, political preferences, and organisational arrangements influence decision quality and performance.Footnote ²⁰ And legal prescriptions for human in the loop rarely to attend those details.

Responses to the empirical unevenness of human in the loop requirements fall into two general classes. One group of commentators suggest the human in the loop’s failures stems from being not human enough. A more ‘human-centric’ approach is necessary to recuperate the human in the loop,Footnote ²¹ prompting examination and articulation of the conditions for achieving deeper human-centrism in decision automation. These proposals draw from human centred design and strategies for ‘function allocation’ that seek to leverage the systematic differences in human and machine capacities to improve outcomes, mitigate biases, or overcome institutional obstacles. Marion Oswald, for instance, suggests effective human discretion is obtainable so long as humans use their knowledge of factors not included in automated assessments, and use procedural and tacit knowledge along with their experience.Footnote ²² Ben Wagner similarly suggests that for human oversight to be meaningful, human decision-makers must be given adequate time; must be adequately qualified to fulfil their oversight task; be adequately psychologically and socially supported; have sufficient access to relevant information to make the correct decision; and have sufficient authority to change a decision if necessary.Footnote ²³

Other scholars read human in the loop’s limited empirical success as a more fundamental challenge to the value of human-centric approaches to governing automation. One branch of this critique challenges the liberal biases associated with law’s construct of the human. Connal Parsley, for instance, argues that the uninterrogated assumptions around the self-determining moral status and rational capacities of the human tasked with oversight have become a conceptual bottleneck to well-considered regulation of automated decision-making.Footnote ²⁴ Scholars like Matilda Arvidsson, analysing the human in the loop in ‘intelligent warfare’ and international humanitarian law, similarly question law’s gendered presumptions around ‘human’ capacities and characteristics in the context of digitally augmented posthuman actors in warfighting conditions.Footnote ²⁵ These critiques show the idealised legal human grating against more materialist accounts of legal agency that challenge clear distinctions between subjects and objects,Footnote ²⁶ and highlight how contemporary regulatory efforts are premised on the faulty assumption that, as Rebecca Crootof and her co-authors put it, the ‘human’ in the loop is ‘a concrete and identifiable entity, and thus an easy regulatory target’.Footnote ²⁷

Other scholars push the critique of human-centrism further, and identify fundamental incompatibilities between human and computational agency. Louise Amoore, for instance, argues that,

Gregor Noll similarly suggests the normative programmes of data-driven automation and human legality are entirely incompatible. He argues that automated decision-making systems are the product of cybernetic ontology that conceptualises human and machines equally as symbolic information processors, whereas legal decision-making derives from (secularised) monotheistic traditions that privilege the human as empowered to rule on earth.Footnote ²⁹ Noll describes human law as having its origin in the divine (word of God) made incarnate through the human processes of study and cognition, whereas automation takes natural life, represented and processed as data, as law’s origin. The computational process of ‘training’ thus replaces human cognition, leaving little room for humans to comprehend the relations between things.Footnote ³⁰ Mark Andrejevic claims in a related way that the logic of automation and its cascades cannot be governed by something that is not equally automated.Footnote ³¹

But these arguments terminate in a conceptual and regulatory impasse. Human oversight of automated decisions may, at times, be ineffective or functionally impossible, but abandoning human agency in decision-making is politically and functionally inconceivable. We might instead ask why the continued proliferation of a regulatory apparatus despite these empirical and conceptual shortcomings? Passage through this dilemma means excavating the broader political work performed by the human in the loop as a regulatory strategy, especially in how it underspecifies its human. This means understanding the human as a highly malleable legal technology, and exploring how differential operationalisations of the ‘human’ stabilise different sets of governing relations. Through that analysis, this chapter argues that the human in the loop does calculated work protecting political actors from the political nature and consequences of the choices that are embedded in technological and automated decision systems.

There is already excellent research describing this dynamic. That decision-making agencies dissimulate responsibility behind technological implementations is well documented.Footnote ³² And human oversight requirements are understood as a mechanism to responsibilise individuals for outcomes they may not control.Footnote ³³ But the jurisprudence outlined below shows what that entails in an administrative context, and the different kinds of political moves it enables. This chapter draws on case law addressing requirements for human decision-makers in different administrative examples to show how, in practice, a human in the loop can provide a governance gloss that both fails to grasp the realities of automated decision systems in operation or the political rationales animating them. These cases do not speak to the (in)efficacy of human in the loop for achieving better outcomes, but rather the systems of administrative governance that develop and proliferate behind the stylised distinction between human and machine endorsed by human in the loop requirements. In particular, these cases offer a lens into the distribution of cognitive tasks and the reconfigurations of accountability associated with automated decision systems used in the exercise of public power, exposing law’s posture towards questions of automation as well as the material stakes of this posture in practice. This chapter argues that the human in the loop paradigm dissimulates the realities of automated decisions along two axes: between humans and machines; and between the individual and the institutional, highlight the political reconfigurations at the heart of ‘governance by data’.

4.3 Human versus Machine: How Human in the Loop Reifies Unrealistic Distinctions Between Humans and Machines in Automated Decision Processes

Assumptions about the human, its moral agency, and the political stakes of human-centrism come into relief around the legal concept of a ‘decision’. The cases described in this section demonstrate how the prescribed role of humans in automated decision-making works to misassign distributions of responsibility between human and technological processes, as well as limit which elements of automated decisions are open to interrogation and contestation. An example is the Australian case of Pintarich v Deputy Commmissioner of Taxation, which considered whether a basic process-automation system had made a ‘decision’ that could be the subject of judicial review.Footnote ³⁴

Pintarich discussed by telephone the amount of tax liability he owed with an officer at the Australian Tax Office (ATO). The officer then keyed information into a ‘bulk issue letter template’ system, triggering a letter to be sent to Pintarich outlining his tax liability. The applicant used that letter to obtain a loan from a bank in order to pay the ATO. However, the ATO officer subsequently claimed that he never agreed to the level of liability discussed on the telephone nor did he intend to send the letter specifying that amount was acceptable. He instead sent another letter stipulating a higher amount of tax liability. The applicant brought judicial review against that second ‘decision’, claiming a ‘decision’ had already been made for the lower amount.

The majority judgment followed precedent established under non-technological decision-making conditions that ‘there needs to be both a mental process of reaching a conclusion and an objective manifestation of that conclusion’ for a decision to occur.Footnote ³⁵ In this case, the majority found no evidence of a mental process reaching a conclusion as to the lower amount of liability – the decision-maker had not undertaken any of the requisite deliberation, assessment, or analysis necessary for decision-making and appeared to trigger the letter by mistake (despite including his signature). Because it was not accompanied by the requisite human cognitive process, the letter sent by the automated system, that is, the objective manifestation of a conclusion, was deemed insufficient to independently constitute a decision in law. This finding is intuitive in many ways and accords with arguments around the centrality of human agency to the grounding of legal authority. Will Bateman, for instance, argues that fundamental public law doctrines assume that legal powers will be exercised by a particular kind of decision-making agent with the cognitive capacity to understand their task and the information relevant to the exercise of that task.Footnote ³⁶ This by implication requires engagement with an intellectual process.

The dissenting Pintarich judgment, however, presented an alternate analysis, rejecting the abstract two-part definition of a decision adopted by the majority. Noting the growing centrality of automation to contemporary governance, Justice Kerr suggested abstract, universal accounts of decisions are inappropriate where, in cases such as this, the actual decision process had been (at least) partially automated. Justice Kerr drew on other authorities indicating that legal decisions are frequently made without explicit mental engagement, that decisions need not be coincident with intention, and that fully automated decisions often represent the position of a state agency. Ultimately, he described the requirement of human mental processes for legal decision to be ‘an artefact of the past’, ‘challenged by automated “intelligent” decision systems’.Footnote ³⁷ While evacuating legal decisions of human moral agency may undermine some administrative law principles (such as those identified above by Bateman), Justice Kerr noted it recuperates others such as fairness and administrative certainty.

That unfairness and administrative uncertainty identified by Justice Kerr were the product of authorities recanting a decision on the basis that it was not informed by adequate human cognition even though human action caused it to be communicated to the decision subject. In other words, the majority’s finding meant the human decision-maker would not be responsible for how their interaction with an automated decision system stipulated and communicated an outcome to a decision subject if it did not match their subjective intention, despite any harm it might cause. The human in the loop requirement thus became a mechanism to prevent unpacking how decisions actually happen and how to understand responsibility in the context of ongoing interactions between a human and an automated system that affects people’s rights. Justice Kerr’s call to pay attention to the reality of decision-making in automated contexts is thus a call to (1) reconsider the normative prescriptions associated with human cognition and agency within broader normative goals of public authority, and (2) understand the complex reconfigurations of responsibility associated with automation in governance.

This notion that explicit human cognition is necessary for legally meaningful decisions is particular to administrative law and the availability of judicial review. Other legal regimes, such as in Article 22 of the EU General Data Protection Regulation,Footnote ³⁸ have long recognised the possibility of fully automated decisions with legal effects.Footnote ³⁹ Whereas the Australian administrative jurisprudence suggests no decision occurs without a human cognitive process, the European data protection regime suggests, alternatively, that decision subjects have a right to have humans supervise or participate in otherwise automated decisions in certain contexts such as if they produce legal or similarly significant effects. That is, fully automated decisions do occur, but there is a right to not be subjected to them – or otherwise demand some human participation in their making. But even this regulatory arrangement produces strange distributions of responsibility and dissimulates critical political arrangements into technological artefacts. An example is the Dutch SyRI judgment.Footnote ⁴⁰

The System Risk Indication (SyRI) system aggregated personal data from various government sources to detect social security fraud. SyRI was a hybrid decision system, with a computational component aggregating and analysing data to produce risk profiles and assessments, flagging individuals for investigation, with a human making the final decision to investigate any particular person. SyRI was abandoned after being found to breach the Right to Private Life in Article 8 of the European Convention of Human Rights for inadequate transparency around how the system was deployed, who was notified, and the content of its risk models. The case was lauded as a victory against the automated administrative state.Footnote ⁴¹ On close analysis however, the judgment also exposed how Article 22 (and its human in the loop prescription) worked to bracket the computational and automated dimensions of decision-making from proper review or contestation.

That judgment considered the human in the loop question in the context of whether the automated creation of risk reports constituted a form of decision that might be contested under Article 22 of the GDPR.Footnote ⁴² Although not directly concluding on the point, the Court accepted that identifying individuals as possible fraud risks did not constitute a decision with legal or similarly significant consequences. Risk profiling was not sufficient to constitute ‘automated individual decision-making’ for the sake of Article 22. That threshold was reached only after the subsequent decision to investigate was made by a human.Footnote ⁴³ On the one hand this makes sense, it would be peculiar to specify that a human participate (more than they always already implicitly do) in each step of a machine learning driven decision process. But the human in the loop rule in this case, by limiting reviewability and contestability through deeming only the human input as legally meaningful, misrepresents the nature of technically inflected decision-making. It renders opaque how an automated system has ‘pre-selected’ those to be investigated further, how a human interprets and interacts with a risk report, and the range of human political decisions that go into the specification and operation of that automated system. It explicitly prevents interrogation of how a decision process is distributed amongst various actors, human and non-human, on the basis that there is an accountable human actor choosing whether or not to investigate any particular individual. The human in the loop rule therefore inhibits understanding the contribution of risk profiling and ranking systems to the exercise of public power, and how responsibility might be tracked through those systems.

Risk ranking is increasingly central to public administration and numerous other governance regimes. Automated systems rank huge populations for various types of risk, with the top hits becoming subject to further human scrutiny. Despite claims such systems are merely efficiency tools with no influence over outcomes, they meaningfully challenge the human-centric notion of ‘decision’ embedded in the human in the loop regulatory paradigm. There is, of course, guidance, on what level of human oversight is necessary for these systems to be acceptable,Footnote ⁴⁴ and the administrative law of decision attempts to delineate ‘stepping stones’ from ‘decisions’ that determine legal rights or entitlements, but human in the loop rules reify this already problematic distinction by drawing attention away from the influence of automated systems and their management of process on final outcomes. It obscures much of what constitutes the actual decision in practice.

As Michael Veale and Reubin Binns astutely point out, decisions in automated contexts are stratified through various stages with different levels of human involvement and intervention in different models such as ‘triaging’, ‘decision-support’, ‘summarisation’, ‘flagging’, and ‘risk scoring’. They describe how these realities problematise the location of any decision within a technical system in ways that undermine the logic of Article 22.Footnote ⁴⁵ Jen Raso similarly demonstrates how automated decision-support systems displace decision-making authorityFootnote ⁴⁶ into a matrix of human and non-human entities in ways that ‘quickly destabilizes concepts common to public law scholarship’.Footnote ⁴⁷ This includes drop-down menus, comments in case management software, and algorithmically generated client letters. Rather than unpack the influence of this type of process automation, the human in the loop regulatory paradigm elides its influence on decision outcomes. It obscures the legal significance of automated processes that select individuals for evaluation, marshal data for analysis, and impose obligations on decision subjects to prove or disprove entitlements or obligations.Footnote ⁴⁸ By holding humans and machines apart, the human in the loop supports the problematic distinction between decision and decision-making that has become increasingly untenable.

4.4 Individual–Institutional: How Human in the Loop’s Responsibilisation of the Individual Obscures Institutional and Organisational Politics

Laws requiring a human in the loop draw from paradigmatic models in human–computer interaction of humans interacting with computer consoles and interfaces. This is an awkward construct, however, considering the complexity of automated systems’ contribution to decision outcomes in legal and administrative contexts. Decisions are made by individuals, informed and contoured by automated systems, all situated within complex organisations. Requiring human oversight to salvage the legality challenged by automation obscures how automation is itself an expression of complex institutional prerogatives, policies, and negotiations intended to affect different people in different ways. Ensuring the presence of a responsibilised decision-maker focuses on adequate governance for individual decision outcomes, but directs regulatory attention away from the influence of institutional policies and their responsibility for outcomes. The Services Australia Online Compliance Intervention (Robodebt) debacle offers a clear example of how a legal focus on human in the loop concealed how problematic political imperatives came to be embedded in technical processes.

Robodebt used data matching of self-reported income against tax records to flag prima facie possibilities of welfare overpayment. The system thus automated the process of debt verification previously performed by a human decision-maker who had to satisfy themselves as to the existence of a debt by looking at payslips and speaking directly with employers. Instead, discrepancies from data matching triggered letters to be automatically sent to welfare recipients, obliging them to update their payment information, with failure to do so resulting in a legal debt. Much of the analysis around Robodebt has focused on the unfairness and obvious inaccuracy of an automated process that used income averaging without a human reviewing or checking the consequent presumption of a welfare debt. These human in the loop arguments focus on how removing a human from the decision-making process diminished the quality of outcomes.

This framing also anchored challenges to the system’s legality. Data matching to flag potential debts had been used since the early 1990s. But in 2017, the Australian Administrative Appeals Tribunal determined that a welfare recipient’s failure to provide information disputing the finding of a debt identified by data matching did not absolve the department of its obligation to pursue third parties for further information and satisfy itself of the debt’s legitimacy.Footnote ⁴⁹ The results of data matching could not constitute a decision of consequence independently. It only raised doubt about the accuracy of past payments warranting the exercise of subsequent powers of inquiry. Despite this finding, the Australian Department of Social Services continued to deploy a larger-scale debt recovery programme, using the same methodology to automate the process of raising debts against welfare recipients. Litigation against the Robodebt system then confirmed that income averaging was not sufficient to raise a debt,Footnote ⁵⁰ resulting in a massive class action in negligence and unjust enrichment and consequent settlement agreement in 2021.Footnote ⁵¹

Robodebt was one of a number of welfare recovery automation systems deployed around the world,Footnote ⁵² however the Robodebt litigation and subsequent Royal Commission spotlighted the institutional dynamics behind of this type of automation. Beyond simply automating the finding of a debt, Robodebt is better understood as a programme for shifting the burden of demonstrating the existence or non-existence of a social security debt from the decision-maker to the decision subject.Footnote ⁵³ Rearranging the labour associated with debt verification away from the government agency enabled a debt recovery programme that could operate at massive scale. This was not a problem of decision automation – the data matching process was simply a solution to the organisational problem of resourcing the debt recovery processes. There were, in fact, humans still threaded through the decision process, but their job had changed from verifying the debt to managing client interactions – a much lower cost form of human oversight. Achieving that labour outcome meant decision subjects were unwittingly looped into the process, burdened with the onus of disproving their wrongdoing with extreme emotional consequences.Footnote ⁵⁴

In approving the class action settlement, Justice Murphy noted that ‘the proceeding has exposed a shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration’,Footnote ⁵⁵ and that ‘it should have been plain that in such circumstances the automated RoboDebt system may indicate an overpayment of social security benefits when that was not in fact the case’. But there was no admission of liability by the government, and Justice Murphy found it unlikely that plaintiffs could prove that government officials actually knew (despite nearly thirty Tribunal decisions rejecting income averaging as a basis for asserting a social security debt)Footnote ⁵⁶ that the system raised illegitimate debts. Despite acknowledging that actual knowledge of illegality in this situation would be a paradigm case for aggravated and exemplary damages, Justice Murphy held that actual knowledge that the automated system raised debts that were not actually owed was unlikely to be demonstrable.Footnote ⁵⁷

The judgment thus awarded financial damages premised on the illegality of removing the human from the decision-making process – but not for the broader institutional policy of reversing the onus of debt verification to scale a punitive programme. The existence or non-existence of a human decision-maker is thus a red-herring. More important was how automating away the initial decision-maker’s obligation to prove overpayment facilitated a debt recovery exercise of a scale otherwise impossible. Focusing on the degraded accuracy from data matching allowed the debacle to be narrated as a technical matter rather than a political one. According to this narrative, the department heads and decision-makers associated with implementing the system were deresponsibilised by the automation. Their political choices were dissimulated into critique of the inaccuracy of an automation process. In a strange mirror of the Pintarich outcome, the Department sought to recover money on the basis of decisions made without the application of a human mental process, but then disavowed itself of responsibility for the outcomes of that decision on the same basis – that nobody actually knew about the inaccuracy and illegality of the system. Interrogating those institutional realities has required a Royal Commission, but ought to be the content of quotidian regulation of automated systems used in the exercise of public (as well as private) power.

In other jurisdictions, there have been cases investigating the mechanics of regulatory agency function throughout the decision process (rather than focusing predominantly on decision outputs). The US Michigan Integrated Data Automated System (MiDAS) litigations, for example, presented the possibility of interrogating responsibility for such systems on a broader institutional register. That litigation concerned a similar, and perhaps more pernicious, social security fraud detection and recovery system used in the state of Michigan from 2013.Footnote ⁵⁸ The numerous opinions and judgments demonstrated willingness by the courts to interrogate structural distributions of accountability and responsibility associated with the organisational processes of decision automation, with little ultimate success. In the end, compliance with human in the loop requirements shielded system designers from liability for the conception and deployment of an abusive programme.

The MiDAS rendered automated determinations of fraudulent conduct (i.e. misreporting income) with a data matching process that used an ‘income spreading’ formula. When an individual claimant was flagged for possible fraud, they were notified but not provided further information about the basis for the agency’s suspicion. Instead, they were provided a cryptic questionnaire about benefits, with a positive response to almost any question triggering a fraud determination. If claimants failed to respond within thirty days, a fraud determination followed without further human assessment. The claimant’s rights to unemployment benefits were then terminated, monetary penalties were applied for fraudulent misrepresentation, and processes commenced (wage garnishing and interception of tax returns) to recover amounts of up to four times the benefits received. Even the letters demanding repayment did not inform recipients of the factual basis for the fraud determinations. A human only participated in the decision process if a recipient commenced an appeal against the determination, few of which were overturned.

The programme was, like Robodebt, highly inaccurate. The Michigan Auditor General found that of 22,000 determinations reviewed, more than 90 per cent of them did not involve fraud. That finding, alongside a legal determination that this type of auto-adjudication was not permissible, triggered reforms in 2015 requiring a greater amount of human involvement in the fraud determination process. In other words, the system was adjusted to conform with human in the loop requirements giving individuals more opportunity to rebut allegations prior to a final fraud determination. But these opportunities remained highly constrained, and though it reduced the false positive rate considerably, it did not shift the broader punitive and unfair nature of the system.

Initial findings by the court against Michigan agency employees were promising. They directly considered the responsibility of specific government officials for particular actions that deprived individuals of constitutional due process rights. The court allowed actions to be brought against officials for their roles in the MiDAS system’s violation of individuals’ rights to procedural due process by terminating their eligibility for unemployment benefits and seizing their tax refunds without any meaningful (pre-deprivation) process.Footnote ⁵⁹

Those agency officials were also joined in the litigation by executives of the private firms that consulted, specified, and provided the software that deprived social security recipients of their constitutional due process rights. Plaintiffs claimed those executives exercised control over the relevant officials; caused them to commit the relevant acts; encouraged the use of ‘income spreading’ as the basis of fraud determinations; encouraged the Agency to change its culture with respect to fraud; suggested problematic changes to the fact-finding questionnaire sent to recipients; built a system that automatically issued determinations based on existing information when the recipient had not yet responded to the questionnaire; and encouraged replacing a system that provided adequate human discretion with one that included far less human oversight. Including individuals from the software developers in the litigation made tracing accountability and authority throughout the procurement and development pipeline a possibility.

At first instance, the agency officials argued that they could not be held responsible for actions performed by an automated system. But this argument was quickly characterised by the court as agency defendants attempting to ‘hide behind MiDAS’. The court noted ‘this argument conveniently ignores the fact that the Individual Agency Defendants implemented and oversaw MiDAS, and prescribed its operation. MiDAS did not create itself. And it did not enforce the false fraud determinations that it automatically rendered – the Individual Agency Defendants did.’Footnote ⁶⁰ However, as the litigation proceeded, agency officials were eventually released from liability. After the 2015 update, the court accepted that MiDAS no longer automatically made fraud determinations, but created fraud flags, meaning there was no deprivation of rights without proper notice, process, or opportunities to appeal.Footnote ⁶¹ In other words, the human in the loop rearrangement assuaged agency liability for the development and deployment of a programme that, like Robodebt, shifted the onus of proof onto claimants in a highly abusive way. Much like Justice Murphy’s finding in the Robodebt class action, the court maintained that this rearrangement did not penetrate the officials’ qualified immunity – the shield to liability for government officials in the performance of their discretionary functions.

A dissenting judgment argued that qualified immunity was never intended for defendants such as these who had masterminded the entire MiDAS programme, including its auto-adjudication functions. That dissent suggested, looking at the specifics of the income averaging process that identified possible fraud and the decision trees associated with the questionnaires provided to claimants, that whatever opportunity to appeal claimants had existed only after their constitutional rights to due process had already been deprived. However, the majority maintained that claimants had the right to challenge the finding of fraud before deprivation occurred, irrespective of the technology used or how abusively it performed the process. There was also no finding of liability against the private firms that provided the technology. They collectively then settled the case,Footnote ⁶² paying out $180,000 to claimants on a $47,000,000 procurement. Michigan now uses a new system developed by Deloitte for identifying welfare fraud.

4.5 Configuring the Human

The human in the loop expresses an unresolvable tension in the control of complex systems. Automation serves to remedy the emotional, unreliable, and feeble human, at the same time as the human serves to corrects clunky, inflexible, uncreative and heartless automation. Similar tensions pervade principles of legality in decision-making vis a vis competing priorities inherent to rule of law thinking, including: universal rules versus individual justice; scale versus inclusion; equality versus discretion; legitimacy versus quality; substance versus procedure, and so on. It is sometimes glossed over that rule of law thinking emerged in opposition to administrative discretion,Footnote ⁶³ despite advocates for the human in the loop frequently associating discretion with individual justice.Footnote ⁶⁴ The point is that the current proclivity for human in the loop ought to be understood as less about preserving unassailable rule of law principles concerning the moral qualities of decision-making officials, but rather as reflecting a political posture, described by Kristen Rundle, suggesting ‘the pendulum has swung too far such that the human being and distinctly human cognitive and moral capacities need to be pushed to the foreground’.Footnote ⁶⁵ Viewed this way, automation does not break legality, it merely rehearses and amplifies the rule of law’s unresolved tensions, which are then papered over by symbolic human oversight rules. Human in the loop becomes a shortcut that institutes the human decision-maker (or supervisor) as an accountable and responsible actor, and situates legal authority in the human inheritance of divine power.Footnote ⁶⁶ Whereas one strand of rule of law thinking demands these qualities to legitimise the exercise of political authority, as noted by Rundle, other strands of rule of law thinking are more concerned with the structure of the relationships between official and decision subject, and the possible exercise of arbitrary power more than the identity of the decision-maker themselves.Footnote ⁶⁷

The political posture favouring human in the loop laws is understandable, because the human in the loop helps smooth over the destabilising effects of technologically assisted decisions. Beyond being a conceptual bottleneck to the appropriate regulation of automated systems,Footnote ⁶⁸ this chapter suggests that underspecifying the contours and features of the human decision-maker does calculated political work. Although identifying non-explicit intentions or institutional logics behind regulatory strategies is always fraught, it is not controversial to conclude that the human in the loop, at least, stabilises a set of governing relations that avoid troubling the status quo of risk profiling by government agencies. A more sceptical reading might suggest the human in the loop enables political operatives to make a double move: on the one hand, pushing responsibility for making political choices into the relation between an authorised decision-maker and the software ‘tool’ they autonomously oversee. Here, accountability becomes a question of the existence or non-existence of the human at the local level – who is the individual responsible for this decision that automation failed to adequately perform? On the other hand, by not prescribing any actual responsibility to the human within the technical apparatus it monitors, it becomes possible to claim the fallible human was bamboozled by automation and responsible for nothing, dissimulating responsibility into the technical system itself and creating ‘orphan decisions’ for which no one is accountable.Footnote ⁶⁹ Either way, by enabling differentiated articulations of human capacities and differentiated operationalisation of associated normativities, the human in the loop strategically elides questions around the different ways humans are threaded into decision systems and attendant attributions of responsibility. It separates the allocation of accountability from the complex distributions of decisional authority that come with the deployment of automated systems, and insists on the existence of human sites of authority and authorship that can be identified and sanctioned. In so doing, it distracts from the need to face the consequences for higher-level political action (decisions about distributions, labour, and intention) or understanding the tools in operation.

A human in the loop needs is only one strategy amongst an emerging mosaic of regulatory tools for managing AI and automation. New instruments like the EU Artificial Intelligence Act include new obligations for the design, development, operation, and monitoring of automated systems.Footnote ⁷⁰ But alongside requirements like transparency and auditing, human oversight remains a fundamental prescription for regulating high-risk systems under those regimes.Footnote ⁷¹ And while emerging requirements will improve the quality and safety of automated systems, they will not alter the human in the loop’s functioning as a versatile legal technology that, on the one hand, enables emerging political and economic formations to coalesce around automation; and on the other, enables complex political action to be expressed through automated decision systems, whose technical rather than political content becomes the focus of regulatory scrutiny.

Rather than rely on a human in the loop, the goal should be understanding the complex redistributions of responsibility that trace the distribution of decision-making elements into automated systems and the authority to specify them, including the influence of companies that consult, design, and build relevant software. These are broader questions of political authority, rather than local concerns about the agentic quality of any particular decision-maker.

5.1 Introduction

Public administrators around the world are enthusiastically seeking to ‘harness the power of data’, reflected in the growing use of algorithmic decision-making (ADM) systems within governmental organisations, eager to replicate their successful application in commercial contexts.Footnote ¹ Many of these networked digital applications include those developed through the application of machine learning (ML) techniques to score, sort and rank individuals to inform how public services and functions are administered, including those which purport to evaluate an individual’s predicted ‘risk’.Footnote ² A combination of technological advances and socio-economic factors have fuelled digital transformation projects within public administration (in a tendency I have identified elsewhere as the ‘New Public Analytics’).Footnote ³ This chapter focuses on how a particular set of promises, assumptions, beliefs and rationalities (collectively referred to as ‘logics’) prevalent in the commercial sphere are being translated and applied to public sector environments in ways that have fuelled the take-up of these ML-based citizen scoring tools in government.Footnote ⁴ Although the use of so-called big data analytics (now more commonly referred to as ‘Artificial Intelligence’ or AI) in government for the purposes of categorisation, assessment and prediction at both individual and population level are relatively recent,Footnote ⁵ the practice of individual risk assessment through actuarial methods has been employed by public authorities for several decades.Footnote ⁶ By the early 1990s, important state functions conventionally undertaken in highly industrialised states, particularly decisions about whether to release inmates serving jail sentences or in pre-trial custody, were already being discussed and characterised in terms of ‘risk management’ at the same time as the English language publication of Ulrich Beck’s ‘risk society’ thesis in 1992.Footnote ⁷

The claimed purpose of these citizen scoring tools, both then and now, is to ‘optimise’ the allocation of public sector resources. Yet the current wave of ML-based citizen scoring tools and systems are touted as superior to the actuarial techniques that preceded them, claiming to offering better, more accurate and ‘objective’ or ‘evidence-based’ predictions, based on the success of ML applications in commerce.Footnote ⁸ My analysis is structured around three celebrated promises invoked to justify the enthusiastic take-up of ML-driven prediction models, notably: (a) more accurate prediction, (b) the generation of ‘actionable insight’, and (c) the value (and superiority) of ‘early intervention’. In so doing, I address three questions.

First, whether ML-based scoring, which has proved successful in the private sector, can be expected to generate ‘more accurate’ predictions to inform decisions made by public authorities about individuals, in comparison with actuarial methods that have long been used by public authorities to assess an individual’s ‘risk’. Second, in what way has the promise of ‘actionable insight’ that fuelled adoption of AI in the commercial sector been invoked in public sector contexts? Third, to what extent have assumptions about the value of ‘early intervention’ to prevent or avoid a future outcome associated with a specific individual contributed to the enthusiastic embrace of citizen scoring, and how can those claims be reconciled with foundational constitutional principles, particularly basic respect for the freedom, dignity and agency of the individual?

I will argue that these three central promises, which have fuelled ML’s enthusiastic adoption by commercial firms, are being unthinkingly adopted by public administrators (at the national, European and global level), failing to recognise that the role of the state is fundamentally different from, and antithetical to, the role of private actors (including for-profit commercial organisations) within liberal democratic states. While the latter are free to pursue private gain provided that they abide by law, public authorities are duty-bound to serve the interests of the public in accordance with, and constrained by their legislative mandate and subject to the safeguards and constraints imposed by constitutional principles, including respect for human rights and doctrines of administrative law. This unthinking and inappropriate translation of ML’s promises from commercial applications into public decision-making tools and systems results in a proliferation of opportunities for injustice and the abuse of public authority in ways that interfere with the rights of individuals while restricting opportunities for affected individuals to understand and contest these decisions. Yet because it is the most vulnerable members of society, for example those convicted of criminal offences, who are parsed and made subject to ‘early interventions’ by public authorities, the use of ML scoring has not attracted significant public resistance. The unjust administration of rights-intrusive interventions and, in turn, the allocation of public benefits and burdens, are being silently yet effectively redistributed through the adoption of ML tools without adequate public oversight, democratic debate or public consent. I demonstrate how this political foreclosure happens, not so much within the space of mathematical measurement and computational calculation, but in the unthinking and misplaced translation of ML’s promises from commercial contexts into the realm of public administration.

5.2 The Promise of More Accurate Predictions

The promise of more accurate prediction, compared with conventional statistical tools, continues to fuel the enthusiastic embrace of ML models across many and varied domains. For commercial organisations whose success depends on attracting and retaining individual customers online as repeat customers, or digital platforms for which greater user ‘engagement’ is directly proportional to the advertising revenue generated,Footnote ⁹ the promise of accurate predictions concerning an individual’s tastes, preferences and behaviours has been seductive.Footnote ¹⁰ Like their retail marketing counterparts, public officials are also keen to accurately predict the future behaviour of individuals, yet their interests and motives are rather different. Since the late 1920s, public officials have sought to predict the ‘riskiness’ of individuals, particularly in the US criminal justice system. Although initially risk was understood as ‘risk of absconding’, it now tends to be understood primarily in terms of ‘predicted dangerousness’.Footnote ¹¹ While so-called ‘first generation’ risk assessment relied upon unstructured clinical assessment, these approaches were succeeded by ‘second generation’ prediction tools during the 1990s when actuarial tools commonly used in the insurance industry were taken up by US criminal justice officials, particularly in seeking to manage the crisis in prison overcrowding resulting from President Nixon’s ‘tough on crime’ policies.Footnote ¹² While these prediction tools relied on ‘static’ risk factors (i.e., factors that cannot be changed) drawn from empirical research, ‘third generation’ risk assessment tools later entered into use, incorporating ‘dynamic’ risk factors (i.e., factors that can be changed either slowly or relatively quickly over time, such as marital status, employment and education level) which advocates argued would increase their efficiency by allowing public officials to target and monitor their risk-reduction efforts.Footnote ¹³ The most recent ‘fourth generation’ risk assessment tools build on third generation tools, but also include case management plans, seeking to match each individual’s unique profile to an evidence-based ‘treatment plan’ by combining risk management efforts with individual case planning.Footnote ¹⁴

Scholarly interest in individual risk prediction tools by public authorities are enjoying a revival with the development of tools based on ML prediction models.Footnote ¹⁵ The capacity to accurately predict a person’s future behaviour which features in the commercial rhetoric surrounding ML’s touted economic value is increasingly evident in public sector contexts, particularly for public administrators already engaging in individual risk assessment practices to inform decisions. Although the promise of accurate prediction is understandably beguiling, public administrators have largely accepted at face value the underlying assumptions and conditions upon which this promise is based. In the following discussion, I argue that this promise, and the conditions upon which its fulfilment depends, do not withstand critical scrutiny.

For public authorities, ML’s promise of more accurate predictions about an individual’s future behaviour holds out the possibility of overcoming the ‘individualised justice objection’, one of the most sustained and significant criticisms of actuarial tools that rely on conventional statistical techniques.Footnote ¹⁶ In particular, the use of actuarial predictions which underpin the practice of ‘risk profiling’ to inform the treatment of persons, particularly in criminal justice contexts, has been persistently criticised on the basis that it fails to attend to the relevant person’s specific characteristics, context and circumstances, thereby departing from the principle of individualised justice which posits that each person has a basic right to be treated as a moral agent with dignity and respect.Footnote ¹⁷ Instead, actuarial assessments treat individuals on the basis of characteristics shared with other members of a group.Footnote ¹⁸ The use of ML techniques applied to very large datasets, however, is often portrayed as overcoming this limitation. As Cukier and Mayer-Schönberger, authors of the popular science book that first brought ‘big data’ into public consciousness, claim:

This statement rests on a belief that ‘more granular predictions for each individual’ overcomes the individualised justice objection levelled at individual risk assessments produced through conventional statistical techniques while reflecting prevailing commercial rhetoric about the promise of ‘more accurate’ predictions of individual behaviour. Yet this belief is mistaken. Although ‘accuracy’ in predictions concerning an individual’s future behaviour are possible through the use of some types of ML models under certain conditions and depending upon the specific behaviours that it purports to predict, the kinds of ML individual risk-prediction tools that public authorities have begun to embrace continue to be based on the projection onto that individual of the profile of a group into which a given individual is allocated. In other words, ML techniques, as they have been deployed in the public sector to ‘score’ a person’s ‘risk’, continue to rely on the same logic of group projection that underpins actuarial statistical methods they are often claimed to supersede. As Imrey and Dawid explain in relation to ‘recidivism risk’ predictions, unless one observes data on recidivism specifically from a new offender (D), then statistical inference about D’s individual risk must be based on assumptions linking D to a population or process from which recidivist outcome data are available to serve as a source of inference that can inform predictions about D’s individual recidivism risk.Footnote ²⁰ They continue:

Furthermore, when confronted with claims that ML prediction models offer greater prediction accuracy, we must attend carefully to the phenomenon purportedly predicted and over what time horizon. This is particularly important when our concern is with predicting a person’s behaviour, given the enormous variation of real-world human behaviours of interest. The more specific, proximate and directly observable through automated means the candidate behaviour is, the greater the level of attainable prediction accuracy. For example, those engaging in online advertising will typically seek to maximise the number of user-clicks on the web-content links automatically served up to individual users. Hence the kind of human behaviour that these advertisers seek to predict concerns whether a person will ‘click’ on a website link. Impressively accurate predictions are possible now that users’ click-stream data is routinely automatically and continuously collected and parsed via ML models applied across a population of users alongside real-time, automated feedback on whether each prediction about a user’s click through behaviours was successful or not.Footnote ²²

In contrast, consider the challenge faced by child protection authorities in predicting whether a particular child is ‘at risk’ of abuse or gross neglect seeking to use algorithmic prediction tools to allocate preventive family support services or to make child protection screening decisions. The interpretive vagueness that surrounds the notion of child abuse, combined with the dearth of reliable data on actual incidents of child abuse and neglect, means child protection prediction algorithms are more often trained on so-called ‘proxy’ data which may be only weakly related (if at all) to abuse incidents across a population.Footnote ²³ This generates what Hildebrandt refers to as the ‘reality gap’ due to an ML function failing to correctly define the real-life events that the data supposedly represent. She attributes this gap to the ‘low hanging fruit’ problem, referring to circumstances in which the availability of big data may appear to enable reliable inferences, whereas in fact the data is not relevant or incomplete.Footnote ²⁴ These errors can also be understood in more conventional statistical parlance, understood in terms of the failure to adopt a representative sampling frame for the purposes of statistical assessment. For example, Kendall notes that child abuse prediction algorithms are often trained on administrative data on reported abuse. But much child abuse and neglect are never reported, and many families that are reported may be subject to both surveillance and personal biases meaning the sample data generated by child protection systems is not representative of the prevalence of child abuse across a population. This, in turn, profoundly undermines the ability of those predictive tools to accurately identify those at risk of abuse, while lending weight to claims that racial and class-based discrimination is built into them.Footnote ²⁵

The ‘reality gap’ is compounded when relevant factors and outcomes are omitted from the training data. In particular, data upon which individual risk prediction algorithms are trained cannot capture ‘counterfactual’ outcomes. For example, if a child was taken into care, then the intervention will preclude the production of data on whether that child would have been neglected and abused had that child had not been taken into care. Conversely, if an abused child is not taken into care and that abuse remains unreported, then the reported data will indicate that the prediction recommending against taking the child into care was correct even though it was not.Footnote ²⁶ Prediction accuracy is eroded further due to the omission of other types of salient information which is not included in the training data, such as changes in the level of informal social support available, which is a key factor in child abuse development.Footnote ²⁷

These difficulties remind us that the ‘quality’ of an ML-based prediction tool is not only a product of the quality, size and richness of the data upon which it is trained, but also of the ‘interpretive work’ associated with the collection and processing of data in the service of useful knowledge production.Footnote ²⁸ To be regarded as data, a data point must relate accurately, directly, reliably and consistently to some phenomena that it purports to represent. If data collected for administrative purposes is the source of ‘ground truth’ data upon which an ML individual risk prediction model is trained and developed, great care is required in making assumptions and assertions about what the resulting predictions are taken to represent, especially when those predictions inform rights-critical decisions about how individuals are treated, particularly by public authorities.Footnote ²⁹ Yet the celebratory rhetoric associated with AI has resulted in a tendency to downplay or largely ignore these inconvenient realities.

Furthermore, care is needed when making generalised claims about all ML techniques, given their wide variety and the different ways in that prediction models are configured within socio-technical systems. As König and Krafft observe, it is striking that algorithmic risk assessments are framed in public debates as ‘Artificial Intelligence’ in a manner that has been partly decoupled from actual practice.Footnote ³⁰ Even well-informed scholarly accounts may refer to ‘AI’ when discussing individual risk assessment tools of the kind that purport to predict an individual’s ‘recidivism risk’, failing to recognise that they closely resemble actuarial prediction models produced by conventional statistical approaches and bear little resemblance to the automated dataveillance systems now routinely employed by digital platforms to predict, generate and distribute online content that fuelled the meteoric rise of the Digital Tech Titans who now dominate the platform economy.Footnote ³¹

Instead, the ML prediction models used by public authorities to generate individual risk predictions are typically static models trained on historic administrative data and do not provide automated feedback about whether a specific prediction was successful or not.Footnote ³² Yet there remains a problematic (albeit understandable) tendency, even amongst well-regarded scholars, to make generalised claims about ML techniques, failing to recognise that capabilities and criticisms that apply to some kinds of ML techniques and prediction models cannot be appropriately generalised and applied to others.Footnote ³³ Online prediction tools seek to predict very specific, narrowly defined, directly observable individual behaviour that is reliably and consistently monitored and measured (as indicated by user ‘clicks’) that are dynamically, automatically and continuously updated from feedback on whether the automated predictions were successful or not. For these algorithmic systems, it may be legitimate to assume that past data patterns are a reliable predictor of future behaviour because we have both high-quality longitudinal data (i.e., granular data over time in relation to a specific person) and lateral data (i.e., it extends widely across a population) to enable individual predictions about a very specific, narrowly defined and proximate future behaviour (whether a user will ‘click’ on a link).Footnote ³⁴ In contrast, the individual risk prediction models used, for example, to assess ‘recidivism risk’ by US criminal justice officials, or by child protection agencies, bear greater similarities to conventional actuarial prediction models and for which their real-world prediction accuracy is unknown.Footnote ³⁵

These concerns underscore the need for caution when interpreting the meaning, reliability, quality and significance of ML outputs, particularly claims of ‘prediction accuracy’. For data scientists, ‘accurate prediction’ is conventionally defined in narrow, technical terms based on the model’s performance in accurately predicting outcomes from a portion of data that formed part of the original dataset from which the training data was taken and specifically set aside for the purposes of assessing prediction accuracy (the ‘test set’). Accordingly, the resulting accuracy metrics are not necessarily indicative of accurate prediction in real-world settings: they pertain exclusively to the original dataset used to train, validate and evaluate the model. In this respect, Tobias Blanke rightly argues that ‘high accuracy figures for decision-making algorithms should never be enough to reassure us that these algorithms are correct and make wise pre-emptive decisions’.Footnote ³⁶ Moreover, an analysis by König and Krafft of individual risk assessment tools used by US states in pre-trial settings reveals that the predictor outcomes themselves vary considerably across tools. They found that although these tools always claim to predict ‘failure to appear’, some claim to predict ‘new criminal activity’ while others offer predictions of ‘being rearrested’, while still others blend substantively different concepts offering a composite measure of both ‘failure to appear’ and ‘new crime/arrest’ to produce a single numerical score.Footnote ³⁷ While these scores might assist in the ranking of candidates relative to each other, whether the score itself bears any consistent and reliable relation to social reality is seriously questionable.

It may be tempting to respond that these problems are fundamentally reducible to the lack of high-quality data pertaining to individuals of the kind that Imrey and Dawid claim preclude the prediction of what they call an individual’s ‘intrinsic risk’. Their account might imply that if relevant recidivism data about that individual were available, the individual’s actual ‘intrinsic’ risk of recidivism could be statistically inferred. Yet this belief is also mistaken, based on a fantasy which computer scientist and philosopher Reuben Binns refers to as ‘Laplace’s omniscient demon’.Footnote ³⁸ Binns points out that even patterns in mouse movements, social graphs, recent purchases and so forth are often framed and taken as equivalent to an omniscient demon’s knowledge of the position and forces of nature. He remarks that:

In other words, as Mireille Hildebrandt has emphasised, the human self is ‘fundamentally incomputable’. No algorithm can be trained on future data: the temporality that grounds us in our humanity and the natality that it entails confronts ML with the fundamental uncertainty of the real world.Footnote ⁴⁰

While achieving perfect accuracy in predicting human behaviour will forever remain an impossible dream, it is alive and well in the minds of those peddling data-driven ‘solutions’ to cash-strapped public authorities. Although the latest risk-prediction models are, like their earlier counterparts, aimed at facilitating more efficient public sector resource allocation, their novelty lies largely in the normalisation of this objective combined with a flawed belief that ML algorithms and the data deluge accompanying the digital revolution enables the routine and accurate prediction of individual behaviour of all kinds, with scant critical reflection.Footnote ⁴¹ Both then and now, the harnessing of statistical tools in the service of more efficient public resource allocation has often been a pragmatic response to crises such as managing prison overcrowding, or easing the fiscal crisis facing contemporary public services.Footnote ⁴² It is precisely this ‘more for less’ challenge that elevates the importance and attraction of the promise of ‘actionable insight’ for public authorities, to which I now turn.

5.3 The Promise of ‘Actionable Insight’

Directly tethered to the promise of more accurate predictions is the promise of ‘actionable insight’, based on a belief that if an organisation can predict the likely behaviour of a user, they can take pre-emptive action to shape that behaviour to serve organisational interests. For digital platforms, predictive analytic systems that track the online behaviour of individuals across a population are now commonly used to sift, sort and score individuals, enabling the automated provision of targeted content.Footnote ⁴³ Such consumer-facing digital services are essentially a ‘barter’ model in which users agree to continuous monitoring of their online behaviour and the collection of digital traces (including meta-data) thereby generated, in return for services.Footnote ⁴⁴ Shoshana Zuboff and others have emphasised how reliance on these post-industrial forms of capitalist production, which Zuboff dubs ‘surveillance capitalism’,Footnote ⁴⁵ requires the continuous and pervasive surveillance of individuals, channelling and controlling flows of personal information while converting them (via the use of AI technologies) into flows of profit, in ways that are obscured from users.Footnote ⁴⁶

These contemporary forms of digital capitalism have spawned a new modality of production which I have referred to elsewhere as ‘mass predictive personalisation’. Like late nineteenth-century industrial modes of production, mass predictive personalisation has the capacity to operate at scale (and thus on a ‘mass’ basis), but differs from industrial production in significant ways, three of which are directly rooted in the promise of ‘actionable insight’.Footnote ⁴⁷ First, rather than generating identical units of production, services are personalised to each user’s predicted tastes, interests, preferences, lifestyle and behaviours. Second, service provision operates, by default, on a predictive basis due to the application of algorithmic profiling to infer and predict each user’s service preferences with the aim of anticipating user needs and ‘pushing’ appropriate personalised services. Third, services are continually and automatically reconfigured based on feedback gleaned from monitoring the recipient’s responses without requiring active and intentional feedback concerning the user’s interest in the service provided.

5.4 Conclusion

In this chapter, I have shown how logics that fuelled the successful adoption of ML techniques within commerce are being inappropriately and unthinkingly translated into public sector contexts in pursuit of operational efficiencies in the form of citizen scoring systems. Not only are these systems being embraced by public authorities without any clear, demonstrable evidence that they deliver on their promises, let alone enhance public value, but they are producing policy change by stealth while generating injustice at scale in ways that are difficult to see and often practically impossible to contest. The deeply problematic manner in which the promises of more accurate prediction, of actionable insight, and of early intervention are being relied upon to support the take-up of citizen scoring systems by public administrators is ultimately rooted in a failure to recognise that the role and authority of the state is entirely different from, and antithetical to, the role and freedom enjoyed by private persons (including for-profit organisations) within liberal democratic states operating within a capitalist economy. While the latter are free to pursue private gain provided that they abide by law, public authorities are duty-bound to serve the interests of the public in accordance with, and constrained by, their legislative mandate and subject to the safeguards and constraints imposed by constitutional principles, including respect for human rights and doctrines of administrative law.

The unreflective translation of ML’s promise from commercial contexts into public administration has proceeded on the basis of assumptions, beliefs and forms of reasoning that are portrayed as straightforward and self-evident. They are not. By framing the difficult and often fraught judgements that must be made in the course of carrying out public functions, such as child protection and criminal sentencing, as dreary matters of procedure and process that can be improved by embracing ML systems which offer more precise, accurate targeting and evaluation of individual risk objects, the embrace of citizen scoring systems conceals contestable political viewpoints. Although some ML prediction models may appear to offer dazzling capabilities, reflected in the fanfare accompanying the roll-out of large language models such as ChatGPT, research demonstrates that ML predictions about the life outcomes of individual persons are extremely poor.Footnote ⁸⁰ Accordingly, the unilateral imposition by public authorities of ‘early interventions’ based on citizen scoring systems informed by ML prediction models, particularly when they are adverse and/or unwanted by those singled out for attention, are not only damaging, dehumanising and rights-violating but anti-democratic in a more general sense. They can undermine the rule of law, while excluding those scored and thence adversely affected from opportunities for meaningful participation or contestation, while reinforcing and deepening existing inequalities between groups that have been historically marginalised and those which have not.

In other words, the unthinking and inappropriate reliance on the promises of ML predictions in the form of citizen scoring systems to inform ‘early intervention’ decisions by public authorities offers a concrete site in which ML systems foreclose politics by side-stepping debate and deliberation about how collectively the state should act (on behalf of the community). This reliance also conceals the political transformation of relations between state and citizen in ways that are adverse to the latter, and to which the general public has not consented. It is striking that early interventions justified in the name of the precautionary principle which accompanied the transatlantic genetically modified organism (GMO) debates attracted intense political controversy given the contestable and speculative normative choices they entail, while those administered against individuals on the basis of ML-based citizen scoring systems attract so little public attention or political concern. These systems are redistributing power, benefits and burdens in ways that are hidden in plain sight, parading under the mundane rubric of business process optimisation, and underpinned by misguided promises of more accurate and precise risk assessment in the quest to enhance the efficiency of public sector operations. These systems are being used to sort, score and unilaterally intervene in the lives of the most poor and vulnerable in society, who lack political clout and resources to contest this sorting, let alone mobilise support to resist and oppose it, flouting constitutional principle while side-stepping the requirement to justify interventions or subject them to public discussion and debate.

6.1 Introduction

In algorithmic culture, the problem of race is a problem of proxies. Though racism still continues in many familiar forms – through structural exclusion and oppression, through systems and policies that serve the neocolonial and/or white supremacist state, through words, actions and attitudes that demean and dehumanise – the tools and techniques that further these racist legacies and turn bodies into racialised subjects now operate in very different registers.Footnote ¹ In part, this is an effect of what Frank Pasquale calls the ‘black box society’: a normalised culture of opacity in which large corporations shield their interests behind ‘technical complexity, real secrecy, and trade secret laws’.Footnote ² This culture of opacity makes it near impossible to know if and how racial (and other) discriminatory practices are taking place, whether intentionally – through, for example, the targeting or exclusion of ads for housing and education based on users’ race;Footnote ³ or unintentionally – through, say, a differential pricing strategy that results in racialised and low-income groups consistently receiving higher prices for the same product.Footnote ⁴

For some, this culture of opacity is an issue to be solved through better techniques of accountability and transparency. The aim is to render black boxes more explainable and observable; to make private interests and private algorithms more legible to public scrutiny.Footnote ⁵ Yet as many critical data scholars have argued, opacity isn’t a straightforward condition that can be remedied through frameworks of ‘shedding light’ or ‘opening up’, but is in many cases an inherent aspect of the complex and fragmentary nature of algorithmic infrastructures themselves.Footnote ⁶ As political geographer Louise Amoore elegantly writes, ‘the condition of giving an account that is never transparent or clear sighted is already the ethicopolitical condition with which we must live’.Footnote ⁷ That is to say, opacity defines the always partial, always situated, always mediated practices of representation and perception, algorithmic or otherwise.Footnote ⁸ In this context, knowing racism (and more importantly, resisting racism) means contending with these partial and opaque infrastructures, not necessarily as problems to be ‘found and fixed’ but as part of the many processes that mediate racialised life.Footnote ⁹

Of interest for us here are the tensions that emerge when these opaque practices make use of ostensibly neutral data points to create clusters and associations that reconstitute categories of race in new and unusual ways. In previous work, we have claimed that systems driven by artificial intelligence (AI) are changing the nature of race as a category and racialisation as a process.Footnote ¹⁰ Alongside scholars such as Marion Fourcade and Kieran Healy,Footnote ¹¹ Oscar Gandy Jr,Footnote ¹² Wendy Chun,Footnote ¹³ Louise AmooreFootnote ¹⁴ and Marie Petersmann and Dimitri Van Den Meerssche,Footnote ¹⁵ we have been interested in how modes of social sorting are today mediated through new techniques of supervised and unsupervised machine learning (ML). Whether placing people within market driven ‘classification situations’,Footnote ¹⁶ assessing and reassessing them along the unstable lines of racialised ‘deep border’ politics,Footnote ¹⁷ or differentiating them using the segregationist logic embedded in network science principles like ‘homophily’,Footnote ¹⁸ AI and ML systems inaugurate new ways of seeing and sorting bodies that has implications for understanding what a category like race is, how it functions, and how it is felt.

We can think of this as a post–big-data politics of race: what we have elsewhere called ‘racial formations as data formations’.Footnote ¹⁹ Our focus has been to bring an attentiveness to the technicity of race’s production to the study of race and algorithmic culture. This means grasping the specific apparatuses and the specific material modes through which race is mediated and produced. In this chapter, we build on this previous work by focusing on a central though rarely analysed technique: the use of proxies. In data science, a proxy is a feature that is used ‘in place of a variable that cannot be measured’.Footnote ²⁰ Proxies act as substitutes that can be used to infer, and therefore to act upon, an unknown piece of data. A proxy is thus a technique for measuring that which is in excess of perception. It reaches across the gap between the social and the computational not to make sense of the world, but to make it actionable within regimes of recognition, prediction and classification. Taken individually, these substitutions might not amount to much. Cascaded together, scaled up, and applied in concrete situations using AI and ML techniques, though, this use of proxies constitutes a logic by which the world can be approximated or modelled.

In what follows, we draw on a range of material from disparate fields, including critical race studies, media studies, science and technology studies, critical data studies, and the technical and legal literature on large-scale data processing to argue that this proxy logic provides us with a framework for understanding the situation we find ourselves in. This is a situation in which tools ostensibly designed to ‘solve’ the injustices caused by human racial bias have instead resulted in their recapitulation, and at worst, in new forms of discrimination. In this paradoxical scenario, attempts to remove race as an actionable data point results only in its entrenchment in a novel form. While algorithmic infrastructures promise to do away with crude categories like race in favour of more precise and emergent forms of clustering, race’s differentiating logic nevertheless continues to shape experience.Footnote ²¹ We argue that proxy discrimination emerges as a consistent by-product of big data processing techniques, generating novel forms of ‘colourblind’ racism.

We then turn to the political implications of the proliferation of proxy logics. As scholars working with concepts like ‘algorithmic governance’Footnote ²² or ‘governance by data’Footnote ²³ have noted, the increasing delegation of decision-making to AI and other automated systems effectively shields activity that is fundamentally political from scrutiny.Footnote ²⁴ Here, the focus is often on the problem of opacity – what causes it, how to resolve it, and the risks it poses for democratic ways of life. But for us, it’s also illuminating to see algorithmic governance as a mode of governance that operates by proxy.Footnote ²⁵ The decisions we delegate to AI and ML systems are rendered opaque not only by the sheer complexity of their underlying infrastructures, the overwhelming volumes of data that they process, or their arrogation of decision-making responsibilities by machines, but, more fundamentally, because they rely on proxy logics.

These logics embed another kind of political transformation: they arguably change the paradigm in which contemporary racial politics operates. Proxy logics use implicit forms of feedback and complex statistical transformations on our data to sort us into contingent groupings – what are sometimes called ‘clusters’Footnote ²⁶ or ‘neighbourhoods’.Footnote ²⁷ That is, one need only to have expressed the wrong combination of preferences or taken the wrong series of actions or have originated from the wrong place to be situated within the wrong kind of algorithmic space. In these instances, race is enacted not through modes of identification grounded in the reading of phenotype or genotype, but by latent associations between features of data.

Contemporary racial politics is associated, fairly or not, with what’s often disparagingly called identity politics. On a personal level, each of us is committed to a political programme encompassed by this term. But we also want to argue that the abstract statistical vector spaces of AI and ML systems operate in an entirely incommensurable racial-political paradigm. To cluster or sort individuals into neighbourhoods, such systems don’t discern our ‘identity’. Rather, they draw inferences and approximations, creating clusters that group people together not on ‘common characteristics, or meeting specified criteria’ but instead on ‘a spatialised proximity or distance’.Footnote ²⁸ What becomes of race within this new paradigm? A paradigm which is defined not by a politics of identification and identity, but on a politics of approximation and proximity – in other words, when race is constituted by proxy?

6.2 On Proxies

A proxy is a substitute or a surrogate, a representative or a stand-in. The term proxy is used to describe a host of relations, from authorial power between different groups, actors and entities (e.g. delegates, legal intermediaries, members of a representative democracy), to forms of military and political power that bolster imperial statecraft (e.g. proxy wars and proxy forces), to the technical roles between different components in an infrastructural system (e.g. proxy servers, standardised measurements).Footnote ²⁹ In its most basic form, a proxy signifies the act of delegation. It allows something to be treated as if it were another thing.Footnote ³⁰ As media historian Dylan Mulvin has incisively argued, this seemingly simple process actually constitutes a key function of modern knowledge production. Identifying case studies and exemplars, working with samples, using testbeds and simulations: these methods all rely on proxy logics that create ‘accessible representation’ of otherwise chaotic or unknowable entities or events, which can then be acted upon as if they were the real thing.Footnote ³¹ ‘This is the promise of proxies’, writes Mulvin, ‘the promise of creating controllable renditions of an unpredictable and unknowable world’.Footnote ³²

It shouldn’t come as a surprise then that proxies feature prominently in AI and ML systems. These are systems that use big data-driven methods to calculate probabilities, model predictions, and ultimately, to tame uncertainty.Footnote ³³ Here, proxies make the act of algorithmic prediction possible. Based on statistical methods, AI and ML attempt to solve problems through a logic of optimisation, a kind of rationality, to use Dan McQuillan’s phrasing, that ‘asserts that condensing [a context] into a calculation provides a superior kind of solution’.Footnote ³⁴ Yet this rationality engenders a different kind of problem. As McQuillan pithily puts it, ‘[v]ery often we can’t directly measure the things that matter the most, like happiness or health’.Footnote ³⁵ To be able to render these kinds of phenomena amenable to calculation, ‘algorithms have to rely on proxies’.Footnote ³⁶

Take, for example, recommender systems. The aim of a recommender system is to present people with things they want, to predict (or arguably, to manufacture) desired behaviour.Footnote ³⁷ Amazon recommends products you might like to buy. Netflix recommends movies you might like to watch. And Twitter recommends people you might like to follow. But these systems cannot produce a complete understanding of your preferences or desires. Actually determining what people want is much more complicated than it seems because people rarely have a coherent, rational or even consistent sense of what they want. Desire is variable and fluctuating, liable to be influenced by others and to change with context. It can contradict our professed values or be harmful to us, or can sometimes be mysterious or inexplicable – often, we simply don’t know what we want. Desire, as it turns out, is elusive and immeasurable. So, in the face of the impossibility of knowing desire, recommender systems instead make a substitution based on other signals – usually behaviours that can be observed and measured in computational terms.

For most platforms, this takes the form of a wide range of explicit and implicit signals. These range from types of explicit engagement – such as clicks, views, shares, reactions or ‘dwell time’ – through to other kinds of implicit feedback – such as inferences drawn from people in your network. In data science, this is called determining ‘revealed preference’, and is premised on the assumption that observable actions can be used to reconstruct the motivation and preferences of a person. This approach is informed by now outdated nineteenth and twentieth century economic theories of consumer behaviour, specifically the characterisation of the consumer as a rational actor whose actions are driven by frameworks like informed decision-making and a utilitarian agenda.Footnote ³⁸

Determining revealed preference operates through a series of proxy substitutions: clicks as a proxy for engagement; engagement as a proxy for rational behaviour; rational behaviour as a proxy for desire; desire as proxy for identity and the self. While desire might not be amenable to calculation and to optimisation, its proxies are. These proxies allow otherwise mutable and amorphous phenomena to be the subject of calculative techniques. In the words of media scholar Wendy Chun, proxies allow us to ‘touch the unknown: they extend the unknowable, by capturing or “synching up with” what is not there’.Footnote ³⁹ Proxies are thus the inevitable effect of tools that seek prediction despite unknowability, that manufacture order amidst chaos. They substitute what we want to measure with what we can measure, allowing us to act upon data about the world as if it represented the world itself.

On one level, we could argue that all data are proxies. They are representations (often numerical values) that are taken as stand-ins for certain phenomena in the world. Making knowledge with data is, then, always already a process of making substitutions. With AI and ML systems, the vast array of applications, the scale of tasks, the complexity of processes, and the increasing stakes of outcomes now assigned to data-driven systems have new implications for lives caught within these regimes. To understand the harms and risks of ‘global governance by data’, then, one must also engage with the opacities and complexities inherent to proxies.

6.3 Proxy Discrimination As Colourblind Racism

One of the cruel ironies of the data-driven age is that systems designed to solve or reduce harms, like racism and racial discrimination, have in practice entrenched or expanded their effects. A key example of this is proxy discrimination.

Direct discrimination on the basis of gender, race, sexuality or other protected attributes is usually prohibited by law. For instance, the use of race as an explicit variable in the determination of decisions regarding credit eligibility, employment, or housing. But discrimination can also occur indirectly through the use of correlations between protected classes and other attributes. For instance, the use of postcodes as a proxy for race, income as a proxy for gender. This is known as proxy discrimination, and while it has historically been understood as a kind of direct discrimination,Footnote ⁴⁰ the widespread application of algorithmic and automated tools into a growing number of high-stakes decision-making processes has seen a new kind of proxy discrimination emerge.

Here, discrimination is a by-product of AI models that draw associations, inferences, and other statistical relationships between correlated sets of data.Footnote ⁴¹ For example, a 2012 Wall Street Journal investigation found that the American retailer Staples adjusted online prices for users based on their proximity to competitor stores. The intention was that customers living within twenty miles of a rival store, such as Office Depot, would receive greater discounts and deals. What was overlooked, however, is that access to infrastructure, such as retail stores, is highly correlated with geography and postal codes, which in turn are correlated with income and race. The result was that low-income and racialised neighbourhoods consistently received higher prices for Staples products.Footnote ⁴²

In another infamous example, Amazon introduced an AI-powered recruitment tool to identify promising employees based on their resumes. The tool was quickly retired, however, after the company found it systematically downgraded female applicants based on proxy indicators. Applicants from historically female colleges, who named certain hobbies, or who even used the word women (such as ‘women’s chess champion’) in their resumes were given lower scores.Footnote ⁴³ In both these instances, neither gender nor race were included as explicit variables, but the patterns discovered by the AI and ML tools nevertheless produced discrimination against women and racialised groups as a second order effect.Footnote ⁴⁴ Even as race recedes, discrimination returns, in novel, algorithmic, gendered and/or racialised guises.

How do we address these pernicious forms of sexism and racism? On the one hand, we could argue, as many scholars have, that these racist and sexist outcomes are not simply the ‘unintended consequences’ of working with big data, but are the all too predictable result of systems designed to recognise existing patterns in social relationships.Footnote ⁴⁵ They reveal what Solon Barocas and Andrew Selbst call ‘the simple fact of inequality’.Footnote ⁴⁶ As such, any efforts to ‘fix’ these systems by creating more precise models or capturing more data will merely replicate, or in many cases exacerbate, the prejudices and injustices that define the institutions governing life as we know it.Footnote ⁴⁷

On the other hand, we might take a broader look, acknowledging that these AI systems have themselves been introduced as a solution to the problem of good ol’ fashioned human racism that pervades institutional decision-making. With this in mind, we can examine the almost naive logic behind this approach. Chun describes this succinctly as the assumption that ‘racism and discrimination naturally stems from human recognition, and thus the cure must be the erasure of all visible markers of difference. If only we got rid of the markers of race – it is presumed – then all would be good’.Footnote ⁴⁸

Building on the work of sociologist of race Eduardo Bonillo-Silva, Chun argues that this approach constitutes a kind of ‘colourblind racism’ that over-invests in the neutralising force of technology. Like Chun, we also wish to interrogate this ‘race-free equals racism-free’ formula.Footnote ⁴⁹ Where Chun looks historically at histories of eugenics and segregation and their role in shaping the principles of network science, we wish to stay with the question of race as it is mediated and constituted today. Specifically, we wish to understand how processes of racialisation and race-making are now contingent on proxies and logics of proxification, and the implications this has for racialised subjects who find themselves on the receiving end of these ‘unintended’ racist effects.

This is a significant conceptual move because problems like proxy discrimination have proven themselves to be incredibly vexed issues for computer scientists, policy makers, and critical scholars alike. Try as we might, we can’t seem to stop proxy discrimination from happening, even when – or, perhaps, explicitly when – race is actively excluded as a feature or variable within a dataset. Despite the great equalising promise of these technological systems, racism seems to always find a way to manifest.

Indeed, even if we look at efforts to remediate proxy discrimination through technical procedures of verification and audits, we find proxies everywhere. For example, in a widely cited paper on ‘Auditing Black-Box models for indirect influence’, computer scientist Philip AdlerFootnote ⁵⁰ and his team developed a method to determine whether specific features in a dataset (like gender or race) influence the outcomes of a model’s prediction, and if so to what degree. Their focus was specifically on proxy discrimination in black box models. To do this, they used two techniques: (1) synthetic data, and (2) models of models. Both these techniques can be interpreted as themselves relying on proxy logics.

Synthetic data is an artificially manufactured dataset. It is data that is created algorithmically rather than derived from actual real-world measurement, and is often used as a stand-in for tests to validate or train ML models. It looks and has the attributes of a real-world dataset, but has the added bonus of being already cleaned as well as side stepping issues to do with data privacy. So, for example, synthetic data is often used in healthcare contexts because it allows for the public use of record-level data without having to worry about patient confidentiality because the features represented are not derived from real people but proxy subjects.Footnote ⁵¹

Similarly, models of models are used as a stand-in when one is precluded from accessing a model’s backend. They are specifically used in cases of black boxes, when a researcher is unable to tweak and rerun an algorithm to see what effect changing this or that variable will have. Instead, the approach is to create a model of a model: a kind of approximation of how you think the model works based on what is observable. For example, in their study examining the visual clustering techniques of the social media platform Instagram, Nic Carah, Daniel Angus and Jean Burgess created their own machine vision system to undertake an unsupervised classification task on over 13,000 images posted to the platform.Footnote ⁵² They then analysed the features of the clusters created in an effort to understand how images become operationalised within ecologies driven by personalisation and recommendation for online advertising. They described this technique as a kind of ‘critical simulation’ that allows them to perform ‘a speculative analysis’.Footnote ⁵³ Similarly, Adler et al used this technique to audit the now infamous Northpoint COMPAS software to try and examine exactly how race, as a feature, impacted on outcome scores for recidivism.Footnote ⁵⁴ Their method examined the interaction between model and data, measuring the differences and sensitivities when certain features were obscured. This technique of auditing a proxy model as a stand-in for a black box system can then be used within broader calls for investigations into potential indirect discrimination.

What this example illustrates is that even in our efforts to observe or measure proxies and their effects, we rely on techniques that are themselves premised on proxy logics. When it comes to knowing and being with algorithms, then, it really is proxies all the way down.

6.4 The Proxification of Race

In the slippage between absence and presence, between the erasure of race and the persistence of racist effects, we find the kernel of what we want to call a logic. This logic helps us to explain not only how racism continues to manifest within systems designed to remedy against prejudice, but to understand precisely how race is being acted upon and transformed in this process. We want to call this logic ‘proxification’. Proxification operationalises the as if. It puts it to work in systems and structures that seek to render the world into computational terms to make it more amenable to order and control. While the logic of proxification is present in algorithmic systems that implement rule-based sorting mechanisms, it is supercharged by the spread of AI and ML techniques.

Of particular interest to us is the role of unsupervised ML in surfacing latent and unrecognised connections. Carah, Angus and Burgess provide an efficient explanation distinguishing between supervised and unsupervised ML systems, describing supervised ML as being ‘oriented toward classification, whereas unsupervised approaches are oriented toward association’.Footnote ⁵⁵ Writing specifically on machine vision systems, they state:

Because they draw on data’s ‘latent properties’, this inductive form of pattern recognition means that associations and connections can be generated in ways that are often both strikingly novel and extremely difficult to reverse-engineer. According to Adrian Mackenzie, these latent connections arise when the features that data exhibit are mapped on to volumes in ‘vector space’.Footnote ⁵⁷ In this vector space, data need not be grouped together on the basis of explicit shared characteristics. Data might contain so many dimensions – variables – that what they do or do not share might not be particularly clear. Rather than sorting data into groups based on unambiguous characteristics, then, a data subset’s ‘similarity and belonging’ must instead be determined by ‘measures of proximity or distance, on flat loci that run as vectors through the space’.Footnote ⁵⁸

As Louise Amoore notes, this kind of statistical transformation allows AI and ML techniques to apprehend emergent ‘clusters’ defined by multiple shared characteristics.Footnote ⁵⁹ The use of clusters transforms the logic by which subjects are sorted into groups, because the problem frame is no longer centred strictly around identification but around alignment and association. She writes:

Clusters thus enact new forms of grouping. Objects are grouped based on their proximity to the centre of the cluster and classified depending on threshold values. This is how an individual might be assigned a risk score, or a number that indicates the degree to which they might have a propensity to default on a loan, defraud welfare services, commit a crime, or pose an existential threat to the state.Footnote ⁶¹

Proxies are fundamental components of the measures of proximity that AI and ML systems use to draw inferences and make determinations. But while clusters might generate particular statistical predictions rendered as measures – of, for instance, probability, or of confidence in a probability measure – their outputs have to be understood as imperfect approximations. Here, the slippage between what we want to measure – a determinate state of affairs – and what we can measure – on the basis of the data we have available, the infrastructures we have established, the compute resources we have access to, and so on – is what transforms the proxy from a minor substitution into an ordering logic.

This logic emerges in the slippage between measurement and representation. While AI and ML systems purport to measure the world, what they actually offer is an increasingly granular statistical representation of that world. Amoore captures this slippage when she notes that neural networks ‘hold[] out an alluring promise to the state; it too could decompose and represent everything, if only it could harness high-dimensional data to resolve the intractable difficulty that is politics’.Footnote ⁶² Amoore’s wry observation recalls the age-old representational problem of resolution: a map would become the territory if it were rendered in full detail; politics, by analogy, would no longer be necessary if one’s model was robust enough.

While we know that AI and ML systems can’t re-render the world whole and complete, producing it identically, the slippage between their incapacity to do so and the confidence we place in them to make decisions that actively shape the world has effects of obfuscation. We use them to make not only single, important decisions. We also use them to make countless small, seemingly inconsequential decisions that, in the aggregate, have profound effects. In the process, we lose sight not only of how these effects might scale to produce emergent and unpredictable outcomes, but also how they introduce new forms of difference into the world at large.

McQuillan highlights this capacity when he describes AI as a ‘technology of racialization’ which only ‘knows how to discriminate into output classes’ by ‘calculating distances, by determining some abstract metric of difference as distance’.Footnote ⁶³ Distance differentiates. Distance carving undifferentiated wholes into groups. Distance is a measure of dissimilarity. Distance, in other words, is able to operate not only as a converse measure to proximity, but as a means of discriminating between groups. Distance is, itself, able to operate as a statistical proxy – for, amongst other things, race. The kinds of discrimination, the feeling of these racist effects aren’t necessarily new – but the technical means that they are produced are. To conclude, we want to connect these novel modes of racialisation to governance, examining how proxification situates race right at the heart of the transformations that governance itself has been undergoing.

6.5 Conclusion: From a Politics of Identity to a Politics of Proximity

In his influential book on the impact of statistical reasoning on the emergence of modern statecraft, the historian of science Alain Desrosières argued that statistics provided governments with the capacity to produce an ‘equivalence space’ in which a state’s resources – people, resources, armies, trade goods, capital – could be made commensurable and, therefore, calculable.Footnote ⁶⁴ We can consider AI and ML as continuing this operational function, expanding and transforming this equivalence space into a vector space from which order is created and decision-making is derived.

These transformations have several implications for understanding contemporary governance. Numerous scholars have analysed how digitisation, the proliferation of algorithmic systems, the explosion of data collection techniques and datasets and the emergence of the platform as a key twenty-first-century infrastructure have transformed how governance is conducted by states and broadened who might be said to govern contemporary society.Footnote ⁶⁵ These techniques are increasingly integrated into formal systems of governance, promising to either make its institutions more efficient or streamlined,Footnote ⁶⁶ or else to mediate the distribution of increasingly scarce resources in increasingly stringent times.Footnote ⁶⁷ They shape our shared sense of social realityFootnote ⁶⁸ by introducing novel modes of sorting and ordering individuals into social collectives.Footnote ⁶⁹ But if we understand governance to be a process of social ordering that is constrained by rulesFootnote ⁷⁰ that isn’t solely carried out by states,Footnote ⁷¹ we can begin to see how seemingly innocuous techniques – such as the use of algorithms to control the content we see online – fundamentally reshape what governance is and how it operates.

In their formative work on algorithmic governmentality, for instance, Antoinette Rouvroy and Thomas Berns have claimed that algorithms usher in a mode of governance that doesn’t act on political subjects. Instead, by operating via statistical means, algorithms enable a mode of governance that attempts to ‘act on’, and thereby shape, the ‘environment’ that such subjects inhabit.Footnote ⁷² In this, it is supposed to constitute ‘governance without a subject, but not without a target’,Footnote ⁷³ and can be linked to governance techniques derived from behavioural science, such as ‘nudging’.Footnote ⁷⁴ In a more recent intervention into this debate, Fleur Johns has also argued that governments’ access to gigantic – and ever-increasing – pools of data is itself having a transformative effect on governance. Johns calls this alternative concept ‘governance by data’, arguing that this phrase ‘denotes instead the propensity for the gathering, assemblage, formatting, analysis, transmission, and storage of digital data to have governance effects’.Footnote ⁷⁵

We are agnostic about ascribing these transformations in governance to algorithms or to data – or to another agent altogether, such as the platform.Footnote ⁷⁶ These frameworks are all compelling; importantly, they aren’t mutually exclusive. What we want to focus on is not what ultimately determines this transformation in governance, but what it means for how politics is done – and more specifically, how racial politics play out.

These systems subject politics itself to a logic of proxification. In his recent work on machine learning, Ramon Amaro makes an illuminating observation about AI and ML systems. AI and ML systems’ power to discriminate between data, surface latent connections, identify clusters, and automate the production of outputs amounts to a form of proxy politics, because such systems are often implemented as a proxy for ‘collective and individual decision making’.Footnote ⁷⁷ By acting on data rather than subjects, this mode of governance can operate as if its effects were rational outcomes of objective processes.

This is why Rouvroy and Berns argue that this emergent form of governance crystallises a ‘certain type of (a)normative or (a)political rationality founded on the automated collection, aggregation, and analysis of big data so as to model, anticipate and pre-emptively affect possible behaviours’.Footnote ⁷⁸ In general terms, this proposition points to how the implementation of algorithmic systems changes the way that governance operates. Rather than taking individuals – whether construed as citizens or users – as the direct subjects of its operations, Rouvroy and Berns argue that this mode of governance affects the relations between the data produced by subjects.Footnote ⁷⁹

By delegating a portion of our freedom not only to decide, but to deliberate over the justness of decisions to the AI and ML systems that constitute algorithmic governance, we are not just eliding the deeply political nature of what these systems do but encoding new forms of inequality. As Shiv Issar and Aneesh Aneesh argue, as these systems become embedded, they constitute a new ‘societal infrastructure’ that ‘allow[s] inequalities to become durable’.Footnote ⁸⁰ For some scholars and commentators, such inequalities might take the form of unequal access to resources, or emerge when unfair decisions are occluded by AI and ML’s a-political and a-normative veneer. For others, they derive from data that represents already-existing, concrete inequalities, which are reproduced by AI and ML systems. This is bad enough, but our claim is that the stakes are even higher.

Earlier, we described proxies as bits of data that can be used in place of variables that cannot be measured. We also referred to these bits of data as ‘minor substitutions’ that, in and of themselves, don’t necessarily produce outsized political effects. But as the examples we’ve outlined in the prior sections illustrate, such minor substitutions collectively conspire to perform a sleight of hand. The more AI and ML arrogate decision-making powers, the more they reconstitute political problems within paradigms of clustering, proximity, association, probability, and pre-emption, the more they risk embedding new apparatuses of domination and discrimination into our digitised societies.

What does this mean for contemporary racial politics? In a recent essay on the impact that deep learning is having on the governance of borders, Louise Amoore argues that ‘the deep learning techniques of our contemporary moment are making race signify something different: a set of emergent proximities and distances generated through clustering’.Footnote ⁸¹ We not only agree with this statement, but think AI and ML extend its logic beyond the borders demarcated by the ‘border’. Insofar as AI and ML are made of proxies all the way down, they remain open to the possibility of being corrupted by racialised discrimination from the inside out.

As we have noted, the populations calculated and constructed by AI and ML systems are premised not on discrete, shared characteristics – on ‘identity’ – but on measures of alignment determined using complex, automated machine learning techniques: on what’s referred to as the ‘proximity’ between individuals,Footnote ⁸² which might be derived from a huge number of parameters including not only an individual’s characteristics (whom they are like), but, increasingly, preferences inferred using behavioural signals.Footnote ⁸³ It’s in this novel mode of calculating persons that a new racial politics takes root: as proxies cascade, they reorder the relationship between cause and effect that has heretofore underpinned the modern construction of race. Where it was once the body that determined the pattern (genotype or phenotype that acted as the signal for racial identification and belonging), it is now the pattern that determines the body (patterns of behavioural data that are used to create clusters, associations, correlations with a label representing identity).

Beyond engendering shifts in how communities or polities are governed, algorithms change the politics of race because they change how we – the collectivity, understood as broadly as possible, of those subjected to racialisation – are seen and represented and how we can act or be acted upon. This is why the interplay of opacity and representation or perception that Amoore highlights is the necessary vanishing point for the analysis we want to present in this essay. What proxies – and the AI and ML systems which rely upon them – engender then is a shift from a politics of identity to a politics of proximity.

We think a fair amount of paranoia is warranted in this situation. Rather than dealing with a biased state institution or service, or a biased monolithic corporation, our contemporary situation confronts us with countless services, premised on data processing using techniques like AI and ML, that may subject us to incorrect racialisations with unfair, discriminatory, or unequal outcomes. This influence is produced, enabled, and made durable, in novel form, by the very proxy logics on which these systems rely.

Using proxies allows engineers to declare their AI and ML systems ‘colourblind’, because they don’t contain data on categories such as race. And yet racism persists in these systems, mutating – via proxies – to take on new and even more opaque forms. As proxies cascade, turning minor substitutions into a broader logic and normalising data-driven techniques of discriminating between different entities, they increase the risk that we submit ourselves to a whole new racial paradigm.

In this way, proxies should not be understood as a problem of algorithms, but rather the intractable condition that makes algorithmic life possible – and new, racialised, modes of data-driven discrimination inevitable. The key point we want to make in concluding is this: to resist this inevitable situation, we need to learn a new set of political techniques equal to a politics of proximity. This doesn’t mean that identity politics is now redundant, or that its aims and goals should be discarded. The two are hardly mutually exclusive, and racial discrimination is so multiplicitous and varied that we need all the political strategies we can get. But it does mean that the tactics of identity politics aren’t equal to the task of resisting new modes of racialisation and racial discrimination premised on logics of proxification.

This sleight of hand contained within the proxy is both an act of disavowal and erasure. To stay with race, we need to refuse its logic. The form that this refusal will take is, for us, perhaps the most urgent question we, that racialised collectivity, need to answer today.

These three chapters offer critical diagnoses of how algorithms govern through their interactions with and beyond data. Each focusses on a particular technical practice of algorithmic governance, provides an account of its logics, and seeks to reveal the consequences of embedding such practices in assemblages of governance. For Phan and Wark, the technical practice is ‘proxification’: the ‘substitut[ion of] what we want to measure with what we can measure, allowing us to act upon data about the world as if it represented the world itself’. They provide an account of the ways in which ‘proxy logics’, and associated practices of ‘clustering’, transform racial categories and racialised governance. For Goldenfein, the relevant practice is that of the ‘human in the loop’ (HitL), understood as both a ‘regulatory strategy’ and a ‘highly malleable legal technology’. His chapter explores the ways in which the reconfiguration of the role and nature of the ‘human’ in novel socio-technical governance assemblages both redistribute accountability and preclude critical interrogation. Yeung’s focus is on the use of commercial techniques of machine learning (ML) – specifically risk profiling and citizen-scoring – in public administration and management. Through their misleading promise of accurate prediction, she suggests, these techniques ‘redistribute power, benefits and burdens in ways that are hidden in plain sight’ and at the same time ‘foreclose politics by side-stepping debate and deliberation about how collectively the state should act’.

In this short intervention, we offer, first, some reflections on what these chapters share along three distinct axes – the analytical, the methodological and the political – before, second, seeking sympathetically to extend them, including by identifying novel questions raised by their analyses.

Analytically, all three chapters are concerned with the constitutive effects of algorithmic governance on – and perhaps, for Phan and Wark, as – the collective life of a polity or social body. Phan and Wark discuss the ways in which technological routines associated with algorithmic proxification, including ‘clustering’, rework the substance and forms of race as a socially constitutive category. Their primary interest, in this chapter, is in commercial firms’ practices of consumer profiling, marketing, price differentiation and so on (without necessarily identifying these practices with ‘private’ action). By contrast, the other two chapters examine the reconstitution of the polity through the reconfiguration of routines of public administration. For Goldenfein, a HitL constitutes a ‘public’ as a political entity that is only in conversation with ‘humans’ in algorithmic systems – which in turn produces institutional and political occlusions, and concomitant lack of accountability. For Yeung, it is precisely the use of private strategies of risk management by public administrations that is the problem. In her account, the ‘public’ and ‘private’ are already constituted and distinct spheres organised around fundamentally different forms of political relationship. She traces the deleterious transformative effects on those political relationships of transposing ML from private to public, via the promise of ‘prediction’, ‘actionability’ and ‘early intervention’.

As such, the chapters share a core analytic: thinking about the relationship of algorithmic governance to institutional and organisational routines that make up day-to-day governing, such as the identity and functioning of the HitL, or the interactions between ML and public sector organisations. The chapters thus also share in a more-or-less explicit effort to expand critical approaches to algorithmic governance: to expand the aperture of critical analysis to bring into focus the relationship between algorithmic routines and organisational routines, especially in the projection of public power.

This has real merit, including at a methodological level. For one thing, this is an approach which demands attentiveness to the specificities of the relation between technological practices and organisational routines – including the particular ways in which the ‘new’ is articulated with the ‘old’. It follows that algorithmic governance is not one thing but many and varied things, demanding fine-grained analysis of institutions and organisations alongside technologies in order to manifest and understand these changes. For another thing, this is a methodological approach that allows a nuanced appreciation of continuity and change. All three chapters note that much of what is happening is at some level reminiscent of what has gone before – the history of analogue and early digital governance by proxy; the enduring presence of token humans in increasingly automated or mechanistic governance; the long history of depoliticisation through the embedding of private sector practices – including ones of prediction, actionability and early intervention – in public management. This enables the chapters to give us a sense (without being too prescriptive) of what might be different today, and whether it is a matter of degree or kind.

The chapters offer perspectives on two related changes in particular. The first is a set of transformations in the speed, scale and degree of automation – specifically, the increasing use of technologies of measurement, analysis, inference and prediction which operate semi-independently of, and rework, directed human cognition. The second is a (possibly) consequent set of changes to the relationship between governing, data, and truth claims, which result in practices of governance that both produce and are produced by knowledge claims which are constitutively provisional, unsettled and partial. Phan and Wark note, for example, the way that proxification both enables semi-automation through the use of AI and ML systems to draw inferences and make determinations, on the one hand, and at the same time produces modalities of decision-making predicated upon rapid and iterated approximation, on the other. These two developments are connected: it is precisely because proxies are always necessarily imperfect and provisional measures of ‘the real world’ that algorithmic governance emerges as a form of governance predicated on an ‘as-if’ and shifting construction of the social world. Similarly, Goldenfein draws attention to the practice of ‘red flagging’ in his description of the HitL. ‘Red flags’ arguably have a similar epistemic character: not a final determination but a provisional and uncertain indicator of a state of affairs, establishing a provisional and ‘good enough’ reality until specifically disproven. This epistemic shift does real work: it mobilises wilful or strategic ignorance of a range of contextual facts as a resource of an administrative system; and it redistributes resources by inverting the burden of proof. Finally, Yeung’s chapter shows the flip side of provisionality. As she notes, one can observe in many contexts an unthinking trust on the part of public administrators, especially in the capacity of ML to perform something called ‘prediction’, that they presume to be ‘accurate’, and a source of truth about the social world. But ‘prediction’ here means something different from its ordinary usage – ‘accurately predicting outcomes from a portion of data that formed part of [an] original data-set from which the training data was taken’ – with the consequence that accuracy (in the sense of the ‘truth to life’ of predictions) is ultimately unmeasurable.

Finally, the three chapters offer a shared sense of political disquiet. They allow the reader to explore how algorithmic governance shapes the conditions of possibility for individual and collective politics through describing the political transformations that it effects. Phan and Wark, for example, trace a shift from a politics of identity to politics of proximity (although whether it is a politics of proximity tout court, or a politics of identity made through algorithmic approximation, is not yet clear). Goldenfein similarly points to an accountability politics around ‘orphaned’ – or perhaps ‘not-quite’ – decisions. Yeung points to new distributions of power and practices of domination which bypass democratic debate and even escape public awareness. All three also encourage the reader to imagine the other forms of politics that these transformations foreclose. Their political spirit is thus not programmatic, prescriptive or policy-oriented. Rather, by refocusing analytic and methodological lenses of critique, they call for broader political reengagement or reinvestment.

We now turn to our reflections on the chapters in the same vein – to extend them analytically, methodologically and politically. Analytically, at least two of the chapters rely on the possibility and existence of some sort of constitutionalised or constitutionalisable collectivity in order to generate new politics and to produce the potential for the accountability of algorithmic governance regimes. This is most explicit for Yeung, who sees the use of ML as mapping onto an existing public/private divide (where ML travels from private to public, with negative effects). Here, each side of the divide has different logics and political and economic imperatives. It is also explicit for Goldenfein, where the ‘human’ in the loop is – and is not – answerable to an unproblematised general citizenry. Both of these chapters, then, in their own way suppose the possibility not just of a politically salient collectivity, but of one which in some fundamental sense can have a degree of autonomy from algorithmic governance itself. Phan and Wark are slightly differently oriented. Rather than focusing on a political ‘public’, they worry about the ways in which the socio-technical reconstitution of social relations through proxification generates new hierarchies, which might in turn shape possibilities for individual and collective political imaginations. And because these imaginations are crucial to ‘the task of resisting new modes of racialisation and racial discrimination premised on logics of proxification’, it will be necessary ‘to learn a new set of political techniques equal to a politics of proximity’.

Yet what are the conditions of possibility for any such ‘public’ – or sociality – to be potentially external to or a check on algorithmic governance? One of the consequences of the transformation of governance into ML routines might be to erode the political distinction between private and public subjecthood, into streams of actionable data. In other words, the movement of ML from private to public might be an effect of how ML transforms political subjectivity – a very different situation from one in which ML causes a transformation of political subjectivity as it moves from private to public. If the former is so, it would require an analysis which brings the mechanisms by which people are transformed into data into that very analysis, along with the use of data by ML. Furthermore, what does algorithmic governance do to the epistemic conditions of possibility for collective politics as traditionally understood? We noted above that algorithmic governance deals not in shared ‘truths’ but ‘as-if’ realities, and that the social collectivities it produces (e.g. ‘clusters’) can take the provisional and unsettled form of quasi-digital artefacts. To what extent, in what form, and in which spaces can social relations of solidarity survive this transformation of the epistemic foundations of the political and civic sphere?

Methodologically, we emphasise the importance of dwelling on the organisational routines of the ‘public’ sector – especially as a means of understanding how public–private relations are reconfigured through algorithmic governance. From time to time (and with good reason) the chapters assume the practices and logics of these routines. Yeung, for example, notes that for ‘digital platforms, the promise of actionable insight rests on the ability to pre-empt and personalise services offered to users, given that the size of a platform’s user-base is typically directly proportional to its market valuation. But for public administrators, actionable insight from risk prediction tools promises to optimise the allocation of public resources’. For Goldenfein,

The human and its routines are however less of sociological and political interest here, than they are of interest in terms of the ‘distribution of cognitive tasks and the reconfigurations of accountability’.

But understanding how algorithmic and governance routines couple is crucial. These chapters show us that the logics of algorithmic governance emerge not just from digital techniques themselves, but their operationalisation in an all-too-familiar and predictable context of human and institutional ‘limitations’: data gaps, fallible practices of approximation, a desire to avoid responsibility or reward allies, impossible time and resource constraints and so on. And more can be done to attend to their articulation in specific contexts. As we have noted elsewhere, the politics of big data governance (a fortiori algorithmic governance) is found not only in its internal operations, but also in the relationship between it and ordinary state institutions and routines; in how different practices imagine reworking that coupling; and in how different imaginaries of remaking organisational routines legitimated the application of big data, algorithms and the like, to state planning.Footnote ¹ So, for Goldenfein, ‘automation does not break legality, it merely rehearses and amplifies the rule of law’s unresolved tensions, which are then papered over by symbolic human oversight rules’. What, exactly, is contained in the act of papering over these tensions is enigmatic, but of real political import.

We also argue for centring reflexivity and reflexive practices further in the analysis of algorithmic governance. By ‘reflexivity’ here we mean the capacity – whether individual, organisational or institutional – for systematic self-criticism and responsive adaptation (whether in the mode of recursion, or the ‘learning’ and rewriting of pattern matching algorithms and so on). Importantly, each of the chapters identify particular forms of reflexivity as crucial to algorithmic governance: after all, one of the central attractions of algorithmic governance is its promise to enable a governance regime to adjust or adapt swiftly and at scale to the world through an apparently endless capacity to collect, analyse and process more data. At the same time, these chapters also share a critique of algorithmic governance as lacking in reflexivity in certain ways – each criticises algorithmic governance as producing unwarranted faith in algorithmic prediction, and ignorance of its flaws, limitations and weaknesses. How comfortably do such critical claims fit with the epistemic shifts noted above – in which algorithmic governance is associated with truth claims that are in some sense always already unsettled, and ways of making choices which are only ever on the basis of ‘as if’ realities?

It may not always be the case that algorithmic governance is associated with an absence of scepticism. In other contexts (and in line with the idea that there are continuities as well as changes in the novelty of algorithmic governance), we can see how some systems that have the same features that these papers diagnose have reflected on and adapted to some of the critiques. For example, development practice suffers from technologisation, and with it, an emphasis on prediction, actionability, early intervention and so on – just as Yeung’s ML does. But we also see work from within development critiquing the transposition of technical and standardised operational regimes into the public sector.Footnote ² Going back to Yeung’s terms, this shapes the political consequences of the application of algorithmic governance, in the sense of producing or foreclosing political horizons – by engaging with, reflecting on and critiquing the ways that algorithms and governance relate, from within and as part of bringing algorithmic and governance routines together. Framed in this way, enquiries into algorithmic governance might encompass questions and related approaches around the forms that reflexivity takes in algorithmic governance; how algorithmic governance shapes what counts as reflexive practice, and its distribution in governance arrangements; and how these arrangements might be historicised.

Finally, politically, we turn to whether and how the politics of algorithmic governance is found not only in transforming individual and collective politics, but also underdetermining them, thinning them out, and deferring them. In other words, and in an echo of some of the reflections above, it appears from the chapters that algorithmic governance is powerful not only for how it changes the forms of collective existence, but also how it reworks its substance, and potentially even erodes its conditions of possibility. In particular, it seems to us that algorithmic governance at least as often brackets and displaces the resolution of questions of truth and value, as it purports to resolve them. For Phan and Wark, for example, proxification constitutively involves looking away from the (unmeasurable) real world to a second-best world of available (but incomplete and poor quality) data, and measurable (but ambiguous) outcomes or behaviours. In continually approximating – and reworking its own techniques of approximation – is technology ‘opaque’ and politically neutralising, or could it be described as a means of bracketing and politically provisionalising ultimate questions of truth and value? Similarly, for Goldenfein, the human decision-maker is a spectral presence – the point of decision-making always taking place somewhere else within a complex and mysterious-by-design regime of distributed cognition.

This recalls the politics of governance projects committed to the desacralisation of the collective:Footnote ³ the enduring suspicion of determining shared matters of value in a stable fashion, and with it, the relentless critique and reconstitution of its expressions.Footnote ⁴ And as many have noted, political responses to these projects of resistance and reinvestment risk being bracketed and coopted along with all the others – becoming yet another competing expression of ultimate value.Footnote ⁵ In other words, the chapters remind the reader that critiquing algorithmic governance requires ‘breaking brackets’, as much as a resistance to ‘political foreclosure’.