4.1 Introduction

In 2017, the Australian Department of Immigration and Border Protection commenced procurement of a new technology platform for processing immigration and VISA decisions.Footnote ¹ Resistance quickly emerged around the risks of automating critical decisions affecting individual’s rights.Footnote ² Responding to those questions, the Department Secretary Michael Pezullo clarified that:

In a theological flourish, this was described by Pezullo as the ‘Golden Rule’, effectively expressing law’s implicit belief that only humans can be intentional moral agents.Footnote ⁴ In the language of technology governance, Pezullo was specifying the protective operation of a ‘human in the loop’ – a concept that has increased in popularity as a regulatory solution for accountability problems accentuated by automation.

In government decision-making contexts, automation typically takes the form of software systems that process specific elements of decisions or guide decision-makers through decision processes. As they have proliferated, these systems have provoked anxieties about inconsistency with rule of law principles. A substantial literature has diagnosed the ways automation challenges notions of legality on multiple registers. Some question the effects of automation on transparency, equality, and accountability;Footnote ⁵ others dig deeper into the relationship between automation and ideas in legal philosophy.Footnote ⁶ These analyses describe how digital technologies challenge the very media of our governing systems,Footnote ⁷ and undermine critical philosophical precepts such as the undecidable horizons of justice.Footnote ⁸ Frequently, these diagnoses conclude with an of appeal to re-humanise automated decisions in order to preserve their alignment with rule of law prerogatives. Beyond automated administrative decision-making, similar appeals emerge wherever automated systems operate. Across these domains, the ‘human in the loop’ has become a central pillar of humanising efforts and an ascendant regulatory orthodoxy designed to ensure the primacy of human values over mindless technocracy. This chapter argues the human in the loop does more than simply ameliorate technology’s disruptive potential. Through analysis of cases invoking human in the loop in administrative contexts, this chapter argues that, in the name of recuperating rule of law values, a human in the loop is used as a versatile legal technology that enables problematic distributions of accountability and inhibits a richer legal conceptualisation of automated decision systems.

4.2 What’s Wrong with a Human in the Loop?

Although laws demanding human oversight for automated decisions have existed for some time,Footnote ⁹ calls for a human in the loop have recently intensified as both ethical requirement and regulatory prescription.Footnote ¹⁰ This is not surprising considering its intuitive appeal as a regulatory strategy. If automated systems and software are opaque by design, the human renders decisions transparent. If automated decisions risk subordinating humans to technocracy, the human provides the requisite dignity of human consideration. If automated systems discriminate on account of the way data encodes social bias, the human corrects errors and ensures decision quality. If automation produces non-reflective forms of mindless governance, the human grounds the legitimacy and authority of legal commands.Footnote ¹¹ If software dissolves accountability into the procurement, design and engineering of a decision system, the human provides a coherent locus of responsibility. If automation includes emergent and generative qualities, the human garners trust, and normalises regulatory strategies premised on risk management.Footnote ¹²

These intuitions mean the human in the loop has become a sort of Talisman of the appeal to re-humanise – a symbolic regulatory apparatus.Footnote ¹³ At the same time, however, the human in the loop’s capacity to satisfy those demands has been questioned on a number of registers. There are empirical challenges to the human in the loop challenging its capacity to improve decisions,Footnote ¹⁴ satisfy dignitarian demands of human rights instruments,Footnote ¹⁵ or facilitate meaningful transparency.Footnote ¹⁶ Ben Green and Yiling Chen, for instance, shows that human oversight of data-driven judicial decisions does little to mitigate discrimination.Footnote ¹⁷ De-Arteaga, Fogliato and Chouldechova offer more mixed results, identifying a limited human capacity to intervene against malfunctioning algorithmic risk prediction tools in the child welfare context.Footnote ¹⁸ But the impact of the human in the loop on decision quality is, at best, uneven. This is not to suggest that technical systems make better decisions without human oversight. Humans are, of course, capable of improving technologically inflected decision processes. Scholars like Jennifer Raso detail how humans actively manipulate decision technologies, even beyond their intended input parameters, to achieve better outcomes by clinical standards.Footnote ¹⁹ But there is immeasurable complexity in how human interaction with different sorts of data-driven and algorithmic systems, influenced by behavioural biases, political preferences, and organisational arrangements influence decision quality and performance.Footnote ²⁰ And legal prescriptions for human in the loop rarely to attend those details.

Responses to the empirical unevenness of human in the loop requirements fall into two general classes. One group of commentators suggest the human in the loop’s failures stems from being not human enough. A more ‘human-centric’ approach is necessary to recuperate the human in the loop,Footnote ²¹ prompting examination and articulation of the conditions for achieving deeper human-centrism in decision automation. These proposals draw from human centred design and strategies for ‘function allocation’ that seek to leverage the systematic differences in human and machine capacities to improve outcomes, mitigate biases, or overcome institutional obstacles. Marion Oswald, for instance, suggests effective human discretion is obtainable so long as humans use their knowledge of factors not included in automated assessments, and use procedural and tacit knowledge along with their experience.Footnote ²² Ben Wagner similarly suggests that for human oversight to be meaningful, human decision-makers must be given adequate time; must be adequately qualified to fulfil their oversight task; be adequately psychologically and socially supported; have sufficient access to relevant information to make the correct decision; and have sufficient authority to change a decision if necessary.Footnote ²³

Other scholars read human in the loop’s limited empirical success as a more fundamental challenge to the value of human-centric approaches to governing automation. One branch of this critique challenges the liberal biases associated with law’s construct of the human. Connal Parsley, for instance, argues that the uninterrogated assumptions around the self-determining moral status and rational capacities of the human tasked with oversight have become a conceptual bottleneck to well-considered regulation of automated decision-making.Footnote ²⁴ Scholars like Matilda Arvidsson, analysing the human in the loop in ‘intelligent warfare’ and international humanitarian law, similarly question law’s gendered presumptions around ‘human’ capacities and characteristics in the context of digitally augmented posthuman actors in warfighting conditions.Footnote ²⁵ These critiques show the idealised legal human grating against more materialist accounts of legal agency that challenge clear distinctions between subjects and objects,Footnote ²⁶ and highlight how contemporary regulatory efforts are premised on the faulty assumption that, as Rebecca Crootof and her co-authors put it, the ‘human’ in the loop is ‘a concrete and identifiable entity, and thus an easy regulatory target’.Footnote ²⁷

Other scholars push the critique of human-centrism further, and identify fundamental incompatibilities between human and computational agency. Louise Amoore, for instance, argues that,

Gregor Noll similarly suggests the normative programmes of data-driven automation and human legality are entirely incompatible. He argues that automated decision-making systems are the product of cybernetic ontology that conceptualises human and machines equally as symbolic information processors, whereas legal decision-making derives from (secularised) monotheistic traditions that privilege the human as empowered to rule on earth.Footnote ²⁹ Noll describes human law as having its origin in the divine (word of God) made incarnate through the human processes of study and cognition, whereas automation takes natural life, represented and processed as data, as law’s origin. The computational process of ‘training’ thus replaces human cognition, leaving little room for humans to comprehend the relations between things.Footnote ³⁰ Mark Andrejevic claims in a related way that the logic of automation and its cascades cannot be governed by something that is not equally automated.Footnote ³¹

But these arguments terminate in a conceptual and regulatory impasse. Human oversight of automated decisions may, at times, be ineffective or functionally impossible, but abandoning human agency in decision-making is politically and functionally inconceivable. We might instead ask why the continued proliferation of a regulatory apparatus despite these empirical and conceptual shortcomings? Passage through this dilemma means excavating the broader political work performed by the human in the loop as a regulatory strategy, especially in how it underspecifies its human. This means understanding the human as a highly malleable legal technology, and exploring how differential operationalisations of the ‘human’ stabilise different sets of governing relations. Through that analysis, this chapter argues that the human in the loop does calculated work protecting political actors from the political nature and consequences of the choices that are embedded in technological and automated decision systems.

There is already excellent research describing this dynamic. That decision-making agencies dissimulate responsibility behind technological implementations is well documented.Footnote ³² And human oversight requirements are understood as a mechanism to responsibilise individuals for outcomes they may not control.Footnote ³³ But the jurisprudence outlined below shows what that entails in an administrative context, and the different kinds of political moves it enables. This chapter draws on case law addressing requirements for human decision-makers in different administrative examples to show how, in practice, a human in the loop can provide a governance gloss that both fails to grasp the realities of automated decision systems in operation or the political rationales animating them. These cases do not speak to the (in)efficacy of human in the loop for achieving better outcomes, but rather the systems of administrative governance that develop and proliferate behind the stylised distinction between human and machine endorsed by human in the loop requirements. In particular, these cases offer a lens into the distribution of cognitive tasks and the reconfigurations of accountability associated with automated decision systems used in the exercise of public power, exposing law’s posture towards questions of automation as well as the material stakes of this posture in practice. This chapter argues that the human in the loop paradigm dissimulates the realities of automated decisions along two axes: between humans and machines; and between the individual and the institutional, highlight the political reconfigurations at the heart of ‘governance by data’.

4.3 Human versus Machine: How Human in the Loop Reifies Unrealistic Distinctions Between Humans and Machines in Automated Decision Processes

Assumptions about the human, its moral agency, and the political stakes of human-centrism come into relief around the legal concept of a ‘decision’. The cases described in this section demonstrate how the prescribed role of humans in automated decision-making works to misassign distributions of responsibility between human and technological processes, as well as limit which elements of automated decisions are open to interrogation and contestation. An example is the Australian case of Pintarich v Deputy Commmissioner of Taxation, which considered whether a basic process-automation system had made a ‘decision’ that could be the subject of judicial review.Footnote ³⁴

Pintarich discussed by telephone the amount of tax liability he owed with an officer at the Australian Tax Office (ATO). The officer then keyed information into a ‘bulk issue letter template’ system, triggering a letter to be sent to Pintarich outlining his tax liability. The applicant used that letter to obtain a loan from a bank in order to pay the ATO. However, the ATO officer subsequently claimed that he never agreed to the level of liability discussed on the telephone nor did he intend to send the letter specifying that amount was acceptable. He instead sent another letter stipulating a higher amount of tax liability. The applicant brought judicial review against that second ‘decision’, claiming a ‘decision’ had already been made for the lower amount.

The majority judgment followed precedent established under non-technological decision-making conditions that ‘there needs to be both a mental process of reaching a conclusion and an objective manifestation of that conclusion’ for a decision to occur.Footnote ³⁵ In this case, the majority found no evidence of a mental process reaching a conclusion as to the lower amount of liability – the decision-maker had not undertaken any of the requisite deliberation, assessment, or analysis necessary for decision-making and appeared to trigger the letter by mistake (despite including his signature). Because it was not accompanied by the requisite human cognitive process, the letter sent by the automated system, that is, the objective manifestation of a conclusion, was deemed insufficient to independently constitute a decision in law. This finding is intuitive in many ways and accords with arguments around the centrality of human agency to the grounding of legal authority. Will Bateman, for instance, argues that fundamental public law doctrines assume that legal powers will be exercised by a particular kind of decision-making agent with the cognitive capacity to understand their task and the information relevant to the exercise of that task.Footnote ³⁶ This by implication requires engagement with an intellectual process.

The dissenting Pintarich judgment, however, presented an alternate analysis, rejecting the abstract two-part definition of a decision adopted by the majority. Noting the growing centrality of automation to contemporary governance, Justice Kerr suggested abstract, universal accounts of decisions are inappropriate where, in cases such as this, the actual decision process had been (at least) partially automated. Justice Kerr drew on other authorities indicating that legal decisions are frequently made without explicit mental engagement, that decisions need not be coincident with intention, and that fully automated decisions often represent the position of a state agency. Ultimately, he described the requirement of human mental processes for legal decision to be ‘an artefact of the past’, ‘challenged by automated “intelligent” decision systems’.Footnote ³⁷ While evacuating legal decisions of human moral agency may undermine some administrative law principles (such as those identified above by Bateman), Justice Kerr noted it recuperates others such as fairness and administrative certainty.

That unfairness and administrative uncertainty identified by Justice Kerr were the product of authorities recanting a decision on the basis that it was not informed by adequate human cognition even though human action caused it to be communicated to the decision subject. In other words, the majority’s finding meant the human decision-maker would not be responsible for how their interaction with an automated decision system stipulated and communicated an outcome to a decision subject if it did not match their subjective intention, despite any harm it might cause. The human in the loop requirement thus became a mechanism to prevent unpacking how decisions actually happen and how to understand responsibility in the context of ongoing interactions between a human and an automated system that affects people’s rights. Justice Kerr’s call to pay attention to the reality of decision-making in automated contexts is thus a call to (1) reconsider the normative prescriptions associated with human cognition and agency within broader normative goals of public authority, and (2) understand the complex reconfigurations of responsibility associated with automation in governance.

This notion that explicit human cognition is necessary for legally meaningful decisions is particular to administrative law and the availability of judicial review. Other legal regimes, such as in Article 22 of the EU General Data Protection Regulation,Footnote ³⁸ have long recognised the possibility of fully automated decisions with legal effects.Footnote ³⁹ Whereas the Australian administrative jurisprudence suggests no decision occurs without a human cognitive process, the European data protection regime suggests, alternatively, that decision subjects have a right to have humans supervise or participate in otherwise automated decisions in certain contexts such as if they produce legal or similarly significant effects. That is, fully automated decisions do occur, but there is a right to not be subjected to them – or otherwise demand some human participation in their making. But even this regulatory arrangement produces strange distributions of responsibility and dissimulates critical political arrangements into technological artefacts. An example is the Dutch SyRI judgment.Footnote ⁴⁰

The System Risk Indication (SyRI) system aggregated personal data from various government sources to detect social security fraud. SyRI was a hybrid decision system, with a computational component aggregating and analysing data to produce risk profiles and assessments, flagging individuals for investigation, with a human making the final decision to investigate any particular person. SyRI was abandoned after being found to breach the Right to Private Life in Article 8 of the European Convention of Human Rights for inadequate transparency around how the system was deployed, who was notified, and the content of its risk models. The case was lauded as a victory against the automated administrative state.Footnote ⁴¹ On close analysis however, the judgment also exposed how Article 22 (and its human in the loop prescription) worked to bracket the computational and automated dimensions of decision-making from proper review or contestation.

That judgment considered the human in the loop question in the context of whether the automated creation of risk reports constituted a form of decision that might be contested under Article 22 of the GDPR.Footnote ⁴² Although not directly concluding on the point, the Court accepted that identifying individuals as possible fraud risks did not constitute a decision with legal or similarly significant consequences. Risk profiling was not sufficient to constitute ‘automated individual decision-making’ for the sake of Article 22. That threshold was reached only after the subsequent decision to investigate was made by a human.Footnote ⁴³ On the one hand this makes sense, it would be peculiar to specify that a human participate (more than they always already implicitly do) in each step of a machine learning driven decision process. But the human in the loop rule in this case, by limiting reviewability and contestability through deeming only the human input as legally meaningful, misrepresents the nature of technically inflected decision-making. It renders opaque how an automated system has ‘pre-selected’ those to be investigated further, how a human interprets and interacts with a risk report, and the range of human political decisions that go into the specification and operation of that automated system. It explicitly prevents interrogation of how a decision process is distributed amongst various actors, human and non-human, on the basis that there is an accountable human actor choosing whether or not to investigate any particular individual. The human in the loop rule therefore inhibits understanding the contribution of risk profiling and ranking systems to the exercise of public power, and how responsibility might be tracked through those systems.

Risk ranking is increasingly central to public administration and numerous other governance regimes. Automated systems rank huge populations for various types of risk, with the top hits becoming subject to further human scrutiny. Despite claims such systems are merely efficiency tools with no influence over outcomes, they meaningfully challenge the human-centric notion of ‘decision’ embedded in the human in the loop regulatory paradigm. There is, of course, guidance, on what level of human oversight is necessary for these systems to be acceptable,Footnote ⁴⁴ and the administrative law of decision attempts to delineate ‘stepping stones’ from ‘decisions’ that determine legal rights or entitlements, but human in the loop rules reify this already problematic distinction by drawing attention away from the influence of automated systems and their management of process on final outcomes. It obscures much of what constitutes the actual decision in practice.

As Michael Veale and Reubin Binns astutely point out, decisions in automated contexts are stratified through various stages with different levels of human involvement and intervention in different models such as ‘triaging’, ‘decision-support’, ‘summarisation’, ‘flagging’, and ‘risk scoring’. They describe how these realities problematise the location of any decision within a technical system in ways that undermine the logic of Article 22.Footnote ⁴⁵ Jen Raso similarly demonstrates how automated decision-support systems displace decision-making authorityFootnote ⁴⁶ into a matrix of human and non-human entities in ways that ‘quickly destabilizes concepts common to public law scholarship’.Footnote ⁴⁷ This includes drop-down menus, comments in case management software, and algorithmically generated client letters. Rather than unpack the influence of this type of process automation, the human in the loop regulatory paradigm elides its influence on decision outcomes. It obscures the legal significance of automated processes that select individuals for evaluation, marshal data for analysis, and impose obligations on decision subjects to prove or disprove entitlements or obligations.Footnote ⁴⁸ By holding humans and machines apart, the human in the loop supports the problematic distinction between decision and decision-making that has become increasingly untenable.

4.4 Individual–Institutional: How Human in the Loop’s Responsibilisation of the Individual Obscures Institutional and Organisational Politics

Laws requiring a human in the loop draw from paradigmatic models in human–computer interaction of humans interacting with computer consoles and interfaces. This is an awkward construct, however, considering the complexity of automated systems’ contribution to decision outcomes in legal and administrative contexts. Decisions are made by individuals, informed and contoured by automated systems, all situated within complex organisations. Requiring human oversight to salvage the legality challenged by automation obscures how automation is itself an expression of complex institutional prerogatives, policies, and negotiations intended to affect different people in different ways. Ensuring the presence of a responsibilised decision-maker focuses on adequate governance for individual decision outcomes, but directs regulatory attention away from the influence of institutional policies and their responsibility for outcomes. The Services Australia Online Compliance Intervention (Robodebt) debacle offers a clear example of how a legal focus on human in the loop concealed how problematic political imperatives came to be embedded in technical processes.

Robodebt used data matching of self-reported income against tax records to flag prima facie possibilities of welfare overpayment. The system thus automated the process of debt verification previously performed by a human decision-maker who had to satisfy themselves as to the existence of a debt by looking at payslips and speaking directly with employers. Instead, discrepancies from data matching triggered letters to be automatically sent to welfare recipients, obliging them to update their payment information, with failure to do so resulting in a legal debt. Much of the analysis around Robodebt has focused on the unfairness and obvious inaccuracy of an automated process that used income averaging without a human reviewing or checking the consequent presumption of a welfare debt. These human in the loop arguments focus on how removing a human from the decision-making process diminished the quality of outcomes.

This framing also anchored challenges to the system’s legality. Data matching to flag potential debts had been used since the early 1990s. But in 2017, the Australian Administrative Appeals Tribunal determined that a welfare recipient’s failure to provide information disputing the finding of a debt identified by data matching did not absolve the department of its obligation to pursue third parties for further information and satisfy itself of the debt’s legitimacy.Footnote ⁴⁹ The results of data matching could not constitute a decision of consequence independently. It only raised doubt about the accuracy of past payments warranting the exercise of subsequent powers of inquiry. Despite this finding, the Australian Department of Social Services continued to deploy a larger-scale debt recovery programme, using the same methodology to automate the process of raising debts against welfare recipients. Litigation against the Robodebt system then confirmed that income averaging was not sufficient to raise a debt,Footnote ⁵⁰ resulting in a massive class action in negligence and unjust enrichment and consequent settlement agreement in 2021.Footnote ⁵¹

Robodebt was one of a number of welfare recovery automation systems deployed around the world,Footnote ⁵² however the Robodebt litigation and subsequent Royal Commission spotlighted the institutional dynamics behind of this type of automation. Beyond simply automating the finding of a debt, Robodebt is better understood as a programme for shifting the burden of demonstrating the existence or non-existence of a social security debt from the decision-maker to the decision subject.Footnote ⁵³ Rearranging the labour associated with debt verification away from the government agency enabled a debt recovery programme that could operate at massive scale. This was not a problem of decision automation – the data matching process was simply a solution to the organisational problem of resourcing the debt recovery processes. There were, in fact, humans still threaded through the decision process, but their job had changed from verifying the debt to managing client interactions – a much lower cost form of human oversight. Achieving that labour outcome meant decision subjects were unwittingly looped into the process, burdened with the onus of disproving their wrongdoing with extreme emotional consequences.Footnote ⁵⁴

In approving the class action settlement, Justice Murphy noted that ‘the proceeding has exposed a shameful chapter in the administration of the Commonwealth social security system and a massive failure of public administration’,Footnote ⁵⁵ and that ‘it should have been plain that in such circumstances the automated RoboDebt system may indicate an overpayment of social security benefits when that was not in fact the case’. But there was no admission of liability by the government, and Justice Murphy found it unlikely that plaintiffs could prove that government officials actually knew (despite nearly thirty Tribunal decisions rejecting income averaging as a basis for asserting a social security debt)Footnote ⁵⁶ that the system raised illegitimate debts. Despite acknowledging that actual knowledge of illegality in this situation would be a paradigm case for aggravated and exemplary damages, Justice Murphy held that actual knowledge that the automated system raised debts that were not actually owed was unlikely to be demonstrable.Footnote ⁵⁷

The judgment thus awarded financial damages premised on the illegality of removing the human from the decision-making process – but not for the broader institutional policy of reversing the onus of debt verification to scale a punitive programme. The existence or non-existence of a human decision-maker is thus a red-herring. More important was how automating away the initial decision-maker’s obligation to prove overpayment facilitated a debt recovery exercise of a scale otherwise impossible. Focusing on the degraded accuracy from data matching allowed the debacle to be narrated as a technical matter rather than a political one. According to this narrative, the department heads and decision-makers associated with implementing the system were deresponsibilised by the automation. Their political choices were dissimulated into critique of the inaccuracy of an automation process. In a strange mirror of the Pintarich outcome, the Department sought to recover money on the basis of decisions made without the application of a human mental process, but then disavowed itself of responsibility for the outcomes of that decision on the same basis – that nobody actually knew about the inaccuracy and illegality of the system. Interrogating those institutional realities has required a Royal Commission, but ought to be the content of quotidian regulation of automated systems used in the exercise of public (as well as private) power.

In other jurisdictions, there have been cases investigating the mechanics of regulatory agency function throughout the decision process (rather than focusing predominantly on decision outputs). The US Michigan Integrated Data Automated System (MiDAS) litigations, for example, presented the possibility of interrogating responsibility for such systems on a broader institutional register. That litigation concerned a similar, and perhaps more pernicious, social security fraud detection and recovery system used in the state of Michigan from 2013.Footnote ⁵⁸ The numerous opinions and judgments demonstrated willingness by the courts to interrogate structural distributions of accountability and responsibility associated with the organisational processes of decision automation, with little ultimate success. In the end, compliance with human in the loop requirements shielded system designers from liability for the conception and deployment of an abusive programme.

The MiDAS rendered automated determinations of fraudulent conduct (i.e. misreporting income) with a data matching process that used an ‘income spreading’ formula. When an individual claimant was flagged for possible fraud, they were notified but not provided further information about the basis for the agency’s suspicion. Instead, they were provided a cryptic questionnaire about benefits, with a positive response to almost any question triggering a fraud determination. If claimants failed to respond within thirty days, a fraud determination followed without further human assessment. The claimant’s rights to unemployment benefits were then terminated, monetary penalties were applied for fraudulent misrepresentation, and processes commenced (wage garnishing and interception of tax returns) to recover amounts of up to four times the benefits received. Even the letters demanding repayment did not inform recipients of the factual basis for the fraud determinations. A human only participated in the decision process if a recipient commenced an appeal against the determination, few of which were overturned.

The programme was, like Robodebt, highly inaccurate. The Michigan Auditor General found that of 22,000 determinations reviewed, more than 90 per cent of them did not involve fraud. That finding, alongside a legal determination that this type of auto-adjudication was not permissible, triggered reforms in 2015 requiring a greater amount of human involvement in the fraud determination process. In other words, the system was adjusted to conform with human in the loop requirements giving individuals more opportunity to rebut allegations prior to a final fraud determination. But these opportunities remained highly constrained, and though it reduced the false positive rate considerably, it did not shift the broader punitive and unfair nature of the system.

Initial findings by the court against Michigan agency employees were promising. They directly considered the responsibility of specific government officials for particular actions that deprived individuals of constitutional due process rights. The court allowed actions to be brought against officials for their roles in the MiDAS system’s violation of individuals’ rights to procedural due process by terminating their eligibility for unemployment benefits and seizing their tax refunds without any meaningful (pre-deprivation) process.Footnote ⁵⁹

Those agency officials were also joined in the litigation by executives of the private firms that consulted, specified, and provided the software that deprived social security recipients of their constitutional due process rights. Plaintiffs claimed those executives exercised control over the relevant officials; caused them to commit the relevant acts; encouraged the use of ‘income spreading’ as the basis of fraud determinations; encouraged the Agency to change its culture with respect to fraud; suggested problematic changes to the fact-finding questionnaire sent to recipients; built a system that automatically issued determinations based on existing information when the recipient had not yet responded to the questionnaire; and encouraged replacing a system that provided adequate human discretion with one that included far less human oversight. Including individuals from the software developers in the litigation made tracing accountability and authority throughout the procurement and development pipeline a possibility.

At first instance, the agency officials argued that they could not be held responsible for actions performed by an automated system. But this argument was quickly characterised by the court as agency defendants attempting to ‘hide behind MiDAS’. The court noted ‘this argument conveniently ignores the fact that the Individual Agency Defendants implemented and oversaw MiDAS, and prescribed its operation. MiDAS did not create itself. And it did not enforce the false fraud determinations that it automatically rendered – the Individual Agency Defendants did.’Footnote ⁶⁰ However, as the litigation proceeded, agency officials were eventually released from liability. After the 2015 update, the court accepted that MiDAS no longer automatically made fraud determinations, but created fraud flags, meaning there was no deprivation of rights without proper notice, process, or opportunities to appeal.Footnote ⁶¹ In other words, the human in the loop rearrangement assuaged agency liability for the development and deployment of a programme that, like Robodebt, shifted the onus of proof onto claimants in a highly abusive way. Much like Justice Murphy’s finding in the Robodebt class action, the court maintained that this rearrangement did not penetrate the officials’ qualified immunity – the shield to liability for government officials in the performance of their discretionary functions.

A dissenting judgment argued that qualified immunity was never intended for defendants such as these who had masterminded the entire MiDAS programme, including its auto-adjudication functions. That dissent suggested, looking at the specifics of the income averaging process that identified possible fraud and the decision trees associated with the questionnaires provided to claimants, that whatever opportunity to appeal claimants had existed only after their constitutional rights to due process had already been deprived. However, the majority maintained that claimants had the right to challenge the finding of fraud before deprivation occurred, irrespective of the technology used or how abusively it performed the process. There was also no finding of liability against the private firms that provided the technology. They collectively then settled the case,Footnote ⁶² paying out $180,000 to claimants on a $47,000,000 procurement. Michigan now uses a new system developed by Deloitte for identifying welfare fraud.

4.5 Configuring the Human

The human in the loop expresses an unresolvable tension in the control of complex systems. Automation serves to remedy the emotional, unreliable, and feeble human, at the same time as the human serves to corrects clunky, inflexible, uncreative and heartless automation. Similar tensions pervade principles of legality in decision-making vis a vis competing priorities inherent to rule of law thinking, including: universal rules versus individual justice; scale versus inclusion; equality versus discretion; legitimacy versus quality; substance versus procedure, and so on. It is sometimes glossed over that rule of law thinking emerged in opposition to administrative discretion,Footnote ⁶³ despite advocates for the human in the loop frequently associating discretion with individual justice.Footnote ⁶⁴ The point is that the current proclivity for human in the loop ought to be understood as less about preserving unassailable rule of law principles concerning the moral qualities of decision-making officials, but rather as reflecting a political posture, described by Kristen Rundle, suggesting ‘the pendulum has swung too far such that the human being and distinctly human cognitive and moral capacities need to be pushed to the foreground’.Footnote ⁶⁵ Viewed this way, automation does not break legality, it merely rehearses and amplifies the rule of law’s unresolved tensions, which are then papered over by symbolic human oversight rules. Human in the loop becomes a shortcut that institutes the human decision-maker (or supervisor) as an accountable and responsible actor, and situates legal authority in the human inheritance of divine power.Footnote ⁶⁶ Whereas one strand of rule of law thinking demands these qualities to legitimise the exercise of political authority, as noted by Rundle, other strands of rule of law thinking are more concerned with the structure of the relationships between official and decision subject, and the possible exercise of arbitrary power more than the identity of the decision-maker themselves.Footnote ⁶⁷

The political posture favouring human in the loop laws is understandable, because the human in the loop helps smooth over the destabilising effects of technologically assisted decisions. Beyond being a conceptual bottleneck to the appropriate regulation of automated systems,Footnote ⁶⁸ this chapter suggests that underspecifying the contours and features of the human decision-maker does calculated political work. Although identifying non-explicit intentions or institutional logics behind regulatory strategies is always fraught, it is not controversial to conclude that the human in the loop, at least, stabilises a set of governing relations that avoid troubling the status quo of risk profiling by government agencies. A more sceptical reading might suggest the human in the loop enables political operatives to make a double move: on the one hand, pushing responsibility for making political choices into the relation between an authorised decision-maker and the software ‘tool’ they autonomously oversee. Here, accountability becomes a question of the existence or non-existence of the human at the local level – who is the individual responsible for this decision that automation failed to adequately perform? On the other hand, by not prescribing any actual responsibility to the human within the technical apparatus it monitors, it becomes possible to claim the fallible human was bamboozled by automation and responsible for nothing, dissimulating responsibility into the technical system itself and creating ‘orphan decisions’ for which no one is accountable.Footnote ⁶⁹ Either way, by enabling differentiated articulations of human capacities and differentiated operationalisation of associated normativities, the human in the loop strategically elides questions around the different ways humans are threaded into decision systems and attendant attributions of responsibility. It separates the allocation of accountability from the complex distributions of decisional authority that come with the deployment of automated systems, and insists on the existence of human sites of authority and authorship that can be identified and sanctioned. In so doing, it distracts from the need to face the consequences for higher-level political action (decisions about distributions, labour, and intention) or understanding the tools in operation.

A human in the loop needs is only one strategy amongst an emerging mosaic of regulatory tools for managing AI and automation. New instruments like the EU Artificial Intelligence Act include new obligations for the design, development, operation, and monitoring of automated systems.Footnote ⁷⁰ But alongside requirements like transparency and auditing, human oversight remains a fundamental prescription for regulating high-risk systems under those regimes.Footnote ⁷¹ And while emerging requirements will improve the quality and safety of automated systems, they will not alter the human in the loop’s functioning as a versatile legal technology that, on the one hand, enables emerging political and economic formations to coalesce around automation; and on the other, enables complex political action to be expressed through automated decision systems, whose technical rather than political content becomes the focus of regulatory scrutiny.

Rather than rely on a human in the loop, the goal should be understanding the complex redistributions of responsibility that trace the distribution of decision-making elements into automated systems and the authority to specify them, including the influence of companies that consult, design, and build relevant software. These are broader questions of political authority, rather than local concerns about the agentic quality of any particular decision-maker.