1. Introduction
Bureaucracy is both the promise and the peril of modern administration: It promises rules applied consistently, decisions reasoned and recorded, like cases treated alike. Its peril is an iron cage that conceals discretion, diffuses responsibility and leaves subjects bewildered. Automated decision-making (ADM) generally, and ADM using what we call artificial intelligence (AI) in particular, intensifies both. In public administration, AI is often promoted as a way to improve efficiency, reduce error, expand access to justice and even personalize government service delivery. Yet it also risks entrenching new forms of opacity – interfaces that obscure the real sources of decision, and responsibility, and obfuscate established pathways of review and contestation.
Franz Kafka’s parable “Before the Law” captures this danger with the metaphor of a gate that appears open, but in practice bars entry through the inscrutable posture of its keeper. In this paper, I argue that the crucial surface of contestation lies not in abstract principles of AI ethics (like “transparency” or even “explainability”), nor in technical fixes, so much as in the design of the human–machine interactions (HMI) that mediate administrative decision-making and in the organisational structures within which those interactions occur. Lawyers have a distinctive contribution to make at this surface: to articulate how administrative law values (such as reason-giving, accountability and the preservation of human judgment) can be encoded into the various layers of the complex “socio-technical stack” of public administration in a digital society.
My argument unfolds in six moves. First, I introduce the idea that technical mediation can lead to administrative architectures (and “user” experiences) that are a new idiom for pathologies inherent within bureaucratic administration itself. Second, I sketch a short intellectual and institutional history of bureaucracy to locate today’s debates about automation within a longer history of administration and information technology (IT). Third, I engage with the conceptual core of “decision-making” in administrative law – what it means for an official to reach a conclusion based on premises of facts and norms, and give it effect – and show how that concept fares in ADM and AI-driven contexts. Fourth, I describe the technical and socio-technical stack through which automated administration actually operates, from data and models to orchestration, applications, interfaces and the organizational setting that gives those parts force. Fifth, I examine the limits of human-in-the-loop (HITL) as slogan, drawing on human factors and recent policy work to show when nominal oversight fails and where accountable workflows must do the real work. Finally, I draw out some design implications and doctrinal pressure points, indicating how administrative law’s demands can be (re)implemented as operational constraints in AI-mediated administration rather than left as abstract aspirations.
2. Gates and gatekeepers
In Kafka’s famous novella The Trial [Der Prozess] (Kafka, /Reference Kafka and Parry2000), Josef K. is arrested without explanation by unidentified agents for an unnamed (probably nonexistent) crime. Although he is told he is free to go about his life, the court system he confronts sprawls through unlikely locations populated by indifferent officials who offer no clear guidance. He tries desperately to engage with the system as a rational agent. The novel proceeds through episodic encounters, none of which bring clarity. His efforts are met with evasions, referrals and gestures that mask the absence of procedural coherence. Joseph K. is ultimately executed “like a dog” without ever learning what (if anything) he did wrong.
The story presents a system that mimics legality but denies the values and procedural integrity we intuitively expect of a “court”, particularly in terms of its procedural norms. Kafka is a complex writer whose work admits of many interpretations. One strand of the story is the vague but pervasive sense that we are all somehow transgressive, and that everyone is part of the apparatus that judges us. Another, less existential, strand is more obvious to a public lawyer: The Trial is a case study in the perversion of Rechtsstaatlichkeit (roughly: rule of law) in the late-Habsburg bureaucratic state. This is this strand that I wish to use as an in-road to understanding problems of AI in public administration, albeit with a focus on executive government agencies (rather than courts).
As its German title suggests, The Trial is a story of perverted process. It focuses on judicial process in a Civilian jurisdiction, but many of the problems it highlights are generalizable across administration, including by executive officials. Nested within the novella is a vignette that was published independently in another collection of Kafka’s short stories. This is incorporated into the second half of The Trial in the course of a conversation between Josef K. and a priest, who is also, of course, somehow part of the court. “Before the Law” [Vor dem Gesetz] distils Kafka’s critique into a stark allegory.Footnote 1 It commences:
Before the Law stands a doorkeeper. To this doorkeeper there comes a man from the country and prays for admittance to the Law. But the doorkeeper says that he cannot grant admittance at the moment. The man thinks it over and then asks if he will be allowed in later. “It is possible”, says the doorkeeper, “but not at the moment”. Since the gate stands open, as usual, and the doorkeeper steps to one side, the man stoops to peer through the gateway into the interior. Observing that, the doorkeeper laughs and says: “If you are so drawn to it, just try to go in despite my veto. But take note: I am powerful. And I am only the least of the doorkeepers. From hall to hall there is one doorkeeper after another, each more powerful than the last.” (Kafka, Reference Kafka and Parry2000)
The man waits his entire life, submitting to arbitrary delay, focusing only on this gatekeeper and making every attempt to convince (and connive) him to grant access. In true Kafka style, the man dies at the gate, after one final exchange:
“Everyone strives to reach the Law,” says the man, “so how does it happen that for all these many years no one but myself has ever begged for admittance?” The doorkeeper recognizes that the man has reached his end, and, to let his failing senses catch the words, roars in his ear: “No one else could ever be admitted here, since this gate was made only for you. I am now going to shut it.” (Kafka, Reference Kafka and Parry2000)Footnote 2
In the reading I propose, the gatekeeper becomes the algorithmic interface – apparently a neutral, rule-bound conduit – while the man represents the subject seeking recognition or recourse. Intermediation is inevitable in any complex social system, but every intervention we can think of to keep (or make) intermediation healthy and sound is subject to its own pathologies. On the one hand, automation technologies promise efficiency and even personalized access to justice – our very own door to the law, always open. On the other hand, they threaten to leave us at a deserted gate, interacting with a gatekeeper who defies our understanding and seems impervious to human reasons. On a deeper level, the system’s opacity is not merely technical but epistemic and moral: the man cannot know whether his failure stems from his own inadequacy, the logic of the system, or sheer arbitrariness. In this framing, Before the Law serves as a cautionary tale for those designing systems of public administration. The tragedy is not that the law is lost, or invisible, but that it is rendered unreachable by the structures built to facilitate our access to it.
2.1. Bureaucratic rationality and the fabric of law
Understanding the automation of administration requires some framing of “administration,” and its relationship to law. Here, as in prior work, I use three points of reference to do so. For the sake of discussion, I assume a fairly standard administrative state typical of the contemporary British and Commonwealth legal family: a Westminster core with a growing role for executive government agencies, and a broadly Thatcherite and New Public Management overlay whereby many functions of “public administration” are delivered by hybrid bodies or even private contractors. Trying to bring the modern apparatus of administration to conceptual heel is no mean feat, and this potted account cannot provide more than a sketch. (For example, I have left the American New Deal out of my historical gloss for the sake of convenience.) But these outlines are sufficient (and sufficiently accurate) to provide a baseline for my discussion.
At its heart sit the principles of parliamentary sovereignty and broad statutory delegations to ministers and arm’s-length bodies. Administration runs on delegated legislation and regulator rulebooks, perhaps with an increasing scope of non-statutory executive action, policed after the event by courts (under principles like legality, procedural fairness, Wednesbury reasonableness and proportionality where a rights instrument applies) as well as by alternative mechanisms including ombudsmen, tribunals and audit offices. Since the 1980s, the state has been disaggregated into executive agencies and regulators, with outsourcing, procurement and performance contracts replacing in-house delivery. Accountability now moves through budgets, freedom of information regimes, audit and regulator handbooks as much as through classic civil service hierarchies, so ministerial responsibility survives but is thinner. Commonwealth variants add rights charters in Canada and New Zealand, and codified judicial review and merits review in Australia under the Administrative Decisions (Judicial Review) Act 1977 (Cth)and the Administrative Appeals Tribunal. The family resemblance remains: a rule of law bureaucracy delivering at scale through statutes, contracts, data and platforms, with legality negotiated among courts, parliaments and regulators.
2.2. Administration and office
Although the figure of the “official” is sidelined in modern public law theory, the concept of office still grounds actions by employees, sub-contractors and lower civil servants in the authority granted, whether by statute or common law (including prerogative), to some office. In principle, officials are accountable through political and legal channels. The administrative model just described was not conceived immaculately, and while space precludes any comprehensive analysis, a brief description of three main points in its historical development is convenient to underline the point.
My starting point is the emergence, in the nineteenth century, of a rationalized administrative state in Britain: a shift from patronage and ad hoc governance to centralized, rule-bound administration under parliamentary control, supported by specialist bodies and a professional civil service (Chester, Reference Chester1981). This institutional settlement established the core features of modern administration – impersonality, hierarchy and decision-making through rules – that continue to structure public law.
A second strand, most clearly visible in the early Australian colonies, shows that colonial governance was already highly statutory, board-driven and oriented toward large-scale economic coordination through public enterprises and bureaucratic machinery (Finn, Reference Finn1987). This complicates the standard Westminster narrative: administration at scale, grounded in statute and exercised through officials, was a constitutive feature of Anglo-Commonwealth governance.Footnote 3
The state and its administrative apparatus grew rapidly in the post-World War II period. The “welfare state” transformed administration into service delivery on a new scale. Tax-financed social insurance and universal services in health, housing, education, etc., required systems that could process millions of similar claims. The legal architecture shifted accordingly. In short, bureaucracy became a mass production line for entitlements, operating through standard forms, eligibility rules, actuarial logics and routinized discretion. That settlement gave us both the legitimacy story (“social citizenship”) and the operating model (files, forms and targets) that automation would later plug into (Esping-Andersen, Reference Esping-Andersen1990; UK Parliament, 1942).
The third point of reference is a countervailing trend. “Thatcherism” retooled the administrative machine considerably, albeit without abandoning its welfare mission entirely. The provider state was disaggregated into agencies; in many cases, service delivery was contracted-out or exposed to quasi-markets, often utilizing some kind of hybrid entity such as a “government-owned business” or a “quasi-autonomous non-government organization”. Public utilities were transformed from boards and commissions into ordinary, private corporations to harness the putative efficiencies of that form, albeit within incomplete market conditions (i.e., as natural monopolies). In this framework, accountability moved from procedure to metrics, contracts and audits. Civil servants became purchasers, contract managers and regulators. Legally, internal administrative rules ceded ground to public contracts and sectoral regulators; practically, “key performance indicators” and benchmarking redefined what counted as “good administration” (Hood, Reference Hood1991; Pollitt & Bouckaert, Reference Pollitt and Bouckaert2004). Recently, there has been a sober reappraisal of the efficacy of this ideological trend in practice, but its core logic remains widespread and often goes unquestioned (Roland, Reference Roland2008). For present purposes, the logic of standardization and quantification in New Public Management laid perfect rails for things like workflow engines, rules engines and ADM in public administration. In doing so, it also laid the groundwork for pathologies such as over-reliance on proxies, box-ticking “human oversight”, and accountability diffused across vendors and dashboards.
Through all of these changes, the concept of public service as an “office” runs through all forms of administration and provides a basic key for understanding its function and pathologies (Allen, Reference Allen2022b). Administration rests on an implicit claim of authority – for example, to grant (or deny) some entitlement, impose some burden, or exact a punishment (Furedi, Reference Furedi2013). Officials (high and low) are the agents that carry out the work of administration, primarily through the act of decision-making – that is, applying the applicable rule-set to the fact-scenario at hand, such that a license (for example) is granted or refused.
In Max Weber’s typology, the ideal-type of “bureaucratic rule” (Herrschaft der Verwaltung, literally “rulership of administration”) is contrasted with charismatic and traditional rulership. It is anchored in the written word and the writing desk (from French bureau), and is organized through the office [Amt]: a fixed jurisdiction (i.e., a defined scope of responsibility and corresponding power), hierarchical subordination, rule-bound procedure, written files, separation from the person (evident in things like salary, tenure, non-personal “office” resources, etc.), recruitment by technical qualification, and an ethos of fact-oriented professionalism [Sachlichkeit], which connotes impersonal (and therefore impartial) decision-making (Dunleavy et al., Reference Dunleavy, Margetts, Bastow and Tinkler2006, p. 23; Weber, Reference Weber1978).
In German, an “official” is a Beamter, one who occupies an Amt. Accordingly, for Weber, “bureaucracy” is found when the claim of authority is anchored in rational-legal office-holding rather than in personality, property, or charisma. The office defines a competence; rules and written procedures channel how that competence is exercised; decisions are issued “as of right” to those who meet stated criteria, recorded in files and justified by reference to the rule-set without personal animus. Hierarchy, specialization and career officials are the machinery that makes routine application possible.
In this frame, administration is essentially rule-governed decision-making at scale: like cases are treated alike, reasons are set out on the record, predictable outcomes are encouraged (or guaranteed), and control is achieved through regular mechanisms like audit and review. In a bureaucratic order, who the official is should not matter; what the office authorizes; and how the relevant rule is to be applied matters. Although their evolution predates modern bureaucratic administration, it is not inaccurate to say that the mechanisms of administrative law (such as judicial review) are meant to ensure that bureaucratic processes meet the normative demands of administrative law, which rest in an open-textured but coherent set of values and principles.Footnote 4
The promise of this form of rule is impersonality and predictability; the risk is drift into mechanized judgment, goal displacement and the masking of discretion behind forms and checklists. Tellingly, Kafka’s absurdist tale in The Trial is a psychedelic caricature of a bureaucratic state gone wrong. The pathologies of a personal, relationship-based system of justice are combined with the pathologies of the mechanistic bureaucratic system – to yield something much worse than either. This is what explains, in my view, the tragedy in Before the Law. Josef K. is at once trapped within Weber’s “iron cage” of a bureaucratized society and subjected to the personalism and arbitrariness of an amorphous community of officials who animate it, as he would be in a traditional or charismatic system of rule.
2.3. ADM, AI and “digital transformation”
This framing brings into relief what is being “transformed” in “digital transformation” of public administration: the processes of bureaucratic rule-application to cases with consequences for those subject to the law. The mass deployment of IT systems reflects their capacity to support human officials through affordances such as sensing, analytical processing, discriminatory sorting and, increasingly, generation of text-based outputs and execution of tasks by AI “agents.” Automated administration introduces the possibility of intensification, at industrial scale and velocity, of bureaucracy’s core traits of standardization, record-based reasoning and routinized decision-making, with new possibilities for intermediation and complex psychological dynamics in which officials and subjects interact with AI and ADM systems.
Inter alia, this means that different types of humans have become involved in public administration in new roles. As Zouridis, van Eck and Bovens explain, using the classical dichotomy between “system-level” and “street-level” bureaucrats:
The use of information technology (IT) has caused the discretionary freedom within large-scale public executive organizations to shift from professional case managers to programmers and data analysts. This is particularly true of the “decision factories”, those large-scale bureaucracies that routinely make decisions on social benefits, licences, tax returns, fines, subsidies and, to an increasing extent, permits. The same shift also seems to occur in law enforcement organizations that apply data science to optimize their resources and intervention strategies. Whereas previously, these organizations employed massive amounts of “street-level bureaucrats”, today these organizations are more properly understood as “system-level bureaucracies”. In these “system-level bureaucracies”, the basic principle of “unit production”, which involved human judgement for each individual case, has been replaced by “continuous process production”. Decision making by automated decision systems based on algorithms has pushed aside human judgements based on rules of thumb. In colloquial terms: it is the computer that says “yes” or “no”. (Evans & Hupe, Reference Evans and Hupe2020, p. 313)
“Computer Says No” is a recurring sketch from the controversial comedy series Little Britain. It has entered popular idiom as a shorthand for institutional unreason, especially in contexts where IT systems are used to justify arbitrary or obstructive decisions. (In each sketch, a customer makes a request or encounters a problem with an administrator named Carol Beer, who consults her computer. After a moment of typing and an expressionless pause, Beer responds with the deadpan phrase.) This is a satirical indictment of systems where digital decision-making blocks healthy human discretion, leading to Kafkaesque outcomes cloaked in technological inevitability. The gatekeeper of some social good (paradigmatically, an official) is limited in their scope of action by an IT system. They might, for example, lack a field to enter some relevant data or lack the admin privileges to override a feature that creates a perverse outcome.
To understand how bureaucratic administration has changed through the use of digital technologies, we can draw a distinction between three ideas. Digitization involves format conversion – i.e., turning analog records into bits, creating a bureaucracy that still assumes the workflows of documents moving between human hands and before human eyes. Digitalization involves the reengineering of routines around networked digital infrastructures (i.e., workflows, platforms, data standards) so that the underlying processes themselves actually change. In the last decade, a more thorough conception of digital transformation has emerged. As Mergel et al. observe, where previous waves of digitization focused on the transition from analog formats to parallel digital services to increase efficiency and effectiveness, digital transformation aims to redesign and reengineer government services more fundamentally (Brennen & Kreiss, Reference Brennen, Kreiss, Jensen, Rothenbuhler and Pooley2016; Dunleavy et al., Reference Dunleavy, Margetts, Bastow and Tinkler2006; Mergel et al., Reference Mergel, Edelmann and Haug2019; Mergel et al., Reference Mergel, Kattel, Lember and McBride2018; Tilson et al., Reference Tilson, Lyytinen and Sørensen2010; Vial, Reference Vial2019).
The next step is to understand the types of systems that might be deployed in a digital transformation agenda. The current focus is on AI-based ADM, but this is a movable feast.Footnote 5 As Floridi and Cowls observe, “AI” comprises several branches of technology that each attempt, in some way, to meet the classical Dartmouth Conference definition of “making a machine behave in ways that would be called intelligent if a human were so behaving” (McCarthy et al., Reference McCarthy, Minsky, Rochester and Shannon1955, as cited in Floridi & Cowls, Reference Floridi and Cowls2019). Currently, the focus is squarely on machine learning (ML) systems, and neural networks and language models in particular. Earlier, key debates on AI-concerned systems that many would today call software. For this reason, without papering over the crucial differences between rules-driven systems and data-driven systems, it is helpful to keep the broader trend in view rather than the specific technology, or set of technologies, driving that trend. In the early 1990s, Greenleaf et al. used “computerization of law” to embrace both (i) the “computerization” of legal source materials (e.g., statutes, judgments and policies) and (ii) the “computerization” of the various aspects of legal practice itself (Greenleaf et al., Reference Greenleaf, Mowbray and Tyree1991, p. 219). In the same way, we should keep our eye on the cluster of technologies currently being deployed to automate the bureaucratic processes – and decision-making in particular.
Although a few years old, the 2018 “AI Knowledge Map” presented by Francesco Corea still provides a useful visual overview (Corea, Reference Corea and Corea2019). This is a two-axis schema that separates what a system tries to do from how it tries to do it. The horizontal axis lays out AI paradigms, from symbolic and rule-based methods, through statistical and probabilistic learning, to sub-symbolic approaches like deep learning, reinforcement learning, evolutionary search and multiagent systems. The vertical axis lists problem domains that range from perception, through reasoning and knowledge, to planning and communication. There are at least three benefits to plotting concrete families onto this matrix – e.g., expert systems, decision networks, probabilistic programming, neural networks and generative models, reinforcement learning, robotic process automation and affective computing. First, tasks and techniques are separable: more than one paradigm can be applied in a given problem domain. Second, dominant clusters (and their limits) become visible, which helps explain strengths and failure modes given the current (evolving) state of the art. Third, gaps and overlaps stand out, so we can locate a real application on the matrix instead of calling everything “AI.” For public administration, this might help map a project’s tasks and approach, choose tools purposefully and anticipate governance issues.
Another important dimension of analysis looks at the level of autonomy of a system – its ability to complete some task without human input – in relation to its other technical features, in particular its opacity. Experience in earlier generations of automation points to the importance of deploying AI systems as “components of broader systems in which humans are positioned not only to bear liability but to add natural intelligence” (Allen, Reference Allen and Kerrigan2022a, p. 151). It is meaningless to ponder the problems of opacity without considering the manner of deployment. An opaque tool used by an expert under close, well-considered supervision is categorically different from the use of such a system by a lay end-user. The tendency for one community to focus on the technical artifact, and for another to focus on the articulation of abstract governance principles, misses the really important surface: the mode and context of interaction.
This brings us to a fundamental point: digital transformation is a matter of humans using new tools to do their work in new ways. In a recent survey of local government authorities, for example, Vogl et al. observed instances of straightforward automation, but they also observed new relationships between public administrators and their AI systems. They described the result as “algorithmic bureaucracy”, stressing the interaction between offices of public administration (street-level and system-level) and computational algorithms (Vogl et al., Reference Vogl, Seidelin, Ganesh and Bright2020, p. 946). Two key insights from Dunleavy et al. are convenient to repeat here. First, bureaucracy as such has co-evolved with IT. The organization of information processing has always been key to bureaucratization, in the sense of rationalization of human conduct. As such, the practices of bureaucracy change as the technological state of the art changes. In the 18th and 19th centuries, state-of-the-art IT comprised fiber-based paper, double-entry bookkeeping and printing. In the 20th century, photocopiers, fax and telephones, and the first generation of Internet affordances such as email were introduced. Today, it comprises ubiquitous networked computers with “intelligent” capabilities. Second, and even more importantly, bureaucracies are socio-technical systems, that is, systems with social and technical components. The interactions between those components are just as important as the artifacts that make up the technical layer of bureaucracy or the agents that make up its social layer (Albrow, Reference Albrow1970, p. 28; Dunleavy et al., Reference Dunleavy, Margetts, Bastow and Tinkler2006, p. 23).
2.4. Law and/as: text, code, data
Bureaucratic administration, I have said, involves the rational application of rules by officials acting impersonally to individual cases at scale, using some form of IT. The law’s relationship to language is significant because we use language (broadly defined) to encode rules and information. For this reason, most bureaucratic processes are fundamentally linguistic – and, in particular, documentary. Law’s relationship to language helps understand how the computational processing of language impacts the administration of the law by officials.
Here, I follow the typology presented by Mireille Hildebrandt of text-driven, code-driven and data-driven law. These are different modes of “doing” law that harness the affordances of different IT systems (Hildebrandt, Reference Hildebrandt2016, Reference Hildebrandt2018, Reference Hildebrandt, Deakin and Markou2020a, Reference Hildebrandt2020b). Law as text is most familiar. The rules that govern bureaucratic administration are encoded in human language – viz., that version of natural language sometimes referred to as “Legalese”.Footnote 6 Human languages comprise semantic content (i.e., the meaning of words), syntax (i.e., structures within the grammar of the language) and pragmatics (i.e., semantics interpreted in context). Rules of syntax not only constrain how one uses the relevant language but embody the logic that is implicit within the language. (Syntax relating to case, for example, expresses logical relations between subject, object and verb; syntax relating to tense expresses logical relations between past, present, future and possible but uncertain, e.g., subjunctive, states of the world.) Every natural language has its own grammar, but there are regularities even across language families that reflect some observable truths about the world we live in (Allen, Reference Allen2017, pp. 402, 404).
Language allows us to express propositions (such as legislative provisions) with normative content that comprise “the law”. These declarations make up a system of rules intended to change the way that human agents behave (typically in the future), and provide predictable answers to problems that arise over time. Harnessing the implicit logic of language is one of the most important parts of the legal craft. It is also the core task of the “bureaucrat” in the Weberian model: applying law to facts through processes such as “subsumption” (loosely: syllogistic logic) or analogical reasoning.
We can now move from law as text to code-driven law. It is not accidental that code denotes a systematically organized corpus of legal rules. Code in the modern sense is a kind of text with properties of heightened formalism both in syntax and semantics. Coders generally tend to strive for mathematical levels of precision in their code-based “expression”, because that code is also (through one or more layers of compilation) translated into the pure (binary) logic of machine code. That is, the language can become a medium of execution in a computer system as well as the mode of expression for some idea (in the sense of reflecting the intention of its coder or the normative content of the natural language text it is based on). Even when we code in a language like Python (which looks funny, but is still comprehensible as “language” to a human), ultimately that must correlate predictably to an on/off state in a silicon chip. This is a question not of “meaning” but of electrons moving across silicon in space. Perhaps for this reason, coders tend to ignore (or intentionally try to exclude) pragmatics, which is so important in giving meaning to utterances in natural human languages.
The transition from text-driven to code-driven law can fail because coders make mistakes in their translation of (natural language) text-based law into code. Where things go wrong, it will often be because coders try to “flatten” the semantic space captured by the natural language into something more formally constrained, in order to promote properties like predictability and certainty. This process itself is problematic, as Lawrence Lessig’s early work makes clear, because coders encode their interpretation of what the natural language text (e.g., legislation) means (Lessig, Reference Lessig2000). They are often grossly unqualified to do this, and may have naïve and unsophisticated notions of what “the law” is and how it works, as well as lacking any relevant domain expertise in terms of the area of application. This basic danger is enhanced by the possibility of failure in the various layers of the compilation process, and also by the attempt to expunge the “encoded law” of pragmatics.
We can now move to data-driven law, which replaces ex ante specification with ex post inference based on historical data: instead of writing down the rule, we train a model on observed pairs of inputs and outcomes and then apply its learned associations to new cases. The attraction is obvious. These systems travel well across messy, high-volume domains; they adapt, and they can surface regularities that no drafter anticipated. Their affordances are statistical rather than logical: they interpolate, they generalize, they rank and score. But those same affordances come with inherent limitations such as the inherent risk of “hallucination”. On a more basic level still, they come with a change in how “reasons” and “meaning” are produced (or simulated). Where text invites reasons in the language of rule application, and code at least purports to implement that language, data-driven systems tend to substitute a confidence score or a feature weight for a reason. The result is a persistent tension between statistical association and normative justification.
Of course, text-, code- and data-driven law interact; many ADM systems, for example, will combine code-driven software with some data-driven components (Hildebrandt, Reference Hildebrandt, Deakin and Markou2020a). This matters for administration because any automated legal process carries an implicit claim: namely, that the progression from facts to outcome is governed by norms, that like cases are treated alike for reasons that can be stated, and that an affected person can contest both facts and norms against a stable record. These implicit claims are reflections of the basic values underlying classical mechanisms of administrative law, and many of them have constitutional importance. A data-driven pipeline can only satisfy those claims if we design for it intentionally. That will require a different and likely evolving set of methods (in the technical layers) and practices (in the social layers). For example, that might mean defining the goal, admissible inputs, thresholds at which action is triggered, and escalation paths for hard cases. It might mean fixing decisions in time with a record of data, retrievals, prompts where applicable, model identity and version, parameters, and any human interventions. And it might mean a system that interacts with its user such that the user produces valid reasons along the way, so that a reviewer can trace how the legal conclusion followed from the inputs under the applicable framework without attributing normative force to the operation of a statistical engine.
Failure looks different across the three modes of doing law. In text-driven systems, the danger is indeterminacy and the quiet expansion of discretion off the page. In code-driven systems, the danger is over-specification: the flattening of legal meaning to what the programmer thought the words must mean, with context squeezed out. In data-driven systems, the danger is under-specification: targets that proxy the wrong thing, labels that embed past error, shifts in population or policy that make last year’s regularities misleading, and an explanatory surface that offers importance weights where the law demands reasons. Code- and data-driven systems also present novel problems of automaticity and cadence that can render systems inhuman, in a very real sense.
All three types of automation can be lawful and effective – but only if their characteristic errors are anticipated and controlled. The paradox, then, is not that rigid rule-engines or statistical tools are used to administer normative decisions, but that we expect them to deliver determinate, auditable, and legitimate (or authoritative) outcomes without giving adequate thought to the legal-institutional “wrapper” that makes that possible. The reconciliation is thus institutional rather than technical: we must bind models to policy through documented targets and thresholds; require decision artifacts that expose the path from input to output; reserve space for principled overrides and record their use; subject outcomes to monitoring that feeds back into both model and procedure; and ensure that contestation routes are real, with access to the record.
Seen this way, “Responsible AI” (RAI) is a design brief for a socio-technical system in which legal demands are performed by the ensemble: by texts that state aims and limits, by code that implements what can be formalized, by models that supply disciplined predictions where rules run out, and by institutions that keep the whole aligned with legality in practice. That, rather than a taste for any particular technique, is what will make automated administration fit for deployment in a rule of law state.
3. Beyond RAI and HITL as slogans
Technological questions easily predominate discussions of digital transformation. Issues are raised by the novel functionalities and affordancesFootnote 7 of a new technology, so the first step is to understand what the technology is and what it does. For this reason, however, discussions of RAI can quickly take one of two directions: they can get lost in the cloud-land of “ethics” and “principles,” or they can get lost in a quest for engineering solutions that is, ultimately, doomed.
From the lawyer’s perspective, neither of these trajectories is satisfactory. As Florian Gamper makes clear in his contribution to this Special Issue, lawyers have neither the luxury of navel-gazing – law needs to deliver an answer, even an imperfect one, that will impact real people’s real lives – nor the credulity to believe that problems with complex socio-technical systems will ever find a purely technical fix. What gets obscured in the middle is, too often, the crucial surface of interaction between humans (typically, officials and subjects) and the machines they use. There is presently insufficient (or ineffective) effort to tie features of the technical stack to the specific governance issues those features are meant to address. My suspicion is that technical features are designed with some engineering novelty in mind rather than with a clear understanding of the messy governance problem that the tool is meant to address. The “interface” layers between the technical and social components of the stack are often overlooked.
3.1. Guns don’t kill people
Much AI governance literature is concerned with the HITL principle. This seems to have arisen in Cold War and post-Cold War defense automation debates before entering the AI governance discourse from the 1990s (Sheridan, Reference Sheridan1992).Footnote 8 While it is still most prominent in those areas where machine action has serious and/or irreversible consequences,Footnote 9 it has become a principle of good AI system design of general applicability. As public administration touches many of our most important interests, up to and including vital civic, political and economic interests and indeed human rights, HITL would seem prima facie to be a necessary condition for “RAI in public administration” – especially where automation involves systems with any degree of opacity or unpredictability.
Despite the narratives of “augmentation” and HITL, many technical systems are still engineered with end-to-end automation assumed. This is partly true of the Dartmouth conception of AI itself: AI is, definitionally, a system that simulates what human cognition can do, such that those designing AI systems have their eye on the goal of mimicking ever more challenging human outputs rather than automating tasks that humans are less good at. The problem with this is twofold. First, it makes the engineering design brief impossibly hard. Second, it excludes the things that humans are naturally good at from the design of the socio-technical system that will actually emerge once the technical artifact is deployed. Take a large language model (LLM)-driven Chabot and information tool, for example. If we want to make it 100 percent accurate, we have to solve the “hallucination” problem, which not even be possible. And then we do not think about designing LLMs and their user interfaces (UI) in a way that switches the human user’s brain on, rather than off, despite our knowledge of cognitive offloading and automation bias. On the other hand, if we accept that LLMs are inherently unreliable, but very good at generating some types of text very quickly given the right inputs, the question turns to the manner of deployment, including UI design and the profile of the user. Sometimes, inherent “unsafety” is a feature of a useful technology that is mitigated by insisting on experienced and qualified operators. We do not hand them out in schoolyards or prisons, but we want surgeons to have sharp scalpels and soldiers to have guns that kill. It is crucial to consider the actual use-case, as a more granular instance of one of the problem domains, as well as the actual features of the application layer of the AI system itself – including, in particular, its UI design.
In most public deployments, the system is not a monolith but a set of modules that work as a stack. The various layers of that stack interact with the system’s human elements, too. Mindful of my technical limitations, I will sketch my understanding of this “socio-technical stack”. At the base sit the data assets and records infrastructure. (For example, a curated, structured and metadata-enriched corpus of legislative and case law text is an infrastructural layer that does not currently exist.) Above that might sit a model layer, which in different projects may be a rules engine, a scoring model, or a generative model. Between data and the model is an orchestration layer that handles retrieval, prompts and glue logic. On top of the model sits the application layer, where business rules, case routing and guardrails are implemented, followed by the interface through which officials actually work. Alongside all of this, we might find an audit and logging layer that fixes each decision in time.
The parts are, in principle, substitutable – some of them might be designed in-house, others bought off the shelf, others designed bespoke. Schemas, interfaces and vendor choices create path-dependence, such that governance has to be specified at the seams between modules, as well, potentially across modules from different developers with different governance assumptions. How are data validated and retained? How are models evaluated, versioned and rolled back? How are prompts, retrievals and parameters frozen for a decision and placed on the record? How are overrides enabled and counted? What do contracts and terms of use require by way of disclosure, evaluation artifacts and incident response? Those with experience of IT in an organization of any scale and complexity know just how unwieldy things can become, and it is foolhardy to suppose that AI-driven ADM systems in the public sector are any different. An important side-note here is the importance of policy considerations at the procurement phase, because many value-relevant assumptions are embedded in the technology procured (Mulligan & Bamberger, Reference Mulligan and Bamberger2019).
Different technical configurations can relocate risks. Retrieval-augmented generation systems make indexing, for example, and corpus curation crucial to accuracy. Rules combined with risk scores put thresholds and calibration at the center. More agentic tools make tool access, rate limits, fine-grained logging and HITL/kill-switch functions more important. The point is not to find a technical “silver bullet” but to specify the operating conditions under which an administrative decision can be produced by the configuration chosen in conformity with the requirements of administrative law.
Bureaucratic systems are, at base, socio-technical systems. Around the technical artifact sit mandate and policy goals, organizational structure and incentives, procurement and vendor management, training and standard operating procedures, workflow and interface design, internal audit and ethics review, and the external controls of tribunals, ombudsmen, auditors and courts. This is the terrain described by the socio-technical literature, and it is where many failures originate: not in the model alone but in the way roles, time pressures, dashboards and incentives shape judgment. Put simply, one might say that our goal is to “tune” the social layers to compensate for the limits of the technical layers, and vice versa. In this exercise, UI design (and the HMI assumptions behind it) keeps the two aspects glued together. As George Tian et al. write:
The inherent effect of employing ADM systems, and the one which gives rise to its most essential legal consequences, is that it turns a purely human decision-making enterprise into a sociotechnical decision-making system; that is, a system comprising both human beings and technological artifacts and in which “humans” and “machines” both contribute to decision outcomes. (Tian, McFarland & Guo, Reference Tian, McFarland and Guo2025, p. 277)
3.2. Keeping human brains in the loop
As we have seen, the moment of decision-making by an official is conceptually central to the idea of bureaucratic administration. Administrative law’s core commitments must be implemented effectively as design constraints in the socio-technical systems in which decisions are made. One of the main ways to achieve this is to keep humans involved in the process that leads to a “decision”. The mere presence of a human operator does not prevent maladministration. Getting HITL design right requires careful thought. The “Computer Says ‘No’” problem provides just one example: a human official interacts with the subject, but the official’s hands are tied by the architecture of the computer system she is using. She is given no override button to intervene – even where the computer system yields a perverse result.
How, then, does administrative law define a “decision”, and how does official use of AI systems affect matters?
Efforts by Australian courts to grapple with this question provide an illustration. In Semunigus v Minister for Immigration and Multicultural Affairs,Footnote 10 Finn J stated:
For present purposes I am prepared to hold that the making of a decision involves both reaching a conclusion on a matter as a result of a mental process having been engaged in and translating that conclusion into a decision by an overt act of such character as, in the circumstances, gives finality to the conclusion – as precludes the conclusion being revisited by the decision-maker at his or her option before the decision is to be regarded as final.
On this view, a “decision” is a (i) conclusion (ii) reached by a human mind, followed by (iii) an overt act that gives it effect, such as communication. Although the definition was proffered in a specific factual and statutory context, this has (in my view rightly) been taken as fundamental.Footnote 11
This articulation runs through subsequent Australian authorities, but it raises some difficult questions. The Full Federal Court majority applied the Semunigus formulation in Pintarich v Deputy Commissioner of Taxation.Footnote 12 That case concerned a computer-generated “template bulk issue letter” that was sent without the relevant Australian Taxation Office official having actually formed the conclusion stated in the letter. The Full Federal Court held that this was not a “decision” for the purposes of judicial review in Australian law. While Pintarich advances the attractive proposition that only human mental processes can ever give rise to a “decision”, this test can boomerang: in end-to-end automated settings, it risks producing a gap where judicial review is most needed, because no human mental process can be identified to satisfy the threshold (Tian et al., Reference Tian, McFarland and Guo2025).
Kerr J’s dissent in Pintarich pressed this point, urging a focus on the objective manifestation in context rather than on locating a subjective mental state. The majority’s approach might create perverse incentives to automate away reviewability. On the other hand, however, to allow an end-to-end computer process to count as a “decision” would erode an important fundamental principle that keeps bureaucratic decision-making “human”. This puts us on the horns of a dilemma.
The basic point, however, is structural: the decision comes first (a mental process), the declarative instrument later (communication). If we reverse the order, the official is not “deciding”; they are auto-notifying the subject of a legal state of affairs that may not actually exist. This must be avoided if we expect AI to play a significant role in the administration of a bureaucratic state that respects the rule of law. Deeming provisions illustrate the same tension. Section 495A of the Australian Migration Act 1955 (Cth), for example, authorizes the minister to arrange for computer programs to make decisions under the minister’s control and then deems such outputs to be the minister’s decisions. Such a statutory attribution regime does not, by itself, establish that any human actually engaged with the case as a decision-maker in the sense contemplated by administrative law: it is a legal attribution device, not a metaphysical account of who actually reasoned about the case. If the system runs end-to-end and the official’s role is perfunctory, the legal fiction may say “minister” but the administrative reality is that the “decision” (or apparent decision) is just a machine output. That is why the design and use of the system – including the human–machine interface and the reality of oversight – remain the critical objects of analysis.
3.3. From HITL-as-slogan to accountable workflows
Ben Green has identified two basic flaws common in policies requiring HITL. First, evidence suggests that humans are actually unable to perform the desired oversight functions: policy drafters overestimate what busy officials can actually scrutinize in context. Second, as a result, HITL policies serve to legitimize deployments of faulty and controversial algorithms. In effect, such policies “launder” accountability for systems that perhaps ought not to be deployed in their current form at all. Thus, HITL policies can provide a false sense of security and enable vendors and agencies to shirk accountability for algorithmic harms. Green’s suggestion is not to abandon human roles or even to minimize HITL as a design principle, but to relocate the center of gravity toward institutional oversight that tests whether a proposed human role can work in practice (Green, Reference Green2022, p. 105681).
The “human factors” literature helps to explain why the HITL slogan underdelivers. Studies on automation bias document a predictable tilt toward deferring to system outputs, even when contradictory cues are present; this effect shows up across domains and is strengthened by workload, multi-tasking and the complexity of verifying automated recommendations. In a classical study from the late 1990s, Skitka et al. established the basic pattern: participants with an automated decision aid made both errors of omission (i.e., missed events when not explicitly prompted about them by the aid) and of commission (i.e., did what an automated aid recommended, even when it contradicted their training and other valid and available indicators) (Skitka et al., Reference Skitka, Mosier and Burdick1999, p. 991). A more recent review by Alon-Barkat and Busuioc investigated automation bias and selective adherence (i.e., selective adoption of algorithmic advice when the advice corresponds to stereotypes) in public sector applications specifically (Alon-Barkat & Busuioc, Reference Alon-Barkat and Busuioc2023, p. 153).
Closely related are bureaucratic forms of deference to systems; where the interface frames the recommendation as default-correct, officials rationally conserve attention and adopt it. Experimental work in public administration also finds that explanations and transparency shape perceived trust in automated decisions without necessarily improving judgment. Such effects are well-documented in domains that have informed public-sector interfaces. For example, another early study found that vigilance drops when verification is costly; mental workload that is too high invites errors, but so does underload when routine monitoring dulls attention (Molloy & Parasuraman, Reference Molloy and Parasuraman1996, p. 311). The takeaway is that oversight roles that presume spare cognitive bandwidth, or that bury the effort of checking behind multiple screens and clicks, will not perform as anticipated.
Empirically, mixed performance in human–AI combinations reinforces the point that design and workflow placement matter. A recent meta-analysis of more than a hundred studies found that, on average, human–AI combinations performed significantly worse than the best humans or AI systems working alone (Vaccaro et al. Reference Vaccaro, Almaatouq and Malone2024). While human–AI combinations produced gains in tasks that involved content generation, decision-making tasks showed significant losses (Mosqueira-Rey et al., Reference Mosqueira-Rey, Hernández-Pereira, Alonso-Ríos, Bobes-Bascarán and Fernández-Leal2023, p. 3046). Likewise, in a recent LLM-based medical diagnostic application, McDuff et al. found that AI could assist humans to make better decisions given the right interface – but that the AI systems alone outperformed the human–AI combination (McDuff et al., Reference McDuff, Schaekermann, Tu, Palepu, Wang, Garrison, Singhal, Sharma, Azizi, Kulkarni, Hou and Natarajan2025, p. 451). The lesson for administration would seem to be that gains can appear where the collaboration is engineered for the task, and cannot be expected where a tool is just dropped into a busy workflow. In the worst case, generic end-to-end systems are deployed that are grossly unsuited to the task at hand; the human “in the loop” serves as a rubber stamp (and liability sponge if things go wrong), but could actually make the automated system worse in substantive terms.
This underscores the further point that we need to design tools with the needs of administration in mind. That means ensuring that those designing and developing systems – and their mode of deployment in bureaucratic organizational settings – have an adequate understanding of administrative law values and the way those values might be embodied in the system itself. Again, HMI assumptions and UI design are crucial. In a relatively recent study, Veale et al. found a “disconnect” between organizational realities and the desiderata expressed in the literature. In response, they identified a few design opportunities, such as building usable transparency tools (Veale et al., Reference Veale, Van Kleek and Binns2018). Design practitioners are not completely naïve, let alone uninterested, in values like fairness and other ethical issues. However, “issues like fairness have been shown to come with technically difficult to reconcile, or even irreconcilable trade-offs” (Veale et al., Reference Veale, Van Kleek and Binns2018, p. 10). This frames their call for interdisciplinary collaboration. Domain-specific, organizational and contextual factors are crucial to improve the fairness and accountability of algorithmic decision-support. Only so much can be learned from studying systems in the abstract; real-world institutional constraints, high stakes and crossed lines of accountability all bear on the task. As a result, those interested in fair and accountable ML must study these processes in the messy, socio-technical contexts in which they inevitably exist, and must design for those deployment contexts. The real world, they continue, is characterized by institutional factors, political winds, technical lock-in and often old, withering infrastructure. They conclude with a call for HMI-focused work:
Rarely have the issues [HMI] concerns itself with been as directly involved in steering choices related to the use of governmental power as much as they are today. As we involve more advanced decision-support, and even decision-making, systems in the workings of the state, this field might even be the “difference that makes a difference” to the rights and freedoms of vulnerable societal groups. We believe that making this difference is possible, but only in close collaboration with different disciplines, practitioners and affected stakeholders. Future research must engage with not only with the new questions and avenues of exploration such research brings, but also the practical constraints that come with studying politically charged settings and developing workable social and technical improvements within them. (Veale et al., Reference Veale, Van Kleek and Binns2018, p. 10)
In a recent review of the HITL state of the art, Mosqueira-Rey et al. offer a map of what HITL actually means. They distinguish modes according to who holds control over the learning process: active learning with the system querying a human labeler, interactive ML with tighter back-and-forth between user and model, and ML where the human designs the concept-space and training curriculum (Mosqueira-Rey et al., Reference Mosqueira-Rey, Hernández-Pereira, Alonso-Ríos, Bobes-Bascarán and Fernández-Leal2023, p. 3006). Usefully, they also situate related strands like curriculum learning and “explainable AI” within this control frame. In short, human involvement is not a checkbox but a design choice that carries different information needs and governance implications. The survey concludes with three trends to watch in current developments. The first is a trend toward increasing importance of interactivity: as we move toward more human control, we move toward greater levels of interaction. Second, however, there is a trend toward the automation of the majority of tasks where ML is being applied (Mosqueira-Rey et al., Reference Mosqueira-Rey, Hernández-Pereira, Alonso-Ríos, Bobes-Bascarán and Fernández-Leal2023, p. 3045). Finally, there is a trend toward focusing on the domain-specific problem and not so much on the technique needed to implement it. In closing, they note a general shift toward what “human-centered” AI, which they welcome:
[T]he first two waves of AI failed not only due to the lack of mature technologies, but also because they left human needs unsatisfied. AI is starting to satisfy them and provide a positive user experience (UX) for a variety of application scenarios in the third wave. It is also starting to deliver mature business models with useful AI in which people started to consider the inclusion of human aspects such as ethics, interpretability, fairness, etc. Thus, the third wave of AI can be characterized by its technological improvement but also by its human-centered approach. (Mosqueira-Rey et al., Reference Mosqueira-Rey, Hernández-Pereira, Alonso-Ríos, Bobes-Bascarán and Fernández-Leal2023, p. 3046)Footnote 13
Taken together, current HITL trends point in a useful but legally awkward direction. More interactivity turns officials into teachers and quiet co-designers of systems, which blurs the line between applying policy and making it. More automation of the pipeline through cloud services and “machine learning operations”Footnote 14 improves reproducibility on paper, yet shifts decision boundaries in the background and scatters the artifacts needed for reasons and record-keeping. More human-centered, machine-teaching approaches promise better alignment with domain expertise, but risk hard-coding tacit heuristics and local workarounds without the usual procedural safeguards. Layered onto this are the familiar human-factors effects: automation bias, bureaucratic deference to default recommendations, and cognitive overload that makes independent verification costly. In administrative settings, the center of gravity therefore lies not inside the model but around it. The practical work includes inter alia specifying roles and information surfaces, fixing decisions in time with a stable record, and designing interfaces that support accountable reasoning rather than routinized adoption. When those conditions are missing, the HITL slogan tends to be empty.
4. Conclusion
In a public lecture for this project, an audience member asked the panel whether administrative law itself must change as the modalities of administration change. It is a fair question. But it tends to invite a reflex of either fatalism or denial. Yes, digital systems interpose models, datasets and interfaces between facts and outcomes, and that tests the existing setup of administrative law. But this is not because administrative law values are obsolete, but because they must now operate through an increasingly complex socio-technical stack. My discussion has attempted to trace that stack from earlier bureaucratic forms and earlier generations of IT, and to show why the legal concept of an official decision – a human judgment followed by an act – remains the conceptual hinge today. Modern administration is a machinery of routine, impersonality and files; Weber’s account explains why office, competence and rule-governed procedure remain the grammar of “lawful” administration. Against that background, automation is really just an intensification of bureaucracy’s core traits.
Problems arise where any given tool (or set of tools) threatens to displace the human judgment the law presupposes, leavingthe appearance of legality. The Australian cases Semunigus and Pintarich illustrate both edges of the doctrine: demanding a genuine (human) mental process risks creating a reviewability gap where end-to-end systems are concerned; relaxing that demand risks dressing up notifications as “decisions”. Delegation and deeming provisions solve attribution on formally but not substantively, doing little to address the basic question of whether anyone has actually reasoned about the case.
One locus of reform is UI design: what the official sees and when, how disagreement with a recommendation is enabled and logged, what artifacts fix the path from inputs and parameters to outcome, and how those artifacts travel to internal reviewers, in the first instance, and thence to tribunals and courts. Administrative law perennials such as natural justice, reason-giving and institutional accountability remain crucial; the task is to (re)implement them as operational constraints in systems that are inherently socio-technical. That means carrying the concept of an official decision – a human judgment followed by an act – through the stack; fixing each case to a stable record of inputs, model states and human interventions; and designing workflows and interfaces that support accountable reasoning rather than routinized adoption. On this view, the architecture of administrative law still provides a stable frame. But no slogan – neither RAI nor HITL – will prevent a Kafkaesque “algorithmic state”. We need systems that are meaningfully designed to support accountable judgment. Human factors research tells us why: under real workloads, officials defer to defaults, suffer automation bias and lose vigilance when verification is costly; conversely, when systems outperform humans, hybrid arrangements can still underperform unless roles and information surfaces are engineered with the task in view.
Kafka’s parable ends with the gatekeeper revealing that the entrance to the Law was made only for the man who waited before it all his life. The tragic irony of the story lies in the structure of mediation that made the law inaccessible while preserving the appearance that it remained open. Automated administration risks producing similar architectures. Digital systems can replicate the outward forms of legality – files, notifications, explanations and review procedures – while the actual reasoning that generates outcomes remains obscure. The challenge for public law is therefore not to resist technological mediation as such. Bureaucracy has always been a technology of rule, in which the technology of record-keeping and documentation is structural. The task is to ensure that the architectures through which administrative decisions are produced preserve the conditions under which law can meaningfully be applied, understood and contested. If administrative systems are designed in this way, the gate to the law may remain mediated, but it will not become illusory.
Acknowledgements
This research was supported by the Singapore Ministry of Education Tier 1 (Category C) Grant 23-SOL-SMU-004, “Operationalising ‘Responsible Artificial Intelligence’ (RAI) in Public Administration”. The substance of this paper was presented at project workshops held at the Asian Law Schools Association Law & Technology Chapter Annual Conference at UNSW (Sydney) on 11 July 2025 and at a workshop convened at SMU Yong Pung How School of Law (Singapore) on 7 August 2025. I would like to thank participants at those events for their feedback. Thanks also to my co-editors Florian Gamper and David Lo, to Brian Lim for our various conversations on this topic, and to the Journal’s anonymous reviewers for their comments. All errors remain my own.
Author contributions
CRediT Taxonomy Jason Allen Conceptualization-Equal, Data curation-Equal, Formal analysis-Equal, Funding acquisition-Equal, Investigation-Equal, Methodology-Equal, Project administration-Equal, Writing - original draft-Equal, Writing - review & editing-Equal.
Open access
