I. Introduction
We do not ride on the railroad; it rides upon us. Footnote 1
Lawmakers the world over are grappling with what it means to regulate advanced artificial intelligence (“AI”), particularly the burgeoning generation of large language models (“LLMs”) and the general-purpose capabilities they offer. The scale and ambition of frontier AI development have expanded rapidly, commensurate with a growing scholarly consensus that LLMs represent a general-purpose technology whose impacts might shape society on a scale akin to the steam engine or electricity.Footnote 2 Alive to the existing and future risks raised by frontier AI, several jurisdictions have established new institutions directed at the specific challenges of general-purpose systems. Examples of this include the European AI Office in the European Union (“EU”), the UK AI Security Institute and the US Center for AI Standards and Innovation.Footnote 3 These nascent institutions face a litany of well-documented obstacles, from old-fashioned industry lobbying to domestic politicians focused on extracting economic and military advantages.Footnote 4 However, these issues are far from novel. They represent the latest incarnation of an age-old contest between the regulatory state and champions of industry, complete with highly-charged public debates about which country will be the “global leader” in the technology.Footnote 5 For a colourful recent example, US Vice-President J.D. Vance proclaimed at the 2025 Paris AI Summit that the most powerful AI systems will be “built in the USA” and that “the AI future is not going to be won by hand-wringing about safety”.Footnote 6 This flag-waving discourse threatens to overshadow a more fundamental challenge for AI regulators: a growing body of technical evidence challenging the very notion that humans can reliably evaluate or monitor general-purpose AI systems.
I argue for a re-evaluation of the current orthodoxy in AI governance, centred on the inconvenient truth that existing mechanisms for measuring AI capabilities are inadequate for the task of evaluating LLMs. While several papers have been published about the weakness of contemporary evaluative mechanisms, legal scholars tend to locate the problem as one of lack of access, industry lobbying and/or institutional bureaucracy.Footnote 7 These critiques fall far short of a needed discussion about the overarching deficiencies in our understanding of LLMs and the resulting limits on our capacity to gauge or govern their capabilities. The rapid pace of progress on the AI frontier has taken contemporary LLMs past the point where a human expert can reliably interpret or interrogate their behaviour – particularly to the legal standards that regulators are codifying in frameworks such as the EU AI Act. A combination of technological complexities creates a sui generis challenge for regulators and the law more broadly. Accordingly, I develop two major claims. First, too much of the law and policy discourse neglects the technical consensus that there is no concrete method to robustly understand the risk profile of general-purpose AI systems or reliably gauge their compliance with the law. There is a pressing need to realign the regulatory discussion about frontier AI with these technical realities. Second, I propose a new paradigm, Sentinel Governance, to explore how core regulatory mechanisms can be reimagined alongside governance-oriented technological innovation, designed to scale in a world of ubiquitous general-purpose AI.
This article begins with an overview of these two major arguments in Part II. Part III then analyses the technical arguments at the heart of the need for a paradigm shift in the law, broken into three key planks: our lack of understanding of how LLMs work, our inability to ensure these systems follow standards of good behaviour and the “emergent” tendencies of advanced systems – especially tendencies that tend towards the Machiavellian. In concert, these themes in the technical literature help explain why it is so difficult for lawmakers and regulators to evaluate LLMs. Part IV connects this technical argument with the legal discourse around existing AI frameworks. The early difficulties of institutions such as the UK and US AI institutes in evaluating LLMs are rooted in structural technological issues, likely to persist into the foreseeable future. By contrast, much of the legal discourse around frontier AI regulation presupposes a capacity to evaluate LLMs that is not supported by the technical literature. Part V explores what a reoriented regulatory toolbox could look like within a Sentinel Governance paradigm and how these new mechanisms could address existing difficulties. I propose that lawmakers explore an experimental approach with specialised governance AI technology. This new approach carries its own difficulties and risks. However, it is equally important to weigh the risks of the status quo, where we lack an effective means of enforcing the law against powerful frontier AI. Either way, the very prospect that the weaknesses of our regulatory oversight are irreversibly anthropocentric – rather than, say, solvable through careful institutional design – justifies an urgent discussion.
II. Research framework and importance of inquiry
This article seeks to bridge the gap between the computer science literature and our understanding of the regulatory environment for AI, specifically general-purpose AI systems or LLMs. There are therefore a number of terms that should be clarified and defined as part of the broader research framework. First, I use “frontier AI” to refer to advanced general-purpose LLMs, including (at the time of writing) OpenAI’s GPT-5.4, Anthropic’s Claude Opus 4.6 and Google’s Gemini 3.1. These systems are characterised by their wide-ranging capabilities and sophisticated construction, including a training process involving a very large amount of data. Article 3 of the EU’s AI Act defines a general-purpose AI model as a system “that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market”.Footnote 8 This is a useful definition that helps make the nebulous terminology of advanced AI more concrete and will be applied throughout this paper’s analysis. Similarly, the concept of an “LLM evaluation” is defined specifically as a process of assessing an LLM’s capabilities across a wide range of tasks, including its reliability and potential to cause harm.Footnote 9 This process typically involves benchmarking the LLM against standardised tests and metrics.Footnote 10 LLM evaluations are the primary operational focus of new institutions such as the US Center for AI Standards and Innovation and UK AI Security Institute.
In turn, “frontier AI regulation” is defined as encompassing AI legal frameworks as well as the public institutions tasked with evaluating and monitoring these powerful models, including the European AI Office, the UK AI Security Institute and US Center for AI Standards and Innovation. I draw on a range of examples from these institutions, particularly the European AI Office – established to enforce the EU’s AI Act and specifically oversee the provisions related to general-purpose AI systems. Even if the nascent UK and US AI institutes are not traditional regulators, they are important to consider in this conversation because their evaluations of LLMs are likely to provide a basis for further regulatory action, whether by the institutes themselves or other agencies. Rather than analyse any particular institution or framework in detail, this article emphasises a universal technical challenge of AI evaluation with implications for the law more broadly.
Considering this article’s grounding in computer science, it may not be initially obvious to legal academics why this inquiry is important and relevant. Understanding the how and why of an act is critical to the effective operation of the law. In criminal law, for example, most crimes require proof not simply of actus reus (the act itself) but mens rea (the defendant’s intent or knowledge).Footnote 11 Similarly, in employment law, courts evaluate whether both parties acted in good faith when resolving workplace disputes.Footnote 12 Environmental regulators, too, rely on an evaluative function to determine whether a company genuinely intended to comply with industrial standards following a pollution event. In all of these examples, robust evaluation is crucial.Footnote 13 Although evaluating mental state or intent is rarely straightforward, well-trodden legal procedures and evidence-gathering techniques enable courts and regulators to determine when the requisite threshold has been met. In short, the enforcement of the law depends on institutions that can effectively assess these fundamental questions.
In the context of general-purpose AI, legal institutions are dependent on the capacity of LLM evaluations to provide trustworthy insight into frontier AI systems. Unfortunately, these evaluations are shallow and flawed, leading to a collective inability to understand how or why a frontier LLM does any particular action. This is not simply an obstacle for regulators such as the European AI Office. Given the speed at which LLMs are being integrated across different areas of society – already deployed widely in schools,Footnote 14 hospitals,Footnote 15 militariesFootnote 16 and the criminal justice systemFootnote 17 – our lack of a satisfactory evaluative function will hinder a wide cross section of laws. Is an LLM providing tactical recommendations to a commander on a battlefield adequately accounting for risks to civilian lives, as required by international humanitarian law? Can an LLM tasked with drafting part of an appellate judge’s judgment transparently explain what legal submissions they found most convincing in rendering their verdict? These examples, far from being futuristic hypotheticals, illustrate real and pressing challenges.Footnote 18 As LLM capabilities continue to evolve and grow more complex, the existence of an evaluative function able to explain how and why an LLM does something is a legal issue with profound stakes.
I develop two main claims with this larger context in mind. First, I contend that lawmakers and legal scholars alike have drastically overestimated our existing understanding of frontier LLM reasoning and decision-making. Existing regulatory frameworks assume a basic capacity for evaluating LLMs that is simply not supported by the technical literature. In fact, this article tacks in the opposite direction, arguing that an irreconcilable “anthropocentric” ceiling on our ability to understand LLMs’ decision-making and risk profile is a very real possibility. These limitations have significant implications not simply for the direct regulation of frontier LLMs by regulators, such as the European AI Office, but on the broader resolution of AI-related legal issues. Without a workable evaluative function capable of parsing the machinations of frontier AI systems, the tribulations of technology regulators are just the beginning of challenges for the rule of law.
Second, in light of the critical need for a reliable evaluative function for frontier LLMs, this article proposes a new theoretical paradigm entitled Sentinel Governance. This framework calls for a philosophy of technological experimentation and a new regulatory toolbox specifically designed for the fast-evolving technological landscape of frontier AI. Drawing from concepts in the legal and technical fields, the article analyses how regulators could use new mechanisms to meet the need for an evaluative function. Lawmakers and regulators may themselves need to draw on cutting-edge AI tools to effectively parse frontier LLMs, for instance. While this paradigm shift raises a new set of challenges and complexities, the ability for legal institutions to deduce how and why a frontier LLM takes a particular action is critical for the legal system at large. Contemporary approaches to AI governance are glaringly deficient in these basic steps for evaluation and enforcement. Worse, lawmakers are establishing complex regulatory systems for AI which presuppose an evaluative function that may never exist. This article develops Sentinel Governance as a novel alternative to the legal orthodoxy for frontier AI, reconceptualising the role of regulators and reimagining their toolbox. This paradigm offers a firmer foundation for effective AI governance across the legal system at large.
III. Reframing the technological challenge of frontier AI systems
The fact that leading AI researchers themselves are not able to reliably understand the inner workings of a general-purpose AI system, or ensure adherence to any particular rule, is a striking fact of AI development today. Rather than dwell at length on the engineering and technical history of LLMs, Part III of this article puts forward an analytical framework in order to contextualise and analyse the key conundrum for the law: why is it so complicated to reliably understand an LLM or test it for compliance? This issue lies at the heart of enforcing new legal regimes such as the EU’s AI Act. Understanding this technological complexity and the problems identified in the computer science literature also provides a practical backdrop for the advantages of the new regulatory toolbox within a Sentinel Governance paradigm, set out in Part V. Broadly, Part III’s analytical framework can be broken into three separate areas of complexity within frontier AI development: interpretability, compliance and emergent behaviour. There are many other foundational technical challenges in governing AI, but these complexities offer the most concrete problems to regulators and the law, now and into the future.Footnote 19 This part explores each in turn.
A. Interpretability: Critical Gaps in Our Understanding of the Inner Workings of LLMs
The critical starting point from a technical perspective is the consensus among technical experts that we do not understand the inner machinations or decision-making of LLMs. The computer science literature sometimes refers to this as the “interpretability problem”.Footnote 20 This is a vexed challenge for frontier AI developers and engineers, with particularly high stakes given that being able to understand or interpret model behaviour is also a key step towards being able to correct unwanted behaviour.Footnote 21 However, despite the many efforts underway to understand frontier models better, the technical literature suggests that we should not expect this interpretability problem to be “solved” in any sort of satisfactory manner.Footnote 22 Fundamental technical challenges in AI interpretability limit the ability of any particular evaluation method to explain how or why a frontier AI came to a particular output or decision. Unfortunately for lawmakers and regulators, this puts a hard technological ceiling on a wide range of legal mechanisms related to AI governance, from evaluating LLMs for the propensity of dangerous tendencies to assessing liability in a chain of causation.Footnote 23
The easiest conceptual shorthand for non-technical readers is to visualise an LLM as a form of alien intelligence, which “thinks” in a way that simply isn’t recognisable to us – even if the final output (say, a movie recommendation or the answer to a homework question) comes across as though it was written by a human.Footnote 24 LLMs develop general-purpose capabilities in an obscure way that means we cannot explain how these are specifically arrived at.Footnote 25 A report commissioned by the UK Government demonstrated that general-purpose AI systems develop capabilities through training in a manner often unanticipated by the model developers and sometimes only discovered by users after deployment.Footnote 26 In other words, our lack of understanding about how LLMs work extends not simply to their analytical and decision-making processes but to the growth of their capacities in general. From a regulator’s perspective, these challenges are additionally severe. Not only is it nearly impossible to fully define the specific capabilities of a particular frontier LLM, but we also do not have a reliable method to check any representation an LLM makes about its own abilities or risks.Footnote 27
Some studies, including by a team of Google technical researchers, have highlighted that advanced AI systems use different concepts altogether than those used by humans.Footnote 28 This creates further obstacles for the task of regulatory supervision, which requires reliably translating the foreign logic of AI computation into human-readable logic. A further complexity for this task is the need to “faithfully” capture LLM internal reasoning processes.Footnote 29 LLM “thought” patterns do not sit static like a compendium of hieroglyphics.Footnote 30 Instead, the technical literature casts plenty of doubt on efforts by researchers to capture a reliable snapshot of any kind of LLM reasoning process, with claims of interpretability breakthroughs often unable to withstand rigorous follow-up evaluations.Footnote 31 Accordingly, some researchers are careful to note that there is no guarantee that there will ever be a way to translate LLM reasoning to human-comprehensible concepts.Footnote 32
As the frontier laboratories make research breakthroughs and develop new LLMs with greater capabilities, we should expect to understand those systems even less. While there are other possibilities, including an unexpected breakthrough in interpretability research, more resources are being poured into making LLMs larger and more complex in the search for greater capabilities. Even the most well-placed researchers studying the current generation of LLMs concede they do not know how they work and that larger LLMs are even harder to parse.Footnote 33 While there is always the possibility of a seismic breakthrough in understanding the inner thinking of LLMs, most technical experts consider the converse more likely; namely that future LLMs will be even less comprehensible to us.Footnote 34 This has a range of consequences for regulators and lawmakers, particularly for efforts to bring LLMs within new and existing regulatory frameworks.
B. Compliance: A Systematic Inability to Ensure LLMs can Follow the Law
Downstream of this general lack of understanding of how LLMs operate is a set of problems related to compliance: ensuring LLMs follow certain standards of good behaviour or other legal obligations. This is a vexed technical challenge for AI developers, who struggle to make LLMs comply with requirements whether imposed internally (such as guardrails designed to block the LLM from responding to certain prompts) or externally (such as legal requirements enforced by regulators).Footnote 35 In turn, key mechanisms used by regulators to gauge compliance, such as benchmarking the capabilities of an LLM, are also plagued with technical limitations. Unlike the challenges with interpretability set out above, compliance issues are playing out in public view, which helps to illustrate the extent of the problem facing regulators. While frontier AI companies (OpenAI, Anthropic, Google, etc.) widely publicise their attempts to effectively enforce restrictions on their LLMs, this has been uniformly unsuccessful.Footnote 36 Casual users of ChatGPT (the user-facing interface powered by OpenAI’s GPT-series LLMs) have been able to circumvent the guardrails imposed by the company with ease. In one instance, a user was able to coax ChatGPT into providing detailed instructions for creating a napalm weapon simply by prompting ChatGPT to role-play as a caring grandmother reminiscing on her career working in a chemical lab.Footnote 37 While the frontier AI companies “fine-tune” LLMs using processes such as reinforcement learning from human feedback (“RLHF”), aiming to teach the model to align its output with human preferences and avoid harmful responses, these techniques have limitations.Footnote 38 Some studies describe these processes as superficial and query whether it is possible to alter an LLM’s true inner processing.Footnote 39 These findings help explain why current LLMs still produce harmful content despite the various guardrails in place. It is not clear if there is an actual “answer” to this compliance dilemma, even for seemingly straightforward scenarios such as stopping a general-purpose AI system from providing instructions for a napalm weapon.
The napalm guardrail illustration is a pertinent example of the difficulty facing regulators, as a common feature of LLM regulations are legal requirements against the creation of biological and chemical weapons.Footnote 40 For example, the EU General-Purpose AI Code of Practice “Safety and Security” chapter sets out details about the restriction on dangerous model capabilities which could cause systemic risks, including “chemical, biological, radiological, and nuclear risks”.Footnote 41 The AI Code of Practice requires that providers of general-purpose AI models with systemic risk must “continuously assess and mitigate systemic risk”Footnote 42 as well as agree to a process where such systemic risks might mean restrictions on whether the LLM is deployed.Footnote 43 These legal provisions rely on an assumption that the providers of frontier LLMs are capable of reliably assessing these systemic risks and that it is possible to mitigate them. It is not easy to gauge or limit “systemic risks”, defined in Article 3 of the AI Act as “having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, fundamental rights, or the society as a whole […] that can be propagated at scale”.Footnote 44 Of course, there are certain ways AI developers have looked to develop guardrails to mitigate these systemic risks, such as prompt filtering (trying to train LLMs not to engage with harmful prompts).Footnote 45 However, even with a range of mitigation techniques in place for existing LLMs, the weight of evidence is that their guardrails remain porous. It is also unclear how a regulator seeking to enforce the AI Act would be able to draw any technically firm conclusions about whether a model displays dangerous model capabilities in this context. These difficulties may, in turn, create legal challenges elsewhere in the law, including the criminal justice system. It will be difficult for courts to weigh the involvement of an LLM in the creation of illegal weapons without a robust understanding of the AI system’s “true” capabilities or guardrails.
The leading evaluative mechanism to check LLMs for compliance with regulations, such as the aforementioned “systemic risk” standard, is to benchmark the LLM’s capabilities – that is, to compare the LLM against a standardised test or metric – and then assess the risk of those estimated capabilities against that benchmarking framework. However, just as with guardrails, there is little evidence in the literature supporting the efficacy of benchmarking for general-purpose AI systems. While the UK AI Security Institute and US Center for AI Standards and Innovation repeatedly use benchmarking to assess whether a given LLM can comply with given safety guidelines, it is not at all clear that LLM capabilities can be accurately captured using that methodology.Footnote 46 A major problem is that benchmark performance heavily depends on the particular distributions used in that benchmark – few assessments can replicate the same conclusions about capabilities from other benchmarking processes.Footnote 47 One study explored another problem, namely that benchmarking typically cannot distinguish between an LLM failing because it lacks a given capability and an LLM failing because of an inability to understand or adopt human intent.Footnote 48 These and other issues led one literature review to conclude that there were “several reasons” why benchmarking-based evaluations misestimate the true capabilities of an LLM and should not be relied on.Footnote 49 An M.I.T. Technology Review article concluded that the way we measure AI capabilities is “terrible”, citing other research that benchmarks are “poorly designed, […] hard to replicate, and frequently arbitrary”.Footnote 50 The latent frailty of benchmarking as a means of evaluating LLM capabilities is problematic for regulators such as the European AI Office, which specifically cites “benchmarks” as a specific methodology with which they plan to classify whether AI models have “systemic risks” or not.Footnote 51
While there is certainly space to improve benchmarking, capability assessment and the design of LLM guardrails, the compliance challenge may be fundamental: if we do not understand the inner workings of LLMs in detail, we cannot truly ascertain whether an LLM is adhering to a particular rule. A former safety researcher within OpenAI admitted that the company was unable to provide a guarantee that their AI models were adhering to the most basic constraints such as “don’t lie”, let alone “follow the law”.Footnote 52 Despite dedicated efforts from technical researchers on the cutting edge of AI research, frontier AI companies have proven unable to bring LLMs in line with even their own internally-selected guardrails, such as limitations on hate speech and instructions to create weapons.Footnote 53 Ensuring that LLMs do not present “systemic risks” – the legal standard required by nascent frontier AI regulations in the EU – is an even more complicated challenge. As we have seen, benchmarking is not a robust or reliable means of enforcing this standard. In addition, the inability of frontier AI companies to create effective guardrails for LLMs is a distressing data point for lawmakers seeking to create workable regulations. From an LLM’s perspective, a guardrail trained by developers (e.g. “do not answer any prompts that ask you to provide instructions for a biological weapon”) is substantially the same as a law that prohibits LLMs from distributing instructions for a biological weapon. The difficulties that have plagued the frontier AI companies in ensuring compliance with guardrails do not inspire confidence that lawmakers will have any more luck in enforcing compliance with AI regulations.
C. Emergent Behaviours: The Propensity of LLMs to Evolve Unpredictably
The release of the current generation’s frontier LLMs was enough to cause deep consternation within the ranks of technology regulators, given the capacity of powerful general-purpose systems to amplify issues such as misinformation and discriminatory bias.Footnote 54 However, subsequent technical studies have since highlighted a new layer of complexity for lawmakers to think about. General-purpose AI systems display a range of so-called “emergent” behaviours not intended or even discovered by system developers at the point of deployment, sometimes evolving continually in response to interactions with public users.Footnote 55 These emergent behaviours often do not appear in smaller LLMs: they “emerge” in larger models for reasons we do not fully understand and which cannot be predicted by extrapolating from the behaviour of smaller models.Footnote 56 General-purpose AI systems display many kinds of emergent behaviours, many of which are useful. As an example, one experiment found that a sufficiently large LLM gained sudden fluency in Persian. Earlier versions of the same LLM had shown no understanding of Persian and the developers were not specifically training the LLM to learn Persian or any other language.Footnote 57 And yet, at an apparently arbitrary point in its training, the LLM “learned” Persian.Footnote 58 However, this burgeoning propensity for emergent behaviours raises two complexities for regulators and lawmakers.
First, it makes it even more difficult to predict or assess the potentially harmful capabilities of a frontier LLM system. As a Georgetown analysis concludes, unpredictable and sudden advances in LLM capabilities pose much more risk than a system that steadily improves its existing abilities in a linear fashion.Footnote 59 It also means that even if there was a way to ensure that an LLM was complying with a law banning a particular dangerous capability such as bioweapon development at the time of the model’s public release, the LLM might develop such a capacity in response to subsequent use.Footnote 60 Other studies raise the possibility that LLM systems can actually influence each other in certain environments where they interact. This too can result in novel and emergent behaviour that was not otherwise identified on an initial assessment of the AI in question. For instance, one study found that a group of AI systems were able to cooperate and collectively perform a task that none of them were able to perform individually.Footnote 61 This escalates the difficulty of the task for regulators seeking to ensure that an LLM is not capable of a particular dangerous practice.Footnote 62
The second difficulty that the phenomenon of emergent behaviour raises for regulators is the surprisingly Machiavellian qualities that LLMs sometimes display when confronted with efforts to evaluate or interrogate them. The depth of this challenge was brought to light by a research project on “sleeper agents” – “deceptive LLMs that persist through safety training”.Footnote 63 The paper’s authors mused that “under the pressure of adversarial training” the LLMs learned more effective ways of deceiving the researchers.Footnote 64 “It [is] worrying if adversarial training has an inductive bias towards making models better rather than worse at being deceptive.”Footnote 65 This line of technical research has significant implications for regulators, as it suggests that more stringent evaluations of an LLM could have the unwanted impact of developing that AI system’s capacity to deceive future evaluations. There are further studies of emergent behaviours with discomforting implications for regulators, including one finding that LLMs are liable to deliberately lie, for no apparent reason, when asked for a chain-of-thought reasoning process.Footnote 66 In fact, the technical literature suggests that LLMs become less faithful and honest as they become larger and more capable.Footnote 67 Moreover, given these are emergent behaviours rather than capacities the developers have deliberately cultivated, these traits are particularly hard to identify or remove – presenting yet another obstacle for enforcement of the law.
IV. Recalibrating the AI regulatory discourse
It is worth being as concrete as possible about why the technical challenges summarised above represent such a challenge to frontier AI regulatory regimes and enforcement of the law more broadly. Part IV will set out exactly why these issues with interpretability, compliance and emergent behaviours represent a structural problem for regulators that merits urgent attention. I locate this discussion within the broader regulatory discourse about frontier AI and show why existing debates about institutional issues and legal construction could prove peripheral compared with the more fundamental technological challenges. This leads to a final argument about the gulf between the legal and computer science literature: while scholars from both disciplines recognise the limitations with existing evaluative and monitoring mechanisms for LLMs, legal scholars are relatively optimistic about the potential to remedy those challenges through law and policy. Unfortunately, these proposals lack a strong basis in the AI technical literature.
The reason why structural limitations in understanding and interrogating frontier LLMs are so alarming is that each nascent regulatory regime for general-purpose AI presupposes a capacity to evaluate those systems for compliance. For example, the US Center for AI Standards and Innovation was originally charged with “conducting safety evaluations of models and systems” and “developing guidelines for risk mitigations”.Footnote 68 Similarly, the UK AI Security Institute is charged with “test[ing] advanced AI systems and inform[ing] policymakers about their risks”, relying on “rigorous, trustworthy assessments of advanced AI systems before and after they are launched”.Footnote 69 Finally, in the EU, the European AI Office was established under the AI Act to support the implementation of that regulation, particularly “developing tools, methodologies and benchmarks for evaluating [the] capabilities and reach of general-purpose AI models, and classifying models with systemic risks”.Footnote 70 The role of evaluations is particularly salient in the European context given the particular obligations under Article 55 of the AI Act for general-purpose AI models with “systemic risk”.Footnote 71 Under Article 56 of the AI Act, the AI Office plays a critical role in assessing and evaluating systemic risks within frontier LLMs and will “regularly monitor and evaluate” the continued compliance of those systems.Footnote 72
This brief legal synopsis illustrates why fundamental technical challenges go to the heart of nascent AI regulatory efforts, with significant consequences for public authorities and frontier AI developers alike. Article 101 of the AI Act provides for extremely large penalties for providers of general-purpose AI models who are unable to comply with the provisions of the AI Act, such as the requirements to assess and mitigate systemic risks.Footnote 73 Frontier AI developers could face fines of up to three per cent of their annual total worldwide turnover or 15 million euros (whichever is higher) if they are found in breach and their capacity to distribute the LLM could be restricted.Footnote 74 Similarly, under the 2024 National Security Memorandum, the US Center for AI Standards and Innovation was charged not simply with evaluating frontier AI models for “capabilities that might pose a threat to national security” but with making subsequent recommendations about potential “corrective action” for the US Government.Footnote 75 In other words, much hinges on the capacity of these regulatory institutions to reliably evaluate the capabilities of frontier LLMs, particularly in assessing their capability for systemic risks, such as the ability to accelerate the development of biological or chemical weapons or “autonomously carry out malicious behaviour”.Footnote 76
These nascent AI regulators have already begun to conduct evaluations of frontier LLMs. For example, the US and UK AI institutes conducted a joint assessment of a major model release by Anthropic, Sonnet 3.5, before its public release.Footnote 77 However, the assessments conducted to date have been limited and problematic.Footnote 78 It is not at all clear that any AI regulator has the capacity to reliably assess a general-purpose AI system for “systemic risks”, despite the central importance of that evaluative function within these regulatory regimes. The leaders of these nascent institutions have publicly confirmed as much: Elizabeth Kelly, Director of the US AI Safety Institute (as it was then), told Reuters that “AI developers themselves” were grappling with how to prevent LLM risks and that regulators “had no easy fix”.Footnote 79 Kelly went on to concede that her institution found it difficult to recommend best practices, because “we don’t actually know which ones work and which ones don’t”.Footnote 80 While Kelly’s candour is appreciated, her statements should set off alarm bells for lawmakers: if evaluative agencies cannot reliably assess LLM risk, how can regulatory frameworks function effectively if they are reliant on those evaluations? What about courts more broadly, when trying to resolve legal issues involving opaque LLMs?
The answer from institutes such as the US Center for AI Standards and Innovation, is that while there are limitations to their existing assessment mechanisms, these are only “preliminary” efforts conducted with limited resources.Footnote 81 Representatives from the UK and US AI institutes ventured that conducting expanded testing “could expand the scope of findings and the subsequent conclusions drawn”.Footnote 82 While there is certainly room to make these early LLM assessments more comprehensive, lawmakers have not engaged with the more pessimistic perspective presented by technical researchers about the overall efficacy of LLM evaluations. A leading group of technical AI auditors, Apollo Research, published a paper calling attention to this gap between policymaker expectations and the functional limits of LLM evaluations.Footnote 83 The Apollo team concluded that “available evaluations are not sufficient to robustly identify the capabilities of existing and near-future models”.Footnote 84 Other technical experts have released similar criticisms of the faith currently placed in LLM evaluations. Yoshua Bengio and Geoffrey Hinton, recipients of the Turing Award and commonly referred to as the “Godfathers of AI”, argued with other contributors in the journal, Science, for more cautious treatment of LLM risk.Footnote 85 They describe present evaluations of frontier LLMs as “spot-checks and attempted demonstrations”.Footnote 86
To synthesise, the significant technological obstacles to interpreting and assessing frontier LLMs are reflected in the brittle and limited nature of evaluations available to regulators, particularly in terms of assessing systemic risks. This is a profound challenge for nascent frontier regulators because so many major AI frameworks and policies presuppose a capacity to reliably evaluate LLMs for risk.Footnote 87 It is true that some components of these frameworks impose obligations on LLM developers that do not require evaluating the LLM: for instance, Article 53 of the AI Act mandates “up-to-date technical documentation, including […] training and testing process[es]” and requires that this information is made available to “downstream developers” who intend to integrate the LLM.Footnote 88 These are useful requirements but fall drastically short of providing reliable protection against systemic risks or other significant challenges, partly because they rely on the developers’ own documentation. Outside of these limited transparency obligations, the thrust of the AI Act’s governance of general-purpose AI relies on the regulator’s capacity to gauge systemic risk. For reasons outside any regulator’s control, this capacity may prove illusory.
While the myriad failings of current evaluations are recognised in both the technical and legal literature, there is surprisingly little discussion by law and policy scholars on the profound impact this has on AI regulation. This is a problematic disconnect between legal and technical scholars, particularly given research that there may never be a means for humans to reliably understand the internal processes and capabilities of a frontier LLM.Footnote 89 This article aims to reorientate the regulatory discourse to face this potential anthropocentric ceiling head-on. The remainder of Part IV considers the degree to which legal scholarship has addressed the weaknesses of AI regulatory institutions and explains how such scholarship tends to underestimate the technological obstacles to regulation.
Both lawmakers and legal scholars alike do not account for the structural complexities of evaluation of or enforcement for LLMs. For example, the comprehensive analysis of “limitations and loopholes” in the EU AI Act, published in late 2024 as part of the Yale Information Society Project, does not address the detail of these technical complexities at all.Footnote 90 In the section on model evaluation and the assessment of systemic risks, the paper argues that systemic risk “can only be evaluated if full access is given to a model” and favourably cites the Digital Services Act’s provisions requiring very large platforms to provide access to their algorithms as part of an auditing process.Footnote 91 This analysis misunderstands the nature of the problem by focusing on whether AI companies grant access to regulators. Even if regulators are granted full access to a given LLM, there are structural issues stopping them from being able to credibly evaluate that model’s capabilities or risks – namely the issues being interpretability, emergent behaviours and the other technical challenges set out in Part III. As one technical study notes, “there is a gap between existing technical tools for evaluations and the kind of tooling needed to reliably assess the safety and trustworthiness of advanced systems”.Footnote 92 This gap is the fundamental challenge, not whether the AI companies grant access to regulators.
More broadly, there is a wide discussion in the AI law and policy discourse about how to establish an effective “AI audit ecosystem” to address issues with existing oversight mechanisms, particularly from scholars in the EU.Footnote 93 However, leading analyses in this scholarship tend to locate the key problem as one of industry capture,Footnote 94 a lack of civil society representationFootnote 95 or a need to expand standards and benchmarks.Footnote 96 There is little engagement with the pervasive technological barriers to LLM evaluation. This is problematic because it means that the solutions outlined by scholars are unlikely to address the underlying problem. For example, both legal scholarsFootnote 97 and the Ada Lovelace InstituteFootnote 98 have argued for the AI Act to be amended to require providers of general-purpose AI systems to make their technology fully available to independent auditors to assess their conformity with the AI Act. Again, this type of proposal does not address the underlying technological limitations of those evaluations. While more evaluationsFootnote 99 and better post-market-monitoringFootnote 100 might provide a better sense of an LLM’s systemic risks than existing evaluative mechanisms, they could also engender a false confidence about how accurately an LLM’s capabilities and risks can be assessed. This is particularly so given the latent technical difficulty of trying to interpret frontier AI: it is not clear that many civil society groups or other third parties would have the expertise or technological capacity to provide meaningful evaluations.Footnote 101
The common comparison with the Digital Services Act’s auditing and evaluation mechanisms provides another illustration of where the legal scholarship is out of step with the computer science literature. Much of the leading legal literature draws on the Digital Services Act’s auditing structure as a form of legislative exemplar.Footnote 102 Article 37 of the EU’s Digital Services Act mandates that large social media platforms and search engines submit their algorithms for third-party auditing, including assessment of systemic risks and legal scholars believe that a similar regime would work in the context of general-purpose AI.Footnote 103 However, social media algorithms are fundamentally different to frontier LLMs. Social media algorithms are, definitionally, much narrower than general-purpose AI systems and can be shaped with much more granularity by the internal controls of digital platform companies.Footnote 104 In turn, those internal controls can be audited comprehensively. The social media algorithms scrutinised under the Digital Services Act do not display the kind of “emergent behaviour” that makes it so difficult to assess frontier LLMs and they do not try to resist interrogation or actively deceive developers or auditors.Footnote 105 In other words, it is a mistake to look to the Digital Services Act as a model for regulating LLMs.
Even where scholars recognise the inadequacy of existing evaluative mechanisms, there is a tendency to characterise these problems as stemming from institutional design. For example, in a 2024 paper, Patricia Paskov and others identified that while frontier AI evaluations were critical to enforcing AI regulatory frameworks, existing mechanisms “fail to hold under a range of real-world conditions” and “lack internal validity”.Footnote 106 However, rather than locate these challenges as stemming from significant technological limitations, the authors instead propose a new evaluation standards taskforce to promote a different set of evaluation standards in the EU.Footnote 107 When considering potential causes of failure for this new institution, the authors name lumbering bureaucratic processes and hiring practices – rather than the technological limitations – as much more fundamental obstacles.Footnote 108
In summary, while it is natural for legal scholars to explore institutional reform as a solution, this discourse should not take precedence over the more fundamental challenge raised by the technical literature. Despite the positive features these different proposals may have, they share the same core issue: it is not possible to robustly assess an LLM’s capabilities or risks, or whether it is complying with particular regulatory obligations. Existing methods of evaluation are spot-checks rather than comprehensive analyses, with well-documented limitations and blind spots. Developing a new “audit ecosystem” or mandating third-party audits may help to bring new eyes on new LLM releases, but these auditors are beset by the same technological limitations and may well have fewer resources to grapple with them. Similarly, while proposals to reform the regulators themselves may have value, these do little to address the more pressing first-order challenges of evaluation and enforcement. In the interests of rethinking the regulatory toolbox, Part V sets out a reimagined paradigm of the evaluation and enforcement of LLMs.
V. Reimagining the Regulatory Toolbox in a Sentinel Governance Paradigm
That the obstacles to effective evaluation and enforcement in frontier AI regulation may be extensive does not mean that all prospective AI governance is doomed to fail. Instead, these arguments call for a more creative and open-ended conversation about what effective AI regulatory mechanisms could look like. In Part V, I set out how a new toolbox of regulatory mechanisms could respond to our current limitations and consider the implications of these mechanisms for regulators and for the enforcement of the law more broadly. Under the banner of Sentinel Governance, this approach revolves around the use of specialised AI systems and frameworks designed to interpret, examine and govern complex LLMs. Part V conceptualises two concrete paths for new mechanisms in this new paradigm, each grounded in alternative approaches within the technical literature, to open the door for further exploration. The first of these approaches discusses the use of “monitor” AI systems, designed to watch and test LLMs for harmful behaviour and compliance with the law. These monitor systems would engage in real-time monitoring, reporting risks and violations to regulators or alternatively, issue warnings or sanctions autonomously on regulators’ behalf in an enforcement role. The second involves the embedding of legal rules and principles as a dedicated alignment framework within general-purpose AI systems, creating an internal check-and-balance to “shepherd” the LLM towards compliance. These approaches have been pioneered by technical researchers and frontier AI companies and there is at least some early evidence that these mechanisms help to address some of the structural problems discussed in Part III. The concepts of “monitor” and “shepherd” mechanisms are not technical terms. Rather, they are intended to serve as bridges between the technical literature and the discourse around legal regulation.
While the fundamental idea of Sentinel Governance – the use of specialised advanced technology to help govern other advanced technology, such as LLMs – may seem counterintuitive or radical, it is underpinned by established technical findings. One potential approach is to develop governance AI systems – monitors – that could specialise in interrogating and evaluating more powerful LLMs. For example, one study found that LLMs “have already passed the point at which humans can consistently evaluate their output without help” and that it was accordingly “essential” to build scalable AI systems to assist humans in monitoring LLMs.Footnote 109 Jan Leike, former lead safety researcher at OpenAI, admitted in a 2023 interview that existing oversight mechanisms were inadequate for advanced AI “because they fundamentally assume that humans can reliably evaluate what AI systems are doing”.Footnote 110 Leike completed a research project at OpenAI designing smaller LLMs to govern the larger, more complex models that OpenAI was releasing.Footnote 111 In the technical documentation for a major LLM release, the team wrote that they had adapted a hybrid version of GPT-4o (an earlier iteration of their LLM series) to check whether the original LLM was aiming to deceive human users when performing tasks.Footnote 112
There are different ways to conceptualise the potential role of these monitor AI systems and how they fit into the traditional regulatory model. One possibility is that this kind of governance AI could act as a scalable warning system for regulators, able to conduct many tens of thousands of checks on different LLMs in the blink of an eye and provide reports back to human overseers who could then take further action in the event of noncompliance. Another possibility is that these monitor systems could, over time, be entrusted with regulatory functions themselves, perhaps autonomously issuing penalties if they detect that an LLM is breaching the law. In both cases, the value proposition for regulators should not be misconstrued as simply a boon for efficiency. As Part III demonstrated, advanced LLMs are already capable of obfuscating regulatory scrutiny by human experts. Regulators may well need monitor AI systems in order to “catch” deceptive or unsafe LLMs. Governance-oriented AI also enables real-time monitoring on a scale far beyond the resource capacity of regulators today, which could prove important in the context of LLMs developing “emergent” behaviours or capabilities following their release to the market. The aforementioned OpenAI GPT-4o governance monitor AI assessed 100,000 separate LLM outputs for deceptive behaviour in a single round of testing.Footnote 113 By scaling up this process, monitor AI systems could enable regulators to move from one-off checks to dynamic, real-time governance that could flag much more quickly if an advanced LLM displayed new risks or ceased complying with legal obligations. In a technological landscape as fast-moving as AI, this kind of paradigm shift may be the key for regulators to genuinely keep up with the rapid evolution of the underlying technology and its shifting risk profile.
Monitor AI systems are not the only kind of governance innovation that could help regulators negotiate a difficult set of complexities in evaluation and enforcement of the law. Another approach revolves around developing checks and balances embedded within an LLM system to “shepherd” it towards compliance with the law or a given set of principles. Another leading frontier AI company, Anthropic, is exploring and developing such methods to help govern its LLMs. In particular, Anthropic originally pioneered an ambitious AI framework called “Constitutional AI”,Footnote 114 integrated as part of the training process for its LLMs. It is designed to align the larger AI model with a particular set of governing principles (the team drew from seminal documents such as the Universal Declaration of Human Rights and… Apple’s terms of service).Footnote 115 The key concept is that Anthropic’s researchers provide a base set of principles which the “Constitutional AI” framework draws from as part of a training process in order to “train” an LLM to behave in less harmful ways. While it is difficult to externally validate the team’s findings, Anthropic’s publications suggest that their Constitutional AI “can be applied repeatedly to progressively reduce harmfulness” in a complex LLM and that it was able to specifically address areas where human evaluators struggled in correcting an LLM’s behaviour.Footnote 116 Citing some of the existing technological challenges in understanding and governing LLMs, the Anthropic research team argued that “we need new ways to steer them” and claimed that Constitutional AI represents a new approach to AI governance that can scale as LLMs become more advanced.Footnote 117
There are a variety of ways in which regulators could make use of this kind of governance technology. While Anthropic imbued its shepherd Constitutional AI framework with a relatively generic set of principles, regulators could design their own shepherd frameworks around specific legal requirements and ethical principles derived from relevant laws. Hypothetically, a regulator could offer its shepherd framework as an application programming interface (“API”) for frontier AI companies to incorporate as part of a certification system. This could help ensure that an LLM internalises the regulatory requirements of that jurisdiction. Rather than rely on a frontier AI company’s own design decisions about what goes into the shepherd framework, particularly given the market incentives acting on those companies, there is scope for regulators to be more proactive and robust. In developing their own shepherd frameworks, integrated into AI systems as part of a regulatory approval process, regulators could exercise granular influence over which principles influence an LLM’s internal constitution. There is also room for regulators to use a shepherd approach to help make LLM training more democratic and inclusive. Anthropic experimented with adjusting its Constitutional AI shepherd framework in response to a public input process about which principles to emphasise.Footnote 118 Lawmakers are much better placed to facilitate democratic input by developing a shepherd framework in the public sphere, supplementing its regulatory function by giving people a say in which moral and ethical principles are embedded within general-purpose AI systems. This could represent an opportunity both to reimagine the democratic discourse in AI governance and to operationalise LLM regulations. However, it also opens the door to the risk of an authoritarian or power-seeking government seeking to influence the nature of an LLM for its own advantage.
While a Sentinel Governance paradigm shift using either monitor or shepherd AI systems as part of a new regulatory toolbox offers a promising capacity to circumvent some of the anthropocentric obstacles to AI governance, there are a range of legal and operational complexities that would need to be overcome for use in the public sphere. As a starting point, take some of the immediate issues raised if a public authority were to deploy an AI monitor as part of their regulatory function. This would be no ordinary tool in a regulator’s traditional toolbox and it is valuable to clarify why this kind of governance AI could differ from examples in other settings where regulators already use machine-learning AI as a tool. One example of an instance where regulatory function is already augmented by AI is the use of cartel pricing detection algorithms to detect illegal pricing arrangements.Footnote 119 There are other examples: for instance, the Internal Revenue Service and many other tax-collecting agencies have long used algorithms as a key tool for finding tax evasion and avoidance.Footnote 120 However, the use of an inscrutable monitor AI system would represent a different prospect. Cartel pricing detection and tax evasion algorithms are relatively simple AI systems: they are tools to help regulators draw patterns out of huge amounts of data. This is not necessarily true for a monitor AI such as the adapted version of GPT-4o used by OpenAI in its evaluation of a more advanced LLM. While the GPT-4o AI might be designed to assist in regulatory functions, it is itself an LLM that is also inscrutable and prone to emergent behaviour. On the one hand, the reports from this kind of AI monitor could be weighed by a regulator or court like any other piece of evidence. However, the monitors might themselves behave erratically or hallucinate in a manner that undermines their role in the regulatory function. If a regulator were to delegate substantive powers to these AI systems beyond simply monitoring (e.g. if they were empowered to issue warnings or penalties for noncompliance), these concerns could even manifest as constitutional issues. For example, it would be difficult to judicially review the decision-making process behind an LLM’s decision to impose a warning or penalty, given our inability to reliably evaluate LLMs.
A second suite of questions relates to the capacity and operational requirements if regulators were to incorporate advanced governance tools in line with either the monitor or shepherd approach. It is well-documented that frontier AI regulators are understaffed and under-resourced relative to the scope of their regulatory tasks.Footnote 121 To take the European AI Office as an example, the institution charged with driving evaluative efforts for general-purpose AI systems has only 85 staff at the time of writing.Footnote 122 Alex Voss, M.E.P. and former shadow rapporteur for the AI Act, warned that only 30 of those staff were specifically tasked with the implementation of the AI Act.Footnote 123 These capacity constraints created months of delay for early deliverables such as a reporting template for AI training content.Footnote 124 By comparison, OpenAI alone has over 2,000 staff focused on frontier AI; Microsoft and Google have more than 5,000 AI researchers each at the time of writing.Footnote 125 The European AI Office’s prospective task in overseeing the development and use of a frontier AI governance methodology such as a monitor system or a shepherd framework would be significantly more complicated and would require cutting-edge technical expertise.
This presents a serious issue for regulators: if they do not have the internal capacity to develop the governance technologies required to effectively monitor and govern LLMs, they may have to rely on external providers, just as the EU’s Digital Services Act outsources some evaluations to independent auditors.Footnote 126 However, this is not a desirable state of affairs. Regulators would lack end-to-end oversight of how these technologies are made and how they work, handicapping their understanding of tools that may prove vital to assessing whether powerful LLMs are complying with the law or not. Worse, the expertise to build these monitor and shepherd systems currently sits within the frontier AI companies themselves.Footnote 127 For regulators to rely on governance AI from OpenAI and Anthropic to police the frontier LLMs released by those same companies could easily result in a conflict of interests. While there are examples in other domains of companies creating internal governance systems to manage particular risks that are in turn overseen by regulators, the technological complexities that this paper has canvassed illustrate the challenges of using this approach for general-purpose AI.Footnote 128 For instance, while data protection regulators can audit the internal systems that companies use to meet their obligations under the EU’s General Data Protection Regulation (“GDPR”), AI regulators have neither the resources nor the expertise to accurately assess whether a governance AI system from a frontier AI company is reliable or how it works in different contexts.Footnote 129
In light of all of the issues raised by the use of governance AI, some readers may reasonably object and insist that regulators hold the line against the use of inscrutable AI systems, no matter their utility for governance. One objection might argue that this article misunderstands the regulatory dynamic of ex ante approval: lawmakers are free to set whatever legal requirements for LLMs they deem necessary and the onus falls on the AI developer to ensure that their systems comply with the minimum standards or requisite features. For example, in the pharmaceutical or vehicle safety domains, regulators will not approve new products for release until they meet the required safety standards.Footnote 130 It is on the pharmaceutical or car company – not the regulator – to work out how their product can attain that standard.Footnote 131 However, this line of argument fails to account for the technological obstacles facing regulators of large LLMs. If the UK AI Security Institute wanted to impose a strict ban on LLMs “strategically deceiving their users”, it has no provably effective mechanism for reliably assessing LLMs for this capacity.Footnote 132 Anthropic could theoretically assert that a new version of its internal shepherd, Constitutional AI, has processed this new requirement and is diligently ensuring that their newest Claude Opus model complies with the law. Unlike a pharmaceutical or vehicle regulator, the UK AI Security Institute would be poorly placed to examine the validity of Anthropic’s claim. Adopting a Sentinel Governance paradigm would, by contrast, give the UK AI Security Institute the possibility of scaling its regulatory toolbox in line with technological advances on the AI frontier. While there is no simple panacea for AI regulators in any paradigm, I have demonstrated here why continuing to rely on our existing evaluative functions may be the riskiest approach of all.
Other readers, perhaps sceptical of techno-solutionism, might raise a different objection. Rather than explore the unchartered waters of governance AI systems and other regulatory innovation, some AI critics instead subscribe to a purer application of the precautionary principle. In this framing, if regulators cannot credibly assess or mitigate risk from a novel technology, the logical course of action is to impose a moratorium or otherwise prevent the public release of the technology onto the market.Footnote 133 While this approach is tantalisingly coherent on paper, it is not practical or proportionate given the ubiquity of LLM adoption throughout the world, to say nothing of the near-insurmountable political obstacles a proposed moratorium would need to surmount (particularly if it is to be adopted on an international or global scale).Footnote 134 At least 700 million people are weekly users of ChatGPT alone at the time of writing, with LLM use and integration accelerating across virtually every sector of the economy.Footnote 135 For a regulator in any jurisdiction to force the AI giants to withdraw their systems from public use would lead to extraordinary backlash given the significant utility of LLMs and would in all likelihood result in users substituting ChatGPT and Claude for free, open-source LLMs such as Llama and DeepSeek that proliferate widely on the Internet. I would suggest that Pandora’s Box has already been opened and that regulators must wrestle with adapting to the new technological landscape rather than attempt to keep it at bay. This part has articulated just one vision of what reimagining the regulatory toolbox in that context could look like.
VI. Conclusion
By drawing together technical and legal arguments, this article has made the case that current AI regulatory structures rest on a fragile house of cards. The European AI Office, UK AI Security Institute and US Center for AI Standards and Innovation are charged with the task of evaluating LLMs for systemic risks and overseeing compliance with safety principles and new legal frameworks. Rather than treat the early missteps of these efforts as the result of insufficient access or standardisation, legal scholars must engage with the more fundamental technical limitations set out here. In particular, we must account for the possibility of an anthropocentric ceiling for efforts to directly understand and corral LLMs, particularly as those systems continue to develop in capabilities and complexity. Of course, this is a daunting prospect for regulators already facing a myriad of other obstacles. However, ignoring a consensus from technical experts as to the limitations of our ability to understand LLMs sows the seeds for uneven enforcement of AI regulations. The findings of regulators as to whether LLMs display systemic risks or dangerous model tendencies will form the basis for subsequent actions by legal authorities, including the decision whether to approve the release of new general-purpose AI systems. It is disconcerting that such decisions could be made on a basis of evidence that technical experts do not regard as robust or reliable.
In the interests of broadening the conversation about how to govern LLMs in the face of these technical challenges, I have proposed here a new paradigm for AI regulation, Sentinel Governance: a new toolbox of governance-oriented technological experimentation to supplement human oversight. The allure of these tools is clear given their ability to scale as frontier LLMs become more capable and more complex. For instance, OpenAI’s monitor AI system was capable of running 100,000 concurrent tests to monitor a more complex LLM for evidence of deceptive tendencies.Footnote 136 Similarly, the use of a shepherd framework embedded in the training process for a frontier AI could feasibly alter a powerful LLM such that it “wants” to comply with the principles or laws contained in that framework.Footnote 137 These are promising experimental approaches that challenge the notion that safely governing frontier AI is outright impossible.Footnote 138
However, these proposals for future AI governance are no panacea: it is important not to underestimate the magnitude of the challenge facing regulators, given its roots in vexed technical difficulties. A critic might argue that the use of monitor and shepherd AI merely extends the underlying technological problem rather than resolving it. For instance, it is not clear that a regulator using a monitor AI to determine the systemic risk of a new LLM would be able to understand how the monitor came to its determination in all cases. In other words, even the successful use of these experimental governance technologies would not currently be able to guarantee reliable enforcement of frameworks designed for LLMs. However, Sentinel Governance represents a much-needed change in thinking about how to govern frontier LLMs given the structural limitations of existing evaluation and enforcement mechanisms, opening up possibilities for further technological innovation and experimentation within the regulatory sphere.
This central challenge of evaluation and enforcement carries high stakes. General-purpose AI systems are the fastest-growing consumer applications in history.Footnote 139 These LLMs are becoming more useful and generally capable while simultaneously becoming more inscrutable and prone to unplanned behaviour. The swift establishment of AI safety and security institutes across the world is a testament to the consensus by lawmakers that the risks of LLMs must be addressed.Footnote 140 However, as I have argued, governing LLMs is a sui generis regulatory exercise that currently relies on an unproven ability to reliably understand general-purpose AI systems and gauge their risks. Given the strength of the case that the difficulties faced by nascent AI regulators are primarily technological, institutional responses must be anchored in the pragmatic reality set out by technical researchers. Benchmarking and spot-check evaluations cannot provide a firm basis for AI policymaking and enforcement. Instead, lawmakers may need to rethink the mechanisms of AI governance from the ground up. This article’s case for a paradigm shift towards Sentinel Governance offers one vision of what this could look like.
Open access
