Policy Significance Statement
Public institutions and civil servants are increasingly turning to large language models (LLMs) like ChatGPT to help analyse the vast amounts of textual information—from policy drafts to public consultations—that define their work. This article demonstrates that the choice of how to use these tools is a critical governance decision, not a simple technical one. We present four distinct models of use, ranging from full manual control to scaled automation, and provide empirical evidence on the trade-offs associated with each. Our findings indicate that the most effective and responsible approach is a “human-in-the-loop” model, where AI is utilised to expedite the labour-intensive aspects of analysis, while human experts maintain complete control over interpretation and judgment.
1. Introduction
The practice of public administration faces a defining challenge of Digital Era Governance (Dunleavy et al., Reference Dunleavy, Margetts, Bastow and Tinkler2006): an overwhelming volume of unstructured textual data that threatens the core imperative of timely, evidence-based policymaking. Within public institutions, this volume of information, encompassing for example legislative proposals, international treaties, public consultations, and policy analysis and evaluations, is creating a critical bottleneck that exceeds the limited analytical capacity of the civil service. The integration of artificial intelligence (AI) into core state functions is therefore an emerging necessity, yet it poses a direct challenge to the established norms and epistemic foundations of public administration (Margetts and Dorobantu, Reference Margetts and Dorobantu2019).
Large language models (LLMs) represent the frontier of this transformation, offering powerful capabilities for processing and interpreting natural language at scale, and their potential is being actively explored by public institutions (Government Digital Service, 2025). Consequently, a robust academic and policy discourse has emerged to govern their use, focusing rightly on the critical need for transparency and explainability. However, this conversation has concentrated overwhelmingly on the risks of quantitative, predictive models (Amarasinghe et al., Reference Amarasinghe, Rodolfa, Lamba and Ghani2023; Rehill and Biddle, Reference Rehill and Biddle2024). This focus, though necessary, has left a critical governance question unaddressed: how should the civil service be equipped and governed to use LLMs for the qualitative, interpretive analysis of policy texts—a task central to understanding nuance, context, and semantic commitment? The answer to this question carries deep implications for public trust, as the perceived legitimacy of AI in governance hinges on its procedural fairness and accountability (Starke and Lünich, Reference Starke and Lünich2020).
This article provides the first systematic, empirical answer to that question. Our research is distinct from studies testing the capacity of LLMs to generate new policy analysis (Safaei and Longo, Reference Safaei and Longo2024). It builds upon earlier work that demonstrated the potential of automated methods in policy document review (Alla et al., Reference Alla, Oprescu, Hall, Whiteford, Head and Meurk2018). We address the gap with regard to governance by evaluating four distinct archetypes for AI-assisted policy analysis. These archetypes, which we operationalise as analytical workflows, are applied to a real-world corpus of 63 United Nations policy documents on AI, deploying and testing the ChatGPT-4 model by OpenAI. They include: (1) traditional manual analysis by subject-matter experts; (2) automated topic modelling; (3) a human-in-the-loop workflow using direct LLM prompting; and (4) an institutional-scale workflow using an API-based model.
By systematically comparing the outputs of each archetype, we provide an empirical framework for institutional decision-making. This framework illuminates the inherent trade-offs between analytical speed, financial cost, contextual depth, and the requisite level of human oversight. Ultimately, we argue that the choice of an analytical method is an element of governance itself. This article provides policymakers and civil servants with an evidence-based guide to making that choice responsibly, ensuring that the integration of LLMs strengthens, rather than erodes, the integrity of evidence-based democratic governance. The article proceeds by outlining the state of the debate on LLMs in policy analysis, detailing our comparative methodology, presenting the results, and concluding with actionable recommendations for public institutions seeking to govern the responsible adoption of these AI tools.
2. The state of the debate
The integration of AI into the core processes of the state is a central feature of Digital Era Governance, representing a fundamental shift in the epistemic foundations of public administration (Margetts and Dorobantu, Reference Margetts and Dorobantu2019). This transformation moves beyond mere efficiency gains to alter how evidence is gathered, interpreted, and deployed in the policy cycle. As public institutions increasingly turn to automated systems, they confront a critical challenge to their public legitimacy, which hinges on maintaining procedural fairness, transparency, and accountability in decision-making (Starke and Lünich, Reference Starke and Lünich2020). The central debate, therefore, is not whether to adopt AI, as it is already happening, but how to govern its adoption in a way that enhances, rather than erodes, the integrity of evidence-based policy.
The current frontier of research demonstrates the efficacy of LLMs for analysing policy texts, moving far beyond the capabilities of traditional natural language processing (NLP) techniques (Alla et al., Reference Alla, Oprescu, Hall, Whiteford, Head and Meurk2018). Unlike traditional NLP methods requiring extensive programming skills, LLMs offer an intuitive interface, democratising access to powerful analytical tools and potentially enhancing research and policy analysis efficiency. The literature shows that these models have been deployed for a range of sophisticated tasks, including the generation of plausible, if often generic, policy analyses (Safaei and Longo, Reference Safaei and Longo2024) and quantitative applications, such as the ideological scaling of political documents (Le Mens and Gallego, Reference Le Mens and Gallego2025). This proliferation of use-cases signals an urgent need for governance frameworks that can keep pace with practice (Ornstein et al., Reference Ornstein, Blasingame and Truscott2025). The most significant application for the daily work of the civil service lies in qualitative annotation, or “text-coding,” where LLMs have been shown to dramatically accelerate the analysis of large datasets (Heseltine and Clemm von Hohenberg, Reference Heseltine and Clemm von Hohenberg2024; Nguyen-Trung, Reference Nguyen-Trung2025).
This emerging body of work highlights a core governance trade-off for public administration. On the one hand, LLMs promise to democratise and scale analytical capacity. Their intuitive interfaces and ability to standardise the coding process can improve consistency and support collaborative analysis across teams (Katz et al., Reference Katz, Wei, Nanda, Brinton and Ohland2023; Zhang et al., Reference Zhang, Wu, Xie, Rubino, Graver, Kim, Carroll and Cai2024), while their proficiency in synthesising complex information (Borger et al., Reference Borger, Ng, Anderton, Ashdown, Auld and Blewitt2023; Ray, Reference Ray2023) can aid in producing coherent narratives from diverse and disparate sources (Lund et al., Reference Lund, Wang, Mannuru, Nie, Shimray and Wang2023). On the other hand, there are also fundamental institutional risks. The reliance on models trained on vast, unverified data introduces the peril of amplifying societal biases, while the subjective and error-prone nature of manual analysis (Morgan, Reference Morgan2022; Ortloff et al., Reference Ortloff, Fassl, Ponticello, Martius, Mertens, Krombholz and Smith2023) is replaced by the opacity and potential unreliability of AI systems (Christou, Reference Christou2023). Even in direct performance comparisons, LLMs struggle with the highly nuanced and context-dependent interpretation that is the hallmark of expert policy analysis (Heseltine and Clemm von Hohenberg, Reference Heseltine and Clemm von Hohenberg2024).
This state of the debate reveals a critical disconnect that our study is the first to address. The methodological literature, while vibrant, is primarily focused on performance: can the LLM perform a specific analytical task to a sufficient standard? The governance literature, conversely, has focused on the risks of predictive models, leaving the governance of interpretive LLM applications underexplored. This article bridges that gap. We move beyond a simple performance test to ask a higher-order governance question about institutional modality for deploying these proven analytical capabilities, their respective trade-offs in terms of accountability, cost, and analytical depth. We thus provide a comparative empirical framework for governing the practice of AI-assisted qualitative policy analysis within public institutions.
3. Data and methods: Operationalising four governance archetypes
To evaluate empirically the four archetypes introduced previously, we designed a comparative study that operationalises each archetype as a distinct analytical workflow. This approach allows us to move beyond theoretical discussion and generate concrete evidence on the practical trade-offs of different modes of LLM integration. To ground our analysis in a realistic public administration use case, all four workflows were applied to a single, high-relevance corpus: a set of United Nations policy documents (63, English-language) concerning the governance of AI, sourced from the UN Digital Library between 2017 and March 2024 (Turobov et al., Reference Turobov, Coyle and Harding2024a). Data were collected using the keyword “AI,” excluding non-textual elements like tables and graphs. We are using the thematic analysis framework across all four archetypes, assuming that it is most relevant for real-life scenario policy analysis and for ease of comparison.
A note on the analytical frame, since the unit of analysis in this study is the analytical workflow, not UN AI discourse itself—the corpus functions as a common, high-relevance instrument against which the four archetypes are benchmarked. Valid comparison requires that a single analytical task be held constant across a manual expert team, an unsupervised statistical model (LDA), a prompted LLM, and an API-based pipeline. We adopt thematic analysis (Braun and Clarke, Reference Braun and Clarke2012) for this purpose for three reasons. First, its sequential, well-specified structure can be rendered as an explicit, auditable, and replicable instruction set (Supplementary Appendix S1), which is precisely what makes it operationalisable and governable across both human and automated archetypes, including topic modelling, which yields only distributional term-clusters and admits no positioned interpretation. Second, as an established, widely standardised qualitative procedure, it reflects the realistic public administration use case this study is designed to evaluate, rather than a specialist analytical posture. Third, and most consequentially for our argument, an interpretive approach, for example, a critical discourse analysis, is by constitution theory-laden, analyst-positioned, and reflexive that cannot be held constant across an unsupervised model and an automated pipeline, and, more fundamentally, the positioned and ideological interpretation it entails is the very analytical labour that this article contends must remain human and must not be delegated to or automated by an LLM. The descriptive register of the archetype outputs reported below is thus a deliberate property of the comparative design: the study documents what each workflow produces, and it does not advance substantive claims about UN AI discourse.
Throughout, “analysis” refers to the thematic coding and theme-development task held common across the four archetypes, and “analytical depth” or “in-depth analysis” denotes not discourse-level critique but the degree of context-sensitive interpretation an archetype can sustain. The design does not claim any prior assumptions—the risk of imposing predetermined categories is mitigated by inductive, data-driven coding—the manual archetype is conducted without applying theoretical frameworks or assumptions, and by anchoring every code to a verbatim quotation, which keeps each interpretive step traceable and open to validation.
3.1. The human-in-the-loop archetype: LLM-augmented analysis via prompt engineering
To enhance and accelerate the initial coding phase of thematic analysis—traditionally a manual task—we employed a custom GPT-4 model (GPT-4-turbo-2024-03-13 version). We acknowledge the rapid evolution of LLMs. While newer model iterations may offer different performance characteristics, the fundamental trade-offs between these four archetypes—representing distinct approaches to automation and human oversight—remain structurally relevant as a durable framework for institutional decision-making. The process is presented as follows: the LLM performs the labour-intensive initial coding and clustering, while the human analyst directs the process and retains all subsequent theme development and interpretation. The model followed a three-step process for each document (Turobov et al., Reference Turobov, Coyle and Harding2024b):
-
1. Data familiarisation and initial coding: identifying key text segments, assigning codes, and extracting relevant quotations.
-
2. Aggregating codes into thematic clusters: grouping codes to support manual theme development.
-
3. Generating AI-specific codes: focusing on codes explicitly related to AI.
The effectiveness of this approach hinges on high-quality prompts (Fiannaca et al., Reference Fiannaca, Kulkarni, Cai and Terry2023). Techniques like Few-Shot Learning (provides the model with limited examples—see Zhao et al. (Reference Zhao, Wallace, Feng, Klein and Singh2021)), Chain-of-Thought (encourages the model to articulate intermediate reasoning steps—see Wei et al. (Reference Wei, Wang, Schuurmans, Bosma, Xia, Chi, Le and Zhou2022)), and Role-Playing Scenarios (prompt the model to respond from specific personas or perspectives—see Gao (Reference Gao2023)) guide language models like ChatGPT to produce contextually appropriate and meaningful results (Zhang et al., Reference Zhang, Wu, Xie, Rubino, Graver, Kim, Carroll and Cai2024). We prepared the model by integrating a knowledge base that included a detailed thematic analysis guide (Naeem et al., Reference Naeem, Ozuem, Howell and Ranfagni2023), an instructional guide (Kiger and Varpio, Reference Kiger and Varpio2020), and a chapter on thematic analysis from the Handbook of Research Methods (Braun and Clarke, Reference Braun and Clarke2012).
Supplementary Appendix S1 reproduces the complete prompting protocol verbatim—the role specification, the integrated knowledge base, the three-step instruction script, the additional instructions, and the few-shot exemplars—and is intended as the reproducible record of this archetype. The protocol applied a fixed combination of prompt-engineering techniques (Role-Playing Scenarios, Few-Shot Learning, Chain-of-Thought prompting, and structured step-by-step instructions). For the API archetype, the decoding parameters were a temperature of 0.5 and a 4000-token limit, with the basis for these choices set out below; the custom-GPT interface used for the human-in-the-loop archetype does not expose such controls, which is itself one of the parameter-control trade-offs recorded in Table 6.
3.2. The institutional-scale archetype: ChatGPT API-based automated analysis
Utilising the ChatGPT API for thematic analysis offers significant advantages in control and flexibility. By encoding the analysis process, researchers can create loops and sequences of instructions, reducing the risk of the model overlooking parts of complex commands. The API also allows manual adjustments of parameters like temperature and token limits. Temperature controls randomness in responses: higher values (e.g., 1) yield more diverse outputs, while lower values (e.g., 0) produce more focused results. Limiting tokens manages response length, ensuring outputs remain concise and relevant, enhancing thematic exploration’s precision.
We conducted a fully automated thematic analysis by integrating the ChatGPT API (GPT-4o version) with a Python script. The script used PyMuPDF to extract text from PDF documents and followed steps mirroring traditional thematic analysis (Naeem et al., Reference Naeem, Ozuem, Howell and Ranfagni2023): data familiarisation, code generation with quotations, theme development, and summarising the analysis in a comprehensive table. Initially, the script extracted and pre-processed the text, removing irrelevant sections like tables of contents or references. It then prompted the model to identify significant phrases related to AI, its applications, ethical considerations, and technological advancements. The generated codes were organised into broader themes by clustering similar codes, encapsulating the main ideas in the data. Finally, the findings were structured into a CSV file containing themes, codes, and corresponding quotations.
We defined specific roles and detailed prompts for the ChatGPT model to ensure consistency and replicability. We tested various temperature settings to balance diversity and focus; a temperature of 0.5 was optimal, as lower values (0.2) limited code identification and higher values (0.8) introduced artefacts. We also adjusted token limits to ensure comprehensive coverage without exceeding the model’s capacity, selecting a limit of 4000 tokens given the GPT-4 model’s maximum of 4096 tokens.
3.3. The macro-scale archetype: Topic modelling using LDA
This archetype represents the macro-scale end of the framework’s trade-off: mapping the thematic landscape of a corpus too large to read closely, at minimal cost and oversight. Automated content analysis of this kind is an established practice in policy analysis (Alla et al., Reference Alla, Oprescu, Hall, Whiteford, Head and Meurk2018). It is this triage function (as topic modelling is rarely used as a routine desk task) that the archetype operationalises and presents to the same governance comparison as the other three.
We applied latent Dirichlet allocation (LDA) (Blei et al., Reference Blei, Ng and Jordan2003; Daud et al., Reference Daud, Li, Zhou and Muhammad2010) to analyse the same corpus of UN policy documents. LDA is a generative statistical model that uncovers unobserved groups explaining similarities within data. It assumes each document is a mixture of topics, and each topic is a distribution over words (Jelodar et al., Reference Jelodar, Wang, Yuan, Feng, Jiang, Li and Zhao2019), making it helpful in identifying underlying themes in large text collections.
To enhance input quality for the LDA model, we pre-processed the text data using the “tm” package (Feinerer, Reference Feinerer2013). The text was converted to lowercase, stopwords, punctuation (preserving intra-word dashes), and numbers were removed, and lemmatisation was performed using the “textstem” package (Rinker, Reference Rinker2018) to reduce words to their base forms. We constructed a Document-Term Matrix (DTM) with the “text2vec” package (Stoltz and Taylor, Reference Stoltz and Taylor2022), generating tokens and creating a vocabulary while excluding specified stopwords. The DTM was converted to a sparse matrix suitable for LDA, ensuring meaningful document representation by removing empty rows.
To determine the optimal number of topics (K), we utilised the “ldatuning” package (Nikita, Reference Nikita2016). Figures 1 and 2 in Supplementary Appendix S2 display the results using metrics like CaoJuan2009 (K = 36), Arun2010 (K = 31), and Deveaud2014 (K = 50), which assess model coherence and topic distinctiveness. CaoJuan2009 method evaluates the similarity between topics. A lower value indicates that topics are more distinct from each other, suggesting a better model. Arun2010 method involves computing a divergence measure between the distributions of topics and documents. A lower value suggests that the model better captures the underlying structure of the data. Deveaud2014 metric calculates the average cosine similarity between topic pairs. A lower average similarity indicates better-defined and more distinct topics.
Due to inconsistencies and a significant drop in marginal topic distribution after 10–15 topics, we conducted additional assessments using the “stm” package (Roberts et al., Reference Roberts, Stewart and Tingley2019), employing four metrics (Figure 2 in Supplementary Appendix S2):
-
• Held-Out Likelihood: Measures model generalisation to unseen data; higher values indicate better generalisation.
-
• Residuals: Indicate model fit; lower values suggest a better fit.
-
• Semantic Coherence: Assesses topic interpretability; higher values mean topics are more meaningful to humans.
-
• Lower Bound: Evaluates model stability; higher values indicate greater stability.
Our analysis showed these metrics stabilised in the 15–20 topic range. After manual assessment, we selected 14 topics (despite using an “English only” filter, some UN documents contained duplications in other languages; one of the 15 topics consisted entirely of foreign words, so we excluded it) to balance interpretability, accuracy, and semantic coherence (Weston et al., Reference Weston, Shryock, Light and Fisher2023).
We trained the LDA model using Gibbs sampling with the “topicmodels” package (Grün and Hornik, Reference Grün and Hornik2011), setting the seed for reproducibility and iterating 1000 times. From the trained LDA model, the main topics were identified, and the distribution of terms within each topic was analysed to understand the thematic composition. This approach systematically uncovers hidden thematic structures in UN policy documents, offering insights into prevalent topics and their associated terms.
3.4. The baseline archetype: Manual expert-driven thematic analysis
We enlisted 5 experts with PhDs in International Studies, Psychology, Law/Legal Studies, Political Science, and Organisation Studies to conduct a robust manual thematic analysis. Each expert analysed three UN policy documents, performing an inductive, data-driven thematic analysis without applying theoretical frameworks or assumptions. They focused on AI as the central theme to ensure it remained the focal point throughout their analysis. The experts had 3 weeks to complete the task, allowing for a thorough examination. Participation was anonymous, and unique IDs were utilised for each expert. Assignments were managed through parallel double coding with random assignment.
Parallel double coding involved two experts independently analysing each document to enhance inter-rater reliability and minimise bias. A balanced distribution algorithm assigned documents so that each expert reviewed three documents, maintaining consistent workloads and analytical rigour. We developed this algorithm in R, sampling documents and assigning them to experts using a random selection process and a matrix to track pairings, adjusting assignments for balance. Due to having an odd number of experts and the need for double coding, one expert was assigned two documents instead of three to maintain overall fairness.
After analysis, each expert submitted a table containing the document title, identified themes with corresponding codes, and specific quotations from the text.
4. Results
This section presents the empirical results from applying our four distinct archetypes to the corpus of United Nations policy documents. The findings reveal that while all four workflows successfully identified core thematic areas, such as human rights, ethics, and international cooperation, each produced qualitatively different analytical outputs. These differences highlight the distinct strengths, limitations, and, most importantly, the inherent governance trade-offs associated with each approach to AI-assisted policy analysis.
4.1. The human-in-the-loop archetype: Augmenting the civil servant with LLM prompting
This archetype, representing a collaborative model between a human analyst and an LLM, proved to be a balanced and highly effective workflow. It achieved significant efficiency gains over manual analysis while preserving essential human oversight in the final development of themes. The scale of this gain is concrete: the manual baseline required five subject-matter experts over 3 weeks to cover three documents each, whereas this workflow applied the initial coding stage for each text across the full 63-document corpus, confining expert effort to theme development rather than exhaustive manual extraction. The LLM successfully executed the initial, labour-intensive coding phase, identifying 713 distinct codes and providing specific, context-rich quotations to support them (Table 1). The subsequent human-led analysis produced a coherent and nuanced set of 12 final themes, including “Ethical AI” and the “UN’s Role in AI Governance” (Table 2). This workflow exemplifies a co-production model, empowering civil servants to conduct in-depth analysis on a larger scale without sacrificing interpretive control.
ChatGPT initial coding results
Table 1. Long description
A three-column table presenting a sample of the initial coding output of the human-in-the-loop archetype. Column one gives the source document (United Nations title and year of publication), column two the code assigned, and column three a verbatim quotation from that document exemplifying the code. The table’s purpose is to show that every code is tied to a specific quotation, so that each interpretive step remains traceable to the source text.
Developed themes
Table 2. Long description
A two-column table setting out the twelve final themes produced by human-led theme development on the codes generated in the human-in-the-loop archetype. Column one names the theme; column two gives a short description of its scope, followed by an illustrative quotation drawn from the corpus. The themes span normative concerns (rights, ethics, inclusion), sectoral applications (education, public services, security), and institutional questions (the UN’s own role, national strategy, geopolitical context).
4.2. The institutional-scale archetype: Achieving high-precision automation via API
The institutional-scale archetype, which simulates a more deeply integrated automated system, showcased the power of programmatic LLM analysis. This workflow generated 473 codes with high precision, which were then refined into nine distinct, coherent themes such as “Socioeconomic Development and Inclusivity” and “AI in Military and Security Contexts” (Table 3). A key governance finding from this archetype is the critical importance of human-led parameter tuning and post-processing. Because each document was analysed separately, the raw output comprised 57 aggregated themes; reducing these to the nine reported themes was not an unstructured re-coding but a directed two-step consolidation (a prompted aggregation pass and a prompted overlap-resolution pass, for example, collapsing the several ethics-related themes) with the human role confined to specifying the consolidation criteria and validating the final themes against the underlying codes and quotations. While the API enables analysis at scale, its output required a final human refinement stage to manage thematic overlaps and consolidate findings, demonstrating that even “fully automated” systems necessitate a robust human governance layer to ensure the quality and validity of the final analytical product.
Fully automated ChatGPT API results
Table 3. Long description
A three-column table presenting the output of the institutional-scale, API-based archetype. Column one names the theme, column two lists sample codes assigned under it (given as short semicolon-separated phrases), and column three gives one or more verbatim quotations from the corpus corresponding to those codes. Nine rows correspond to the nine consolidated themes.
4.3. The macro-scale archetype: Rapid triage with automated topic modelling
The macro-scale archetype, operationalised through LDA, demonstrated its utility for providing rapid, high-level situational awareness across the entire corpus. This workflow was effective at identifying broad, recurring topics such as “AI Risks,” “Data Privacy and Protection,” and “Digital Education” (Table 4). For a public institution, this archetype serves as a powerful triage tool, enabling the quick mapping of the thematic landscape of a large document set. Its primary limitation, however, is its lack of contextual understanding—the model identifies clusters of words but cannot interpret their specific meaning or extract actionable, quote-backed evidence, making it unsuitable for detailed policy analysis. This is a boundary of the method rather than a weakness, since the archetype is deployed only for macro-scale analysis, with interpretive depth provided by expert-led manual interpretation.
Topic modelling results
Table 4. Long description
A table reporting the fourteen topics from Latent Dirichlet Allocation analysis. Each row gives an interpretive topic label assigned by the authors, the topic number, and the ten highest-weighted terms for that topic, separated by hyphens; square brackets expand truncated or abbreviated word stems. The output consists only of distributional term clusters: no quotations or document-level context accompany the topics.
4.4. The baseline archetype: The depth and nuance of expert-driven analysis
The traditional workflow of manual analysis by subject-matter experts served as the benchmark for analytical depth. As shown in Table 5, this archetype excelled at capturing nuanced procedural dynamics and complex stakeholder relationships that are often implicit in policy texts. The experts’ ability to perform deep contextual interpretation provided a rich, granular understanding of the documents, and this depth remains the standard against which the other archetypes are assessed. Its constraint is one of reach, not quality: expert analysis is resource-intensive and does not scale to the volume of unstructured text that, as noted at the outset, exceeds the analytical capacity of the civil service. The governance question is therefore not whether scale is preferable to depth (the comparative framework treats neither as intrinsically superior) but how far analytical reach can be extended without sacrificing the interpretive depth that only expert judgement provides.
Experts analysis results
Table 5. Long description
A table reporting the manual, expert-driven baseline archetype, organised as seven sequential blocks. Each block is headed by the title and date of a United Nations document and contains two side-by-side columns, one for each of the two experts who independently coded that document under the parallel double-coding design. Experts are identified by anonymised codes in the range 10061 to 10065. Within each cell, coding is presented as a set of category headings, each followed by a list of associated codes separated by semicolons.
4.5. Comparative analysis: A governance framework for institutional choice
Each archetype provides unique analytical value but comes with a distinct set of trade-offs. Table 6 aggregates these findings into a comparative governance framework to guide institutional decision-making. The results clearly show that the choice is not between “human or machine,” but rather a strategic decision about how to balance the competing demands of analytical speed, cost, contextual depth, and the level of human oversight required to maintain accountability and rigour in the policy analysis process. Scale and granularity are best understood as serving different governance purposes rather than competing standards. Granular expert analysis recovers the taken-for-granted assumptions and implicit relationships that only close contextual reading can surface; macro-scale analysis cannot match that depth, but provides the rapid situational awareness needed across a corpus too large for close expert reading, indicating where deeper scrutiny should then be directed.
Benefits and limitations of each method
Table 6. Long description
A three-column comparative table summarising the trade-offs between the four archetypes. Column one names the method; columns two and three list its benefits and its limitations respectively, each as a series of short semicolon-separated points. The four rows are: ChatGPT with prompt; ChatGPT through API; Topic modeling; and Manual expert analysis. Read across the rows, the table shows that returns in scale and speed are consistently paid for in contextual depth or in the amount of human validation required.
All four archetypes recovered the same core thematic areas—human rights, ethics, security, international cooperation, and sustainable development—so the broad policy landscape proved robust to the choice of method, and the archetypes diverge in granularity. Depth-oriented workflows produced finer, quotation-anchored coding (713 codes resolved to 12 themes for the human-in-the-loop workflow, 473 to 9 via the API), whereas topic modelling yielded 14 coarse clusters and expert analysis a rich but document-bounded reading. Because the task is inductive interpretation rather than classification against a known key, there is no single correct output against which accuracy can be scored: the operative question is not which archetype is most accurate but which best balances the governance demands of a given purpose—the basis on which we recommend the human-in-the-loop workflow.
-
• Expert Manual Analysis offers depth and precision with unique insights through direct quotations; best for detailed analysis of a limited number of texts but is resource-intensive.
-
• Topic Modelling excels in identifying broad themes across large datasets, providing a macro-level view; limited in contextual interpretation, making it less suitable for in-depth analysis.
-
• ChatGPT with prompts balances depth and efficiency, offering context and specific quotations without requiring programming knowledge; suitable for moderate-scale analyses.
-
• ChatGPT via API provides control and flexibility, effective for researchers familiar with programming, balancing breadth and detail.
5. Discussion: Governance implications of AI-assisted policy analysis
Our comparative analysis moves the conversation about AI in the public sector to comparative empirical evidence, contributing to the discourse on “governing with intelligence” (Yar et al., Reference Yar, Hamdan, Anshari, Fitriyani and Syafrudin2024). The results demonstrate that the choice between analytical workflows is not a neutral technical decision, but a fundamental governance choice with direct trade-offs between efficiency, cost, analytical depth, and human accountability. This aligns with the frame reflection approach, which posits that policy choices are shaped by competing frames of a problem (Newman and Mintrom, Reference Newman and Mintrom2023). Our four archetypes can be understood as four distinct institutional frames for AI integration, and the choice between them is a deliberate act of governance that will profoundly shape the nature and quality of the evidence base available to policymakers.
A core finding of this study is that the most promising and responsible path for LLM integration lies in co-production/LLM assistance, rather than replacing human expertise. The “Human-in-the-Loop” archetype (ChatGPT with prompts) offered the most effective balance of speed and analytical rigour. This workflow democratises advanced analytical capabilities, enabling civil servants to conduct sophisticated analyses at scale without sacrificing final interpretive judgment. This finding offers an empirical counter-narrative to simplistic scenarios of automation or replacement. It aligns with a more critical perspective on public sector AI, which warns against a “techno-solutionist” mindset that devalues the tacit knowledge and ethical reasoning of human experts (Paul, Reference Paul2022). Actual progress lies in creating collaborative systems where AI serves as an augmentative tool, enhancing the capacity of the policy analyst who remains central to the process.
Effective co-production, however, requires robust governance protocols. The use of computational methods for policy evaluation is an established practice that has always required careful methodological design and oversight, whether using machine learning techniques in specific domains like agriculture or higher education policy (Ballestar et al., Reference Ballestar, Doncel, Sainz and Ortigosa-Blanch2019; Sánchez et al., Reference Sánchez, Rodríguez and Espitia2020) or deploying modern LLMs. Our experimentation revealed that even advanced models can produce artefacts or misinterpret instructions, underscoring the non-negotiable need for critical human oversight. The practical techniques used in this study, such as clear, step-by-step instructions and role-playing scenarios (best practices to maximise LLM’s potential provided in Supplementary Appendix S1), should therefore be understood not as mere “prompt engineering,” but as essential governance protocols for ensuring the quality and validity of AI-assisted analysis. Public institutions should invest in developing these skills to mitigate the inherent risks of model fallibility.
The “Institutional-Scale” archetype, reliant on a commercial API, highlights a critical long-term governance consideration. While this workflow offers powerful automation, it relies on proprietary, third-party models, raising concerns about data privacy, security, and strategic autonomy. For some institutions, this can be a present constraint: where data-governance rules restrict the uploading of policy documents to an external proprietary model, the prompt-based and API archetypes may be unacceptable irrespective of their analytical merits, making the admissibility of a workflow under an institution’s data regime a prior condition of the choice this framework describes. We therefore propose that the current API-driven model represents a transitional phase. The logical maturation for public administration is a move towards sovereign analytical capacity: the use of open-source or custom-built LLMs hosted locally and fine-tuned on institution-specific data. This approach, although more resource-intensive, would provide public institutions with greater control and the ability to develop models more closely aligned with public values, a crucial step in building trustworthy AI systems for governance (Yar et al., Reference Yar, Hamdan, Anshari, Fitriyani and Syafrudin2024).
The findings of this study should be considered in light of its limitations. Our analysis was conducted using a single model family (OpenAI’s GPT-4) and a specific corpus of UN policy documents. While this controlled design provides strong internal validity, the framework itself is model-agnostic by construction (the four archetypes describe modes of human-machine division of labour rather than properties of a particular model), but its empirical characteristics were established with a single proprietary model family. Establishing whether open-source or locally hosted models, which many institutions will be obliged to use on data-governance grounds, reproduce these characteristics across diverse policy domains is the priority direction for further research. Further research is also needed to test the robustness of our four-archetype framework across a wider range of LLMs and diverse policy domains. Moreover, the practical challenges of LLM deployment, such as token limits for very long documents and the potential for models to amplify biases present in the source material, require ongoing investigation and the development of more sophisticated mitigation strategies. A further limitation concerns the choice of analytical frame. Applying thematic analysis consistently across all four archetypes was necessary for valid comparison, but it sets aside the questions a discourse-analytic reading would emphasise: how policy language constructs authority and legitimacy, and how some framings become dominant while others are marginalised. Whether a discourse-analytic approach could itself be evaluated through a comparable governance lens, given that it does not lend itself to the same step-by-step specification, is a promising direction for further research.
6. Conclusion: Governing the AI reality of evidence-based policy
The integration of LLMs into the analytical work of public administration is a governance act, not merely a technological procurement. The choice of how to deploy these powerful tools—whether as a macro-scale triage system, a tool for co-production, or a fully automated institutional workflow—is a strategic decision with direct consequences for the integrity, accountability, and quality of evidence-based policymaking. Our empirical comparison of four distinct archetypes provides an evidence-based framework for navigating the inherent trade-offs between speed, cost, analytical depth, and human oversight. By moving beyond a simple performance test to evaluate the governance implications of different workflows, this article provides a durable framework for public institutions seeking to adopt AI responsibly.
Based on this evidence, we make four key recommendations for policymakers and public managers. First, public institutions should treat the adoption of AI-assisted analytical tools as a strategic governance decision that requires leadership engagement, rather than delegating it as a purely technical matter. Second, our evidence strongly suggests prioritising “human-in-the-loop” workflows that augment, rather than replace, the expertise of the civil service. This archetype offers the most effective balance of efficiency and analytical rigour, democratising access to advanced tools while preserving essential human judgment. Third, the successful deployment of such workflows requires a parallel investment in the institutional capacity of the civil service. This extends beyond technical literacy to include the development of robust governance protocols for prompt design, output validation, and bias mitigation. Finally, public administration should develop a long-term strategy focused on establishing a robust analytical capacity through the use of tuned open-source or locally hosted models, thereby ensuring greater security, control, and alignment with public values.
The ongoing AI transformation of the state presents both profound opportunities and significant risks. LLMs offer a powerful means to enhance the analytical capacity of public institutions, enabling them to meet the challenges of an increasingly complex, data-rich world. However, the benefits of these technologies can only be realised if they are governed with foresight and a clear-eyed commitment to democratic principles. By making the choices outlined in this framework deliberately and responsibly, public administration can ensure that the adoption of AI strengthens, rather than erodes, the integrity of the evidence base upon which democratic governance depends.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/dap.2026.10100.
Acknowledgements
With enormous appreciation to Professor Helena Farrand Carrapico, Professor Benjamin Farrand, Dr Nina Jorden, Dr Yulia Karpich, and Dr Anna Antonova for their partnership throughout this project and for providing experts manual thematic analysis. For helpful and extensive comments on topic modelling and work with API, we thank Dr Maria del Rio-Chanona.
All errors are our own.
Author contribution
Conceptualisation: A.T.; D.C.; Data curation: A.T.; Formal analysis: A.T.; Methodology: A.T.; D.C.; Software: A.T.; Validation: A.T.; Project administration: D.C.; Resources: D.C.; Writing—original draft: A.T.; review & editing: D.C.
Data availability statement
Dataset “Research data supporting “Using ChatGPT for Thematic Analysis Working Paper: UN policy documents 2017–2024.” with a list of the UN policy documents and press releases, a file with codes generated by a custom GPT model and a ReadMe file available in the Apollo—University of Cambridge Repository. https://doi.org/10.17863/CAM.108401.
Funding statement
This research was supported by a grant from the MacArthur Foundation (Grant No. 23-2306-157778-TPI).
Competing interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical standard
The research meets all ethical guidelines, including adherence to the legal requirements of the study country.
This research received ethical approval from Department of Politics and International Studies (POLIS), University of Cambridge.
Working paper
This article builds upon and expands a previously published working paper: Turobov, A., Coyle, D., & Harding, V. (2024). Using ChatGPT for thematic analysis. arXiv preprint arXiv:2405.08828. https://arxiv.org/abs/2405.08828.
Open access
Comments
No Comments have been published for this article.