Policy Significance Statement
AIMS-QA is an AI-driven tool for policymakers, regulators, academics, and civil society assessing corporate modern slavery statements. By synthesising metrics from academic and civil-society methodologies, it moves beyond legal compliance to assess the quality and depth of disclosures. Its taxonomy distinguishes statutory baseline questions (jurisdiction-specific legal obligations) from beyond compliance questions (stakeholder and civil-society expectations), avoiding the conflation of soft expectations with hard legal duties. AIMS-QA supports scalable analysis of thousands of disclosures across jurisdictions. It is a triage and screening aid that surfaces evidence and reasoning for human reviewers, not a substitute for enforcement: it is not intended to determine legal compliance, but to support structured analysis against both statutory and non-statutory expectations.
1. Introduction
1.1. Background and problem
Modern slavery, including forced labour, human trafficking, and exploitation, remains a critical global issue that affects millions of individuals worldwide (Walk Free, 2023). In response, governments have implemented legislation such as the UK Modern Slavery Act (MSA) (UK Parliament, 2015) and the Australian MSA (Australian Government, 2018), which require large companies to disclose annual statements outlining the steps they have taken to identify and address modern slavery risks in their operations and supply chains. These transparency requirements aim to promote corporate accountability and transparency, while also enhancing supply chain governance.
As of 2025, a decade after the first MSA, tens of thousands of corporate statements have been published and submitted to government registries (Australian Government, Attorney-General’s Department, 2024; UK Government, 2024) or company websites. Compliance assessments by regulatory bodies, academic institutions, and civil society organisations (CSOs) are conducted manually and cover only a small portion annually. The vast majority remain unexamined, as no organisation can manually review all statements each year. Despite rising mandatory reporting, enforcement remains a major challenge, hindering large-scale transparency and accountability.
In addition, to ensure that corporate statements are meaningful and actionable, it is essential to assess not only whether companies comply with the law but also the depth, quality, and relevance of their reported actions. Academics and CSOs have developed open-source methodologies with specific metrics to assess modern slavery disclosures. While informed by mandatory legislative criteria, most incorporate additional metrics reflecting stakeholder priorities, resulting in diverse, often incomparable frameworks tailored to investors, advocacy groups, or industry specialists. For example, investor-driven approaches emphasise risk management and due diligence, whereas CSOs working with survivors prioritise human rights outcomes or remediation. Sector-specific analyses introduce industry-relevant indicators, such as responsible sourcing of conflict minerals.
This variation challenges comparability and accountability. Without a shared framework, establishing high-quality responses or identifying industry leaders is difficult. We use “standardisation” to denote consistency in methodologies employed by stakeholders such as academics, CSOs, and investors when assessing disclosures. While regulatory divergence across jurisdictions echoes broader sustainability disclosure debates, our analysis focuses on the methodological fragmentation, limiting comparability across studies and sectors. Examining legal variation implications remains important for future research.
This fragmented landscape of assessment frameworks, combined with the sheer volume of statements and limited capacity for manual review, underscores the urgent need for scalable, consistent, and adaptable evaluation methods. In this context, artificial intelligence (AI) plays an increasingly important role in disclosure analysis. Existing systems such as AIMS.au (Bora et al., Reference Bora, St-Charles, Bronzi, Fansi Tchango, Rousseau and Mengersen2025b) and AIMSCheck (Bora et al., Reference Bora, Arodi, Zhang, Bannister, Bronzi, Fansi Tchango, Bashar, Nayak, Mengersen, Che, Nabende, Shutova and Pilehvar2025a) demonstrate the potential of fine-tuned large language models (LLMs) and zero-shot prompting for cross-jurisdictional compliance evaluation. However, these approaches remain limited to surface-level binary classification for compliance checks, constrained by static prompts struggling with incomplete or ambiguous evidence, and lack adaptation for granular analysis aligned with diverse stakeholder priorities.
This paper focuses primarily on the Australian MSA and, secondarily, the UK MSA, as these offer the most established, widely scrutinised reporting regimes. Their regulatory maturity, large volume of publicly available disclosures, and sustained academic and civil-society attention make them ideal for developing scalable assessment frameworks. We adopt the Australian regime as our primary anchor for both doctrinal and technical reasons that we develop in Section 2.2; the UK regime is treated as a comparison case, with its Section 54(5) categories used as analogous content domains rather than as equivalent mandatory obligations. Other regimes, such as Canada’s Fighting Against Forced Labour and Child Labour in Supply Chains Act (2023) or California’s Transparency in Supply Chains Act (2010), have generated fewer disclosures or received less benchmarking attention. Yet, while anchored in the AU and UK contexts, the framework developed in this study is adaptable to other legal regimes with similar obligations, contributing to comparative disclosure analysis with broader international relevance.
Aligning diverse stakeholder priorities presents challenges, particularly when reflecting competing normative commitments or regulatory objectives. Full standardisation may be undesirable, potentially overlooking important contextual differences or legitimate variations. Our methodological focus balances comparability needs with recognising that some pluralism may be inevitable and valuable.
To address these limitations and unlock the full potential of AI in this context, there is a need for a systematic mapping of the questions and metrics used across existing benchmarking methodologies. Rather than simply comparing methodologies, research should synthesise their core elements and organise them in a way that distinguishes statutory obligations from stakeholder-driven expectations, so that the resulting framework can speak credibly to both regulators (who are concerned with legal duties) and other stakeholders (who are concerned with substantive impact). Such a framework can help identify areas of consensus, highlight the most valued indicators, and offer clearer guidance to businesses on what constitutes meaningful disclosure beyond basic compliance, without inadvertently presenting soft expectations as hard legal duties. To ensure consistent and accountable assessment across companies over time, the framework should also be adaptable and scalable. When integrated with an AI-driven system that embeds appropriate governance safeguards, it can support efficient, high-quality analysis of large volumes of statements, laying the foundation for a more transparent, consistent, and impactful evaluation of modern slavery disclosures.
1.2. Our contribution
This paper responds to these challenges and makes two key contributions, one to the policy and regulation space and another to the field of data science and AI, as illustrated in Figure 1.
Overview of paper contributions to policy and data/AI.
Figure 1. Long description
The flowchart is divided into two main vertical sections enclosed in dashed purple borders.
1. Left Panel: POLICY & TAXONOMY.
- Top: Literature represented by four document icons with magnifying glasses.
- Middle: A large blue box titled 13 Open Benchmarking Methodology Families. It contains three sub-sections: a grid of 20 colored dots labelled 473 Metrics, a vertical list of 5 shapes labelled 115 Topics, and a branching diagram labelled 7 A U M S A Criteria.
- Bottom: An arrow from the middle box points to a yellow box labelled 209 Streamlined Questions.
2. Right Panel: DATA & A I.
- Top: A document icon labelled Modern Slavery Statement.
- Upper Middle: A yellow box labelled Streamlined Question with a lightbulb icon. This box is connected to the 209 Streamlined Questions from the left panel by a thick grey arrow.
- Lower Middle: A processing stage with two agents. Agent 1: Classifier uses a gear and A I node icon. Agent 2: R A G Watchdog uses an A I node and lightbulb icon. An exclamation mark icon sits between them.
- Bottom: The process concludes with a box for Output Answer + Reasoning, which leads to a final box for Human Reviewer featuring a person icon and a bar chart.
First, this paper provides a comprehensive and systematic synthesis of the current landscape of modern slavery disclosure benchmarks. It identifies and consolidates 473 individual metrics drawn from 13 unique publicly available benchmarking methodologies developed by academics and CSOs. These metrics are mapped to the seven reporting criteria of the Australian MSA and further simplified into 209 distinct questions grouped under 115 thematic topics. We adopt the AU criteria as an analytic benchmark for organisational coherence; the rationale and limits of this choice are discussed in Section 2.2. Within this organisation, each question is tagged as belonging to either a statutory baseline layer (questions corresponding to jurisdiction-specific legal duties such as approval, signature, and the other AU MSA mandatory criteria) or a beyond compliance layer (questions reflecting stakeholder, investor, and civil-society expectations). As part of this contribution, the paper also introduces a prioritisation framework that identifies which questions are most frequently used, across at least three of the independent methodologies. This helps highlight areas of consensus among diverse stakeholders, including investors, CSOs, and policy analysts, and guides both regulators and businesses required to report.
Second, this paper contributes a scalable, AI-driven methodology for evaluating modern slavery disclosures. It introduces a two-stage AI pipeline that begins with a state-of-the-art, pre-trained, sentence-level classification model to extract relevant contextual information (also referred to in this paper as “evidence”) from corporate statements, aligning content with the AU MSA mandatory criteria adopted as our analytic benchmark. This extracted evidence is then passed to a large language model (LLM), guided by domain-specific prompts, to generate structured answers to predefined assessment questions. To improve robustness and address cases where the first stage fails, the pipeline incorporates a Watchdog mechanism based on retrieval-augmented generation (RAG) (Lewis et al., Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela2020), which monitors for missing or insufficient evidence and recontextualises the query before re-evaluating the question. This watchdog component performs question-level (rather than criterion-level) retrieval, complementing Agent 1’s coarser-grained classification, and ensures that the system can recover from unsatisfactory outputs by dynamically retrieving and integrating relevant information from the source statement.
This combination of methodological flexibility, semantic granularity, and a quality-oriented prioritisation framework positions AIMS-QA as a more scalable, transparent, and policy-relevant solution than prior systems, one that moves beyond surface-level binary compliance checks to enable nuanced, evidence-aware analysis at scale, bridging the gap between regulatory intent and actual reporting practice.
2. Background and related work
2.1. Modern slavery laws and disclosures
The International Labour Organization (ILO) defines modern slavery as “situations of exploitation that a person cannot refuse or leave because of threats, violence, coercion, deception, and/or abuse of power” (International Labour Organization et al., 2017). This umbrella concept encompasses forced labour, human trafficking, debt bondage, forced marriage, and other slavery-like conditions, shifting from historical conceptions based on legal ownership towards understanding rooted in coercion, control, and exploitation.
In today’s globalised economy, such practices are embedded within complex international supply chains, making detection and measurement difficult. Modern slavery’s hidden nature challenges evidence-based policymaking and enforcement. The ILO, Walk Free, and the International Organisation for Migration (IOM) jointly publish the Global Estimates of Modern Slavery (International Labour Organization et al., 2022), complemented by the Global Slavery Index (GSI) (Walk Free, 2023), to improve understanding. The 2022 estimates indicate approximately 50 million people lived in modern slavery situations on any given day in 2021, a 10 million increase since 2016.
This increase shows that the pace of interventions is failing to match the scale and growth of the problem, in part driven by compounding global crises, including armed conflict, climate change, and the socioeconomic impacts of the COVID-19 pandemic (Walk Free, 2023). This underscores the urgent need for scalable, data-driven solutions to generate evidence, guide regulatory enforcement, and inform effective policymaking.
International policy response to modern slavery has been reinforced by UN Sustainable Development Goal Target 8.7, which calls for states to “Take immediate and effective measures to eradicate forced labour, end modern slavery and human trafficking and secure the prohibition and elimination of the worst forms of child labour, including recruitment and use of child soldiers, and by 2025 end child labour in all its forms.” (United Nations Department of Economic and Social Affairs, 2015). The UN 2030 Agenda, UN Guiding Principles on Business and Human Rights (Office of the United Nations High Commissioner for Human Rights, 2011), and OECD Due Diligence Guidance for Responsible Business Conduct (OECD, 2018) have prompted increased regulatory initiatives strengthening corporate accountability for human rights.
Legislative frameworks such as the state-level California Transparency in Supply Chains Act (2010) (Harris, Reference Harris2015), the national-level UK MSA (2015) (UK Parliament, 2015), the Australian MSA (2018) (Australian Government, 2018), and more recently, similar laws such as the Canadian Fighting Against Forced Labour and Child Labour in Supply Chains Act (Public Safety Canada, 2023) require large companies to publish annual statements disclosing the steps they are taking to identify and address modern slavery risks within their operations and supply chains.
2.2. Assessment methods
Although these laws impose formal reporting requirements, evaluating their effectiveness requires methodological clarity. From a legal research perspective, two dominant approaches are typically used to study such frameworks: doctrinal legal analysis, which examines the law as written, and empirical legal research, which investigates how the law operates in practice (Davies, Reference Davies2020). Doctrinal analysis entails a systematic study of legal sources, such as statutes, case law, and regulatory frameworks, to uncover their core principles, structure, and interrelationships. This approach is essential for grasping the theoretical basis of legal provisions and how they are applied within various business and human rights laws.
From a doctrinal analysis perspective, researchers examine the key reporting requirements of each law. Importantly, the statutory design varies across jurisdictions, and these differences matter for how disclosures should be assessed. Under the Australian MSA, Section 16 sets out seven mandatory reporting criteria that an entity’s annual statement must address, and Sections 13–14 require the statement to be approved by the principal governing body and signed by a responsible member of that body. Under the UK MSA, by contrast, Section 54(5) is framed in permissive terms: a statement may include information on six categories (organisational structure, policies, due diligence, risk assessment, effectiveness, and training). These categories are therefore best understood as recommended or illustrative content rather than as strictly mandatory disclosure requirements, although other formalities under Sections 54(6)–(7), such as approval by the board of directors (or equivalent management body), signature by a director (or equivalent), and publication on the entity’s website, are clearly mandatory. The Canadian Fighting Against Forced Labour and Child Labour in Supply Chains Act 2023, Section 11(3), prescribes a list of mandatory disclosure items closer in form to the Australian regime.
This paper adopts the Australian MSA, Section 16 criteria as the organising anchor for our question taxonomy, for two complementary reasons.
First, doctrinal tractability. As explained above, among the three regimes, the Australian Act is the earliest of the three to enumerate an exhaustive list of mandatory content requirements (the seven criteria listed below). The UK MSA Section 54(5) categories are permissive (“may include”) rather than mandatory, and the Canadian Act Section 11(3), while prescriptive, was enacted in 2023 and has produced relatively few statements to date. Anchoring on the AU MSA mandatory criteria, therefore, provides the most operationally tractable foundation for a unified taxonomy.
Second, model alignment. The fine-tuned ModernBERT classifier used by AIMS-QA’s first-stage agent (described in Section 3.2) is trained on the AIMS.au dataset, whose sentence-level annotations are themselves aligned to the AU MSA mandatory criteria. Organising the question taxonomy around these same criteria ensures that the classifier’s predictions can be directly used to retrieve relevant evidence for each question. A taxonomy organised around UK or Canadian categories would require either retraining the classifier on a new annotated dataset (which does not currently exist at the necessary scale) or accepting some mismatch between retrieval categories and question categories. Crucially, anchoring on the AU criteria is an organisational choice, not a normative one. We do not claim that the disclosures expected under the AU MSA constitute the universal standard for what a modern slavery statement should contain across all jurisdictions. Where we refer to UK categories alongside the AU criteria, we treat them as analogous content domains for cross-jurisdictional comparison, not as equivalent mandatory obligations. The two-tier tagging introduced in Section 3.1 (Step 5) further ensures that questions reflecting stakeholder expectations beyond the statute are clearly distinguished from those that reflect statutory baseline obligations. Section 16 of the AU MSA requires reporting entities to submit an annual statement, approved by the principal governing body and signed by a responsible member of that body, that addresses seven criteria:
-
• Criterion 1: Identify the reporting entity.
-
• Criterion 2: Describe the entity’s structure, operations, and supply chains.
-
• Criterion 3: Describe the risks of modern slavery practices in the entity’s operations and supply chains, including those of any owned or controlled entities.
-
• Criterion 4: Describe the actions taken to assess and address those risks, including due diligence and remediation.
-
• Criterion 5: Describe how the effectiveness of those actions is measured.
-
• Criterion 6: Describe consultation with any entities the reporting entity owns or controls during the preparation of the statement.
-
• Criterion 7: Any other relevant information.
A comparative mapping of these criteria with the UK and Canadian regimes is provided in Appendix A, Table A1, with markers and a note indicating which items are mandatory and which are permissive in each jurisdiction. For example, the Australian Act uniquely requires consultation processes, whereas the Canadian Act introduces provisions for remediating income loss to vulnerable families. Jurisdictional variations prevent wholesale application of taxonomies across legislative frameworks without adaptation. Our proposed framework is a flexible foundation tailored to local reporting requirements and is not a one-size-fits-all model.
From an empirical legal perspective, research examines how law operates in practice (Davies, Reference Davies2020). In the modern slavery context, this includes examining how corporations respond to disclosure obligations, such as the work of Rogerson et al. (Reference Rogerson, Crane, Soundararajan, Grosvold and Cho2020) and Flynn and Walker (Reference Flynn and Walker2021), and assessing the quality and substance of their reporting, such as the work of Pham et al. (Reference Pham, Cui and Ruthbah2021) and Schaper and Pollach (Reference Schaper and Pollach2021).
Previous empirical research on modern slavery disclosure has predominantly focused on the UK and Australian MSAs, which have been in force long enough to generate a critical mass of corporate statements. These studies have largely fallen into two categories that will be addressed separately in the following sub-sections: manual reviews and more recent approaches that incorporate computational or AI-driven techniques.
2.2.1. Manual review approaches
A growing body of empirical research has employed manual review methods to evaluate corporate responses to modern slavery disclosure laws. These studies vary in scope, with some focused solely on statutory statements under the MSAs, while others analyse additional corporate materials such as annual reports, Environmental, Social, and Governance (ESG) disclosures, and supplier codes. For example, Rogerson et al. (Reference Rogerson, Crane, Soundararajan, Grosvold and Cho2020) and Islam and van Staden (Reference Islam and Staden2022) used interviews with reporting entities and stakeholders to highlight concerns over regulatory effectiveness and limited behavioural change. Thematic content analysis is another common approach. Ras and Gregoriou (Reference Ras and Gregoriou2019) analysed metaphors in corporate disclosures, revealing symbolic narratives around responsibility. Flynn and Walker (Reference Flynn and Walker2021) applied content analysis to UK FTSE firms’ modern slavery statements through an institutional-theory lens, examining the pressures shaping corporate responses. While offering valuable insights, such approaches still rely on manually coded text and often focus on a limited subset of high-profile firms.
Structured content analysis has also been used to assess compliance with legislative requirements and quality of reporting through scoring rubrics or checklists. In the Australian context, Christ et al. (Reference Christ, Rao and Burritt2019) and Rao et al. (Reference Rao, Burritt and Christ2022) established pre-legislative baselines, examining the quantity and quality of voluntary disclosures by large listed companies before the Australian MSA took effect. Following implementation, recurring benchmarking projects led by Pham et al. (Reference Pham, Cui and Ruthbah2021, Reference Pham, Cui and Ruthbah2022, Reference Pham, Cui and Ruthbah2023, Reference Pham, Cui and Ruthbah2024) systematically assessed statutory statement quality across ASX300 and subsequently ASX100 companies over successive reporting cycles.
CSOs and independent research centres have also played an important role. Business and Human Rights Resource Centre (2017) and CCLA Investment Management (2024) have each developed benchmarks to assess disclosure quality across sectors. Other notable initiatives include the Domus 8.7 Index (ACAN, 2022) and the Moving from Paper to Practice report (Australian Council of Superannuation Investors, 2021), which analysed ASX200 company reporting practices under Australia’s MSA. Additional methodologies and empirical assessment frameworks cover the assessment of Australian corporate statements (International Justice Mission Australia, 2022), substantive supply-chain reporting (Schaper and Pollach, Reference Schaper and Pollach2021), the Australian university sector (Unger et al., Reference Unger, Moolchand and Marshall2022), and responses to the California Transparency in Supply Chains Act (Birkey et al., Reference Birkey, Guidry, Islam and Patten2018).
Several initiatives have tracked corporate responses across successive reporting cycles. The Beyond Compliance studies led by Walk Free and WikiRate (2019, 2021, 2022, 2023, 2024), and the benchmarks published by KnowTheChain (2022, 2025) both assess successive rounds of statements against a stable methodology. In the Australian context, Paper Promises (Sinclair and Dinshaw, Reference Sinclair and Dinshaw2022) and Broken Promises (Dinshaw et al., Reference Dinshaw, Nolan, Hill, Sinclair, Marshall, McGaughey, Boersma, Bhakoo, Goss and Keegan2022) evaluated the early impact of the MSA, and Moolchand et al. (Reference Moolchand, Marshall, Coffey and McErvale2023) extended this approach to meat company statements across two reporting years. Together, these studies demonstrate how structured, repeated review can track not only compliance with modern slavery legislation but also whether entities honour the commitments made in their own earlier statements.
Manual assessment methods face significant limitations. They are labour-intensive and difficult to scale as disclosure volumes grow: the Domus 8.7 Index required 70 interns and over 5000 hours to review 1500 statements (ACAN, 2022), and evaluating more than 3300 statements on WikiRate for the Beyond Compliance series took several years. Assessor subjectivity also introduces inconsistency, although this can be mitigated through training, inter-rater reliability measures, and multiple coding rounds. As disclosure requirements expand globally, these constraints make it increasingly difficult for manual review to keep pace, underscoring the need for scalable, AI-driven techniques.
2.2.2. Emerging data and AI-driven approaches
While traditional assessments of modern slavery disclosures have primarily relied on manual methods, a growing body of research has begun exploring the use of data science and AI to scale and automate compliance evaluations.
Machine learning and natural language processing techniques, including unsupervised approaches such as topic modelling, have been used to analyse modern slavery statements from the UK (Bora, Reference Bora2019; Nersessian and Pachamanova, Reference Nersessian and Pachamanova2022). These studies reveal trends in how companies frame disclosures, often linking them to broad human rights narratives. However, such methods lack the precision needed to verify compliance with specific legal criteria.
A larger-scale data-driven initiative was led by Walk Free and The Future Society (The Future Society, 2022). This project analysed over 16,000 statements, leveraging a subset of approximately 2400 annotated disclosures from WikiRate (Walk Free and WikiRate, 2019) to train supervised classifiers. These classifiers aimed to detect the presence of mandatory criteria and beyond (e.g., board approval) using keyword extraction, annotator comments, and n-gram features. However, the authors note that the quality of annotations was inconsistent, and the reliance on surface-level features limited scalability and reliability, particularly for longer and more complex disclosures.
More recent approaches have explored the potential of LLMs for regulatory compliance tasks. For example, Poon et al. (Reference Poon, Rahimikia, Vallavanthra, Wei, Popenici, Rudolph, Ismail and Tan2025) applied LLMs to replicate the Business & Human Rights Resource Centre’s FTSE100 scoring framework. Their work focused on prompt engineering to automate disclosure evaluation, highlighting the promise of LLMs to scale manual benchmarking approaches without fine-tuning. However, their study remains largely exploratory, with limited empirical evaluation.
Yet, it was Bora et al. (Reference Bora, St-Charles, Bronzi, Fansi Tchango, Rousseau and Mengersen2025b) who introduced the first fine-tuned transformer-based models, BERT and LLaMA, trained on AIMS.au, the largest open-source, sentence-level annotated dataset of Australian modern slavery statements. Their models classify compliance with the mandatory reporting criteria, demonstrating how general-purpose LLMs can be adapted to perform precise legal classification tasks. Building on this, Bora et al. (Reference Bora, Arodi, Zhang, Bannister, Bronzi, Fansi Tchango, Bashar, Nayak, Mengersen, Che, Nabende, Shutova and Pilehvar2025a) developed AIMSCheck, a modular, jurisdiction-aware pipeline that integrates fine-tuned models to assess statements’ compliance with reporting criteria across Australia, the United Kingdom, and Canada. Their system benchmarks LLM performance across legal domains, supports cross-jurisdictional compliance evaluation, and incorporates explainability tools and advanced prompt techniques such as Chain-of-Thought (CoT) (Wei et al., Reference Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le and Zhou2022) reasoning to enhance transparency and interoperability.
Follow-up work by Bora et al. (Reference Bora, Zhang, Bashar, Nayak and Mengersen2026b) builds on this foundation and introduces a distillation-based framework that unifies four specialised teacher models, each leveraging a unique method, such as pseudo-labelling, prompt augmentation, contrastive learning, or context-enhanced learning, into a single, efficient student model. Their approach integrates their complementary strengths at the representation level through ensemble distillation. This enables the student model, a fine-tuned ModernBERT variant, to retain high performance while remaining lightweight and deployment-appropriate.
While prior AI-driven approaches have marked significant progress in scaling the assessment of modern slavery statements, they primarily focus on classifying the presence or absence of content aligned with mandatory reporting criteria. These systems rely on fixed, sentence-level classification pipelines that, while effective for compliance auditing, lack the semantic flexibility, depth, and adaptability required for more nuanced, policy-relevant analysis. AIMS-QA builds directly upon this foundation but introduces several critical advancements that move the field forward in meaningful ways. Most notably, it replaces the binary compliance classification paradigm with a two-stage question answering (QA) AI pipeline that enables flexible, evidence-aware interrogation of disclosures. Further distinguishing AIMS-QA is its introduction of a comprehensive, expert-informed question taxonomy with 209 questions derived from 473 metrics consolidated across 13 civil-society and academic benchmarking methodologies. A 37-question priority framework is also introduced, comprising 35 consensus questions that appear in at least three independent methodologies and two statutory baseline questions retained irrespective of frequency.
Together, these innovations make AIMS-QA a substantive advance in data- and AI-driven systems for evaluating modern slavery disclosures beyond compliance. It enables nuanced, evidence-sensitive, and high-throughput analysis that adapts to evolving regulatory and stakeholder needs. By bridging the gap between legal compliance and meaningful accountability, AIMS-QA offers a more scalable and policy-relevant solution than prior AI-driven approaches.
3. Methodology
3.1. Synthesis of disclosure assessment metrics
This synthesis methodology followed a structured, multi-step approach to identify, categorise, and streamline disclosure assessment metrics from the literature.
Step 1: Finding the benchmarking methodologies
To develop a structured foundation for evaluating disclosures related to modern slavery, we built upon the literature review discussed in Section 2.2.1. From the studies identified, we found 13 available benchmarking methodologies, each with associated assessment criteria. Figure 2 visualises the application of these methodologies in the literature.
Overview of benchmarking methodologies from academic and civil-society studies.
Figure 2. Long description
The diagram is organised into color-coded rectangular blocks.
Top-left pink block: Walk Free and WikiRate - Beyond Compliance. Includes five studies: Hotel Sector 2019, Finance Sector 2021, Garment Industry 2022, Renewable Energy 2023, and Electronics Sector 2024.
Top-centre teal block: KnowTheChain - Benchmarking. Includes two periods: 2022 to 2024 and 2025 to 2026.
Top-right blue block: Business and Human Rights Resource Centre 2017.
Middle-right purple block: C C L A - Modern Slavery U K Benchmark 2024.
Bottom-left red block: Monash University - Modern Slavery. Includes four Pham et al. studies: A S X 100 in 2024, 2022, and 2023, and A S X 300 in 2021.
Bottom-centre green block: Paper Promises Series. Includes Sinclair and Dinshaw 2022 - Paper Promises, Dinshaw et al. 2022 - Broken Promises, and Moolchand et al. 2023 - Where is the Beef?
Centre-right blue block: Birkey et al. 2018 - Mandated Social Disclosure.
Centre-right pink block: Christ et al. 2019 - Accounting for Modern Slavery.
Middle-right orange block: Flynn 2021.
Middle-right yellow block: I J M Australia 2022 - Spot Fires in Supply Chains.
Middle-right grey block: Schaper and Pollach 2021 - Modern Slavery Statements.
Bottom-right light grey block: Rao et al. 2022 - Voluntary Modern Slavery Disclosures.
Bottom-right olive block: Unger 2022 - University Modern Slavery Reporting.
Step 2: Extracting the metrics from benchmarking methodologies
We collected 473 metrics from all available benchmarking methodologies. Each unique benchmarking methodology was mapped only once, even if it was applied across multiple subsequent studies. However, if any of these follow-up studies introduced additional metrics not present in the original methodology, those unique metrics were also included. For instance, the Beyond Compliance series is treated as a single methodology, and its common metrics, repeated across studies, were mapped only once. However, when unique metrics, such as the one focused on the renewable energy sector, were found in one of their studies, they were also retained in our synthesis.
Step 3: Mapping metrics to AU MSA reporting criteria as an analytic benchmark
Each metric was independently reviewed by two modern slavery reporting domain experts, who assigned (i) a corresponding reporting criterion from the Australian MSA and (ii) an emergent thematic topic, as illustrated in Figure 3. Each metric was mapped to one of the AU MSA mandatory reporting criteria for the reasons explained in Section 2.2 (doctrinal tractability and alignment with the underlying classifier). This organisational mapping does not imply that every metric mapped to a given criterion reflects a mandatory legal obligation in Australia or in any other jurisdiction. Many metrics, for example those concerning living-wage assurance, supplier disclosure beyond tier one, or named grievance mechanisms accessible to workers, originate in stakeholder, investor, or civil-society expectations rather than statutory text. The two-tier tagging introduced in Step 5 makes this distinction explicit at the question level.
Mapping of the Modern Slavery Act’s mandatory reporting criteria to thematic topics.
Figure 3. Long description
A radial diagram with a central hub connecting to a dense outer ring of text labels via thin grey lines.
At the centre are nine coloured nodes: seven reporting criteria plus approval and signature:
* A U C 1: Reporting (Green)
* A U Sig Entity (Red)
* A U Approval (Orange)
* A U C 6: Consultation (Brown)
* A U C 7: Other (Light Blue)
* A U C 3: Risk and Incidents Identification (Purple)
* A U C 2: Structure, Operations and Supply chains (Yellow)
* A U C 5: Effectiveness of Actions (Pink)
* A U C 4: Risk Mitigation and Remediation (Dark Blue)
The outer ring contains numerous specific thematic topics organised by these categories.
Starting from the top and moving clockwise:
* The Dark Blue section (A U C 4) is the largest, containing dozens of labels related to mitigation and remediation, such as mitigation - due diligence, mitigation - forced labour, and remediation - grievance mechanism.
* The Pink section (A U C 5) includes K P I, effectiveness, and results - remediation.
* The Yellow section (A U C 2) includes supply chain, structure, and operations.
* The Purple section (A U C 3) includes risks - identification, forced or involuntary overtime, and incident.
* The Light Blue section (A U C 7) includes C O V I D - 19, statement, and investor.
* The remaining small sections (Green, Red, Orange, Brown) include reporting entity, signature, board approval, and consultation.
Thematic analysis followed a bottom-up, inductive approach, allowing granular elements of each metric to inform broader conceptual groupings. Experts first worked independently on a subset of the data, followed by collaborative sessions to resolve discrepancies and refine a shared coding protocol. The coding process proceeded iteratively, supported by regular calibration meetings to ensure consistency and coherence. Upon completing the thematic mapping, a single expert conducted a final consistency review across all assignments.
Step 4: Assigning streamlined questions and thematic topics
As many metrics were repeated across the benchmarking methodologies, the experts developed streamlined question texts to represent the essence of each metric. These questions were informed by the original metric language, the assigned theme, and the relevant legislative criterion. This process resulted in 209 streamlined questions organised under 115 thematic topics, forming a unified, interpretable taxonomy of disclosure quality.
Step 5: Two-tier tagging; statutory baseline versus beyond compliance
To preserve a clear conceptual separation between legal obligations and stakeholder-driven expectations, each streamlined question was further tagged by the same two domain experts into one of two layers: statutory baseline (S), questions that correspond directly to disclosure obligations enumerated in the Australian MSA, and beyond compliance (B), questions originating in academic and civil-society methodologies that reflect stakeholder, investor, or sector-specific expectations and that go beyond the text of the statute. This two-tier tagging is not intended to dismiss beyond compliance questions; on the contrary, our framework is designed precisely to surface them. Rather, the tagging makes explicit that AIMS-QA evaluations carry different normative weight depending on the layer: a “No” on a statutory-baseline question signals possible non-compliance with a legal duty, whereas a “No” on a beyond compliance question signals a gap relative to good-practice expectations. This separation also guards against the inadvertent “hardening” of soft expectations into perceived legal duties, particularly when assessment tools are deployed by or on behalf of public bodies.
Step 6: Selecting the priority framework for analysis
The full taxonomy of 209 questions provides comprehensive coverage but is too large for routine human review or rapid AI-assisted triage. We therefore developed a priority framework, a smaller, operationally tractable subset, guided by two principles. First, stakeholder convergence as a robustness signal: we retained questions appearing in at least three of the 13 independent methodologies. Convergence across methodologies developed by different stakeholder groups (investors, civil society, academics, and sector-specific analysts) signals indicators that capture widely shared expectations rather than the idiosyncratic priorities of any single methodology. The threshold of three balances robustness (a higher threshold would exclude emerging indicators) and inclusivity (a lower threshold would be biased towards indicators specific to only one or two methodologies). This yielded 36 questions meeting the frequency threshold, of which 35 were retained in the priority framework. We excluded one time-bound indicator concerning how COVID-19 affected an entity’s modern slavery risks. Although it appeared in four methodologies, this convergence reflects the pandemic-era context in which those methodologies were developed or applied rather than an enduring cross-year disclosure expectation. Second, statutory baseline as a non-negotiable floor: we added two further questions that did not meet the threshold but reflect mandatory AU MSA obligations, signature, and reporting entity identification. Statutory baseline obligations are included regardless of convergence, because failure to address them indicates possible non-compliance with a legal duty rather than a gap relative to good practice. The resulting framework contains 37 questions (Appendix A, Table A2). The priority framework is a practical layer on top of the full taxonomy, not a replacement: rapid triage is well served by the priority framework, while in-depth audits and research applications can draw on the full 209-question taxonomy. Each question is tagged S (statutory baseline) or B (beyond compliance), per Step 5. As frequency-based filtering reflects the priorities of existing methodologies, the framework should be revisited periodically to incorporate emerging indicators and evolving best practice.
3.2. AIMS-QA framework
To operationalise the QA task on complex modern slavery statements, we introduce AIMS-QA, a two-stage AI framework that integrates classification-based reasoning with a RAG approach. This design enables high-resolution, regulation-aligned QA tailored to the nuanced requirements of modern slavery disclosures.
Literature shows that relying solely on LLMs for QA can introduce hallucinations (Asai et al., Reference Asai, Wu, Wang, Sil and Hajishirzi2024) undermining output reliability. RAG (Lewis et al., Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela2020) mitigates this risk by grounding answers in supporting documents, thereby enhancing factual accuracy. Yet, despite its advantages, traditional RAG approaches may still retrieve irrelevant or misleading content (Asai et al., Reference Asai, Wu, Wang, Sil and Hajishirzi2024), due to the retriever’s limited ability to capture fine-grained, domain-specific semantics. In the context of modern slavery statements, accurate responses often hinge on a nuanced understanding of the legal reporting criteria and the stakeholder-driven expectations layered on top of them, granularity that typical RAG systems often miss.
To address these limitations, recent advances have reframed RAG as a conditional verifier or watchdog rather than a default evidence provider. For instance, Ding et al. (Reference Ding, Pang, Wei, Shen and Cheng2024) introduce a consistency-based mechanism that activates RAG only when an LLM’s response lacks coherence. In a related vein, Self-RAG (Asai et al., Reference Asai, Wu, Wang, Sil and Hajishirzi2024) retrieves passages on demand and uses self-reflection signals to critique both the retrieved evidence and the generated output.
Building on this paradigm, AIMS-QA dynamically delegates reasoning tasks across two coordinated agents. In this framework, agents pursue the shared goal of identifying and presenting relevant information from corporate disclosures in response to the given questions. Rather than relying on general knowledge acquired during the pre-training of LLMs, the framework is designed to ground all reasoning in the content of the statements themselves. As illustrated in Figure 4, it includes two cooperating agents, Agent 1 and Agent 2, each using a distinct strategy to achieve this goal.
The proposed AIMS-QA framework.
Figure 4. Long description
The flowchart is divided into two primary dashed purple boxes and a final review stage.
Agent 1: Classifier (Left Panel)
- Inputs: Streamlined Question and a document icon feed into the process.
- A U M S A Criteria box feeds into a Fine-tuned Modern B E R T block.
- The document passes through a Sentence Splitter, which then feeds into the Fine-tuned Modern B E R T.
- The output of the B E R T model is labelled Evidence.
- Evidence is processed by Ge M M A (L L M).
- Logic paths: If No Context or No Answer is found, the flow redirects to Agent 2. If an Answer is found, it proceeds to an Output box containing Answer and Reasoning.
Agent 2: R A G Watchdog (Right Panel)
- Inputs: Receives the No Answer redirection from Agent 1, along with the Streamlined Question and document icon.
- The document passes through a Chunk Splitter.
- The Streamlined Question and chunked data feed into a Fine-tuned Sentence B E R T Retriever.
- The retriever produces Evidence, which is then processed by Ge M M A (L L M).
- This results in an Output box containing Answer and Reasoning.
Final Stage (Bottom)
- Both Output boxes from Agent 1 and Agent 2 point to a final dashed blue box labeled Human Reviewer, depicted with a person icon and a report icon.
Throughout this paper, Agent 1 refers to the evidence-extraction agent, which uses the fine-tuned ModernBERT classifier to identify statement evidence relevant to a given criterion. Agent 2 refers to the RAG Watchdog agent, which is triggered when Agent 1 cannot ground a sufficiently supported answer from the extracted evidence alone.
Agent 1: classifier
Leveraging the structured taxonomy described in the previous section, where each streamlined question maps to a specific criterion under the AU MSA (used here as our analytic benchmark), AIMS-QA first applies a sentence-level classifier to identify relevant evidence. A statement in PDF format is first parsed into individual sentences using a Sentence Splitter, a component that segments long text into separate sentences. These sentences are then assessed by a state-of-the-art classification model built on the fine-tuned ModernBERT backbone (Warner et al., Reference Warner, Chaffin, Clavié, Weller, Hallström, Taghadouini, Gallagher, Biswas, Ladhak, Aarsen, Cooper, Adams, Howard, Poli, Che, Nabende, Shutova and Pilehvar2025; Bora et al., Reference Bora, Zhang, Bashar, Nayak and Mengersen2026b), which is a transformer-based model specifically adapted for identifying relevant content in modern slavery statements. ModernBERT was selected because, among the transformer-based models and LLMs benchmarked in prior work (Bora et al., Reference Bora, Zhang, Bashar, Nayak and Mengersen2026b) on the AIMS.au dataset, it achieved the strongest sentence-level classification performance on the AU MSA mandatory criteria, making it the most reliable backbone for the evidence-retrieval stage of AIMS-QA. This model, trained on a large number of statements from the AIMS.au dataset (Bora et al., Reference Bora, St-Charles, Bronzi, Fansi Tchango, Rousseau and Mengersen2025b), predicts whether a sentence is relevant to a given criterion based on the predefined mapping to legislative criteria. If a sentence is predicted as relevant, an Evidence Retriever gathers both the sentence and its surrounding context (the evidence), specifically, the 100 tokens immediately preceding and following the target sentence, consistent with the context window used to train the underlying classifier in prior work (Bora et al., Reference Bora, Zhang, Bashar, Nayak and Mengersen2026b), and sends them to the Gemma 2 27B LLM for response generation. The LLM produces an output consisting of the direct answer, such as “Yes” or “No,” along with the reasoning behind the decision. If an output is generated, the answer, explanation, and supporting evidence are presented to the human reviewer (the user). If no answer is produced, such as when no evidence is extracted or the model replies with “Not enough context,” the system transitions to Agent 2. The two agents are deployed sequentially rather than in parallel for two reasons. First, computational efficiency: invoking the RAG Watchdog (Agent 2) requires re-encoding the entire statement and running an additional Gemma 2 27B LLM pass, which would substantially increase inference cost if applied to every question on every statement. Second, complementary roles: Agent 1 is calibrated for high precision on cases where the classifier confidently identifies relevant evidence, while Agent 2 is designed as a fallback for ambiguous or evidence-sparse cases where Agent 1 cannot ground an answer. Running both agents by default would not improve recall on the cases Agent 1 already handles confidently, but would substantially increase computational and human-review burden. Future work could explore parallel execution with answer reconciliation, particularly in deployment contexts where computational cost is less constrained than analytical confidence.
An important methodological choice is that Agent 1 retrieves evidence at the criterion level rather than at the question or topic level. The ModernBERT classifier is trained on the AIMS.au sentence-level annotations, which are themselves provided at the criterion level (e.g., C4: Risk Mitigation and Remediation), because no public sentence-level annotated dataset currently exists at the finer question level. Criterion-level retrieval therefore maximises recall: it surfaces all sentences plausibly relevant to a given criterion and lets the downstream Gemma 2 27B LLM, conditioned on the specific question text, perform the finer-grained relevance judgement during answer generation. This design has a known cost: the LLM may sometimes return a correct answer based on partially off-topic evidence, the “right answer for wrong reason” failure mode that is particularly undesirable in oversight settings where explainability and auditability matter. We mitigate this in three ways. First, Agent 2 (the RAG Watchdog) performs question-level retrieval using a SentenceBERT encoder fine-tuned against the question text, providing a finer-grained complement when Agent 1 cannot ground an answer. Second, every output is presented to a human reviewer together with the LLM’s reasoning chain and the cited evidence, so reviewers can detect cases where the rationale does not match the question. Third, our error analysis (Appendix B) explicitly identifies cases where the model arrives at a correct answer based on irrelevant or partially off-topic evidence, surfacing this failure mode for further attention. We discuss in Section 5 how a question-level classifier trained on a future expansion of the AIMS.au annotations at finer granularity could, in principle, enhance Agent 1’s criterion-level retrieval and remove this limitation.
Agent 2: RAG watchdog
When Agent 1 fails to identify relevant evidence or when the initial response from the LLM indicates that no answer can be provided, a secondary agent is activated. Agent 2 implements a RAG approach, which combines document retrieval with generative reasoning to produce evidence-based answers (Lewis et al., Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela2020). We refer to this component as the RAG Watchdog throughout the paper. Its conditional use of retrieval is informed by adaptive-retrieval approaches such as Self-RAG (Asai et al., Reference Asai, Wu, Wang, Sil and Hajishirzi2024) and the consistency-based retrieval mechanism proposed by Ding et al. (Reference Ding, Pang, Wei, Shen and Cheng2024). The statement is first divided into overlapping chunks (multi-sentence text segments) using a Chunk Splitter, with each chunk containing approximately five consecutive sentences to preserve the surrounding context (i.e., a chunk is larger than a single sentence but smaller than a full statement section). To identify the most relevant sections, the system uses a fine-tuned SentenceBERT Retriever (Reimers and Gurevych, Reference Reimers and Gurevych2019). SentenceBERT is a transformer-based encoder that encodes each text segment (in this framework, each chunk) and the input question into numerical vectors and ranks chunks by their semantic similarity to the question. In this framework, the retriever has been fine-tuned on the AIMS.au dataset to select chunks that are most closely related to the compliance analysis questions, performing retrieval at the question (rather than criterion) level. These top-ranked chunks are passed to the same Gemma 2 27B LLM used by Agent 1, which generates a second-round output with answers and reasoning. The resulting output is then presented to the human reviewer (the user) to ensure quality and accuracy.
This design reflects an explicit commitment to human-in-the-loop validation. Rather than positioning the framework as a fully automated decision-maker, AIMS-QA submits each answer to a human reviewer together with the model’s reasoning and supporting evidence. In doing so, the reviewer does not simply receive a binary outcome (e.g., “Yes”/“No”) but also the interpretive chain of reasoning underpinning it. This transparency both mitigates risks of overreliance on automation and preserves human oversight in contexts where disclosures often involve legally and ethically nuanced judgements. In this way, the framework balances the scalability benefits of automation with the governance safeguards of human interpretive authority.
For both agents, we employ the LLM Gemma 2 27B (https://huggingface.co/google/Gemma-2-27b). for response generation, using a structured prompt based on the CoT technique (Wei et al., Reference Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le and Zhou2022), which asks the model to think step-by-step. Gemma 2 27B has a maximum context window of 8192 tokens. Within this budget, we set a self-imposed cap of approximately 2000 words on the evidence segment passed to the model (roughly 2600–2800 tokens depending on tokenisation). This choice was informed by manual assessment, which showed that longer evidence segments consistently led to truncation-related errors. Together with the prompt scaffolding (system role, instructions, task description, and question) and headroom for generation, this cap keeps inputs comfortably below the context limit.
As shown in Figure 5, the prompt includes five components: a system role definition (e.g., “You are an analyst specialising…”), detailed instruction steps (e.g., key rules), task descriptions (e.g., Task), the policy question, and the retrieved evidence. The framework is implemented using Python and deployed locally with Ollama (https://ollama.com/).
Example of the prompt used in the AIMS-QA framework.
Figure 5. Long description
The diagram is titled L L M Prompt Template in a dark gray header bar. Below the header, the content is organized into three main sections.
1. System message. This section defines the role as an analyst specialising in A U modern slavery declarations. It specifies the task of identifying sentences regarding approval by a governing body. It includes a Key rules section with a placeholder for Instructions and a Task section requiring a Y E S or N O answer. It concludes with a placeholder for Context.
2. User message. This section contains a placeholder for Question Here.
3. Answer. This section provides a chain of thought instruction, stating Let’s think step-by-step. In order to provide the correct answer, you need to check if the context matches the key rules. Provide the reasoning and the final answer (Y E S or N O).
By coordinating these two agents, the framework dynamically adapts to the complexity of each query and the availability of relevant evidence. Importantly, the Gemma 2 27B LLM is instructed to rely exclusively on the retrieved evidence and instructions, avoiding responses based on general pre-training knowledge. All outputs, including the answer, explanatory rationale, and evidence, are returned to human experts for review, who retain full interpretive control over final assessments.
4. Results and discussion
4.1. Classification accuracy of fine-tuned ModernBERT (component-level evaluation)
This section reports the standalone performance of the fine-tuned ModernBERT classifier, that is, Agent 1’s first stage in isolation, evaluated on the AIMS.au test set. The end-to-end performance of the full AIMS-QA pipeline (Classifier + RAG Watchdog) on the 37 priority framework questions is reported separately in Section 4.2.
Given that modern slavery statements are often vague and written in corporate jargon or legal language, accurately identifying relevant sentences and evidence remains a significant challenge. In AIMS-QA, the sentence-level classifier addresses this by identifying content relevant to each criterion under the AU MSA. When the question (e.g., Was the statement approved by the principal governing body of the reporting entity?) matches a criterion (e.g., Approval), the sentence predicted by the fine-tuned ModernBERT (Bora et al., Reference Bora, Zhang, Bashar, Nayak and Mengersen2026b) and its surrounding evidence are passed to the Gemma 2 27B LLM for answering, allowing Agent 1 to efficiently narrow down relevant content while answering the question. The classifier’s performance is evaluated using the F1 score, which balances how many selected sentences are correct (precision) and how many correct sentences are found (recall). Ranging from 0 to 1, higher F1 scores indicate better performance. On the AIMS.au test dataset (50 sentence-level annotated statements, Bora et al., Reference Bora, St-Charles, Bronzi, Fansi Tchango, Rousseau and Mengersen2025b), the classifier achieves a macro F1 score of 0.740 across the mandatory compliance tasks (Figure 6), with particularly high scores for Approval (0.866) and C2: Structure, Operations and Supply Chains (0.820), demonstrating Agent 1’s effectiveness in filtering content before triggering more complex reasoning.
F1 score for the fine-tuned ModernBERT classifier on the AIMS.au test set across the AU MSA mandatory compliance tasks.
Figure 6. Long description
The horizontal bar chart uses an x-axis labeled F 1-score ranging from 0.50 to 0.90 in increments of 0.05. The y-axis lists eight categories. A vertical red dashed line indicates a Macro F 1 of 0.740. The teal bars represent the following values from top to bottom.
* Approval: 0.866
* C 1 Reporting Entity: 0.594
* C 2: Structure, Operations and Supply chains: 0.820
* C 3: Risk and Incidents Identification: 0.756
* C 4: Risk Mitigation and Remediation: 0.785
* C 5: Effectiveness of Actions: 0.595
* C 6: Consultation: 0.722
* Signature: 0.785
The highest performance is in Approval, while the lowest scores are in C 1 Reporting Entity and C 5 Effectiveness of Actions.
4.2. AIMS-QA tests
The AIMS-QA framework was evaluated in a simulated deployment setting, in two phases, using ten statements from the AIMS.au test dataset (Bora et al., Reference Bora, St-Charles, Bronzi, Fansi Tchango, Rousseau and Mengersen2025b). In the first phase, five statements were processed and their outputs reviewed to identify system-level errors and question-design problems. Following the adjustments described below, the revised framework was evaluated on a second, disjoint set of five statements not used during that refinement. Two domain experts independently assessed the outputs for each of the 37 priority framework questions on these five statements, yielding 185 question instances (37 questions applied to each of the five statements). All quantitative results reported below derive from this second, held-out set. The unit of evaluation in this paper is the question instance: one question applied to one statement. The figure of 185 question instances therefore refers to the total number of evaluated statement–question pairs (37 priority-framework questions applied to each of 5 statements), not to 185 separate statements.
In the first phase, the outputs of the five diagnostic statements were reviewed, and a detailed error analysis was performed. This process helped identify errors related to both the AI-driven framework and the design of the questions.
-
• For instance, our evidence-handling pipeline introduced evidence-related errors, particularly for questions associated with mitigation. In these cases, the retrieved evidence sometimes exceeded our self-imposed 2000-word evidence cap. Consequently, this limitation led to failure cases in Agent 1, as some relevant evidence was not passed to the LLM.
-
• In addition to system-level limitations, the evaluation revealed several issues stemming from question design. Some questions required rephrasing to improve clarity and ensure alignment with model capabilities. For example, a question about whether the company pays a “living wage” was frequently misinterpreted by the model, which correctly identified references to minimum wage compliance but failed to assess whether the entity ensured wages that meet basic living standards. This underscores the importance of disambiguating terms that have distinct legal and ethical meanings. Furthermore, compound questions containing conjunctions such as “and” introduced interpretive rigidity: if the model found evidence addressing only part of the question, it often dismissed the entire passage as irrelevant. These findings highlight the need for careful question decomposition and precise linguistic framing to support both accurate model inference and reliable human interpretation.
To address these limitations, we modified the framework to automatically split input evidence into multiple chunks when it exceeded a 2000-word evidence cap, with each chunk processed in turn and the resulting partial answers consolidated. In parallel, domain experts revised and restructured several questions to improve clarity and alignment with model capabilities. Following these adjustments, the revised framework was applied to the five held-out second-phase statements.
This section presents a quantitative analysis of the framework’s performance on these five held-out statements, corresponding to 185 question instances. This evaluation enabled two key analyses: the measurement of inter-expert agreement and an assessment of model prediction accuracy.
Table 1 reports the inter-expert agreement and Cohen’s Kappa, which measures agreement between the two experts beyond that expected by chance and ranges from −1 to 1, with higher values indicating stronger agreement. Inter-expert agreement was higher on cases answered by Agent 1 (Classifier) (0.858; Kappa 0.714) than on cases answered by Agent 2 (RAG Watchdog), which reached 0.625 (agreement) and 0.000 (Cohen’s Kappa), respectively. This difference may be attributed to the fact that Agent 2 is only invoked when Agent 1 fails to identify adequate evidence, meaning it tends to handle more difficult or ambiguous cases. Furthermore, as Agent 2 was only invoked in 8 of the 185 question instances (those where Agent 1 returned no answer or “not enough context”), such a small number of instances may not adequately capture the variance between expert annotations, justifying the 0.000 results. Nevertheless, the overall inter-expert agreement (0.848) and Kappa (0.695) suggest that expert evaluations of the system’s outputs were generally consistent.
Expert-vs-expert agreement by source of answer in the AIMS-QA pipeline
Table 1. Long description
The table consists of four columns: Answer source, Agreement, Cohen’s Kappa, and N.
* The first row of data for Agent 1 (Classifier) shows an Agreement of 0.858, a Cohen’s Kappa of 0.714, and an N value of 177.
* The second row for Agent 2 (R A G Watchdog) shows an Agreement of 0.625, a Cohen’s Kappa of 0.000, and an N value of 8.
* The final row for Overall metrics shows an Agreement of 0.848, a Cohen’s Kappa of 0.695, and an N value of 185.
Tables 2 and 3 present the system’s accuracy evaluated against annotations from two domain experts, summarised by compliance criterion and by answer source, respectively. The AIMS-QA framework achieves perfect or near-perfect accuracy on “Approval,” “C1 Reporting Entity,” “C2: Structure, Operations and Supply Chains,” and “Signature” across both experts, reflecting strong performance on these well-defined criteria. In contrast, lower accuracy is observed on “C4: Risk Mitigation and Remediation,” “C5: Effectiveness of Actions,” and “C6: Consultation,” suggesting greater complexity or misalignment between the two experts in these categories (e.g., C6: Consultation). Agent 1 (Classifier) consistently performs well, with 0.847 and 0.909 accuracy against Expert 1 and Expert 2, respectively, supporting the effectiveness of this pipeline. While Table 1 reports agreement between the two experts, Tables 2 and 3 report how often the AIMS-QA outputs agreed with each expert. Thus, the comparison subjects differ, even though the same 185 question instances are used.
System accuracy using Expert 1 and Expert 2 as ground truth
Table 2. Long description
The table consists of three columns: Criterion, Accuracy vs. Expert 1, and Accuracy vs. Expert 2.
* Approval: 1.000 for both experts.
* C 1 Reporting Entity: 1.000 for both experts.
* C 2: Structure, Operations and Supply Chains: 0.971 for Expert 1 and 1.000 for Expert 2.
* C 3: Risk and Incidents Identification: 0.833 for Expert 1 and 0.967 for Expert 2.
* C 4: Risk Mitigation and Remediation: 0.788 for Expert 1 and 0.847 for Expert 2.
* C 5: Effectiveness of Actions: 0.733 for Expert 1 and 0.933 for Expert 2.
* C 6: Consultation: 0.600 for Expert 1 and 0.800 for Expert 2.
* Signature: 1.000 for both experts.
* Overall: 0.838 for Expert 1 and 0.913 for Expert 2.
System-vs-expert accuracy of the AIMS-QA pipeline’s agents
Table 3. Long description
The table consists of four columns and four rows including the header.
Columns from left to right are: Answer source, Accuracy vs. Expert 1, Accuracy vs. Expert 2, and N.
Row 1 (Header): Answer source, Accuracy (vs. Expert 1), Accuracy (vs. Expert 2), N.
Row 2: Agent 1 (Classifier) shows an accuracy of 0.847 against Expert 1 and 0.909 against Expert 2, with an N value of 177.
Row 3: Agent 2 (R A G Watchdog) shows an accuracy of 0.625 against Expert 1 and 1.000 against Expert 2, with an N value of 8.
Row 4: Overall results show an accuracy of 0.838 against Expert 1 and 0.913 against Expert 2, with a total N value of 185.
At the end of this phase, we conducted a detailed error analysis to assess remaining challenges in framework performance and question design. While overall output quality had improved following earlier refinements, several nuanced issues persisted. These included occasional hallucinations, cases where the model provided correct answers for incorrect reasons, and confusion when responding to compound questions. It also struggled with subtle distinctions, such as treating a whistleblowing policy as equivalent to a formal grievance mechanism. Although expert disagreement was generally low, the disagreements that did occur were primarily related to differences in interpreting whether the provided evidence was sufficiently informative or whether the language used in disclosures was too vague. These cases highlight the inherent subjectivity in interpreting complex or non-standardised corporate reporting and underscore the importance of clear reporting guidelines to support consistent evaluation. A full breakdown of these findings is provided in Appendix B.
4.3. Governance considerations for deployment
Because AIMS-QA is positioned for use by regulators and oversight bodies, its deployment raises governance considerations familiar from automated decision-making and administrative law. In line with these concerns, the framework is designed to support triage and screening rather than enforcement. Its outputs are intended to surface potentially incomplete or inconsistent disclosures for further human review, and should not be used on their own to ground adverse legal or regulatory consequences.
To support transparency and accountability, each system output is accompanied by a structured record that includes the relevant question, the extracted evidence, the model’s reasoning, the agent that produced the answer, and the model and prompt version. This documentation is intended to enable both internal quality assurance and, where relevant, external audit, aligning with administrative-law expectations around reason-giving and the contestability of decisions informed by automated tools.
At the same time, the framework incorporates safeguards to mitigate risks of over-reliance on model outputs. In particular, its design assumes a human-in-the-loop process in which reviewers retain interpretive authority and are able to assess both the evidence and the reasoning provided by the system. Where appropriate, deployment protocols can require reviewers to formulate independent answers before consulting model outputs, complemented by periodic sampling audits and targeted testing on ambiguous or adversarial disclosures to identify systematic failure modes. More broadly, the assessment of modern slavery disclosures often involves contested and value-laden judgements, particularly where reporting is vague or ambiguous. Such judgements cannot be resolved by a classifier or language model alone, which highlights the continued role of human interpretation and the importance of clear reporting and annotation standards.
Finally, the distinction between statutory baseline and beyond compliance questions plays an important governance role. By separating legally mandated disclosures from stakeholder-driven expectations, the framework reduces the risk that evaluative outputs are interpreted as statements of legal non-compliance where they instead reflect gaps relative to good practice.
5. Conclusion and future work
This paper introduced AIMS-QA, a scalable, AI-driven framework for evaluating modern slavery statements, grounded in a structured taxonomy synthesised from 13 leading benchmarking methodologies. By consolidating 473 metrics into 209 focused questions organised against the Australian MSA criteria as an analytic benchmark, we developed a harmonised and prioritised question set, explicitly tagged into a statutory baseline layer and a beyond compliance layer, to support consistent, comparative, and policy-relevant analysis. The two-stage evaluation pipeline, combining sentence-level classification with LLM-driven QA, offers a flexible and interpretable alternative to traditional surface-level binary compliance checks. A RAG Watchdog mechanism further enhances system robustness, especially in the face of ambiguous or incomplete disclosures, and provides question-level retrieval as a complement to Agent 1’s coarser-grained classification.
Our contribution to the Data for Policy community lies in demonstrating how legally grounded, expert-informed data structures can be operationalised within scalable AI systems to support regulatory monitoring and accountability, while remaining doctrinally careful about which expectations are mandatory and which are aspirational. AIMS-QA moves beyond surface-level compliance classification and enables fine-grained, question-driven evaluations across both layers of the taxonomy. This work illustrates how AI methods can be integrated into real-world policy infrastructures, offering regulators, CSOs, and researchers a practical tool for assessing corporate transparency in complex, unstructured textual data, while embedding governance safeguards (Section 4.3) appropriate to high-stakes oversight contexts.
In addition to its technical value, AIMS-QA has implications for institutional mechanisms of policy enforcement, regulatory oversight, and corporate accountability. By embedding AI into disclosure assessment, the framework intervenes in broader governance arrangements and raises important questions about the delegation of interpretive authority to machine learning systems in high-stakes regulatory domains. Although our study focuses on methodological consistency, the framework also demonstrates how AI could contribute to the standardisation and harmonisation of policy compliance mechanisms across jurisdictions, particularly where fragmented methodologies hinder comparability. In addition, by generating systematic and scalable assessments, AIMS-QA can feed into policy feedback loops, providing regulators, civil society, and businesses with actionable insights that can inform regulatory reform and shape stakeholder behaviour. In this way, AIMS-QA should be understood not only as a practical tool for scaling disclosure analysis but also as a case study in how data infrastructures and policy systems co-evolve, advancing more evidence-based and data-driven governance models in the field of human rights and ESG regulation.
Through detailed evaluation and error analysis, we identified areas for future improvement, including reducing reasoning misalignment, clarifying ambiguous questions, and better distinguishing nuanced legal and ethical distinctions (e.g., minimum vs. living wage). We also observed that expert disagreement, while limited, reflects the need for clearer reporting guidelines and structured interpretive frameworks.
Looking ahead, we identify several opportunities to enhance the accuracy, adaptability, and policy alignment of AIMS-QA. These include moving from criterion-level to question-level sentence classification once a sentence-level annotated corpus at the question granularity becomes available, exploring Mixture-of-Experts architectures that allow Agent 1 and Agent 2 to operate in parallel rather than as a sequential fallback chain, incorporating reinforcement learning from user feedback supported by human-in-the-loop validation mechanisms and operationalising the governance safeguards of Section 4.3 as standing components of any deployment. Improvements in prompt design, logic auditing, and natural language inference could further strengthen the alignment between model reasoning and outputs. Refining the phrasing of the questions through expert review and controlled templates will help reduce ambiguity, while enhanced annotation guidelines and structured consensus-building tools can address sources of expert disagreement. More details on these future directions are provided in Appendix C.
Data availability statement
The AIMS-QA implementation is archived on Zenodo at https://doi.org/10.5281/zenodo.21945942 (Bora et al., Reference Bora, St-Charles, Arodi, Zhang, Bannister, Bronzi, Fansi Tchango, Rousseau, Bashar, Nayak, Thinyane and Mengersen2026a) (see papers/aimsqa/), with ongoing development at https://github.com/mila-studios/ai4h_aims-au. The Agent 1 classifier weights are available at https://doi.org/10.6084/m9.figshare.33261576 and the Agent 2 retriever weights at https://doi.org/10.6084/m9.figshare.33261639. The 37-question priority framework used in the evaluation is provided in Appendix A, Table A2; the full 209-question taxonomy is available from the corresponding author on request.
Acknowledgements
Portions of this manuscript were refined with the assistance of generative AI tools (specifically, ChatGPT and Claude) to improve clarity and language. Additionally, some figures were generated using the online data visualisation platform Flourish Studio. The authors take full responsibility for the accuracy, integrity, and originality of the content presented. All conceptual, analytical, and interpretive contributions are those of the authors.
Author contributions
Conceptualization: A.E.B. (Equal), D.Z. (Equal), H.T. (Equal), M.A.B. (Supporting), R.N. (Supporting), K.M. (Supporting); Data curation: A.E.B. (Equal), D.Z. (Equal), and H.T. (Equal); Formal analysis: A.E.B. (Equal), D.Z. (Equal), and H.T. (Equal); Investigation: A.E.B. (Equal), D.Z. (Equal), and H.T. (Equal); Methodology: A.E.B. (Equal), D.Z. (Equal), H.T. (Equal), and M.A.B. (Supporting); Project administration: A.E.B. (Lead), R.N. (Supporting), and K.M. (Supporting); Resources: A.E.B. (Lead), R.N. (Supporting), and K.M. (Supporting); Software: D.Z. (Lead) and M.A.B. (Supporting); Supervision: M.A.B. (Equal), R.N. (Equal), and K.M. (Equal); Validation: A.E.B. (Equal) and H.T. (Equal); Visualization: A.E.B. (Lead); Writing—original draft: A.E.B. (Lead), D.Z. (Equal), and H.T. (Equal); Writing—review and editing: A.E.B. (Equal), D.Z. (Equal), H.T. (Equal), M.A.B. (Equal), R.N. (Equal), and K.M. (Equal). All authors approved the final submitted version of the manuscript.
Competing interests
The authors declare none.
Ethical standard
No formal ethical approval was required for this research. However, the research adheres to all ethical guidelines, including compliance with the legal requirements of the study country.
Appendix A: Supporting materials
Note on tier assignment. Tier assignment was undertaken by the same two domain experts who carried out the metric-to-criterion mapping. A question was tagged as statutory baseline (S) where its content corresponds directly to language in the AU MSA, Section 16 (or to the mandatory formalities under Sections 13–14). A question was tagged as beyond compliance (B) where it specifies a particular implementation form or a particular indicator that goes beyond the open-textured language of the statute. We acknowledge that some assignments are debatable on the margin, and we encourage users of the framework to revisit them in light of jurisdiction-specific guidance and law developments. The tier labels are intended as a transparency aid, not as legal advice.
Comparative mapping of reporting requirements and guidelines of the Australian, UK, and Canadian Legislation
Table A1. Long description
The table consists of three columns: Australian Modern Slavery Act 2018 (Cth), Section 16 (M); U K Modern Slavery Act 2015, Section 54(5) (P); and Canadian Fighting Against Forced Labour and Child Labour in Supply Chains Act 2023, Section 11(3) (M).
* Row 1: Australia requires identification of the reporting entity; U K and Canada are N/A.
* Row 2: All three require descriptions of organizational structure, business/operations, and supply chains.
* Row 3: Australia requires describing risks in operations and supply chains; U K focuses on assessing and managing risk (P); Canada requires identifying parts of business with risk and steps taken to manage it (M).
* Row 4: Australia requires describing actions taken to address risks, due diligence, and remediation; U K includes policies, due diligence, and training (P); Canada includes policies, due diligence, remediation measures, and training (M).
* Row 5: All three require describing how the entity assesses the effectiveness of its actions.
* Row 6: Australia requires describing the consultation process with owned/controlled entities; U K and Canada are N/A.
* Row 7: Australia allows for other relevant information; U K is N/A; Canada specifically requires measures taken to remediate loss of income to vulnerable families (M).
* Row 8: All three require approval by the principal governing body or board of directors (M).
* Row 9: All three require a signature from a responsible member or director (M).
Note: M = mandatory, P = permissive (“may include”). Adapted from Bora et al. (Reference Bora, Arodi, Zhang, Bannister, Bronzi, Fansi Tchango, Bashar, Nayak, Mengersen, Che, Nabende, Shutova and Pilehvar2025a). The UK MSA Section 54(5) categories are framed as content a statement may include and are therefore best read as recommended or illustrative content rather than as strictly mandatory disclosure obligations. By contrast, the formalities under UK MSA Sections 54(6)–(7) (approval, signature, publication) are mandatory. The Australian and Canadian regimes prescribe the substantive content as well as the formalities. We adopt the AU criteria as our analytic benchmark and clearly mark mandatory items above as (M) and permissive items as (P).
Prioritisation framework: consensus questions and statutory baseline additions
Table A2. Long description
The table is organised into four columns.
* A U Approval: Board approval, Tier S. Question: Was the statement approved by the principal governing body of the reporting entity?
* A U Signature: Signature, Tier S. Question: Was the statement signed by an appropriate member of the principal governing body?
* A U C 1: Reporting entity, Tier S. Question: Does the entity describe the relevant reporting entity(s) of the statement?
* A U C 2: Structure, operations, and supply chains. This section includes seven entries covering Operations (Tier S), Structure (Tiers B and S), and Supply chains (Tiers B and S). Questions focus on workforce composition, subsidiary structure, and disclosure of tier one and beyond-tier-one supplier locations.
* A U C 3: Risk and incidents identification. This section includes six entries covering Incident (Tier B), Minimum wage (Tier B), Recruitment (Tier B), Risks-caused (Tier B), Risks-description (Tier S), and Risks-location (Tier S). Questions address specific allegations, migrant workforce risks, and where risks are present in the business.
* A U C 4: Risk mitigation and remediation. This is the largest section with 17 entries. Topics include code of conduct, contracts, grievance mechanisms, health and safety, leadership, living wage, M S policy, responsible purchasing, risk assessment tools (audits/surveys), and remediation processes. Tiers vary between S (Statutory) and B (Beyond compliance).
* A U C 5: Effectiveness of actions. Three entries covering Effectiveness (Tier S), K P I (Tier B), and K P I-training (Tier B). Questions ask how effectiveness is assessed and for specific performance indicators like the number of employees trained.
* A U C 6: Consultation: Consultation, Tier S. Question: Did the entity disclose its consultation process among its affiliated entities?
A note specifies that Tier S equals Statutory baseline and Tier B equals Beyond compliance.
Note: Thirty-five questions appear in at least three benchmarking methodologies; two statutory baseline questions (signature and reporting entity identification) are retained irrespective of frequency. Tier (S) = Statutory baseline; Tier (B) = Beyond compliance. See Section 3.1 and the note below the table.
Appendix B: Error analysis
At the end of the second-phase experiment, we conducted a detailed error analysis of the outputs from the five second-phase statements. While no additional straightforward or systematic errors were identified, the review surfaced several persistent limitations of both the agents and the question design, which merit further investigation.
Agent-related issues
-
• A recurring issue was the model’s occasional misalignment between the question and its internal reasoning. For example, when asked a question related to Criterion 4 (mitigation actions), the model’s explanation at times referenced evidence relevant to approval processes instead. This confusion requires deeper diagnostic analysis.
-
• In one instance, the model exhibited hallucination. It generated a justification claiming that “20% of staff received modern slavery training,” even though no such statistic appeared in the source statement. This figure was, in fact, part of an illustrative example included in the prompt, suggesting that prompt content may inadvertently leak into the model’s output.
-
• In several cases, the model arrived at the correct final answer but based its reasoning on irrelevant or incorrect parts of the evidence. This points to a need for stronger alignment between justification and decision-making, especially when interpretability is critical.
Question-related issues
-
• Several questions may require further clarification to reduce ambiguity and support consistent interpretation. For instance, the question: “Did the entity disclose information on how it monitors supplier risk, including engagement with workers and other stakeholders?” was interpreted inconsistently. In cases where engagement with stakeholders was not disclosed, the model sometimes discarded otherwise valid evidence related to supplier risk monitoring. This suggests that experts may interpret the scope of such compound questions differently, reinforcing the need for more collaboration on question design.
-
• Additionally, many companies respond to prompts with negative disclosures (e.g., denials of incidents), which some experts consider valid evidence, while the model often discards such content. For example, in response to the question: “Does the entity describe specific allegations and incidents of modern slavery that it has encountered and/or provide details of products and services affected?”, some companies stated, “We are not aware of any allegations of human trafficking/slavery activities against any of our suppliers or customers in the last financial year.” While this is technically a form of disclosure, models tend to classify it as irrelevant or insufficient, potentially underestimating the completeness of a response.
-
• Finally, the model requires further training to correctly distinguish between overlapping but non-equivalent concepts. For example, a whistleblowing policy was sometimes treated as sufficient evidence of a grievance mechanism, when in fact, such policies may not meet the substantive requirements of an accessible, worker-oriented grievance process. These fine-grained distinctions are critical in legal and policy contexts and require more targeted modelling or post-processing strategies.
Expert disagreement
-
• As shown in our Results Section 4, overall disagreement between experts was low. However, the disagreements that did occur were primarily related to differences in interpreting whether the provided evidence was sufficiently informative or whether the language used in disclosures was too vague. These cases highlight the inherent subjectivity in interpreting complex or non-standardised corporate reporting and underscore the importance of clear reporting guidelines to support consistent evaluation.
Appendix C: Future work
Larger-scale and stratified evaluation
The evaluation reported in Section 4 is based on five statements held out from the AIMS.au test set for second-phase evaluation, contributing 185 question instances. While this sample is sufficient to demonstrate feasibility, it is too small to support strong generalisation claims. Priority directions for future evaluation work include: (i) scaling to the full 50-statement AIMS.au test set, (ii) stratified sampling across sectors, jurisdictions (including AIMS.uk and AIMS.ca), and statement lengths to support cross-segment analysis, (iii) bootstrapped confidence intervals on per-criterion and per-agent accuracy values, and (iv) standalone evaluation of the RAG Watchdog component on a larger sample to disentangle its performance from its current role as a fallback to Agent 1. A natural extension of (iv) could be a Mixture-of-Experts architecture that allows Agent 1 and Agent 2 to operate in parallel and have their outputs reconciled, rather than running as a sequential fallback chain.
Technical and methodological enhancements
Beyond evaluation scale, we identify several technical and methodological enhancements aimed at improving the reliability, adaptability, and policy alignment of the AIMS-QA system. First, if the system is deployed, incorporating reinforcement learning from user feedback can enable the system to evolve over time, refining its performance based on real-world usage. To ensure robustness and guard against unintended biases or degradation, we envision a human-in-the-loop validation process, where user feedback is reviewed by expert moderators before being used for model updates, mirroring established quality control practices in collaborative software environments.
Second, to improve the alignment between model reasoning and conclusions, future work should explore advanced prompting techniques, as well as logic auditing tools that check for consistency between the evidence cited and the model’s final judgement. Given the importance of explainability in regulatory contexts, auxiliary explanation models or structured output formats could further support transparency and stakeholder trust. Future efforts should also consider prompt design under environmental constraints. This includes accounting for prompt length limits, token boundaries, and processing time to ensure that prompts are not only contextually rich but also efficient and scalable. Carefully engineered prompts that anticipate these constraints will help maintain performance in real-world deployment settings where latency and resource usage are critical considerations.
Third, question design remains a critical area for refinement. Many current questions are complex or ambiguous, which can lead to misinterpretation by both models and human experts. Adopting controlled, linguistically simplified question templates and establishing a formal review process involving both legal and linguistic experts can help ensure questions are clear, legally accurate, and computationally tractable. Integrating natural language inference (NLI) capabilities may further enhance the system’s ability to handle subtle distinctions, such as differentiating between “minimum” and “living” wage claims, or recognising when whistleblowing policies fall short of formal grievance mechanisms.
Finally, to reduce expert disagreement and improve annotation quality, we advocate for the development of detailed annotation guidelines with well-defined edge cases. Structured consensus-building tools can be used to resolve interpretive differences, and the resulting adjudicated annotations can serve as high-quality training signals for further model fine-tuning.
Taken together, these future directions aim to build a more adaptive, trustworthy, and policy-responsive compliance evaluation system, one that not only supports current regulatory frameworks, but also serves as a model for how AI and structured data can be responsibly embedded into governance processes. By combining domain expertise, technical rigour, and iterative improvement, AIMS-QA contributes to the growing ecosystem of data-driven technologies advancing policy design, monitoring, and enforcement in high-stakes government contexts.
Open access
Comments
No Comments have been published for this article.