5.1 Introduction
Before national, regional and international legislators considered regulating automation in response to emerging technologies, including artificial intelligence (AI), debates had already emerged about its role in critical areas of public governance, such as law enforcement. While automation is not new for the purposes of crime control (Sommerer, Reference Kimery2022), modern AI systems serve multiple law enforcement purposes, ranging from crime detection and investigation to crime prediction (Europol, 2025). However, their use in criminal proceedings raises concerns about fundamental rights, especially when AI-generated insights appear to influence decisions beyond human oversight (Straus, Reference Knapp and Langill2021). Notably, the capacity of AI systems to operate automatically and at scale (Yeung, Howes and Pogrebna, Reference Кuzmenko, Yarovenko and Perkhun2020) coupled with a higher degree of epistemic opacity (Buihsman, Klenk and van den Hoven, Reference Hassija, Chamola, Mahapatra, Singal, Goel, Huang, Scardapane, Spinelli, Mahmud and Hussain2025; Burrell, Reference Haugeland2016) challenge significantly the responsible use of such systems, particularly given the severity of the consequences for the affected individuals (see Section 5.2).
This chapter contributes to a multi-disciplinary approach to automated law enforcement, examining the example of AI tools intended to be employed for crime analytics purposes. As the cyber-element penetrates even traditional forms of criminal activity and data-driven police investigations grow more complex, law enforcement authorities (LEAs) leverage AI tools to analyse large datasets, identify patterns, uncover hidden relationships and unveil unknown patterns or discover hidden relationships in the data (for the respective definition see Commission, 2021, Annex III, point 6, lit. g). These tools may offer services like translation, entity extraction, video transcription, social media analysis, and relationship mapping through filtering and visualisation, and, thus, may consist of one or more respective components. The ultimate goal of their use is tackling crime more effectively. LEAs may use their output to establish or confirm suspicion, guide investigations, compile prosecution files, determine key details of criminal conduct, and support prosecutors in initiating criminal proceedings.
This chapter explores the impact of AI-powered crime analytics on fundamental rights, focusing on defence rights through the lens of the European Convention on Human Rights (ECHR) and the Charter of Fundamental Rights of the EU (CFR) (Section 5.2). It then critically examines the stance of Regulation (EU) 2024/1689, commonly known as the EU AI Act (AIA) on such AI systems, analysing key provisions within its risk-based approach to AI (Section 5.3). Given that crime-analytics tools as such have been largely excluded from high-risk classification in the AIA, it explores alternative safeguards advocating for a fundamental rights-by-design approach to their development (Section 5.4). Finally, this chapter concludes with a socio-ethical account on bias detection and mitigation tailored to the interconnectedness of micro (technical, individual), meso (organisational), and macro (social) challenges associated with AI-enabled crime analytics (Section 5.5).Footnote 1
5.2 The Impact of AI-Driven Crime Analytics Tools on Defence Rights
The compatibility of AI with fundamental rights has been the focus of intense debates at national, regional, and international levels (Lane, Reference Lane2022), aiming to promote human-centric AI. The affected rights span privacy, data protection, freedom of expression and association, non-discrimination, effective remedy, and fair trial rights (see, for instance, Council of Europe, Reference Hilal, Gadsden and Yawney2017; González Fuster, Reference Jamil2020). Although some rights and risk parameters have received more attention than others, the EU legislators adopted a holistic approach to mitigate AI’s adverse impacts across public and private fields of action. Given the particular features of crime analytics tools, the following analysis focuses on defence rights (see also Sachoulidou, Reference Sachoulidou2026).
The EU Member States shall abide by the fair trial principle and the rights that arise therefrom as enshrined in the ECHR (Article 6) and the CFR (Articles 47–48). According to the jurisprudence of the European Court of Human Rights (ECtHR), one has to evaluate the overall fairness of criminal proceedings (ECtHR, 2024, paragraphs 2–3) – with the concept of ‘criminal charge’ having a meaning that is independent of the categorisations employed by national legislators (ECtHR, 2024, paragraphs 15–16). Next, the notion of charge is defined as the ‘the official notification given to an individual by the competent authority of an allegation that he has committed a criminal offence’ and includes the test of whether the suspect’s situation has been affected substantially (ECtHR, 2024, paragraph 18). The fair trial guarantees, ranging from the right to a fair and public hearing to the right to equality of arms and adversarial hearings and the right to be presumed innocent, apply throughout the entire criminal procedure, encompassing the pre-trial and the trial stage as well as the appeal proceedings.
AI systems used at pre-charge stages, such as those assessing the likelihood of future offending, will typically fall outside the scope of fair trial protections (Sachoulidou, Reference Kimery2023). Conversely, AI-based crime analytics systems may be introduced during criminal proceedings and their output, depending on the applicable national rules on evidence admissibility, may be treated as evidence. As such, their use must align with fair trial guarantees throughout all procedural stages (cf. Palmiotto, Reference Keohane2021). The affected individuals should be able to exercise their right to equality of arms and an adversarial hearing. Equality of arms presupposes that each party is given a reasonable opportunity to present their case under conditions that do not place them at a disadvantage vis-à-vis their opponent (ECtHR, 2024, paragraph 171). Similarly, the right to an adversarial hearing is respected when the parties are able to have knowledge of and comment on all evidence adduced or observations filed with a view to influencing the court’s decision, without having to determine that the failure to access a certain piece of evidence has caused them any prejudice (ECtHR, 2024, paragraph 172). This suggests that the defendant shall be able to challenge evidence that arise from the use of AI-powered crime analytics tools effectively. This presupposes, however, a certain level of transparency regarding the algorithmic output as well as the inner workings of the algorithm.
Further ECtHR guidance supports this approach. Taking the example of expert findings, the Court has stressed that the failure of the prosecution to disclose the technical details an expert report has been based upon may compromise the defence’s ability to challenge the report and, thus, raise an issue of equality of arms (ECtHR, 2024, paragraph 184). Limited access to the case file or other documents on public-interest grounds may have a similar effect – with the Court finding that respect for defence rights requires that access limitations shall not prevent the evidence from being made available to the defendant before the trial and the latter from commenting on it (ECtHR, 2024, paragraphs 185–186). However, the right to disclosure of relevant evidence is not absolute as competing interests, such as police secrecy, shall be taken into consideration and be weighed against the rights of the accused. In such cases, only such measures restricting defence rights which are strictly necessary are permissible under Article 6(1) ECHR, and such restrictions should be sufficiently counterbalanced by the procedures followed by the judicial authorities (ECtHR, 2024, paragraph 191). The importance of the undisclosed material and its use in the trial remains of central importance to assess whether defence rights have been violated (ECtHR, 2024, paragraphs 193–194). As regards cases where the prosecution files include electronic data, the defence shall be able to be involved in the laying down of the criteria for determining what may be relevant for disclosure. Moreover, the refusal to allow the defence to have further searches of identified or tagged data may violate the right to be provided with adequate facilities for the preparation of the defence. Finally, the defence must be able to assess the integrity and reliability of the evidence presented in court (cf. Oerlemans and Royer, Reference Kelly, Kaye and Oviedo-Trespalacios2023; Stoykova, Reference Stoykova2023; Kipker and Bruns, Reference Joh2022). If this is not possible, the burden falls even more heavily on national authorities to subject such evidence to the highest level of scrutiny (ECtHR, 2024, paragraph 195).
The lack of sufficient knowledge and resources to confront evidence against oneself may impact adversely on the right to examine witnesses (Article 6 (3) lit. d ECHR). The term ‘witness’ shall encompass any evidence that is used in criminal proceedings and can support a conviction. In this sense, AI-generated output that supports a criminal investigation and arrives in the courtroom falls into the scope of this definition (cf. Gless, Reference Jackson, Markevych, Bardin and Thomas2020; Palmiotto, Reference Keohane2021). The ability to challenge evidence can also be analysed through the lens of the presumption of innocence (PoI) (Article 6(2) ECHR, 48 CFR). Even if the prosecution or the court formally bears the burden of proof, the defence must be able to contest any (indirect) shifting of this burden, especially in cases where EU secondary law, in line with ECtHR jurisprudence (ECtHR, 2024, paragraphs 409–411), allows for legal and factual presumptions of criminal liability (recital 22 Directive (EU) 2016/343). When such evidence or presumptions stem from an AI system, the primary barrier to meeting the protection standards required by PoI is algorithmic opacity (Sachoulidou, Reference Kimery2023). This opacity may arise from multiple sources: deliberate corporate or institutional self-protection and concealment, including police secrecy; lack of technical expertise among the judiciary and defence to assess AI-generated evidence; a fundamental mismatch between the complexity of AI-driven decision-making and human cognitive capacities (Burrell, Reference Haugeland2016).
Next, the fair trial principle includes the right to a reasoned judicial decision, ensuring that parties have been meaningfully heard. This requires judges to base their reasoning on objective arguments and uphold defence rights (ECtHR, 2024, paragraph 198). Moreover, reasoned decisions are essential for safeguarding the right to appeal. This remains just as critical when AI-generated evidence is admitted in criminal proceedings (Palmiotto, Reference Keohane2021; Franssen and Berrendorf, Reference Keohane2021) – with the lack of sufficient knowledge of the inner workings of such systems posing a significant obstacle to compliance with this rule-of-law principle.
Lastly, reliance on AI-generated evidence may challenge compliance with the in dubio pro reo principle (Article 6 (2) ECHR, 48 CFR). This principle is considered to have been violated when a judicial decision is not sufficiently reasoned as well as when the defence faces an extreme and unattainable burden of proof depriving them of the slightest prospect of success (ECtHR, 2024, paragraph 407). This is particularly relevant for AI-powered crime analytics systems, as their use in criminal proceedings may distort legal reasoning and impose overwhelming challenges on the defence, particularly in terms of the resources needed to contest AI-supported conclusions.
Moreover, the in dubio pro reo principle is put under strain in an environment where AI outputs create the illusion of certainty and objectivity, seemingly surpassing the fallibility of human judgement (cf. Peršak, Reference Kethineni, Cao and Dodge2021). Even when doubt persists (for instance, in cases where the error rate of an AI system is disclosed), it remains unclear how much doubt is required to override the algorithmic output and exonerate a defendant (Sachoulidou, Reference Kimery2023).
5.3 AI-Powered Crime Analytics Tools: The Normative Context
At the EU level, the development and use of AI-powered crime analytics tools should be placed in the normative context shaped by the AIA, as complemented by the EU fundamental rights legislation and the EU data protection acquis. The AIA seeks to balance between innovation and effective protection of health, safety and fundamental rights (Article 1(1) AIA). In this context, it includes a definition of AI systems, confirming that AI is best understood as a socio-technical concept (Barabas, Reference Barabas, Dubber, Pasquale and Das2020). More specifically, AI systems are defined as ‘machine-based system(s) designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infer, from the input (they) receive, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments’ (Article 3(1) AIA). Crime analytics tools may fall into the scope of this definition, provided they are not limited to following predefined, explicit instructions or operations, or they are solely intended for descriptive analysis, hypothesis testing, and visualisation, but rather learn, reason or model at any stage of their lifecycle (European Commission, 2025a, p. 9).
Next, the AIA stands out for its top-down approach to risk regulation (De Gregorio and Dunn, Reference Hua and Belfield2022) that presents varying levels of density (Ebers, Reference Ebers, Di Matteo, Poncibò and Cannarsa2022). At the core of this approach is the assessment of an AI system’s impact on health, safety, and fundamental rights (recitals 7 and 46 AIA). The AIA categorises AI systems into the following risk categories (for an analysis focused on automated law enforcement see Sachoulidou and Vavoula, Reference Sachoulidou, Vavoula, Lucarelli and Sperling2025):
1. Unacceptable risk leads to a prohibition that may be waived under certain conditions (Article 5 AIA; see European Commission, 2025b). In the area of law enforcement, this category includes AI systems that are employed for ‘making risk assessments of natural persons in order to assess or predict the likelihood of a natural person committing a criminal offence, based solely on the profiling of a natural person or on assessing their personality traits and characteristics’ (Article 5(1) lit. d AIA) and ‘real-time’ remote biometric identification systems employed in publicly accessible spaces (Article 5(1) lit. h AIA).
2. High risk results in a series of minimum requirements for providers, deployers and third parties (Articles 8–49 AIA). This includes AI systems that have a ‘significant harmful impact on the health, safety and fundamental rights’ (recital 46 AIA). Such AI systems may be intended to be used as a safety component of a product or may be products themselves and, as such, are required to undergo a third-party conformity assessment with a view to their placement on the market or being put into service (Article 6(1) AIA). Besides this, this category encompasses stand-alone AI systems deployed in the areas of activities listed in Annex III to AIA (Article 6(2) AIA). In the area of law enforcement, point 6 of Annex III to the AIA explicitly classifies as high-risk only those AI systems which are intended to be used to: assess a natural person’s risk of becoming the victim of criminal offences; serve as polygraphs or similar tools; evaluate evidence reliability; assess the likelihood of a natural person of (re-)offending not solely based on profiling of natural persons as referred to in Article 3(4) of the Directive (EU) 2016/680 (Law Enforcement Directive (LED)), or personality traits and characteristics or past criminal behaviour of natural persons or groups; profile natural persons as referred to in Article 3(4) LED in the course of crime detection, investigation, or prosecution.
3. AI systems intended to interact directly with natural persons pose a limited risk and, as such, should comply with transparency requirements (Article 50 AIA); that is, the natural persons concerned should be informed that they are interacting with AI. This does not apply to ‘AI systems authorised by law to detect, prevent, investigate or prosecute criminal offences, … unless those systems are available for the public to report a criminal offence’ (Article 50(1) AIA).
4. The deployers of AI systems that present a minimal risk should create codes of conduct, including related governance mechanisms, to promote the voluntary application of some or all the mandatory requirements applicable to high-risk AI systems (Article 95 AIA).
In the light of these categorisations, AI-supported crime analytics tools, which do not serve any additional purpose out of the above-mentioned ones, nor fall into the scope of point 8 of Annex III to AIA (‘AI systems intended to be used by a judicial authority or on their behalf to assist a judicial authority in researching and interpreting facts […]’), will be governed by flexible standards, an outcome that prioritises an effectiveness-focused approach to automated law enforcement (cf. European Police Chiefs, 2022). Notably, the Council of the EU (2022) edited out Annex III point 6 lit. g AIA, introducing an exception for crime analytics that is meant to address contemporary investigative needs. Despite the disagreement at the level of the European Parliament (2023), this exception was preserved in the final text of the AIA.
Additionally, Article 6(3) AIA introduces exceptions to the automatic classification of AI systems deployed in designated high-risk areas. The primary rationale is that such systems do not necessarily have a significant impact on decision-making, nor they harm substantially health, safety or fundamental rights (recital 53 AIA). This exception encompasses cases where an AI system is intended to: (1) perform a narrow procedural task; (2) improve the result of a previously completed human activity; (3) detect decision-making patterns or deviations from prior decision-making patterns, without replacing or influencing the previously completed human assessment, without proper human review; or (4) perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III. This appears to be the case with crime analytics tools that perform ‘merely’ narrow procedural tasks, such as translation or entity extraction, and ‘only’ preparatory tasks to the ultimate law enforcement and judicial decision-making.
This stance conflicts, however, with the data-intensive processes involved in developing and training AI-powered crime analytics tools. The accuracy and reliability of their output (to the extent that the latter is not merely a visualisation of a certain dataset) can be significantly compromised when such systems are developed/trained on low-quality, biased or otherwise non-representative data (see, for instance, Mittelstadt et al., Reference Kan2016; Ryan, et al., Reference Khullar2019; FRA, 2020), leading to unjust identification of individuals (rather than patterns) (cf. recital 59 AIA). Next, given that crime analytics tools are meant to facilitate complex criminal investigations and, accordingly, inform criminal proceedings, opacity coupled with lack of documentation may significantly undermine the exercise of fundamental procedural rights, particularly if the algorithmic output is admitted as evidence.
Moreover, crime analytics systems will fall outside the protective scope of Article 86 AIA, which grants a right to clear and meaningful explanation only for decisions based on high-risk AI outputs. As a result, affected individuals will have to rely on general legal frameworks for remedies. This is particularly concerning when AI has the potential to influence criminal proceedings. Notably, recital 59 AIA already stresses ‘the difficulty in obtaining meaningful information on the functioning of (AI) systems (employed for law enforcement purposes) and the resulting difficulty in challenging their results in court, in particular by natural persons under investigation’ – but only with regards to high-risk AI systems.
Lastly, the editing out of crime analytics systems does not take into consideration the risk of their (reasonably foreseeable) misuse (cf. Article 3 nr. 13 AIA) in operational settings. Such misuse may take the form of repurposing a crime analytics tool for profiling purposes. Besides this, an even more usual case would be to incorporate crime analytics tools into other existing AI systems; the latter may be high-risk and, thus, higher standards will apply to their development. Yet, crime analytics tools would be originally developed without similar restrictions. Additionally, the crime analytics exception fails to consider the power imbalance that is inherent into the law enforcement environment, even though power imbalance has been listed among the criteria that may be used for the purposes of adding or modifying use cases of high-risk AI systems pursuant to Article 7 AIA.
5.4 How to Merge Socio-Ethical, Legal, and Technical Standards: The Matching Exercise
A risk-based approach is crucial for regulating AI, as reflected in the AIA’s focus on protecting fundamental rights, though its limited guidance as to how to conduct an impact assessment (Mantelero and Marí, 2025) and the complexity of AI systems necessitate diverse, tailored assessment methodologies integrating socio-ethical, legal, and technical standards. Given the special features of AI-powered crime analytics tools and considering that the law enforcement context is not, nor should be, exempted from (public) scrutiny (Mantelero and Fanucci, Reference Mantelero and Fanucci2024), this section presents a model-template, called ‘matching exercise’ (ME) for ensuring a fundamental rights-by-design approach to AI in law enforcement settings. This assessment framework consists of a set of questions designed to address socio-ethical and legal implications of the development and use of AI in the law enforcement environment, combining different methodologies: an Ethical Impact Assessment (EIA), Social Impact Assessment (SIA), and Human Rights Impact Assessment (HRIA). In this way, it aims to enable a holistic approach to AI impact.
An EIA provides a methodological framework to ensure that the development of AI systems ‘align(s) with the promotion and protection of human rights and human dignity, environmental sustainability, fairness, inclusion, and gender equality’ (UNESCO, 2023, p. 5). It includes an ex ante assessment of the development phase of an AI system and an ex post assessment of the commercialisation phase, covering its entire lifecycle. In this context, ethics serves as a complement to legal standards. Given that the profound societal impact of AI cannot be fully mitigated by human rights laws alone, it often requires a nuanced approach that considers both community choices and individual autonomy, thereby pointing to the need of an analysis of socio-ethical values in context (Mantelero, 2022, p. 93). This means an EIA relies significantly on stakeholder engagement, ethical perceptions, and real experiences of those involved (UNESCO, 2023), which requires a context-sensitive and participative analysis of community values where the AI solutions are to be implemented (Mantelero, 2022, p. 93).
Another dimension that complements ethical accounts for human protection consists of the societal consequences of developing AI tools. A SIA encompasses ‘processes of analysing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programmes, plans, projects) and any social change processes invoked by those interventions’ (IAIA, Reference Joh2015, p. 1). It typically relies on local knowledge and participatory processes to assess the concerns of interested and affected parties (Vanclay, Reference Vanclay2003, p. 6), bringing together different stakeholders to monitor the social impact of a planned intervention and analyse its alternatives. The challenge consists of defining comprehensively what qualifies as social impact or social risk as ‘all issues that affect people, directly or indirectly, are pertinent to (SIA)’ (Vanclay, Reference Vanclay2003, p. 8). In the AI context, a SIA should assess technical features that may be inaccessible to the public, professionals, or vulnerable groups affected by an AI system. Moreover, an AI system may have the potential to affect an undefined geographical community or population, making it impossible to determine which concrete stakeholders should participate in SIA procedures.
Alternatively, one may deploy a ‘conceptual and procedural framework (that) guarantees the promotion and protection of human rights in policies, programs, plans and projects’ (IAIA, Reference Joh2015, p. 6). An HRIA can be defined as ‘a process for identifying, understanding, assessing, and addressing the adverse effects of a project or activities on the human rights enjoyment of impacted right-holders’ (The Danish Institute for Human Rights, Reference Kornaros2020, p. 7). Employing an HRIA means: (a) placing human rights and the principles arising therefrom at the centre of attention; (b) requiring accountability and transparency on the part of duty-bearers towards right-holders; (c) fostering empowerment of right-holders and capacity building; (d) ensuring that the meaningful participation of right-holders in development processes and planned interventions is recognised as an intrinsic right; (e) facilitating the non-discriminatory engagement of right-holders and prioritising vulnerable or marginalised individuals or groups (The Danish Institute for Human Rights, Reference Kornaros2020, p. 7).
To effectively implement an HRIA in an AI system under development, there are five steps to follow: (1) plan and define the scope of the system under evaluation; (2) collect data and develop a baseline; (3) analyse impact considering the risks pointed out; (4) mitigate and manage risks; (5) report and evaluate (The Danish Institute for Human Rights, Reference Kornaros2020, p. 7; cf. Mantelero and Marí, 2025). Additionally, any assessment model should develop a self-assessment tool (questionnaire) to ‘identify values, rights and freedoms at risk, as well as relevant groups of rights-holders involved’ (Mantelero and Fanucci, Reference Mantelero and Fanucci2024, p. 7), as well as ‘an ad hoc expert committee’ (Mantelero, Reference Joseph and Bielicki2018, p. 758) to guarantee an independent review of the design choices, the contents of the questionnaire, the appropriateness and clarity of the questions, and propose improvements.
The ME serves as a self-assessment tool for developers and providers of AI-powered crime analytics tools. The development of such a tool requires determining: the relevant variables or risks that should be taken into account; the methods used to assess them and to develop risk indices; the contribution of standard questionnaires to this process along with their limitations; and the importance of expert-based evaluation in conducting the assessment (Mantelero and Marí, 2025, p. 6). Driven by this, the first step in designing the ME is to identify the variables (or risk definition on the grounds) to be tested based on the AIA’s minimum requirements for high-risk AI systems (Articles 9–15 AIA), given the criticism expressed above as to the editing out of AI-enabled crime analytics from Annex III (see Section 5.3).
These requirements are used as the basis for designing a set of questions in the form of an AIA compliance test, which assesses risk management (Article 9), data governance (Article 10), technical documentation (Article 11), record-keeping (Article 12), transparency (Article 13), human oversight (Article 14), accuracy, robustness, and cybersecurity (Article 15) (for an example see Table 5.1). Additionally, socio-ethical considerations, including human rights principles arising from key legal frameworks, such as the ECHR and the CFR, are integrated into the ME design. Questions within these components of the ME are associated with the detection of privacy threats, the impact of false positives and false negatives, and a detailed focus on bias detection and mitigation (see Section 5.5), which can be critical due to their direct social and legal implications in AI-enabled crime analytics.

Table 5.1 Long description
The table outlines how an AI system can meet the E U A I Act’s Risk Management System requirements under Article 9. It includes five columns: E U A I A, Essential requirements, Questions, A I Tool’s Features, Project Deliverable, and Observations. Under Essential requirements, the focus is on the Definition of the risk assessment system. Four key questions are listed: whether the A I system has an established and maintained risk assessment system; whether it provides technical documentation on purpose and foreseeable misuse; whether it identifies possible threats and consequences (including design or technical faults); and whether it informs end-users of existing or potential risks. The remaining columns are empty, intended for recording how an AI tool, project deliverables, and observations address these questions.
To turn the ME into a ‘user-friendly’ assessment tool, a glossary with terms and concepts can be added, including definitions of technical terms and examples as to how to address the questions.
In the second stage, oriented to data collection, this participatory approach implies that the ME should be shared with potentially affected groups or their representatives, partners of the building system (technical, legal), and relevant stakeholders (community, advisers, and so on), who should be selected based on their involvement in the development of the AI system in question and their familiarity with AI as a general concept. Additionally, a meaningful impact assessment should include transparency guidelines that ‘promote multi-stakeholder participation to bridge the gap between ethical principles and concrete practices of developers’ (Mantelero and Fanucci, Reference Mantelero and Fanucci2024, p. 1).
The third stage involves a qualitative analysis of the responses to assess the potential impact of the system’s features and decisions, and identify risk mitigation measures. To generate such insights, deductive and inductive approaches can be useful (Creswell and Tashakkori, 2007). A deductive approach involves checking existing patterns and requirements against the data in a top-down fashion. This ‘top-down’ method helps to assess if the system aligns with existing standards or norms. An inductive approach works from the ‘bottom up’. It can be adopted to identify, analyse, and organise raw data into unified themes of information that indicate the AI system’s potential impact. Combining these approaches over an interpretive textual analysis can help understand contextual specifics of the responses contributing to better tailored measures to mitigate risks and impacts across the different development stages of an AI system. This can be done by analysing the likelihood and severity of an adverse impact to construct a risk index, as identifying the level of risk both prior to and following the application of preventive or corrective actions is necessary to show that the risk has been clearly and effectively managed (Mantelero and Marí, 2025).
The last stage involves a report, or a set of recommendations based on the impact assessment’s results that should be public, promoting transparency and facilitating future revision of the assessment tool and the measures adopted (Mantelero, Reference Mantelero and Fanucci2025).
5.5 Ethics and Fundamental Rights-by-Design: Identifying Bias Detection and Mitigation Measures for Fairer Outcomes
AI-enabled crime analytics systems are prone to various biases that influence algorithmic and human decision-making processes and data interpretation. While the sources of such biases are widely debated, biases associated with AI-enabled law enforcement tools are generally characterised by two dimensions: inaccuracy – where predictions deviate from reality, and undesirability – where AI-generated outcomes reinforce harmful or unfair effects (Zajko, Reference Zajko2021). These dimensions may overlap but should not be conflated, as an AI system’s output can be factually accurate while still leading to undesirable social consequences. This points to the complexities of biases in AI systems which can perpetuate their own biases exacerbating inequalities (FRA, 2022) and flawed decision-making in policing. To mitigate these risks, the AIA mandates strict bias detection and mitigation measures for high-risk AI systems. Article 10 (2) lit. f–g AIA requires developers to assess and address biases to prevent adverse effects on fundamental rights, health, and safety, while Article 10 (5) lit. a–f AIA imposes strict controls on sensitive data processing to balance bias mitigation with privacy protection.
Yet, challenges remain in effectively addressing bias in AI-enabled law enforcement tools manifested in two ways: first, existing bias mitigation measures may be inadequate, as solutions are not yet sufficiently developed to account for the complexities of bias detection and mitigation in these systems. Second, even targeted mitigation efforts may fall short if they fail to consider the broader socio-technical context in which such AI tools operate, including their interdependencies, and human dynamics. Overlooking this context can influence the classification of crime analytics tools, potentially allowing biases to persist and leading to discriminatory outcomes. Moreover, such oversight may compromise the integrity and reliability of evidence derived from these systems in criminal proceedings (see Section 5.2). Therefore, understanding the nature of biases imposed on AI-enabled crime analytics tools and developing robust mitigation measures are necessary to safeguard against discriminatory practices and promote responsible and ethical innovation and governance in the law enforcement environment.
This section explores the complexities of bias in AI-enabled crime analytics tools, and provides a knowledge base for self-assessment tools, such as the ME (see Section 5.4). It identifies the types of biases detected in such AI tools and existing mitigation measures to inform relevant risk management and policy decisions. It builds on results from a scoping review conducted for AI-enabled crime analytics tools within the TRACE project (see Sachoulidou et al., Reference King, Aggarwal, Taddeo and Floridi2024). The results indicated a broad spectrum of biases in AI-enabled crime analytics tools across their lifecycle, including both pre-development (ideation) and post-deployment (governance) phases: individual biases, societal biases, and algorithmic biases. These biases tend to be horizontally applicable across different types of AI tools intended to be used for law enforcement purposes.
Socio-cognitive biases can be induced by police officers’ use of AI systems, which influence their decision-making. Automation bias occurs when officers overly rely on automated suggestions, disregarding external evidence (Alon-Barkat and Busuioc, Reference Alon-Barkat and Busuioc2023). Police officers are more susceptible to confirmation bias – favouring information that supports preferred conclusions, leading to selective and biased information processing, as well as anchoring bias – a cognitive bias that influences decision-making towards information that appears to match (or amplify) their subjective arbitrary reference points (Datta, Whitmore and Nwankpa, Reference Datta, Whitmore and Nwankpa2021). Officers tend to trust AI recommendations that align with their professional knowledge (Selten, Robeer, and Grimmelikhuijsen, Reference Selten, Robeer and Grimmelikhuijsen2023), reinforcing existing stereotypes and creating a feedback loop where biased AI predictions perpetuate biased law enforcement practices (Alon-Barkat and Busuioc, Reference Alon-Barkat and Busuioc2023). This consequence is observed in digital forensics related to the construction or reconstruction of (chain of) evidence that is at the core of AI-enabled crime analytics (Leone, Reference Leone2021).
On a social level, AI biases can stem from broader structural inequalities embedded in institutions and social norms. In AI ethics, much attention is given to discrimination against protected categories such as race, gender, and disability (Zajko, Reference Zajko2021) – often termed ‘discrimination bias’. However, in policing, these biases tend to be examined separately. For instance, racial bias as a byproduct of racial profiling has been found embedded in AI-driven law enforcement tools leading to wrongful identification of marginalised communities (Gentzel, Reference Gentzel2021). Data used in crime analytics, such as report filing (Davis et al., Reference Davis, Purves, Gilbert and Sturm2022) can reinforce such a bias. Gender bias, often intertwined with racial bias, reflects the practice of assigning specific attributes or roles based on gender (OHCHR, 2020). Gender classification algorithms have been shown to perform less accurately for women than men, reflecting biases in police training data (Xiang, Reference Kurshan2022). They can lead to fairness biases, which reflect historical inequalities embedded in AI policing tools (Cardenas and Vallejo-Cardenas, Reference Hearson2019), financial crime analytics (Moldovan, Reference Moldovan2023), and anti-corruption technologies (Odilla, Reference Kanupriya2023).
Biases linked to specific social categories often manifest as algorithmic biases, emerging at various AI system stages, including design, data collection, training, testing, and deployment. Pre-existing bias in training data stems from inherent dataset biases. Crime analytics tools, for instance, may rely on legal datasets that introduce algorithmic biases, as historical inequalities in court data and variations in legal rule programming can produce discriminatory outcomes (Grosso et al., Reference Grosso, O’Brien, Taylor, Woodworth, Lanier, Bohm and Acker2014). This negative legacy bias is further compounded by representation bias, where underrepresentation of certain groups leads to poor generalisation and skewed results (Yan, Kao and Ferrara, Reference Kurzweil2020). In AI tools using large language models, such as crime analytics, framing bias influences causal interpretations and moral evaluations, leading to misclassification or inappropriate semantic categorisation (Zola et al., Reference Zola, Segurola-Gil, Bruse, Galar and Orduna-Urrutia2022). Aggregation bias arises when distinct groups requiring separate treatment are processed uniformly, distorting outcomes, and can be worsened by contaminated data or adversarial manipulation. Consequently, flawed data representation, contextual loss, or oversimplified modelling and evaluation metrics contribute to systematic prediction errors.
When AI-enabled crime analytics tools are applied to real-life conditions, a deployment bias can occur as real-life conditions diverge from training environments, leading to unfair or inaccurate outcomes (Hutiri and Ding, Reference Jofre, Bosisio and Riccardi2022). As AI-enabled crime analytics systems start interacting with multiple users, unintended discriminatory patterns may be reinforced, leading to emergent biases. From a governance perspective, biases have been identified in ethical impact assessments used alongside crime analytics tools, particularly in population composition assessments (Guo and Kennedy, Reference Guo and Kennedy2023), financial decision-making, and algorithmic risk assessments in the criminal justice system (Zilka et al., Reference Labadze, Grigolia and Machaidze2023), making them prone to biases both independently and through their interconnectedness within law enforcement socio-technical systems (Purves, Reference Purves2022).
Mitigation measures align with this perspective, addressing micro- (algorithmic, developer, and user biases), meso- (organisational culture), and macro- (socio-cultural implementation) levels, therefore contributing to a holistic mapping of diverse dimensions to be integrated in relevant assessment tools. At the micro level, no concrete strategies are identified to directly mitigate individual biases. Instead, emphasis is placed on the role of technology developers, highlighting the need for interdisciplinary learning as a prerequisite to understand how race, gender, and systemic inequalities are embedded in society. This ontological engagement is considered critical for developing classification models, which are often tasked with categorising race, gender, and behavioural traits for investigative purposes without necessarily examining how these categories intersect with social hierarchies and historical discrimination (Zajko, Reference Zajko2021). Self-assessments of AI tools could encourage such an engagement in law enforcement settings.
Yet, the focus lies upon algorithmic interventions to bias mitigation, including data pre-processing, in-processing interventions, post-processing adjustments, and fairness-aware algorithm design that primarily target biased training data, model assumptions, and feedback loops. Although a bias-free AI tool cannot be suggested, bias-aware data analytics can aid bias mitigation of AI-enabled crime analytics, and more broadly policing tools through ecologically valid datasets (Hutiri and Ding, Reference Jofre, Bosisio and Riccardi2022), dataset augmentation, as well as re-sampling for diversity (Amini et al., Reference Amini, Soleimany, Schwarting, Bhatia and Rus2019). In crime analytics tools that use natural language processing (NLP) techniques, mitigation strategies can include modifying word embeddings for gender neutrality or augmenting language corpora to break biased associations, albeit indirect discrimination (i.e., discrimination by proxy) may still occur (Fosch-Villaronga et al., Reference Fosch-Villaronga, Poulsen, Søraa and Custers2021). Fairness-aware model design emphasises integrating fairness into training rather than relying on post hoc corrections. For example, Wang et al. (Reference Kurshan and Shen2019) proposed replacing explicit fairness metrics with a secondary classifier trained on human judgements. For real-world applicability, AI models were recommended to be assessed using social-minded quality measures to reduce the reinforcement of societal biases (Du et al., Reference Du, Hargreaves, Sheppard, Anda, Sayakkara, Le-Khac and Scanlon2020), aspects that may be overlooked if rigorous assessment standards do not apply.
That is, technical solutions alone may not suffice; ethical considerations need to be embedded in crime analytics tools. Yet defining ‘fairness’ remains a key ethical challenge (Silva and Kenney, Reference Silva and Kenney2019). Participatory design methods that engage stakeholders can help build trust between data collectors and impacted communities (Xiang, Reference Kurshan2022) and promote a better understanding of what constitutes ‘fair treatment’. For example, AI-enabled policing tools analysing social media data without community input can exacerbate discrimination by misinterpreting cultural and linguistic nuances (Frey et al., 2020). Beyond fairness, adhering to human-centric principles – transparency, accountability, data integrity, and autonomy – is central to effective mitigation (Hung and Yen, Reference Jensen and Iosifidis2021). This necessitates socio-technical approaches which enable human interaction with and correction of biased AI outputs as a systematic, normative practice.
Audits can serve as a critical mechanism towards socially responsible AI-enabled investigative tools. Effective audits extend beyond technical evaluations of data and algorithms to interrogate the theoretical assumptions embedded in model design and their broader social impact (Ugwudike, Reference Kothinti2022). Internal and external audits have increasingly been recognised as essential accountability measures, helping to identify conduits of bias, embed ethical principles in AI systems, and prevent potential harms before deployment (Ugwudike, Reference Kothinti2022). Transparency tools, such as model cards, facilitate collaboration between AI developers, policymakers, and end-users, integrating diverse perspectives into bias assessments (Akter et al., Reference Akter, McCarthy, Sajib, Michael, Dwivedi, D’Ambra and Shen2021). Psychological audits have been proposed to assess AI-driven predictions of human traits and behaviours impartially (Landers and Behrend, Reference Johannesen and Zucman2023), while interactive tools like D-BIAS enable human-in-the-loop interventions to visualise and modify causal structures in datasets (Ghai and Mueller, Reference Ghai and Mueller2022). As AI continues to shape law enforcement decision-making, robust auditing frameworks need to be introduced to mitigate bias in crime analytics tools.
However valuable their contributions to bias mitigation, these measures can only partially influence the outcomes of AI-enabled law enforcement tools. This is because these tools operate within a broader socio-technical system of policing that is not independent from human dynamics. In this sense, establishing appropriate organisational safeguards (meso-level) is crucial when implementing AI in law enforcement (Grimmelikhuijsen and Meijer, Reference Grimmelikhuijsen and Meijer2022), a dimension that has not been properly examined and integrated in relevant regulatory frameworks. Organisational self-monitoring, preparing algorithmic impact statements relative to organisational culture, increasing gender diversity within law enforcement and inter-agency collaboration (Zilka, Sargeant and Weller, Reference Zilka, Sargeant and Weller2022) have been proposed to counteract biases in AI design and oversight. In combination, this evidence suggested that institutional and bureaucratic reforms, standardised operating procedures, transparency in procurement processes, analysing governance limitations, and establishing ex post mechanisms for accountability and redressal can be increasingly needed to mitigate systemic biases in AI-driven law enforcement decision-making, including in the use of crime analytics tools.
On a macro level, bias mitigation measures focus on systemic, legal, and societal interventions to address structural inequalities. A key strategy involves enhancing policing technological capabilities to be inclusive and remain responsive to evolving societal conditions, such as dynamic changes in the people, types and locations of crimes without undermining regulatory constraints (Fosch-Villaronga et al., Reference Fosch-Villaronga, Poulsen, Søraa and Custers2021). This necessitates treating AI-enabled law enforcement as a socio-political, rather than purely technical issue, where interdisciplinary approaches engage with historical and social contexts to transform unjust social structures and drive meaningful change (Zajko, Reference Zajko2021). Moving beyond simply identifying bias to evaluate which AI system’s uses are harmful, to whom, and why, points to the social justice implications of algorithmic decision-making (Drage and Frabetti, Reference Huang and Kisgen2024) and a key dimension for future self-assessment frameworks.
5.6 Conclusions and Recommendations
The use of AI-enabled crime analytics tools can pose concerns for fundamental rights, fairness, and justice, highlighting the need for a multidimensional regulatory approach, informing the current AIA’s approach to them (Sachoulidou, Reference King and Floridi2024). Legislatively, there is an urgent need for specific rules governing automated law enforcement, focusing on complaint and redress mechanisms, greater transparency, and clear evidentiary standards in criminal proceedings where the algorithmic output may be admitted as evidence. Such measures should be complemented by procurement guidelines for AI use in law enforcement settings (European Parliament, 2021, paragraph 17). Soft-law measures, including impact assessments and mandatory periodic audits of their results (European Parliament, 2021, paragraph 18), can help implement a fundamental rights-by-design approach, even where such an assessment is not mandatory (cf. Article 27 AIA). Promoting AI literacy among developers and deployers is essential – not just to understand the technology, but also its ethical risks and limitations (European Parliament, 2021, paragraphs 22–23). This is evident in measures to address bias that require action at the individual (micro), organisational (meso), and societal (macro) levels.

