The AI Act as Part of the EU Product Safety Ecosystem

In reaction to critical assessments of the preceding framework on European product safety, the EU adopted the New Legislative Framework in 2008. This framework was initially comprised of Regulation (EC) 765/2008 and Decision 768/2008, ³¹ and was later amended by Regulation (EU) 2019/1020. ³² The New Legislative Framework provides the basis within which specific legislation regulates the safety of specific categories of products. As such, following the initial Regulation and Decision in 2008, several directives, regulations, and delegated acts were adopted, regulating diverse groups of products such as toys, ³³ machinery, ³⁴ medical devices (MDR), ³⁵ and AI systems (AI Act). ³⁶ Below, we provide a high-level overview of the interaction between the MDR and the AI Act; for practical guidance, the reader is referred to a recent document by the joint Artificial Intelligence Board and Medical Device Coordination Group. ³⁷

Within the New Legislative Framework, the AI Act stands out since it applies “horizontally” across sectors, whereas most of the legislation within the framework is sector-specific legislation, i.e., it applies to groups of products manufactured in specific sectors, such as the aforementioned toys, machinery, and medical devices (“vertically”). The overarching goal of the AI Act is to “foster the development, use and uptake of AI in the internal market that at the same time meets a high level of protection of public interests, such as health and safety and the protection of fundamental rights, including democracy, the rule of law and environmental protection as recognised and protected by Union law” (recital 8). It does so by supplementing existing sector-specific legislation with requirements to address AI-specific hazards and risks for health, safety, and fundamental rights. ³⁸ This means that, for example, a medical device which incorporates AI is subject to both the MDR and, for those AI-specific hazards and risks, the AI Act.

The AI Act connects to sector-specific legislation within the New Legislative Framework as follows. Qualifying a product as a high-risk AI system within the meaning of the Act triggers the applicability of many safety requirements under the AI Act against which, in this article, we will map and contrast the values in the manner described above. There are two grounds for a product incorporating AI to qualify as a high-risk AI system: the product either requires a third-party conformity assessment, as opposed to self-assessment, under sector-specific legislation like the MDR, thereby justifying a high level of risk (AI Act art. 6 ¶ 1), or the product is applied in one of the fields listed in Annex III (AI Act art. 6 ¶ 2), for example, to assess the risk and determine the price for natural persons applying for health insurance (AI Act ann. III art. 5(c)). ³⁹

While the adoption of the AI Act has seen positive appraisal,Reference Niemiec ⁴⁰ it has also attracted criticism. For example, basing the risk classification in the AI Act on the requirement for third-party assessment under sector-specific rules was criticized since sector-specific risk might not be a sufficient proxy for AI risk.Reference Palmieri ⁴¹ The importance of sufficiently clear standards is generally well-recognized,Reference Çubukçu ⁴² although some authors are concerned about (the importance and transparency of) the standardization process.Reference Nevejans and Barfield ⁴³ The initial criticism that the draft AI Act did not include a right of recourse for an injured individualReference Malacka ⁴⁴ has been countered through the introduction of Articles 85 and 86, but some authors argue that this only partially solves the problem.Reference Gornet ⁴⁵ More generally, some authors are concerned about the limited capacity and expertise of the parties regulated by the AI Act, the national authorities enforcing the AI Act, and the notified bodies performing third-party conformity assessments to sufficiently understand and apply the rules contained in it,Reference Aboy, Minssen and Vayena ⁴⁶ although some argue that the difference from existing obligations under the MDR might be minimal.Reference Helle ⁴⁷ Several authors argue that the AI Act may not be sufficient to safeguard patient health and safety due to its general nature, while the MDR might similarly fall short since it fails to address AI-specific risks. For example, non-diagnostic software or software that falls under the “in-house” exemption (Article 5 MDR) does not require third-party assessment and is thus not classified as high-risk AI.Reference Ebers ⁴⁸ Others argue that these regulations are too focused on risk mitigation while overlooking (other) factors that are crucial for trust in health care settings.Reference Solaiman ⁴⁹ Be that as it may, seeing that the MDR does not specifically deal with AI risks and the AI Act does, we will henceforth only use the AI Act to map and contrast the values in the manner described above.

DECIDE as a Putative High-Risk AI System Under the AI Act

In general, product safety laws seek to protect consumers by requiring, inter alia, providers (in most cases: manufacturers or developers) ⁵⁰ to place or make available on the EU market only safe products. ⁵¹ Product safety laws consist of many specific EU regulations and directives depending on the nature of the product concerned. Insofar as AI-CDSSs are concerned, the most relevant product safety laws are the Medical Devices Regulation (MDR) and, specifically with respect to AI, the AI Act and the requirements set out therein concerning high-risk AI systems.Reference Minssen, Solaiman and Cohen ⁵²

For the purposes of this article, we define GPs as the deployers of DECIDE. This designation reflects the organization of health care in the Netherlands, where GPs predominantly own primary care practices (83% of primary care practices in 2022). ⁵³ Where GPs instead work as employees within a larger health care organization, the responsibilities of the deployer under the AI Act will be borne by that organization. This does not materially alter the scope or content of those responsibilities and therefore our findings are generally also applicable in those situations.

We also assume that DECIDE qualifies as a high-risk AI system (AI Act art. 6 ¶ 1 and ann. I § A no. 11), according to which the requirement for the AI component to undergo a third-party conformity assessment usually applies. ⁵⁴ This interpretation is in line with the recent guideline published by the Joint Artificial Intelligence Board and the Medical Device Coordination Group. ⁵⁵ The provider of DECIDE may also voluntarily decide to abide by the requirements of high-risk systems even if the provider is under no legal obligation to do so, for example because the provider firmly believes that it is vital to ensure that the values deemed important by potential users, which align with the designation of high-risk AI systems, are met (see below). The articles in the AI Act pertaining to high-risk AI systems are the most specific articles relating to AI systems in the EU, providing a richer basis to contrast the stakeholders’ values. The requirements of high-risk systems are the most difficult to comply with, and therefore, this analysis would provide guidance in the most stringent regulatory scenario, or if relevant stakeholders (provider, GP, patients, etc.) would like these requirements to apply.

Obligations in the MDR and the AI Act and Enforcement Thereof

Pursuant to the MDR and the AI Act, providers must inter alia ensure that the devices they place on the market or put into service in the EU have been designed and manufactured in accordance with the MDR’s and the AI Act’s requirements (MDR art. 10 ¶ 1 and AI Act art. 16 (a)). Deployers (such as GPs) must, inter alia, use a high-risk AI system in accordance with its instructions for use (AI Act art. 26 ¶ 1) and assign human oversight (AI Act art. 26 ¶ 2). Failure to do so may result in legal action by regulatory authorities, for example, by requiring the provider to bring the system into compliance or recall it (i.e., returning systems that have already been made available) (MDR art. 98 ¶ 1 and AI Act art. 79 ¶ 2). Public safety laws, such as the MDR and the AI Act, are public law instruments. As such, enforcement is up to regulatory authorities. In contrast, private parties (such as patients and GPs) are, in principle, not granted any rights in those instruments to act against other private parties (such as providers of AI-CDSSs).Reference Van Kolfschooten ⁵⁶

Relationship Between Product Safety and Contract Law

However, noncompliance with product safety laws may influence contract law.Reference De Graaf and Veldt ⁵⁷ If a provider sells an AI-CDSS to a GP, sales law generally requires the seller (in our case, the provider) to deliver products ⁵⁸ to the GP in conformity with the contract. See, for example, the criterion “fit for the purposes for which goods of the same description would ordinarily be used”, if it were to be applicable (art. 25 ¶ 1 of the United Nations Convention on Contracts for the International Sale of Goods). Non-conformity may entitle the GP to pursue remedies vis-à-vis the provider, including specific performance (repair/replace), termination for breach (resulting in the provider having to pay back the money the GP paid to the provider, and the GP having to return the AI-CDSS to the provider), and liability for damages. Noncompliance with product safety laws may be a factor in determining whether the AI-CDSS is considered to lack conformity. In business-to-consumer law, EU law even makes explicit that technical standards should be considered when determining whether the object being sold or provided is fit for purpose.Reference Micklitz, Eliantonio, Kokea-aho and Mörth ⁵⁹

If a GP uses an AI-CDSS in furtherance of a medical treatment contract, the AI-CDSS used must be suitable for what it is being applied for, failing which the GP may be in breach of contract vis-à-vis the patient and liable for the damages suffered by the patient as a result thereof (or in some countries liable on account of extra-contractual (tort) liability). In the EU, this is determined by member states’ national laws, which may differ from state to state, and which also impose upon the GP a more general ‘duty of care’ vis-à-vis the patient.Reference Koch ⁶⁰

Relationship Between Product Safety and Tort Law

Noncompliance with product safety law may also influence tort law, i.e., a direct action by the patient against the provider with whom the patient has no contract. If a provider places a defective AI-CDSS on the market or puts it into service in the EU and that defective product causes inter alia personal injury damages to a patient, the provider must compensate the patient for such damages pursuant to the EU Product Liability Directive (PLD), regardless of whether the provider is at fault, i.e., no-fault/strict liability.Reference Graaf, Mak, Tal and Berlee ⁶¹ In order for the patient to be compensated, the patient “is required to prove the defectiveness of the product, the damage suffered and the causal link between that defectiveness and that damage” (PLD art. 10 ¶ 1). However, it may be very difficult, if not impossible, for the patient to prove that the AI-CDSS is defective (PLD art. 10 ¶ 1).Reference Duffourc and Gerke ⁶² That is why the PLD has determined that “relevant product safety requirements” (such as those in the AI Act) need to be taken into account in assessing the defectiveness of a product (PLD art. 7 ¶ 2(f)). The PLD has also sought to alleviate the patient’s burden of proof in the following manner. If the patient demonstrates that the AI-CDSS does not comply with mandatory product safety requirements that are intended to protect against the risk of damage suffered by the patient, the AI-CDSS shall be presumed to be defective, and it will then be up to the provider to rebut this presumption (PLD art. 10 ¶ 2(b) and ¶ 5). This again demonstrates how important it is that the stakeholders’ values, such as the ones identified in this study, are adequately incorporated in the AI Act. If this is not the case, a patient suffering damage due to a defective AI-CDSS may not benefit from the presumption of defectiveness, effectively making it difficult to recover damages from the provider, leaving the patient believing that the values he/she deems important are not adequately protected by law. ⁶³

The proposed AI Liability Directive (AILD), which has since been withdrawn, would have come into play when the PLD does not apply, and (therefore) a national fault-based tort action by a patient against the provider needs to be instigated. Note that it explicitly precludes its application if the AI system only provides information or advice that is taken into account by a GP (AILD rec. 15).Reference Hacker ⁶⁴ The provider of a high-risk AI system within the meaning of the AI Act shall be considered to have failed to comply with its duty of care if it has not abided by a limitative list of specific product safety requirements set out in the AI Act (AILD art. 4 ¶ 1(a)). This includes not designing and developing the AI-CDSS in accordance with requirements relating to (1) human oversight (AILD art. 4 ¶ 2(c)); (2) transparency (AILD art. 4 ¶ 2 (b)); and (3) training techniques and quality of data sets (AILD art. 4 ¶ 2(a)) and accuracy of the AI-CDSS (AILD art. 4 ¶ 2(d)), which in turn contributes to ensuring nondiscrimination and fairness. While the draft AILD has been withdrawn, ⁶⁵ it remains relevant in the sense that it shows that the aforementioned requirements were considered so important to the European Commission (EC), at least at the time the EC drafted them, that noncompliance with those requirements automatically amounted to a failure to comply with the duty of care. This is regardless of whether, as both the PLD and AILD would otherwise require in order to trigger presumptions of defectiveness and causality (PLD art. 10 ¶ 2(b) and AILD art. 3 ¶ 1(a)), the product safety requirement is intended to protect against the risk of damage suffered by the patient.

Compliance Obligations of Providers of AI-CDSS

The legal framework with respect to the development and use of AI, even in the (over)simplified version presented in this section and depicted in Figure 2, is convoluted. However, presenting it has shown that it is important for the provider and GP to respectively develop and use products in compliance with applicable product safety requirements. Failing to do so may cause the provider or GP to face legal action by regulatory authorities or require them to reimburse damages suffered by the GP and patients (respectively) because of such noncompliance. In order not to overcomplicate matters and considering that the AI Act is the most specific product safety law applicable to AI-CDSSs, only the AI Act will be used to contrast the most important values with legal requirements.

Figure 2.

Simplified visual summary of the legal product safety law applicable to DECIDE, and similar AI-CDSSs, and the association with contract law and tort law.

PLD: Product Liability Directive.

MDR: Medical Device Regulation.

Figure 2. Long description

The flowchart is divided into two main sections.

On the left, a vertical column shows three actors connected by dashed lines.

1. At the top is DECIDE’s provider, illustrated as a person with a laptop.

2. Below them is a gray box labeled Contract law: Mostly national laws applicable to relationships between parties.

3. In the middle is a G P, illustrated as a doctor with a stethoscope.

4. Below the G P is another identical gray box for Contract law.

5. At the bottom is the Patient, illustrated as a person with hands raised.

A curved arrow points from the Patient back up to DECIDE’s provider, passing through a gray box labeled Tort law P L D: Ensures that victims can claim compensation from manufacturers when they suffer damage caused by a defective product.

On the right, a large bracket encompasses the entire left-side hierarchy. The bracket points to text stating: Require A I compliance with A I Act and M D R.

To the right of this text are two overlapping circles under the heading Product safety law:

1. The top red circle is labeled A I Act: Fosters trustworthy A I in Europe. Establishes risk-based rules especially for A I providers and deployers.

2. The bottom blue circle is labeled M D R: Maintain M D’s high level of safety. Establishes risk-based classes and rules for manufacturers.

One-on-One Value Sensitive Design Workshop to Uncover Ethical Values Among Providers of Primary Care (the Value Sensitive Design Toolkit)

Multistakeholder Guidance Ethics Workshops

The Guidance Ethics Approach is based on the theory of technological mediation,Reference Verbeek ⁷⁷ which embraces the idea that humans and technology mutually influence one another. It starts with the question of how a specific technology can be a responsible part of our society and how we could strive to accomplish this, but at the same time leaves room for the possibility that conditions for responsible use are unattainable in a specific context, and therefore the development or use of the technology is not justifiable in that specific context. It sees ethics as positive (focusing on what we want), bottom-up (focusing on values identified by stakeholders), and from within (accompanying the design, embedding, and use of technology). The approach has been operationalized mainly in the form of workshops, ⁷⁸ in which stakeholders systematically discuss together, with the help of a moderator, the possible use and ethical aspects of a concrete technology (in our case, DECIDE) in a concrete context (primary care) (Figure 1).

A Guidance Ethics workshop consists of three parts. First, experts with in-depth knowledge of the technology describe the technology and its context, for example, how specific health issues are dealt with in current practice, whether there are unmet health care needs or challenges, the aims and expected benefits of the technology compared to current practice, and how the technology is intended to work in this context (Phase 1: Technology in context). Next, participants identify (a) stakeholders who play a role in and/or could be affected by the development and introduction of the technology; (b) positive and negative effects of the use of the technology, and (c) values inherent in those positive and negative effects. Effects are labelled as “positive” or “negative” when participants consider that something they regard as important (a value) could be positively or negatively affected by the technology (Phase 2: Dialogue). Finally, participants discuss potential strategies or actions that could promote the positive effects, limit the negative effects of DECIDE, and protect the values that have been identified as relevant (Phase 3: Options for action). Three types of options for action are described: technological (related to the technical aspect of the technology in question), environmental (pertaining to the physical and institutional context in which a technology is used), and user-related (concerning things that users of the technology could do or need to be able to do to promote responsible use of the technology).

Two Guidance Ethics workshops were conducted in Leiden, the Netherlands, in February and July 2024. ⁷⁹ Participants were enrolled via the researchers’ networks, looking for a balance between gender and expertise (patient perspectives, design, ethics, legal, and computer science). The first workshop was mostly directed at experts (including patient representatives), while the second workshop sought to emphasize the firsthand perspectives of patients, and specifically those with difficulties in accessing care (e.g., challenges with reading, writing, and counting). The workshops were video-recorded, and photographs were taken of participants’ notes. Recordings were transcribed and thematically analyzed.

Reuse of the Data from the Guidance Ethics Workshops and Value Sensitive Design for the Legal Analysis

In order to align the results of the Value Sensitive Design and Guidance Ethics workshops for the legal analysis, the values as reported by participants of both workshops were categorized (by M.V. and A.K.) according to the trustworthy AI requirements as set out by the AI HLEG. The AI-HLEG Guideline and Assessment List, ⁸⁰ specifically the definitions (Box 1) and descriptions of the trustworthy AI requirements, were used as the basis for this translation task. This decision was made to provide an equal basis for the analysis, because of the aforementioned connection between the AI HLEG 2019 and the AI Act, and because it provided us with the right level of abstraction to subsequently carry out the legal mapping.

The AI HLEG states that trustworthy AI needs to protect fundamental rights recognized by the Charter of Fundamental Rights of the EU. Additionally, it needs to respect the ethical principles of respect for human autonomy, prevention of harm, fairness, and explicability. These principles are translated into the seven trustworthy AI requirements (Box 1). ⁸¹

Human Agency and Oversight

One of the most often identified trustworthy AI requirements resulting from the workshops is human agency (Box 1). In the AI HLEG 2019 Guidelines, human agency is described, inter alia, as “Users should be able to make informed autonomous decisions regarding AI systems. They should be given the knowledge and tools to comprehend and interact with AI systems to a satisfactory degree and, where possible, be enabled to reasonably self-assess or challenge the system. AI systems should support individuals in making better, more informed choices in accordance with their goals” and not “shape and influence human behaviour through mechanisms that may be difficult to detect, since they may harness sub-conscious processes, including various forms of unfair manipulation, deception, herding and conditioning.” ⁹⁰

The Value Sensitive Design workshops provided insights into GPs’ and GP assistants’ perspectives. The requirement of human agency and oversight was associated by the researchers with the participants’ interpretation of the following values as presented in the toolkit: “transparency,” “freedom and autonomy,” “dignity,” “privacy,” “non-maleficence,” “solidarity,” “trust,” and “responsibility and accountability.” The strong presence of the requirement of human agency and oversight (8/10 values) is an indication of how central it is in today’s health care, and specifically when talking about CDSSs. ⁹¹ GPs and GP assistants interpreted freedom and autonomy as the patient being able to refuse, including to refuse to act on DECIDE’s output. Similarly, they considered that GPs may need to retain the freedom not to act on DECIDE’s output. Transparency was, inter alia, interpreted by the participants of the Value Sensitive Design workshops as a (pre)condition for doctors’ and patients’ autonomy and the shared decision-making process. “Freedom and autonomy,” “dignity,” and “privacy” were interpreted as necessary for the shared decision-making process, allowing people to make their own choices, where consent was seen as a must. “Non-maleficence” was associated with the need for consent to opt into the system. Health care professionals explicitly mentioned dignity when speaking about “solidarity,” where they also mentioned the capacity of patients to act on DECIDE’s outcome. “Trust” was related, among others, to the health care professionals’ autonomy to make decisions while using DECIDE, whilst “responsibility and accountability” included aspects related to the freedom of health care professionals to provide advice, and of patients to act on DECIDE’s outcome.

The participants of the Guidance Ethics workshops identified one positive effect of using DECIDE in relation to human agency and oversight, particularly its potential to support individuals in making informed life choices. However, they also mentioned a number of potential negative effects: datafication of the individual, social control, discrimination based on the risk scores, medicalization, added workload for GPs, exclusion of patients with low-risk scores, premature categorization of individuals as patients, stigmatization, lack of explainability and understandability, inappropriate follow-up care, and patient anxiety and stress. The values, according to the participants, related to these effects were summarized as: (a) values of health care professional and patient: autonomy; (b) health care professional-related values: provision of attention/quality time to patients and multiple follow-up moments, to get to know and inform patients better, give more options to patients; (c) patient-related values: individuality, freedom (including negative freedom, freedom of choice even if unhealthy, ability to try out things, not to be judged), self-direction, (right to) self-determination, empowerment, personal power, independence, control (including over image and reputation), access to tailored choices/advice, opportunity to increase knowledge, health as focus of the GP appointment, choice of who to inform, right not to know, preference to receive a letter (over a telephone call), being recognized as a human being, receiving personal and human attention, and opportunity for a group-oriented approach.

From a legal point of view, the AI Act only explicitly mentions human agency in recital 27, stating that it means “that AI systems are developed and used as a tool that serves people, respects human dignity and personal autonomy.” Autonomy is, besides recital 27, referred to in recital 29 in reference to AI-enabled manipulative techniques. The AI Act subsequently embraces human agency by explaining that its purpose is, inter alia, to “promote the uptake of human-centric and trustworthy artificial intelligence” (AI Act art. 1 ¶ 1). More specifically, the AI Act prohibits, in art. 5 para. 1, the following AI practices, which can negatively impact human agency when they cause harm: (a) subliminal techniques or purposefully manipulative or deceptive techniques, and (b) exploiting vulnerabilities of a natural person due to their age, disability or a specific social or economic situation. Human oversight, on the other hand, is more broadly addressed in the AI Act, ⁹² including the explicit obligation that “High-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.” (art. 14 ¶ 1).

The ability to make informed autonomous decisions presupposes that the patient has sufficient information to decide whether to consent to having the GP use DECIDE to determine his/her cardiovascular disease risk and, if the patient consents, to interpret and act upon DECIDE’s output. In our current conceptualization of DECIDE’s implementation, DECIDE will be used by GPs either alone or during the consultation, to provide patients with access to the results. Patients, however, would never use DECIDE on their own. Given this context, the GP must be put into a position to adequately inform the patient about what DECIDE does and answer any questions the patient may have in relation thereto. The values regarded by participants as relevant for DECIDE provide more in-depth understanding of the requirements of human agency and oversight (e.g., dignity and personal autonomy) and also make clear that transparency and explainability are preconditions to achieving human agency. That brings us to the second requirement resulting from the workshops: transparency (which includes explainability).

Transparency

The AI HLEG describes the requirement of transparency (Box 1) as traceability of data, explainability, and communication. Explainability is described, inter alia, as “the ability to explain both the technical processes of an AI system and the related human decisions” and that “Technical explainability requires that the decisions made by an AI system can be understood and traced by human beings…. Whenever an AI system has a significant impact on people’s lives, it should be possible to demand a suitable explanation of the AI system’s decision-making process. Such explanation should be timely and adapted to the expertise of the stakeholder concerned (e.g., layperson, regulator or researcher).” At the same time, the AI HLEG 2019 is mindful that “trade-offs might have to be made between enhancing a system’s explainability (which may reduce its accuracy) or increasing its accuracy (at the cost of explainability).” ⁹³

During the Value Sensitive Design workshops, the participating health care professionals related the value of transparency (which included explainability and interpretability) to the doctor-patient and doctor-DECIDE interactions. They interpreted transparency between doctors and DECIDE as health care professionals being able to “understand” the model, which data are collected, how the “decision” to assign a score to a patient takes place, and the benefits. In the doctor-patient relationship, they interpreted transparency as being able to honestly explain the risk and the most important contributing factors, why a patient group is chosen, and what the consequences are. They also mentioned that transparency should take place not only during the appointment with the doctor, but also when patients are first contacted (e.g., via a letter). Transparency was seen as necessary for trusting the system and for the shared decision-making process. Participants described transparency as a condition for meeting other values such as freedom and autonomy; justice, fairness, and equity; responsibility and accountability; privacy; and trust.

The participants of the Guidance Ethics workshops identified several potential negative effects of using DECIDE in relation to transparency. These included the exclusion of “patients” with low cardiovascular disease risk who might still benefit from DECIDE (e.g., knowing one is leading a healthy life), the unintended consequence of prematurely labelling someone as a patient, causing stigma, stress, and anxiety to patients, and DECIDE being unexplainable (including during phone calls). No positive effects were selected by participants. Relevant participants’ values associated with transparency can be summarized as knowing and informing the patient better (potentially more than once), for which a greater awareness of communication practices is necessary; a right to insight into the risks, and how adjustments (lifestyle and/or medication) would impact that risk; and clear and personalized information and advice, including steps forward, possible treatments, and expected improvements.

The AI Act approaches the requirement of transparency from two perspectives, elaborated on in the next subsections. First, it establishes obligations when a person has been negatively affected by the AI system. Second, it establishes general explainability obligations.

Diversity, Nondiscrimination, and Fairness

The HLEG describes the requirement of diversity, nondiscrimination, and fairness (Box 1) as, inter alia, avoiding unfair bias, explaining that data sets used by AI systems (for training/operation) may suffer from “the inclusion of inadvertent historic bias, incompleteness and bad governance models,” whilst the continuation of such biases “could lead to unintended (in)direct prejudice and discrimination against certain groups or people, potentially exacerbating prejudice and marginalisation.” ¹⁰⁴

The trustworthy AI requirement of diversity, nondiscrimination, and fairness is linked to the value of “justice, fairness, and equity,” “solidarity,” and “responsibility and accountability,” which were identified by health care professionals as relevant for DECIDE during the Value Sensitive Design workshops. Health care professionals saw this requirement as applicable to the provision of care (“to treat everyone”), to the data (data collection and storage, e.g., in the electronic health records), the model, and the developers (who should have an “ethics background”). Diversity, nondiscrimination, and fairness was mentioned by health care professionals as related to the value of solidarity (associated with avoiding the labelling of patients) and accountability (treating everyone with justice).

During the Guidance Ethics workshops, the requirement of diversity, nondiscrimination, and fairness was linked to potential positive and negative effects. As a positive effect, participants identified that DECIDE could provide the opportunity to include individuals normally not included in prevention programs. In contrast, several negative effects were identified: increasing pressure on basic health care costs (potentially limiting equal access to basic care), higher workload for GPs, exclusion of “patients” with low cardiovascular disease risk who might still benefit from DECIDE, premature labelling of individuals as patients, stigma, datafication, social control, and discrimination based on risk scores. According to participants, the inherent values related to these effects included: ambiguity (not fitting standard profiles), freedom not to be judged, equity (including in opportunity, care provision, with emphasis on groups that need care), equality (including in care provision and equal worth of patients), inclusiveness, justice, accessibility to and quality of care, ensuring that one does not automatically get a stamp (e.g., “patient”), protection against negative impacts on health insurance, and access to a broader range of care options.

Historically, diversity, nondiscrimination, and fairness fall within the ambit of, for example, the EU Charter and the GDPR. ¹⁰⁵ What we are more concerned with in this article are specific requirements in the AI Act with respect to high-risk AI systems, which contribute to fulfilling these HLEG requirements.Reference Van Kolfschooten ¹⁰⁶ The AI Act requires the provider of DECIDE, if it is considered a high-risk AI system, to ensure that the system is developed on the basis of training, validation, and testing data sets which meet certain quality criteria (AI Act art. 16 (a) and 10 ¶ 1, 2, and 5). The criteria include data governance and management practices concerning “examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination” (AI Act art. 10 ¶ 2(f)) and “appropriate measures to detect, prevent and mitigate possible biases identified according to point (f)” (AI Act art. 10 ¶ 2(g)). The criteria also include that “Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose” (AI Act art. 10 ¶ 3).

Complementary to aspects directly linked to the requirement of diversity, nondiscrimination, and fairness, the AI Act also establishes requirements of technical robustness. The Act also requires the provider to ensure that the system is designed and developed “in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle” (AI Act art. 16(a) and 15 ¶ 1), more in particular by being “as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or the environment in which the system operates, in particular due to their interaction with natural persons or other systems” and, if the systems continue to learn, being “developed in such a way as to eliminate or reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are duly addressed with appropriate mitigation measures” (AI Act art. 15 ¶ 4). The development of the system in accordance with these criteria could contribute to reducing false positives (unnecessarily determining that a patient falls into a certain cardiovascular disease risk category and inviting the patient to the GP’s office for further examination). It could also help avoid discrimination, stigmatization, and negative impacts on health insurance, and it could ensure fair access to health care. However, the effectiveness of these rules in mitigating these concerns has been debated. ¹⁰⁷

Strengths and Limitations

This study has limitations and strengths. The study was based on the secondary analysis of qualitative and empirical research data, which required the data to be “standardized” prior to the legal analysis. We used the trustworthy AI requirements as an analytic tool for this purpose; as demonstrated, this approach is useful but also carries certain constraints. Nevertheless, empirically identifying these constraints constitutes a valuable contribution to the theoretical analyses previously reported.Reference Hagendorff ¹¹⁰ Additionally, it identifies three complementary requirements to the seven trustworthy AI requirements in the context of AI-CDSSs. Another limitation is that data were collected from citizens from only one country. The number of participants was considered adequate for qualitative analysis, but the application of our results to other settings should be considered with the understanding that we still have no good idea of the extent of the potential generalizability of our results. However, using empirical data as the basis for carrying out a legal analysis is one of the key strengths of the study. The qualitative data produced were very rich; they included a wide range of stakeholders and reflected a serious commitment to providing an effective space to vulnerable populations (represented by participants with difficulties in writing and reading). It is also worth mentioning some methodological considerations of the Value Sensitive Design and Guidance Ethics workshops. The Value Sensitive Design workshops used a deductive approach, yielding more focused but narrower data on participants’ values. In contrast, the Guidance Ethics workshops adopted an inductive approach, and participants selected effects (and corresponding values) they found most important to discuss in detail. A limitation of the study is that the complex EU and national law landscape that regulates product safety for AI in health care was only briefly mentioned in this article for the sake of clarity. However, that did not limit our capacity to achieve our main objective. Additionally, our approach allowed us to identify relevant gaps in the AI Act with respect to the protection of values deemed relevant by individuals in our society. It also allowed us to propose ways to compensate for those shortcomings; for example, by contractual means. However, we limited our analysis to product safety regulation, as clarifying this framework was a necessary preliminary step before undertaking a more comprehensive examination of interactions between different regulatory frameworks. Finally, this study demonstrated the value of integrating ethics, design, and law through a multidisciplinary approach and based on empirical data, an approach that is rarely seen in legal analysis.