In this issue, Hannah van Kolfschooten discusses the importance and shortcomings of the European Union’s Artificial Intelligence (AI) Act, a groundbreaking legal framework for AI that falls short as applied to some health care contexts.Reference van Kolfschooten ¹ The law bans certain AI practices that threaten human autonomy or exploit vulnerability, but allows exceptions for medical or safety purposes. Van Kolfschooten argues that without a legal framework for applying these exceptions, they are effectively loopholes that may be exploited at the expense of vulnerable patients.

The use of AI in health care is not new,Reference Kaul, Enslin and Gross ² but it presents a wide range of risks ³ that are exacerbated by the explosion of generative AI in a relatively short period of time. Even as health care providers and payers incorporate generative AI into their daily practice and patients want to understand how AI is being used in their health care,Reference Platt ⁴ the technology is largely unregulated in the United States. ⁵ The EU’s AI Act is an important first step in establishing principles for the regulation of AI, but may be insufficient in the case of health care exceptions, and in the United States, existing legal and ethical frameworks are also insufficient to address some types of AI use in health care.

Providers have long had to balance the principles of autonomy and beneficence, which are sometimes in conflict. In mental health care, in particular, patients may be deemed incompetent to consent to a particular treatment (so that a provider or family member substitutes their judgment in the patient’s best interest)Reference Chaet ⁶ or a patient may benefit from a treatment that undermines their autonomy, such as a somewhat manipulative behavior modification therapy. Current frameworks depend on the judgment of a human provider to balance competing ethical principles, ensure patient understanding and informed consent, and intervene if the risks outweigh the benefits.

In the case of AI, however, an AI tool may be deployed without a provider’s expert guidance, or the AI tool may make judgments that are hidden from the patient and provider. In those cases, the patient cannot depend on a provider’s professional ethics to ensure compliance with ethical and legal standards. Under the EU’s AI Act, van Kolfschooten argues, the exceptions to the Act’s restrictions on AI may be applied without consideration of therapeutic necessity, proportionality, avoiding undue coercion, and the right to alternate treatments. This is insufficient to protect patient autonomy. Legal and ethical frameworks for the use of such AI tools should ensure that human ethical judgment remains in the process. Current frameworks that assume such human involvement (and liability) may not be adequate for new AI applications.

The lack of transparency in AI tools, especially generative AI, is especially at odds with current frameworks.Reference Kiseleva, Kotzinos and De Hert ⁷ The consent-based model of health care ethics, which is the foundation of legal frameworks governing standards of medical care as well as standards for human subjects research, depends on an individual patient or subject being able to understand the risk of a particular intervention, choosing freely to participate, and being able to revoke that consent if desired.Reference Hariri ⁸ In contrast, the hallmark of generative AI is its ability to customize responses for an individual that may be different from the responses it gives to another individual. This means that a provider or researcher may not be able to know how the AI tool will interact with the patient or subject, and therefore may not be able to adequately characterize the risk presented by the AI-driven intervention. The continuously evolving nature of generative AI tools makes them inherently unpredictable. How can AI tools be deployed as part of patient-centered care if the patient cannot be adequately informed about what they will do, to be able to evaluate risk and decide to opt in or out? How can providers or researchers adequately understand the risk or benefit if they cannot look “under the hood” of AI tools to ensure that they are functioning safely and as intended? The fact that many AI tools are based on proprietary technology may prevent the transparency necessary to capture patient choices, inform patient understanding, ensure informed consent, and understand the impact of AI-driven interventions. Corporate interests that seek to limit liability and protect intellectual property (IP) may work against closing the loopholes in the EU’s AI Act.

Van Kolfschooten calls for the expansion of the legal framework created by the AI Act to ensure that exceptions are only “applied in circumstances where their use is indispensable, proportionate, and accompanied by safeguards capable of offsetting the structural disadvantages faced by patients.”Reference Van Kolfschooten ⁹ In addition, we need to create ethical and legal frameworks for the use of AI in health care that are truly patient-centered. This means ensuring not only informed consent but also accountability and quality,Reference Kiseleva ¹⁰ including safety and equity. Transparency is necessary to understand the impact of AI-driven tools on patients and to understand how such tools align with ethical principles like autonomy and informed consent, and what level of transparency is sufficient may vary from one use case to another. Existing frameworks assume a level of predictability and understanding that may not be present in applications of AI in health care. New frameworks are necessary to ensure that core ethical principles like autonomy are preserved even as AI evolves and its use expands across health care.