Reply

I thank Dr Ryland for his thoughtful commentary on my study of large language model (LLM) performance in MRCPsych-style questions. ^{Reference Ryland1} His call for psychiatrists to remain alert and adaptable as artificial intelligence (AI) evolves to ensure it promotes the best interests of both patients and the profession is apt. Thankfully, various inherent barriers must be overcome before LLMs and AI more broadly can be responsibly integrated into clinical psychiatry. These challenges are naturally slowing adoption, thereby affording the profession the necessary time for adaptation. They are broadly as follows.

First, LLMs hallucinate – generating plausible but fabricated information that poses clear risks to patient safety. ^{Reference Harrer2} While hallucination rates are falling as models evolve, hallucinations might be an intrinsic property of current architectures rather than a remediable flaw. ^{Reference Asgari, Montaña-Brown, Dubois, Khalil, Balloch and Yeung3} Further work to minimise such errors, and collective deliberation on what constitutes an acceptable rate and severity of error, is first required before widespread LLM adoption.

Second, and relatedly, LLMs are poorly calibrated: they cannot reliably distinguish what they know from what they do not. My study found correct and incorrect answers stated with equal confidence. Even when prompted to verbalise their certainty, LLMs are consistently overconfident in their assertions of medical diagnoses and treatments. ^{Reference Savage, Wang, Gallo, Boukil, Patel and Safavi-Naini4} This is not merely inconvenient, since knowing the limits of one’s knowledge is often more important than the knowledge itself in clinical practice.

Third, medical LLMs trained on historical data risk perpetuating and amplifying existing biases, including demographic disparities related to gender and race. ^{Reference Omar, Sorin, Agbareia, Apakama, Soroush and Sakhuja5} In psychiatry specifically, LLMs have been shown to propose inferior treatments when patient race is indicated, with bias most evident in cases of schizophrenia and anxiety. ^{Reference Bouguettaya, Stuart and Aboujaoude6} Such biases threaten to deepen rather than reduce healthcare inequalities.

Fourth, despite advances in reasoning models, the underlying mechanisms of LLMs remain inherently opaque. Interrogating the basis for their outputs is difficult, making it challenging for clinicians to evaluate whether a recommendation is sound or to explain decisions to patients. ^{Reference Mesinovic, Watkinson and Zhu7} This ‘black box’ problem undermines the transparency that underpins informed consent and shared decision-making in psychiatry.

Fifth, regulatory and liability frameworks remain underdeveloped. Clinical decision support software using complex, non-transparent algorithms requires strict regulatory oversight. The Medicines and Healthcare products Regulatory Agency is actively developing guidance for AI as a medical device, ⁸ but when AI contributes to adverse outcomes, the allocation of liability between clinician, deploying institution and technology developer remains legally untested.

Finally, data governance presents unresolved tensions. Patient information entered into commercial LLMs may be stored, processed or used for model training in ways that conflict with data privacy and confidentiality requirements. ^{Reference Jonnagaddala and Wong9} The sensitive nature of clinical data in psychiatry makes these concerns especially challenging.

These are not arguments against integrating AI into clinical practice but preconditions for its responsible adoption. The task for psychiatry is not merely to adapt to technological change but to shape its integration in ways that serve the interests of both patients and the profession. The inherent delays in adopting these technologies provide a valuable opportunity to do so.