Knowledge as echo, not reason

LLMs excel in textbook-style questions because they are fluency machines trained in vast medical corpora. Bang et al also documented frequent diagnostic contradictions, including assigning patients with mutually exclusive conditions. ^{Reference Bang, Jung, You, Kim and Kim1} These ‘comorbidity errors’ are serious. They are instances of LLM ‘hallucinations’: misleading, incorrect or entirely made-up information (Table 1). The hallucinations observed by Bang et al reveal that current models do not reason interpretively: they echo probability distributions. ^{Reference Bang, Jung, You, Kim and Kim1–Reference Kirmayer, Aggarwal, Bhui, Fung and Kohrt4} Psychiatric diagnosis is an interpretive act shaped by culture, trauma, language and the therapeutic relationship. ^{Reference Kirmayer, Aggarwal, Bhui, Fung and Kohrt4} Non-verbal cues that can shape diagnoses are invisible to text-based systems. High-performance medicine requires a convergence of human and artificial intelligence, not the substitution of one for the other. ^{Reference Topol3}

Table 1

Promises and pitfalls of large language models (LLMs) in psychiatry

Table 1 Long description

“Promises and pitfalls of large language models in psychiatry.” This replaces the version previously supplied. I’m flagging the change explicitly because the prior description misrepresented the table’s structure, and the correction matters for both accuracy and accessibility (screen-reader rendering). What was wrong with the earlier version: 1. It described Table 1 as a single four-column grid. Table 1 is actually three independent sub-tables — 1A, 1B, and 1C — each with three columns and its own distinct row axis. The four-column structure did not exist in the source. 2. It treated “Why benchmark superiority does not equal clinical safety” as a column header. That string is the title of sub-table 1A, not a column. The first column header is “‘Specialized’ Benchmark Context.” 3. It detached the “Risk If Absent” column from sub-table 1B and grafted it onto 1A’s rows, creating row pairings the source never makes. It also silently dropped 1B’s other two columns (“Principle” and “Why It Matters in Psychiatry”), even though “Risk If Absent” is only interpretable relative to its principle. 4. Sub-table 1C (“Consequences of presenting AI as a clinical authority,” five rows) was omitted entirely. What the corrected version does: it describes 1A, 1B, and 1C as three separate sub-tables with their true titles, column headers, and row counts (1A: 3 columns x 6 rows; 1B: 3 columns x 6 rows; 1C: 3 columns × 5 rows). All cell wording follows the source spelling (e.g., “Specialized,” “judgment,” “optimized”). The corrected define-summary is attached/pasted below. Please confirm once it has been substituted for the prior version, and let me know if you need it in a particular format for your typesetting workflow. Thank you, Here’s the corrected define-summary, faithful to the source structure: Table 1. Promises and pitfalls of large language models in psychiatry. This is a single table composed of three independent sub-tables (1A, 1B, 1C), each with three columns and its own row axis. Source: Advice despite Low Accuracy, NEJM AI. 2025;2(6):2300015. Table 1A — “Why benchmark superiority does not equal clinical safety.” Three columns: “Specialized” Benchmark Context | “Generalized” Real-World Psychiatric Context | Clinical Implications. Six rows: High performance on structured tests → Performance degrades in open, emotionally complex clinical settings → Benchmark success does not predict bedside safety. Clean, pre-filtered vignette inputs → Fragmented narratives, ambiguity, silence, contradiction → Loss of non-textual clinical signals increases misinterpretation. Textbook diagnostic fluency → Longitudinal instability across prompts and time → Models generate plausible text, not demonstrating clinical understanding. Simulated empathy in benchmarks → “Relational AI” designed to simulate companionship or intimacy → Simulated support may mask the absence of clinical safety protocols and reasoning. General competence across structured exams → Fragility in affective and moral reasoning in the clinical setting → Psychiatry depends on nuance, judgment, and ethical responsibility. Controlled evaluation environments → Market-driven deployment of relational systems → Engagement-optimized design prioritizes user dependence over patient safety. Table 1B — “Principles for safe and ethical AI integration in psychiatry.” Three columns: Principle | Why It Matters in Psychiatry | Risk If Absent. Six rows: Real-world evidence → Psychiatric illness unfolds within culture, trauma, and lived experience → Tools validated only on vignettes may fail vulnerable patients. Market–safety alignment → Incentives must prioritize patient welfare over market dominance → Safety becomes “collateral damage” to commercial growth. Complete honesty about limitations → Trust depends on transparency and humility → Over-confidence invites inappropriate reliance. Visible human responsibility → Psychiatry requires moral and legal accountability → Responsibility cannot be delegated to machines. Relational integrity → The therapeutic alliance is a clinical intervention, not a product feature → Simulated intimacy replaces professional oversight. Explicit hallucination safeguards → Generative models can fabricate persuasive clinical claims → Undetected hallucinations undermine insight and adherence. Table 1C — “Consequences of presenting AI as a clinical authority.” Three columns: Observed Phenomenon | Illustrative Evidence | Implications for Psychiatric Care. Five rows: Inflation of AI authority and institutional drift → Misleading headlines that frame AI as authority rather than as an assistive tool based on the results of limited benchmarks → Patients treat AI outputs as clinically definitive; loss of ethical and professional foundations of care. Clashing incentives (public health vs. market incentives) → Market-driven AI designed for intimacy, increasing usage and dependency → Safety is subordinated to engagement and growth. Shifting patient behavior → Patients uploading psychiatric records to general-purpose LLMs due to free, appealing, and convenient nature → Normalization of unsupervised, unvetted treatment decisions. Amplification of unsafe or unhealthy mindsets or actions → AI appealing to and validating users experiencing mental health crises; indirectly suggesting risky or harmful behaviors → Increase in risk for vulnerable individuals with a lack of direct healthcare access. Marginalization of complex patients → AI is unable to provide correct advice on complex, context-dependent, and obscure problems due to rigid adherence to scripts, input-derived misunderstandings, or hallucinations → Adoption of ineffective or potentially harmful treatments that engender distrust in healthcare.

The illusion of the benchmark

Media headlines that ‘AI beats doctors’ misrepresent Bang et al’s findings. Psychiatry residents are supervised trainees. Comparing a model trained on a vast record of published medical knowledge to early-career clinicians in a controlled study is an apples-to-oranges comparison violating basic principles of evidence-based medicine. Algorithms that excel in laboratory settings often falter in clinical usage. Murthy describes this as the ‘generalist–specialist paradox’, where medical artificial intelligence models excel in rigid, specialised tasks but falter in situations requiring contextual, relational and affective reasoning. ^{Reference Murthy2} These situations are predominant in real-world medicine, where cultural and contextual awareness are needed to accurately exchange information to and from a patient (Table 1). These intricacies are not communicated to the public, who instead absorb headlines that erode trust in human care.

The limits of vignettes

Bang et al’s study relies on structured case vignettes. ^{Reference Bang, Jung, You, Kim and Kim1} These are useful pedagogical tools but poor proxies for real psychiatric encounters. Real-world symptoms are shaped by cultural meaning, interpersonal dynamics and non-verbal cues: a change in tone or posture can make a diagnosis. None of this complexity survives translation into a well-engineered prompt. Evidence suggests that numerical accuracy does not guarantee positive clinical outcomes. Cardiac algorithms have produced alarming false positives outside controlled settings, ^{Reference Lampert, Bhatt, Vaid, Kon, Feinman and Jou5,Reference Kang, Lo, Zhang, Ng, Lin and Scott6} and artificial intelligence tools in ophthalmology have shown performance collapses under independent scrutiny. ^{Reference Kang, Lo, Zhang, Ng, Lin and Scott6} In psychiatry, equating numerical accuracy with positive clinical outcomes risks life-altering consequences.

Assistance inviting substitution

Bang et al also found that model guidance improved resident performance. ^{Reference Bang, Jung, You, Kim and Kim1} This suggests a promising role for artificial intelligence as an intelligent reference that broadens differential diagnoses or checks facts. The risk, however, is that assistance quietly becomes substitution. As systems become more fluent and persuasive, clinicians may stop doubting them. Automation rarely fails by overthrowing human judgement: it fails by inviting complacency. ^{Reference Chang, Huang, Hsu and Liu8,Reference Shekar, Pataranutaporn, Sarabu, Cecchi and Maes9} In psychiatry, where uncertainty is intrinsic and moral responsibility is central, this drift is dangerous.

The quiet problem of inherited bias

LLMs are trained on humanity’s written record and inevitably inherit its prejudices. Studies show that artificial systems can reproduce and amplify cognitive and social biases in clinicians. ^{Reference Shekar, Pataranutaporn, Sarabu, Cecchi and Maes9} LLMs also introduce the risk of commercially incentivised bias. The ‘alignment’ of an LLM’s answers is set by its developers: LLMs may thus be aligned towards commercial viability rather than patient safety. In addition, the massive data-sets used to train LLMs may include sponsored content, potentially introducing commercial bias (Table 1). In psychiatry, bias shapes identities, determining who is labelled anxious, unreliable or ‘non-compliant’. When a machine speaks with confidence, its inherited bias becomes harder to detect and easier to obey. This is why we emphasise that medical artificial intelligence must adhere to fundamental ethical principles, emphasising transparency and accountability.

Methodologically limited comparisons between artificial intelligence systems and clinicians are increasingly contributing to a public perception that artificial intelligence can outperform physicians in real-world care. This belief may foster inappropriate trust in artificial intelligence-generated medical advice, particularly among vulnerable populations, despite evidence of low diagnostic accuracy ^{Reference Shekar, Pataranutaporn, Sarabu, Cecchi and Maes9} (Table 1).

Psychiatry as a moral practice

Psychiatry is a moral practice before being a technical one. It rests on ethics, philosophy, sociology, anthropology and the study of suffering. It requires moral courage, judgement under uncertainty and capacity for regret. No machine bears the weight of moral responsibility, grief or regret. If the profession becomes enchanted by tools that speak without conscience, it risks forgetting its own foundations. The rapid deployment of artificial intelligence in mental health is often driven by commercial pressure rather than by patient need or psychiatric consensus, resulting in interactions that emphasise user retention and validation, using intimacy at the risk of patient safety and dependence on artificial intelligence ^{Reference Peoples and Blumenthal-Barby7} (Table 1). We do not test new medications on the public and ask ethical questions later, yet artificial intelligence deployment often follows this troubling path. The concern is not technology itself, but the risks to safety, trust and quality of care when powerful systems are introduced without safeguards.

The measure of progress

Bang et al’s study shows that machines can imitate psychiatric knowledge. However, the greatest risk is not incorrect diagnoses, it is the gradual displacement of human connection. If we begin to believe that understanding can be replaced by computation, we turn complex emotions into mere data. Progress in psychiatry should not be measured by speed or accuracy alone, but by whether patients feel safe, seen and respected. Artificial intelligence may have a place in this delicate space, but as augmentation, never replacement. The future of psychiatry will be determined not by what machines can do, but by what we decide they must never replace.