The assumption of rational input

Tversky and Kahneman demonstrated that human reasoning is not merely occasionally flawed. but systematically distorted. ^{Reference Tversky and Kahneman1,Reference Kahneman2} Anchoring, the availability heuristic, loss aversion and confirmation bias are not aberrations; they are the default operating mode of human cognition. Beck’s cognitive model further established that clinical populations exhibit predictable patterns of distorted thinking: catastrophising, selective abstraction, overgeneralisation and dichotomous reasoning. ^{Reference Beck3} These distortions are not confined to those with diagnosable conditions. They operate, to varying degrees, across the general population.

The implications for AI are significant. If training data reflect not what people experienced, but what they believed they experienced, filtered through threat miscalibration and negativity bias, then the model is not learning human behaviour. It is learning human behaviour as reported by cognitively biased agents. The output appears rational because the input appeared rational. But the input was generated by minds that systematically deviate from rationality in well-documented ways.

AI systems are already being used to triage mental health referrals, ^{Reference Smith, Taylor, Joyce, Page, Joyce and Cipriani4} assess risk in criminal justice settings ^{Reference Dressel and Farid5} and deliver psychological interventions through automated platforms. If the data driving these systems were generated by individuals operating under cognitive distortions – and the evidence from cognitive science and clinical psychiatry suggests that it routinely is – then the validity of their outputs deserves far more scrutiny than it currently receives.

The gap between presentation and experience

Clinical psychiatric assessment operates on a principle that has no equivalent in AI development: what a person presents may differ fundamentally from what they experience. This is not a limitation of the clinical method. It is the clinical method.

Patients detained under the Mental Health Act 1983, or facing the possibility of detention, may minimise psychotic symptoms, deny suicidal ideation or present a degree of wellness they do not feel. This is understandable. When the alternative is compulsory admission, enforced medication or decisions made without your involvement, managing what you disclose is not deception. It is self-preservation. Any experienced clinician recognises this. Equally, patients may present with exaggerated distress, seeking admission to a place of safety or maintaining attachment to services that have become their primary social structure. In both directions, the gap between presentation and experience is clinically significant.

In medicolegal practice, this discrepancy is at times the entire focus of the assessment. The expert witness evaluates credibility, identifies inconsistency between self-report and collateral evidence, and forms a clinical opinion that may diverge significantly from the account the individual provides. This work exists because the legal system recognises what AI development has not yet confronted: that human self-report cannot be taken at face value.

This gap is not confined to clinical AI. Any platform that processes human input, whether conversational agents, hiring algorithms or financial decision-support tools, operates on the same unexamined assumption: that what the user provides broadly reflects what they think, feel or intend. ^{Reference Palmier-Claus, Ainsworth, Machin, Barrowclough, Dunn and Barkus9} A person seeking advice from a general-purpose AI during a period of acute distress will generate input shaped by that distress: catastrophised, narrowed, driven by threat appraisal rather than balanced reflection. The model responds to the input as given. It has no mechanism for recognising that the input itself may be a symptom rather than a statement.

The populations that AI understands least

The concerns raised above apply to the general population. But there is a further dimension. The people whose conditions make it hardest to contribute meaningful training data are the same people about whom AI-driven decisions carry the highest stakes, and the greatest potential for harm.

Consider the individual experiencing an acute psychotic episode. Their perception of reality is fundamentally altered. Their speech may be characterised by loose associations, tangential thinking or frank word salad. These are patterns that a trained psychiatrist recognises as diagnostically significant, but that a language model trained on coherent, structured text would likely screen out as irrelevant or corrupted input. The very data that carries the most clinical meaning would be discarded as noise. Their capacity for insight and ability to recognise that they are unwell may be entirely absent. This individual cannot generate data that accurately represents their own experience because their illness has disrupted the very cognitive architecture on which accurate self-report depends.

The same applies to individuals in acute mania, whose grandiosity, disinhibition and pressured output may present as confidence and productivity to any system without clinical training. For individuals with severe depression, cognitive narrowing, low motivation, psychomotor retardation and hopelessness may render them unable or unwilling to engage with digital platforms at all.

Neurodevelopmental conditions present a different but equally significant challenge. Individuals with autism may communicate in ways that diverge from neurotypical norms in prosody, pragmatic language and the relationship between expressed and felt emotion. Individuals with intellectual disability may be unable to engage with standard data-collection instruments. These people are absent from the data not because they have nothing to contribute, but because the systems were not built to accommodate how they communicate.

The result is a systematic gap. These populations exist in AI training data primarily through clinical records and research literature written about them by others. Their lived experience is largely missing. AI systems trained on this data do not understand these individuals. They understand how clinicians describe these individuals, which is a fundamentally different thing. ^{Reference Morrin, Nicholls, Levin, Yiend, Iyengar and DelGuidice6}

This matters because AI-driven systems are already making or informing decisions that affect these populations directly: in mental health service provision, benefits and disability assessments, criminal justice risk prediction and safeguarding. The people AI understands least are the people for whom the consequences of being misunderstood are most severe.

The upstream question

The current conversation about AI safety and alignment is dominated by computer science, philosophy and public policy. Recent contributions have proposed frameworks for epistemic trustworthiness in AI agents ^{Reference Marchal, Chan, Franklin, Revel, Keeling and Fischli7} and called for governance structures to manage the risks of increasingly autonomous systems. ^{Reference Christian8} This work is valuable. But it is incomplete.

The frameworks being proposed to evaluate whether AI systems can be trusted to model human behaviour have been developed almost entirely without input from the profession that has spent the longest studying how human behaviour is expressed, concealed, distorted and misrepresented. Psychiatry holds specific, well-evidenced expertise in each of the areas this article has identified: the systematic unreliability of human self-report, the gap between presentation and experience, and the populations whose conditions place them beyond the reach of conventional data collection. This expertise is not theoretical. It is practised daily, in clinical encounters, medicolegal assessments, risk evaluations and the ongoing care of the populations most affected by AI’s behavioural blind spots.

Recent work on AI and mental health has begun to identify some of these concerns, particularly the risk that large language models may validate or amplify delusional content in vulnerable users. ^{Reference Morrin, Nicholls, Levin, Yiend, Iyengar and DelGuidice6} This is an important step. But the focus has remained on the downstream effects of AI interaction on individuals who are already unwell. The upstream question, whether the data on which these systems were trained is itself shaped by the cognitive and clinical phenomena psychiatry routinely assesses, has received far less attention. ^{Reference Stade, Stirman, Ungar, Boland, Schwartz and Yaden10}

The question is not whether AI is useful; there is little doubt that it will transform aspects of healthcare in beneficial ways. The question is whether AI systems that model human behaviour can be considered valid without input from the profession that understands most intimately how human behaviour is produced, performed and misread.