Response

The recent Feature by Cockburn and Large ^{Reference Cockburn and Large1} uses brief vignettes to illustrate what they see as the potential pitfalls of using the OxSATS risk prediction tool for suicide to aid clinical decision-making. In the Feature the authors create eight vignettes, which they then use to calculate risk percentages with the freely available online OxSATS risk calculator (https://oxrisk.com/oxsats/). The vignettes are then interpreted by the authors to make points about the perceived limitations of using the OxSATS tool to predict suicidality. The use of vignettes to explore the acceptability, feasibility and utility of risk prediction tools is well established; however, best practice requires their use in empirical studies, rather than the speculative manner that the Feature article adopts. In this Commentary, we discuss how vignettes are used empirically and how this methodology can enrich our understanding of using risk prediction tools in clinical practice when employed appropriately. In addition, we suggest that Cockburn and Large have misused the vignette approach and outline a practical guide for clinicians to appraise or conduct research of this sort.

Risk prediction tools are widely used in many areas of medicine to support clinical decision-making but remain underused in psychiatry. Various methods are used to develop risk prediction tools, including traditional statistical approaches based on regression techniques and those using machine learning methods. Despite many publications on prediction models, there is limited research on their clinical effectiveness, with little evidence to support the selection of any one approach over another, such as regression modelling in preference to machine learning. ^{Reference Zhou, Chen, Cao and Peng2} The recent proliferation of clinical prediction models reflects the increasing ease with which these algorithms can be developed, with a notable acceleration after 2010 and an estimated total of 248 431 models published by 2024. ^{Reference Arshi, Wynants, Rijnhart, Reeve, Cowley and Smits3} The availability of ‘big data’ allows researchers to generate an abundance of tools without necessarily considering their clinical applications. To demonstrate an impact on clinical outcomes, several stages are needed, starting with external validation in new populations to establish generalisability. This should be followed by implementation and evaluation of effectiveness through suitably designed experimental studies, such as cluster randomised trials. Despite this complex, multi-stage pathway, only a minority of models reach the external validation stage, with even fewer making it to implementation.

Risk prediction tools that do reach practice are typically used by clinicians who are also responsible for interpreting them and, having done so, they may then communicate the results to patients or simply integrate model outputs into clinical decision-making without discussing this with their patients. Either way, it is essential that clinicians are comfortable with using such tools in practice. Understanding implementation has several components, including acceptability, feasibility and utility. Acceptability of risk prediction refers to the readiness of clinicians to use these tools, and feasibility is the practicality of integrating tools within existing practice; in turn, these depend on factors such as the availability of model input information and the existence of easy-to-use risk calculators. Utility considers the usefulness of information generated by a new tool, which leads to actionable results and improved care. Perceived benefits are not limited to clinical outcomes, and may include improved communication, patient experience or workflow.

Mixed-methods studies based on standardised vignettes provide a highly informative way of exploring implementation outcomes with clinicians from relevant services. This can help clarify whether risk prediction models could fit with current practice and aid with the more effective application of tools, for example by refining the presentation of risk information. Using diverse vignettes allows researchers to explore varying scenarios, including those with higher and lower risk scores, with clinicians in a controlled fashion. Vignette-based studies also offer the opportunity to consider questions around implementation in an efficient and ethically acceptable way, without the need to involve real patients. This can represent considerable advantages when some regulatory frameworks consider risk prediction tools as interventions, which can make approval and evaluation expensive and time-consuming. In fact, because most tools do not recommend treatments they can be viewed as low-risk devices.

Vignettes can be developed by adapting and anonymising real cases or deliberately constructing cases that present particular uncertainties. ^{Reference Cowley, Farewell and Kemp4} Involving relevant clinicians or patients in the design of vignettes may help to increase credibility. The inclusion of more vignettes allows a greater range of scenarios to be discussed, but too many vignettes may hamper recruitment by increasing the time burden on participants, who are typically busy clinicians. Participants are often requested to start by assigning a risk rating to each vignette based solely on their clinical judgement. This can be followed by the participant completing a risk tool or calculator themselves, or by simply presenting them with the tool’s output. The former approach enables participants to have a sense of the practicalities of using a tool, and allows for testing of interrater reliability. Clinicians can then be asked whether they would change their management considering the information provided by the risk tool. This can be achieved by asking them to choose between National Institute for Health and Care Excellence-recommended actions ^{Reference Cowley, Farewell and Kemp4} or through less structured reflections on hypothetical options. ^{Reference Whiting, Glogowska, Mallett, Maughan, Lennox and Fazel5}

The concordance between clinician (without use of a tool) and tool score, and perceived usefulness in practice, is complex. ^{Reference Beaudry, Canal-Rivero, Ou, Matharu, Fazel and Yu6} For example, high concordance may reassure clinicians of a tool’s credibility because it aligns with their own judgement. Conversely, lower concordance may be useful to prompt clinicians to reconsider their own evaluation of risk, if it deviates markedly from an empirically derived prediction tool. Although these analyses can yield important insights, a more detailed understanding can be achieved through more formal qualitative methods. Free text boxes in survey-based methodologies may offer one route, but richer data are often obtained using semi-structured interviews. ^{Reference Whiting, Glogowska, Mallett, Maughan, Lennox and Fazel5} The interactive nature of interviews allows researchers to respond to, and explore, the nuances of clinician and patient views. This lack of clinician input is a notable limitation in the Cockburn and Large approach.

Controversy surrounds the use of risk prediction tools for self-harm and suicide. Recent guidance from NHS England recommends against their use, in favour of a safety-based approach. This view is based on dated evidence that overlooks the potential benefits that newer, high-quality risk prediction tools could offer, if correctly used. ^{Reference Seyedsalehi and Fazel7} For example, a 2022 multinational study of a probabilistic tool for suicide risk in mental illness (OxMIS) found that 82% of 60 clinicians interviewed stated that the risk calculator would be practical to use, and that 89% would consider using it in their future practice. ^{Reference Beaudry, Canal-Rivero, Ou, Matharu, Fazel and Yu6} Importantly, in the rest of medicine, a recent evidence review on risk communication has recommended using probabilities rather than imprecise terms such as ‘common’ or ‘unlikely’. ^{Reference Zikmund-Fisher, Thorpe and Fagerlin8} Further evidence about the implementation of such tools is urgently needed, and vignette-based studies offer a powerful way to do this. However, it is essential that researchers adopt an agnostic approach and empirically test tools with clinicians from relevant services, rather than relying on their own interpretation. A recent vignette-based study of the OxSATS tool including 11 UK-based clinicians found that participants valued its ease of use, the objectivity it provided and how it supported structured thinking around risk factors, especially in uncertain cases, alongside potential limitations such as over-reliance on tool outputs. ^{Reference Ouaret Sorr9} These findings contrast with the exclusively negative interpretations of Cockburn and Large, offering a more nuanced and evidence-based approach to thinking about the role of suicide risk prediction tools in clinical practice.