BACKGROUND

Risk assessment and management has become an explicit part of mental health care delivery (Department of Health, 2000; 2007; Morgan Reference Morgan2007) as in many other aspects of corporate and public sector activity (Power, Reference Power2004). Risk in this context usually means a composite measure of probability and severity of possible harm that might occur to oneself, to others or to an organisation as a result of a particular series of events (Undrill, Reference Undrill2007). Harm can result from many possible causes. In mental health practice, the risk assessment process usually relates to the possibility that a patient may commit suicide or cause serious harm to another person. The process of risk assessment has been formalised in the forensic services by the development of specific risk assessment tools such as the Hare Psychopathy checklist (Hare, Reference Hare1991) and the HCR-20 (Webster et al. Reference Webster, Douglas, Eaves and Hart1997). These assessment tools offer improved prediction of violence when used properly on appropriate populations but they are complicated to administer and have poor predictive value both for acute violence in hospital (Grevatt et al. Reference Grevatt, Thomas-Peter and Hughes2004) and when applied to low risk populations (Buchanan, Reference Buchanan2008).

Clinical risk assessment is also influenced by a secondary process in which decision makers seek to protect themselves or their organisation from potential criticism in the event of a poor decision (Power, Reference Power2004). Important decisions such as whether to admit someone to hospital, to detain them under the Mental Health Act and whether they need treatment in secure conditions are usually driven by the perception of both primary and secondary risk.

To this end many mental health trusts have introduced their own risk assessment tools. These vary greatly in content and complexity (Higgins et al. Reference Higgins, Watts, Bindman, Slade and Thornicroft2005). Often there is no evidence that a particular tool is valid or even that there has been any attempt at a validation process (Doyle & Dolan, Reference Doyle and Dolan2002). Many are simple tick box exercises with no guidance on the interpretation of results. Some of these tools are probably useless, some may even cause harm, either by encouraging a false sense of security or by prompting unnecessary restriction (Undrill, Reference Undrill2007). On the other hand the completion of a risk assessment tool may have the benefit of focussing the clinician’s thinking onto risk issues (Undrill, Reference Undrill2007).

It is generally agreed that a comprehensive assessment of risk should take elements from actuarial assessment of risk (a process which seeks to assign a mathematical figure derived from research, to the increased risk associated with particular socio-demographic groups and behaviour patterns) augmented by assessment of dynamic factors such as the presence of environmental triggers, active mental illness and substance use (Buchanan, Reference Buchanan2008). These dynamic variables are particularly important in the psychiatric intensive care unit (PICU), which treats patients identified as being at acute risk of harm as a result of mental illness and whose mental state can change quickly (Brown et al. Reference Brown, Chhina and Dye2008). Most of the risk management decisions in a PICU take place in a well-supervised environment and relate to issues such as changes in observation level, leave and transfer to an acute ward, decisions which are unlikely to be facilitated by long term risk assessment tools. In this context, the Brøset violence checklist identifies six dynamic variables (confusion, irritability, boisterousness, verbal threats, physical threats and attacks on objects) as predictors of violence by patients in the 24 hours following assessment (Woods & Almvik, Reference Woods and Almvik2002).

This paper describes a validation study of a risk assessment tool that was introduced in the Southampton PICU in 2006. The tool was designed by the ward team based on contemporary risk assessment literature (Monahan et al. Reference Monahan, Steadman, Robbins, Silver, Appelbaum, Grisso, Mulvey and Roth2000; Qin et al. Reference Qin, Agerbo and Mortensen2003) and was designed to predict violent behaviour and serious self harm in the two weeks following assessment and to demonstrate that stringent risk assessment was being undertaken. It produces separate scores for risk of suicide and violence to others, based on actuarial risk factors, historical and recent behaviour. We suspected that staff compliance might be poor if the tool was difficult to complete and selected risk factors which were objective and likely to be known to staff even in an uncommunicative patient.

METHOD

Risk assessment forms (see Appendix Fig. A1) were completed prior to the weekly ward round for all patients in the PICU. This was done where possible by the patient's primary nurse and by the nurse in charge of the ward in their absence. The forms were designed to give a risk score in each of three categories (actuarial, historical and recent behaviour) of variables derived from literature review. A simple scoring scheme scoring scheme produced a 10 point score for actuarial risk factors, a 22 point score for historical risk and a thirty five point score for recent risky behaviour. These individual scores were then added to give a total score out of 67 in each of two domains: suicide and assault.

Hospital incident forms were collected in respect of all PICU patients for the two weeks following the risk assessment, including those relating to patients who had moved from the PICU during this period. Each form was coded as positive or negative for violent and for suicidal behaviour, by a researcher who did not work on the PICU and who was blind to the risk scores. The study protocol was reviewed by the Trust director of research and development who decided that it represented service evaluation and did not need ethical committee approval.

STATISTICAL ANALYSIS

The principal analysis used multivariate regression to measure the association between preselected predictor variables (such as drug use, gender and recent violence) and outcome measures (in this case violent behaviour or self harm). The variables were first tested in univariate analysis. Those which showed a significant association were then entered into a multivariate analysis.

A secondary analysis used a ROC curve analysis which plots sensitivity against 1-specificity and measures the area under the resulting curve. In this analysis a large area under the curve represents good performance (an area under the curve of 1.0 represents perfect prediction, an area of 0.5 random results). An effective instrument requires both high sensitivity (proportion of positive incidents correctly predicted by the instrument) and high specificity (proportion of negative incidents – in this case patients who were not violent – correctly predicted by the instrument). The results were analysed using the SPSS statistical programme (SPSS, 2009).

RESULTS

Risk assessment forms were completed on 158 consecutive PICU patients over a three month period. Incident forms identified 18 patients who committed 25 acts of actual physical violence in the two weeks after a risk assessment. Most of the violence was minor; only two incidents scored higher than two on the Trust's four point severity scale.

In univariate analysis recent and historical violent behaviour were positively associated and drug use negatively associated with violent behaviour in the PICU. There were no other significant associations (Table 1). In multivariate analysis (Table 2) a one point increase in the risk score for recent violent behaviour was associated with a 14% increase in risk of violence in the PICU (OR 1.14, 95% CI: 1.05–1.24). Drug use was negatively associated with violent behaviour (OR 0.34, 95%CI: 0.12–0.95).

Table 1.

Univariate logistic regression analysis of variables potentially associated with an episode of violence in the subsequent two weeks

Table 2.

Multivariate logistic regression analysis of variables associated with violence in univariate analysis in the subsequent two weeks

The area under the ROC curve of sensitivity/1-specificity was 0.75 (95%CI: 0.65–0.85), in respect of recent violent behaviour (Fig. 1). The area under the curve increased to 0.82 (95% CI: 0.75–0.90) when the score for demographic variables was added and to 0.83 (95% CI: 0.75–0.90) when the historical violence score was also added – changes that were not statistically significant.

Figure 1.

ROC curve of recent violence score as predictor of violence within next two weeks in PICU patients Area under curve: 0.75 (95% CI: 0.65–0.85)

There were two episodes of very minor self harm during the study period, too few for meaningful analysis.

DISCUSSION

This study found that recent violent behaviour was a significant predictor of violence and drug use a negative predictor for violent behaviour by patients detained in a PICU. No other variables were significant predictors of violence. These results suggest that a complicated risk assessment tool incorporating actuarial and historical data adds very little to the process of predicting violent behaviour in PICU patients. The absence of association with variables known to predict violent behaviour in the general population shows the importance of tailoring a screening assessment to the exact circumstances in which it is used. Evidence derived from population studies, where a small minority of people exhibit violent behaviour, may be unhelpful in assessing a very highly selected group such as PICU patients most of whom have shown significantly aggressive behaviour before admission (Brown et al. Reference Brown, Chhina and Dye2008). Similarly someone who has been violent in the context of drug use may be at relatively low risk of violence when in a drug free environment.

STRENGTHS AND WEAKNESSES

This study was done on a ten-bed English PICU, designed and operated in line with Department of Health (2002) recommendations and with a typical patient group (Brown et al. Reference Brown, Chhina and Dye2008). The data were collected from a complete series of consecutive risk assessments (sometimes of the same individuals if they were still on the PICU). The forms were completed as part of standard ward activity and should have been completed professionally. There is a degree of subjectivity in the decision whether or not to complete an incident form but the procedure is guided by Trust protocols and it is likely that they would be completed for violent incidents. Data about recent behaviour were probably of higher quality than data on historical behaviours as the latter were often collected without access to old case records or informant history. A ROC curve score of 0.75 suggests an effective instrument.

The major weakness was the relatively small sample size which leaves open the possibility that the presence of atypical patients may have skewed the results. The use of the risk assessment tool may also have influenced clinical decisions and altered the rate of violent behaviour (Doyle & Dolan, Reference Doyle and Dolan2002). Furthermore the study reports a locally developed risk assessment tool and it is arguable that another tool might perform better. Specifically we acknowledge that the need to produce a user friendly tool led to a scoring scheme that did not accurately score the excess risk associated with a particular variable. Nevertheless while you may disagree with particular details of this risk assessment form, it is worth pondering whether the tools used in your organisation are any better.

IMPLICATIONS FOR TREATMENT

There has been much argument about the validity of structured risk assessment tools in the management of mental illness (Buchanan, Reference Buchanan2008) but it is clear that they are ‘here to stay’ (Undrill, Reference Undrill2007). The most effective way of reducing risk is to perform the core activity (in this case providing effective treatment of mental illness) to a high standard (Undrill, Reference Undrill2007). In patient-centred practice, risk should be considered primarily in terms of risk to the patient and risk assessment and management undertaken as a consensual process in a manner which addresses the patient’s underlying anxieties (Undrill, Reference Undrill2007). This should be a guiding principle even though the process may be difficult in a seriously disturbed patient.

There is reasonable evidence that a structured risk assessment tool (Woods & Almvik, Reference Woods and Almvik2002) can reduce the incidence of violence and of coercive interventions in acute psychiatric wards (Abderhalden et al. Reference Abderhalden, Needham, Dassen, Halfens, Haug and Fischer2008), even if the confidence intervals on many tools are too wide to usefully predict individual aggression (Hart et al. Reference Hart, Michie and Cooke2007). The use of a risk assessment tool may also help to focus staff on the process of risk assessment and thereby facilitate production of an effective risk management plan (Carroll, Reference Carroll2007). It probably also reduces anxiety about the possibility of harm to the organisation or personal career as a result of a poor decision. Organisations should however consider both that poor quality tools may not improve risk management and that excessive paperwork may reduce the quality of staff-patient interactions.

In this context it is instructive that such important changes in clinical practice have been introduced without the evidence base that would be required for a new drug or specific therapy. We suggest that interventions such as risk assessment tools should be subject to the same validation process that would be required for any other clinical initiatives and that validated tools, such as the HCR-20, should only be used in the environment and for the purpose for which they were validated. Clinicians should resist the introduction of tools that cannot demonstrate adherence to these criteria.

CONCLUSION

Risk assessment needs to be performed and understood as an important component of clinical risk management rather than an end in itself. The best risk management probably comes from a combination of actuarial and clinical indices. The assessment of these variables may be helped by a risk assessment tool but any such tool must be properly validated for use in the particular environment and patient group. A dysfunctional risk assessment tool has the potential to worsen risk management. Mental health patients and services would probably also benefit from a culture which demanded that service changes, such as the introduction of risk assessment tools, be subject to proper piloting and evaluation.