To date there is little research about how successfully trusts are implementing guidance from the National Institute for Health and Clinical Excellence (NICE), but what there is suggests that implementation is inconsistent (Reference Sheldon, Cullum and DawsonSheldon, 2004). A large amount of information on evidence-based implementation methods is published by NICE to support trusts with this task (National Institute for Health and Clinical Excellence, 2006). In order to determine the degree of progress and the key implementation processes used, the National Collaborating Centre for Mental Health (NCCMH) carried out a survey of mental health trusts, analysing data alongside findings from the Healthcare Commission audit of the implementation of the schizophrenia guideline.
Method
We used data from two sources to unpack progress with implementation: data from a bespoke questionnaire tool, and a meta-analysis using this alongside data from the Healthcare Commission audit of the implementation of the NICE schizophrenia guideline. The questionnaire items were derived from two sources, to answer two key questions: first, how far have trusts followed NICE's principles of implementation; and second, what evidence is there that trusts are implementing guidelines? Question one is answered by assessing the extent to which trusts gave positive responses to specific questionnaire items derived from the NICE principles. To answer the second question, expert advisors to the project have identified key implementation markers, which show that a trust is fully engaged with implementation. Questionnaire items were mapped to the principles summarised in Box 1. The forms were sent to all chief executives and medical, clinical and clinical governance directors at all National Health Service (NHS) mental health trusts (including primary care trusts providing mental health services) in England as well as combined trusts in Wales (a total of 488 questionnaires sent to 79 trusts).
Data from the recent audit of implementation of the schizophrenia guideline (Healthcare Commission, 2006), carried out as part of the Healthcare Commission joint review of community mental health services, were re-analysed alongside data from the present study. The audit identified a number of key implementation markers, and gathered data to assess adherence with these: availability and use of family interventions and cognitive–behavioural therapy (CBT), provision of information and occupational needs, physical health checks, prescribing (atypical antipsychotics, depot formulations, treatment-resistant schizophrenia), monitoring, medication and advance directives. A proxy measure of guideline implementation was created within the data-set by summing the trust-level scores for each of the indicators in the audit.
Data were analysed using SPSS version 14 for Windows. Correlations and one-way analyses of variance (ANOVAs) were used to identify significant univariate relationships; linear regression was used for multivariate effects. In order to examine fully the effect of corporate commitment and leadership, a new variable was constructed comprising identified executive lead, identified clinical director lead, clinical champion, team to support the clinical champion, and reports to trust board. The new variable showed an acceptable Cronbach's μ value of 0.68.
Results
A total of 209 usable forms were returned, an adjusted response rate of 47% (209/(488–44)). Data are available for the various professional groups sampled: 59% of forms were returned from chief executives, 42% from clinical directors, 47% from medical directors and 59% from directors of clinical governance. Data were returned from 69 of the 79 trusts sampled (87%), ranging from 0% to 100% of forms returned (mean 48%).
Box 1. The National Institute for Health and Clinical Excellence (NICE) six principles of implementation
-
1. Board support and clear leadership
-
2. Provision of a dedicated resource (a NICE manager)
-
3. Support from a multidisciplinary implementation team
-
4. A systematic approach to financial planning
-
5. A systematic approach to implementing guidance
-
6. A process to evaluate uptake and feedback
Descriptive results
Corporate and local leadership (mapped to NICE principle 1)
-
• 56% indicated that the trust had identified a clinical director-level lead for guideline implementation
-
• 46% indicated that the trust had identified a clinical guideline implementation champion; of the clinical champions, 57% had a team to support them
-
• 85% indicated that the trust had a committee to oversee implementation of guidelines
-
• 57% indicated that trust boards routinely received reports on guideline implementation.
Staffing and resources (mapped to NICE principles 2 and 3)
-
• 32% indicated that the trust had appointed staff specifically to implement NICE guidelines
-
• 42% indicated the trust had calculated likely resource implications for guidelines
-
• 38% indicated the trust had committed funds to implementation
-
• funds committed ranged from £10 000 to £1.2 million (mean £34 000).
Commissioner resourcing (mapped to NICE principle 4)
-
• 84% indicated that recent NHS funding problems had prevented (or would be likely to prevent) implementation of guidelines
-
• 71% rated the trust's support from their commissioning primary care trust in implementing health technology appraisals as poor, very poor or non-existent
-
• 82% rated the trust's support from their commissioning primary care trust in implementing guidelines as poor, very poor or non-existent.
Accessibility of guidelines (mapped to NICE principle 5)
-
• 70% indicated that NICE guidelines were easily available on the trust intranet
-
• 67% indicated that the trust also made guidelines accessible in the course of clinical practice in another way
-
• 86% had looked for implementation information on the NICE website.
Operational implementation (mapped to NICE principle 5)
The use of outcome measures is a proxy for implementation, because it is a precursor for appropriate recording of patient information and an indicator that diagnosis is being recorded. Respondents were asked if clinical teams routinely used outcome measures. Responses are shown in Fig. 1. Seventy per cent of respondents indicated that their trust used a computerised clinical information system for patient details.
Monitoring (mapped to NICE principle 6)
-
• 73% indicated that the trust had audited health technology appraisal implementation
-
• 84% indicated that the trust had audited guideline implementation.
Percentage of respondents indicating that clinical teams routinely use outcome measures.
Inferential results
Commissioner support of health technology appraisal implementation
One-way analysis of variance showed that a lower commissioning score is associated with:
-
• lower likelihood that a trust would have calculated the resource implications of implementation (F=2.464, d.f.=4, P=0.047)
-
• lower likelihood that resources had been committed (F=4.217, d.f.=4, P=0.003).
Correlation showed that a lower score for commissioning support is linked to a lower level of commitment and leadership (see below; R=0.180, P=0.048).
Commissioner support of guideline implementation
Our hypothesis was that, as with support for health technology appraisals, commissioning support of guideline implementation might exert an effect on other variables. One-way ANOVA showed that poorer commissioning support of guidelines was associated with:
-
• higher level of health technology appraisal audit (F=2.471, d.f.=4, P=0.046)
-
• poorer access to guidelines other than on the intranet (F=3.184, d.f.=4, P=0.005).
NHS funding problems
Our hypothesis was that funding problems might curtail implementation. This proved not to be the case for audit of health technology appraisals, where reported funding problems were associated with:
-
• higher likelihood of audit taking place (F=14.667, d.f.=1, P < 0.001);
-
• poorer level of access to guidelines other than on the trust intranet (F=7.375, d.f.=1, P=0.007).
Commitment and leadership
One-way ANOVA showed that better levels of commitment and leadership were associated with:
-
• staff appointed to support implementation (F=2.987, d.f.=12, P=0.001)
-
• resource implications calculated (F=2.153, d.f.=12, P=0.016)
-
• resources committed (F=3.025, d.f.=12, P=0.001)
-
• audit of health technology appraisals undertaken (F=2.702, d.f.=11, P=0.003)
-
• existence of a clinical information system (F=1.940, d.f.=12, P=0.032).
Correlation showed that better commitment and leadership is associated with more prevalent use of outcome measures (R=0.194, P=0.008).
Commissioner support, and corporate commitment and leadership
Correlation shows a positive relationship between commissioner support for health technology appraisals and commitment and leadership (R=0.180, n=181, P=0.016).
Analyses of the Healthcare Commission audit
A series of analyses were carried out to look at the relationship between previously identified ‘causal’ factors and implementation as measured by the Healthcare Commission audit of implementation of the schizophrenia guideline.
Descriptive results
Items within the audit were collapsed into a number of key markers showing adherence levels (Table 1).
Adherence with key markers from the Healthcare Commission audit
| Compliance, % | |||
|---|---|---|---|
| Marker | Minimum | Maximum | Mean (s.d.) |
| Depot used where preferred or adherence issue | 82.00 | 100.00 | 96.99 (4.10) |
| Responses/side-effects documented in care plan | 60.0 | 100.0 | 96.10 (5.71) |
| Polypharmacy not given | 67.0 | 99.0 | 92.20 (4.64) |
| Care plan has advance directive or continuity plan | 49.0 | 100.0 | 81.60 (10.63) |
| Physical health review | 9.0 | 97.9 | 80.98 (11.15) |
| Written material offered | 48.0 | 92.0 | 67.81 (8.66) |
| Occupational status assessed | 49.0 | 99.0 | 64.82 (11.46) |
| Clozapine prescribed for treatment-resistant schizophrenia | 30.8 | 90.3 | 61.28 (11.90) |
| Family therapy offered | 7.1 | 93.2 | 55.17 (17.30) |
| Cognitive—behavioural therapy offered | 20.0 | 96.0 | 46.86 (15.08) |
| Care plan has advance directive or continuity plan and choice of antipsychotic | 0 | 83.1 | 29.27 (13.10) |
Inferential results
In order to look at statistical relationships, the individual scores for the audit were combined to give an overall proxy measure for implementation. Correlations showed that better levels of implementation were associated with the following factors (data at trust level):
-
• better corporate commitment and leadership (R=0.342, n=57, P=0.009)
-
• existence of a committee to oversee implementation (R=0.429, n=57, P=0.001)
-
• commissioner support for health technology appraisal (R=0.459, n=57, P=0.000)
-
• less likelihood of guideline audit having been carried out (R=–0.270, n=57, P=0.042).
The data were submitted to a step-wise linear regression. All variables from the NCCMH survey were input as independent variables, the schizophrenia proxy as the dependent. The model yielded a reasonable R2 adj of 0.284 using three variables, model ANOVA significant (F=8.410, d.f.=3, P < 0.001), and collinearity statistics were within acceptable range (condition index=7.72). Variable (β) weights showing direction and strength of effect in the model were:
-
• commissioner support for TA, β=0.39
-
• corporate support, β=0.26
-
• identified clinical director lead, β=0.23.
This interaction between commissioner support and corporate support has a powerful predictive effect on implementation.
Discussion
Limitations
The response rate was below 50%, and thus the findings must be seen as illustrative rather than definitive. To present the results from the survey we used all individual responses, as opposed to overall trust responses. There is, in some cases, variation between responses from different individuals from the same trust (although analysis of variance showed no significant results). Since the study focused only on mental health trusts, findings are not generalisable to other healthcare settings; the survey also provides a snapshot only of the current situation. The use of a self-report method (for both the questionnaire and the schizophrenia audit) will inevitably bring in elements of bias.
NICE guidance implementation and key implementation indicators
The NICE principles of implementation (Box 1) are not being closely followed by many trusts. It is encouraging, however, that a majority of trusts seem to have some identified prerequisites for implementation in place, showing engagement with guideline implementation, although it is arguable that these might be seen as prerequisites for an effective clinical service, and that their existence owes more to this than any link to guideline implementation.
Markers from the schizophrenia audit
Some of the markers show high levels of adherence (use of depot medication, documentation of response and side-effects in care plan, avoidance of polypharmacy), whereas others are poorly adhered to (notably provision of CBT and family interventions). The high levels of adherence on the top five markers may be due to coincidence with established best practice rather than to change driven by guidance. The low levels of CBT and family interventions are possibly due to lack of availability and funding rather than commitment from clinical teams to implementation.
Key findings
Hypothesis testing shows a clear link between corporate commitment and leadership and implementation of guidelines. Corporate commitment is linked to many key implementation markers - where it is lacking, implementation will not be very far advanced. This hypothesis is clearly supported by the secondary analyses of the Healthcare Commission data on NICE schizophrenia guideline implementation. There is a clear link between corporate commitment and leadership and implementation of this specific guideline, from both the correlation and the regression analyses, the latter showing the interactive effect of commissioner support and commitment and leadership (lack of commissioner support is linked to poorer corporate commitment and leadership).
It is interesting to note that a higher level of audit of health technology appraisals is linked to poor commissioning support and funding problems. This may be a function of two possibly unrelated effects: it may be that commissioners are insisting that an audit of an appraisal is undertaken as a prerequisite of funding being allocated; it is also possible that this audit is being conducted as a lobbying activity to persuade commissioners to allocate funding (via a demonstration of need).
Implications
This study shows empirically that in order for implementation to succeed, the trust's board and senior managers must be engaged. Further, it shows that commissioners must demonstrate support. These findings give ammunition to trusts and to NICE to enable pressure to be put on commissioners in order to improve support of NICE guidance and of its implementation. Further, these data show that some trusts have advanced well in implementing guidelines, providing identifiable best-practice models for others to follow.
Declaration of interest
None. The study was funded by the National Collaborating Centre for Mental Health.
Open access
eLetters
No eLetters have been published for this article.