Strengths and limitations

Although we have demonstrated that SAFE is a useable and reliable measure, our study has a number of limitations. First, the candidate-item selection process was not systematic. Instead we conducted a selective but focused review of the implementation science literature. It is possible that a wider systematic review would have identified additional implementation barriers and facilitators in relation to characteristics of the intervention. Further to this, the review was restricted to mental health services within the NHS. Although this may limit the tool’s applicability to other healthcare settings, a number of systematic reviews have identified similar implementation barriers and facilitators in other settings (such as in the USA) and for other long-term health conditions. ^{Reference Cabana, Rand, Powe, Wu, Wilson and Abboud17} Furthermore, a number of included reviews assessed the implementation literature on a broader scale. Specifically, for a review to be included in the thematic analysis, it needed to present data that was applicable, but not restricted, to the UK.

A second limitation was the small-scale pilot and psychometric evaluation. Twenty interventions were included in the psychometric evaluation. These were rated by up to four different reviewers, with one reviewer rating each paper a week later to assess test-retest reliability. Although the number of studies was limited, the papers included in the evaluation covered a broad range of interventions (including many featured within NICE clinical guidance). The focus of the psychometric evaluation mirrored the areas important to a systematic review used for evidence appraisal. For example, within good-quality systematic reviews, multiple reviewers will rate included papers (interrater reliability), with the aim of systematic reviews to be reproducible across time (test-retest reliability). The psychometric properties evaluated in this study were selected to reflect these features. Future work could look at evaluating the use of SAFE within an evidence review procedure such as a health technology appraisal or guideline development process.

Finally, the methods used to develop the reporting guidelines were limited in their scope. Moher and colleagues suggest a method for developing reporting guidelines that includes a review of the literature followed by a Delphi exercise and face-to-face consensus meeting. ^{Reference Moher, Schulz, Simera and Altman26} As the reporting guidelines in this study focus specifically on allowing the rating of SAFE within evidence appraisal and decision-making processes, a more pragmatic approach to the development process was undertaken, in that each item in SAFE was constructed as an item in the reporting guidance. Future work could look at expanding these reporting guidelines to include other areas outside of mental health services and implementation features in addition to the characteristics of the intervention.

Despite these limitations, one strength of the study was that the psychometric evaluation indicated that SAFE is useable and reliable. The ease of use of SAFE suggests it could be easily appended to current evidence review processes across a range of different contexts. The associated reporting guidelines also have the potential to have a positive impact on the quality of interventions reported in peer-reviewed journals, thus providing systematic reviewers and policy makers with the information needed to evaluate likely implementation.

Comparison with the literature

Over the past decade implementation science has become a rapidly evolving area of interest with research attention turning to the implementation and sustainability of programmes and innovations within routine clinical care. ^{Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale27} Within their review of the literature, Wiltsey Stirman and colleagues ^{Reference Wiltsey Stirman, Kimberly, Cook, Calloway, Castro and Charns28} identified 125 studies investigating sustainability, including 20 studies within the mental health domain. They found that innovation characteristics including fit with current practice, ability for the innovation to be modified and effectiveness were important influences on the sustainability of the innovation being assessed in the individual studies. Furthermore, features such as resources, working culture and training and education requirements also had an impact and match items included in the SAFE scale.

Although SAFE is a novel tool for assessing the feasibility of an intervention at the evidence review stage, other attempts have been made to assess and characterise the barriers to routine translation of evidence into practice. In their review of implementation measures, Chaudoir and colleagues identified 62 available measures assessing different aspects of implementation. ^{Reference Chaudoir, Dugan and Barr8} None of the identified measures specifically focused on the characteristics of an intervention associated with feasibility, instead the measures were either restricted to evaluations of specific interventions, focused on guideline implementation or including assessment of the innovation alongside other areas such as staff attitudes, political context, organisation factors, all of which would not be possible to assess at the evidence appraisal phase. Furthermore, unlike SAFE, which has demonstrable interrater and test-retest reliability, the majority of measures in the review were not psychometrically evaluated. ^{Reference Chaudoir, Dugan and Barr8} Although not included in the Chaudoir et al review, ^{Reference Chaudoir, Dugan and Barr8} the NHS Institute for Innovation and Improvement has recently developed the Spread and Adoption tool, which aims to help staff increase the sustainable implementation of innovations within the NHS. ²⁹ This online-based tool asks individuals to rate their agreement with a number of statements grouped into three categories: people, innovation and context. Although providing a summary assessment, the tool does not specifically focus on rating the feasibility of the intervention and instead covers a broader range of contextual factors; furthermore, it lacks a clear empirical basis. Finally, Slaghuis and colleagues ^{Reference Slaghuis, Strating, Bal and Nieboer30} have also developed a framework and instrument to measure the sustainability of new work practices being implemented in long-term care. They identify ‘routinisation’ and ‘institutionalisation’ as the two elements of sustainability. However, as with the measures included in the Chaudoir et al review, ^{Reference Chaudoir, Dugan and Barr8} the framework and measure are designed to evaluate practices within clinical use, rather than at the evidence review stage. By contrast, SAFE assesses individual intervention papers during the policy-making process.

Relevance for practice and policy

To support implementation in clinical practice, an understanding of the factors that facilitate or hinder successful evidence utilisation is required. At present, healthcare improvements have often been targeted at factors related to individual healthcare practitioners, such as their knowledge, routine and attitudes. ^{Reference Grimshaw, Thomas, MacLennan, Fraser, Ramsay and Vale27,Reference Lawrence, Fossey, Ballard, Moniz-Cook and Murray31} However, successful implementation is influenced by components occurring at multiple ecological levels of the healthcare system, such as the individual, social, organisation, economic and political context and patient beliefs and behaviour. ^{Reference Damschroder, Aron, Keith, Kirsh, Alexander and Lowery7,Reference Grol, Wensing and Eccles32-Reference Sanders and Haines34} Implementation is a complex social process linked with the context in which it takes place.

The SAFE scale specifically focuses on one factor indentified as important to successful implementation, namely the characteristics of the intervention. Within this complex process of implementation, rating feasibility based on the characteristics of that intervention offers a circumscribed and useable source of information for both reviewers and policy makers when making decisions about evidence recommendations. Guideline development processes make use of systematic reviews of best available evidence as part of the decision-making process, alongside other rating systems such as GRADE (Grading of Recommendations Assessment, Development and Evaluation), which makes statements about the overall quality of the evidence. Recently, there have been further suggestions that the GRADE process should incorporate other features of the evidence and intervention including resource allocation. ^{Reference Guyatt, Oxman, Kunz, Jaeschke, Helfand and Liberati35} It is at this stage in the evidence review process that SAFE could be used to help clinicians and guideline panellists with the decision-making process.

A number of papers have focused on the implementation of NICE clinical guidelines for mental health conditions. Despite a range of initiatives, implementation within routine care, particularly of psychological therapies and interventions focusing on physical healthcare, has remained low. ^{Reference Berry and Haddock14,Reference Gagliardi36,Reference Jolley, Onwumere, Kuipers, Craig, Moriarty and Garety37} For instance, uptake of both family intervention and cognitive-behavioural therapy (CBT) for psychosis has been low, with estimates suggesting that less than 30% of eligible patients receive these interventions. ^{Reference Prytys, Garety, Jolley, Onwumere and Craig38} These findings are not restricted to schizophrenia - Rhodes and colleagues ^{Reference Rhodes, Genders, Owen, O'Hanlon and Brown39} found that although the majority of clinicians were aware of and using NICE clinical guidance for depression, only 20% felt confident in their use of the guidelines. Many clinicians stated that resource implications, lack of time and availability of training had a negative impact on their routine utilisation within clinical practice. Using SAFE within the evidence review process could help to highlight areas of interventions that make their implementation more difficult. This would allow for the strategic targeting of resources and the tailoring of implementation strategies at an early stage in the dissemination process to overcome these issues and hence maximise routine implementation. As well as the clinical gains, the cost savings arising from higher levels of implementation are potentially significant. For example, Vos and colleagues ^{Reference Vos, Haby, Magnus, Mihalopoulos, Andrews and Carter40} indicated that if recommended treatments that are currently underutilised, such as CBT for depression and anxiety and family interventions for schizophrenia, were implemented then significant cost savings would be made, in addition to improvements in the health status of individuals.

The second aim of the paper was to produce a checklist for authors to use when reporting interventions. The pilot study indicated that a number of areas are at present poorly reported in both trial protocols and in randomised controlled trial publications. For instance, despite economic costs and staff time constraints being identified as two main barriers to implementation, few trial publications and protocols reported details of these areas. One way to improve the consistency of reporting within journals is the use of reporting guidelines. Hopewell and colleagues ^{Reference Hopewell, Ravaud, Baron and Boutron9} have recently demonstrated that the implementation of CONSORT has led to improvements in the abstracts of articles published in a number of high-quality medical journals. Although the SAFE reporting guidelines have not been developed using a formal framework, ^{Reference Moher, Schulz, Simera and Altman26} they are empirically supported and will support improved characterisation of feasibility.

Future research

Given the interest in implementation science and the increasing evidence to suggest low implementation of evidence within clinical practice, it is imperative that future work continues to assess not only the barriers to implementation but how these can be overcome. The results presented here represent a pilot study and small psychometric evaluation of a new measure and reporting guideline. Larger-scale work is needed to assess the utility of SAFE within systematic reviews such as those used within the guideline development process. Additionally, work could focus on adapting and modifying SAFE so that it is applicable to other areas of healthcare and other non-UK settings. In particular, implementation influences may differ across settings, and the degree of commonality is unknown - future research using the same methodology with different clinical populations and service settings will be needed to establish whether the same influences, and hence SAFE, apply.

Implications

The SAFE scale represents a novel approach to assessing the feasibility of different interventions. It has the potential to be used alongside efficacy and health economic evidence to assist commissioners, policy makers and guideline developers with their decision-making processes. This comes at a time when mental health services worldwide are faced with increasingly difficult decisions regarding resource allocation and implementation priorities. Furthermore, the identification of reporting guidelines for feasibility provides a mechanism for standardising the reporting of this aspect of interventions within high-quality peer-reviewed publications.