Introduction
The National Institute for Health and Clinical Excellence (NICE) recommends the use of psychological therapies as first-line treatments for mild-to-moderate mental health difficulties (National Institute for Health and Clinical Excellence (NICE), 2009, 2011a ). In the Republic of Ireland, the lack of access to psychological therapies often results in general practitioners (GPs) relying on psychotropic medication to manage such difficulties (Grace et al. Reference Grace, Rowe, Saunders and Cullen2012). This is not only inconsistent with best practice, but also with service users’ preferences for talking therapies (Baumeister, Reference Baumeister2012; McHugh et al. Reference McHugh, Whitton, Peckham, Welge and Otto2013 c). Furthermore, there may be a greater reliance on secondary care mental health services, resulting in more inappropriate referrals to these services (McHugh et al. Reference McHugh and Byrne2013b ). Thus, there is clear need to increase access to psychological therapies in primary care, while also ensuring that such therapies are high throughput, cost-efficient and clinically effective (Kazdin & Blase, Reference Kazdin and Blase2011; Fairburn & Wilson, Reference Fairburn and Wilson2013).
The Roscommon Primary Care Adult Psychology Service was established in 2009 to meet the need for psychological interventions in primary care (Hughes et al. Reference Hughes, Byrne and Synnott2010; Kierans & Byrne, Reference Kierans and Byrne2010). Similar to the stepped care structure of STEPS in Glasgow (White et al. Reference White, Joice, Petrie, Johnston, Gilroy, Hutton and Hynes2008) and Improving Access to Psychological Therapies (IAPT) in the United Kingdom (Clark, Reference Clark2011; O’Shea & Byrne, Reference O’Shea and Byrne2014), this service aimed to provide accessible, low-intensity psychological interventions for adults with mild-to-moderate mental health difficulties. Brief cognitive behavioural therapy (CBT) was the main intervention offered by the service, in addition to some basic psychoeducation (e.g. bibliotherapy). Previous evaluations of the service demonstrated the clinical effectiveness of brief CBT, although the average waiting time of 2–3 months did not meet service users’ need for early access to treatment (Bourke & Byrne, Reference Bourke and Byrne2012; McHugh et al. Reference McHugh, Brennan, Galligan, McGonagle and Byrne2013 a; McHugh et al. Reference McHugh, Gordon and Byrne2014).
With pilot funding from the Office of the Assistant National Director of Mental Health, the Roscommon Primary Care Adult Psychology Service was developed into Access to Psychological Services Ireland (APSI) in 2013 (see Fig. 1 for service structure). APSI had a number of significant developments from the original service, based on recommendations from previous service evaluations. In particular, the service aimed to maximise accessibility by offering a next-day assessment to all new referrals and an immediate screening to those attending the service’s ‘Walk-in Clinic’. In seeking to increase throughput, greater availability to the low-intensity, evidenced-based intervention of guided self-help was provided (Cuijpers et al. Reference Cuijpers, Donker, van Straten, Li and Andersson2010; Haug et al. Reference Haug, Nordgreen, Ost and Havik2012). Lastly, the service sought to work more closely with existing community services. This included coordinating a shared referral protocol with Counselling in Primary Care (CIPC), as well as organising concurrent treatment of service users with secondary care mental health services.
Structure of Access to Psychological Services Ireland (APSI) service and coordination with other community services. CBT, cognitive behavioural therapy; GP, general practitioner.
The current research presents a service evaluation of APSI in its first year of functioning. The clinical effectiveness of the service was evaluated by examining service user outcomes from the APSI interventions. The cost-effectiveness of the service was evaluated by examining medical cost-offset and employment facilitation associated with treatment.
Method
Evaluation design
A repeated measures design was used to evaluate service users’ clinical, health and economic outcomes. Psychological distress as measured by the Clinical Outcomes in Routine Evaluation–Outcome Measure (CORE-OM) was compared at assessment, post-therapy and 3-month follow-up. Depressive symptomatology as measured by the Patient Health Questionnaire-9 (PHQ-9), anxiety symptomatology as measured by the Generalised Anxiety Disorder-7 (GAD-7) and health status as measured by the EQ-5D-3L were compared at assessment and post-therapy. Service users’ economic outcomes as measured by the Eco-Psy were compared at assessment and 3-month follow-up.
Service users
Those suitable for the service were adults (>18 years) with mild-to-moderate mental health difficulties. Exclusion criteria included the presence of a severe or complex mental health difficulty (e.g. schizophrenia, bipolar disorder); personality disorder; significant cognitive impairment; substance abuse disorder; and active risk of suicide, violence or self-neglect. These exclusion criteria were screened for within the referral information and during the initial assessment session.
A total of 198 service users completed at least one intervention with the service. These service users consisted of 55 (27.8%) males and 143 (72.2%) females with a mean age of 39.7 years (s.d.=13.2). Concurrent attendance of secondary care mental health services was indicated by 32.3% (n=64) at initial assessment. Suicidal ideation (with no intent) was present for 19.7% (n=39). Table 1 presents the primary mental health difficulty of treatment completers at initial assessment. Information from the initial assessment interview and psychometric measures were used to make a provisional diagnosis of an anxiety or depressive disorder based on the criteria of the Diagnostic and Statistical Manual of Mental Disorders (5th edn; DSM-5; American Psychiatric Association, 2013). Service users presenting with mild levels of anxiety or depression were classified as having subthreshold anxiety and subthreshold depression, respectively. A secondary or tertiary mental health difficulty was present for 35.4% (n=70) of treatment completers.
Primary mental health difficulty of treatment completers at assessment
Interventions
Guided self-help treatment was delivered over a 6-week period, consisting of an initial face-to-face session, two mid-treatment telephone sessions and a final face-to-face session. At the first session, service users were provided with psychoeducational materials relevant to their mental health difficulty, with subsequent sessions providing support and advice in utilising these materials. The psychoeducational resources provided to service users were acquired from freely available National Health Service online resources.
Psychoeducational groups consisted of five 90-minute sessions conducted on consecutive weeks. The content of the groups focussed on developing positive coping strategies based on CBT models. A psychoeducational focus was retained and service users were not required to talk about their personal difficulties. As the group intervention was established towards the end of the service year, only two groups were delivered during the evaluation time period.
One-to-one brief CBT consisted of six sessions delivered on consecutive weeks. Common treatment strategies included the use of thought diaries, cognitive restructuring, behavioural activation and behavioural experiments (Beck, Reference Beck2011). Exposure therapy was also commonly used for managing anxiety disorders.
The intervention initially offered to service users depended on the severity of their mental health difficulties at assessment, consistent with NICE guidelines (NICE, 2009, 2011a ). Those scoring in the mild or moderate range on the CORE-OM were offered guided self-help, with a subsequent option of completing psychoeducational groups. If the service user showed no clinical improvement from these interventions, brief CBT was offered. For those service users scoring in the moderate-to-severe range on the CORE-OM, but whose presentation was not severe enough for secondary care mental health treatment, brief CBT was initially offered.
Practitioners
The service was delivered by six mental health practitioners who were based in each of the six primary care team areas in county Roscommon (population of ~64 000). Three of the practitioners had a higher diploma in psychology, two had a master’s degree in applied psychology and one had a research PhD in psychology. Before delivering treatment, practitioners attended training workshops on the assessment and treatment of mental health difficulties in primary care, including workshops on delivering guided self-help, psychoeducational groups and brief CBT. This training was delivered by a senior clinical psychologist who also provided weekly group supervision.
Measures
CORE-OM
The CORE-OM is used to measure the clinical outcomes of a mental health intervention by assessing changes in the level of psychological distress of service users (Barkham et al. Reference Barkham, Margison, Leach, Lucock, Mellor-Clark, Evans, Benson, Connell, Audin and McGrath2001). This self-report 34-item questionnaire assesses the four domains of subjective well-being; problems/symptoms; life/social functioning; and risk. The CORE-OM shows good reliability (α=0.94) and validity, and can be used in a wide range of mental health settings (Evans et al. Reference Evans, Connell, Barkham, Margison, McGrath, Mellor-Clark and Audin2002).
PHQ-9
The PHQ-9 is a self-report measure used to assess depressive symptomatology based on the nine DSM-5 symptoms of a major depressive disorder. This measure has been shown to have high reliability (α=0.89), sensitivity (88–92%) and specificity (78–88%; Kroenke et al. Reference Kroenke, Spitzer and Williams2001; Gilbody et al. Reference Gilbody, Richards and Barkham2007; Furukawa, Reference Furukawa2010). In addition to acting as a screen for major depressive disorder, the PHQ-9 can be used as a continuous measure of depression.
GAD-7
The GAD-7 is a self-report measure used to assess the seven symptoms of a DSM-5 diagnosis of generalised anxiety disorder (GAD; Spitzer et al. Reference Spitzer, Kroenke, Williams and Lowe2006). Although the measure was initially used as a screen for GAD, it has been shown to have good screening properties for panic disorder, social anxiety disorder and post-traumatic stress disorder, with area under the curve (AUC) values in the range of 0.91–0.83 (Kroenke et al. Reference Kroenke, Spitzer, Williams, Monahan and Lowe2007). As such, the GAD-7 may be used as a measure of non-specific anxiety and was used for this purpose within the current study.
Eco-Psy
The Eco-Psy is a self-report measure designed to assess economic outcomes associated with mental health treatment (Twomey et al. Reference Twomey, Byrne and McHugh2013). It consists of 12 open- and closed-ended items that examine healthcare utilisation, employment status and work productivity. This measure is in the process of being validated.
EQ-5D-3L
The EQ-5D-3L is a widely used self-report measure of overall health (Rabin & Charro, Reference Rabin and Charro2001). It consists of the five dimensions of mobility, self-care, usual activities, pain/discomfort and anxiety/depression. Each dimension has three answer categories representing three levels of functioning (i.e. no problem, some/moderate problems, extreme problems).
Data analysis
Repeated measures t-tests were used to compare service users’ scores on the clinical and health measures from pre- to post-therapy. In evaluating the effect of concurrent secondary care mental health treatment on clinical outcomes, a mixed factorial analysis of variance (ANOVA) was used. Secondary care mental health treatment was inputted as the between-subjects variable with time inputted as the within-subjects variable. With regard the Eco-Psy data, a Wilcoxon signed-rank test was used to analyse the non-normally distributed healthcare appointment data from assessment to follow-up. McNemar tests were used to compare the proportion of services users taking medication or in employment at assessment and follow-up.
The recovery rate calculated was based on the proportion of treatment completers achieving reliable and clinically significant change on the CORE-OM (Jacobson & Truax, Reference Jacobson and Truax1991). With this criteria, a service user was considered to be recovered if their CORE-OM score moved below the clinical cut-off score of 10 and this reduction was of five or more points (Connell et al. Reference Connell, Barkham, Stiles, Twigg, Singleton, Evans and Miles2007). Those service users that were already below the clinical threshold on the CORE-OM at assessment were excluded from this analysis.
Results
Pathway of referrals
A total of 714 referrals were made to APSI during the period of August 2013 to July 2014, with GPs accounting for the majority of referrals (64.8%, n=463), followed by secondary care mental health (14.8%, n=106) and self-referral (11.9%, n=85). An initial assessment appointment was attended by 73.8% (n=527) of referrals, with 10.2% (n=54) of these accepting the offer of a same-day or next-working-day appointment. The median time between referral receipt and assessment was 9 days.
Following assessment, 8.2% (n=43) of attendees were deemed inappropriate for primary care treatment and 19.5% (n=103) had a preference for a counselling intervention with CIPC (see Fig. 2: both APSI and CIPC referrals were assessed by APSI). The remaining 381 service users were assessed as suitable for APSI, with 52% (n=198) of these completing an intervention with the service. Treatment completion was defined as attending all the planned sessions of the intervention. Almost a quarter of service users (24.1%, n=92) dropped out after their initial assessment; of those who started guided self-help, 39.7% (n=73) dropped out, with a corresponding dropout rate of 21.6% (n=16) for brief CBT.
Pathway of service users through Access to Psychological Services Ireland (APSI). CBT, cognitive behavioural therapy; CIPC, Counselling in Primary Care.
Interventions completed
Of the 198 individuals that completed an intervention, 69.7% (n=138) completed guided self-help, 41.9% (n=83) completed brief CBT and 5.8% (n=8) completed psychoeducational groups (see Fig. 2). A majority completed only one intervention (85.9%, n=170), 12.6% (n=25) completed two interventions and three service users (1.5%) completed all three interventions. The most common combination of treatments was guided self-help followed by brief CBT (53.6%, n=15).
Clinical outcomes
Complete data on all three clinical measures (i.e. CORE-OM, PHQ-9, GAD-7) were available for 85.8% (n=168) of treatment completers. Table 2 presents the mean CORE-OM scores at pre- and post-therapy for those completing guided self-help only (n=95), brief CBT only (n=48) and multiple interventions (n=25). Repeated measures t-tests showed a statistically significant reduction in CORE-OM scores for all groups. A small number of service users had complete CORE-OM data at 3-month follow-up (n=34) and showed a statistically significant reduction from pre-therapy (M=14.9, s.d.=6.8) to follow-up (M=7.3, s.d.=7.9), t (33)=4.8, p<0.001, d=1.0.
Comparison of mean pre- and post-therapy Clinical Outcomes in Routine Evaluation–Outcome Measure (CORE-OM) scores for each intervention
CBT, cognitive behavioural therapy.
Service users’ GAD-7 and PHQ-9 scores were analysed to determine changes in anxiety and depressive symptomatology, respectively. Comparing service users mean GAD-7 scores at pre-therapy (M=11.3, s.d.=5.6) and post-therapy (M=4.6, s.d.=4.3), a repeated measures t-test showed a statistically significant reduction, t (167)=16.0, p<0.001, d=1.3. Comparing service users mean PHQ-9 scores at pre-therapy (M=11.4, s.d.=6.1) and post-therapy (M=4.9, s.d.=4.8), a repeated measures t-test also showed a statistically significant reduction, t (167)=14.9, p<0.001, d=1.2.
Clinical recovery was assessed for the 137 services users scoring in the clinical range on the CORE-OM at assessment. Clinical recovery was observed for 67.9% (n=93) of these service users who showed reliable and clinically significant change on the CORE-OM. A reliable reduction in CORE-OM scores was observed for 81% (n=111). Comparing the outcomes across interventions, a recovery rate of 69.3% (n=52) was observed for those completing guided self-help only, 59.5% (n=25) for those completing brief CBT only and 80% (n=16) for those completing multiple interventions.
A mixed factorial ANOVA was used to compare the change in CORE-OM scores of service users concurrently engaged with secondary care mental health services (33.9%; n=57) and those engaged with APSI treatment only (66.1%; n=111). The main effect of treatment was significant, F (1,166)=285.53, p<0.001, η 2=0.632. The interaction of treatment and concurrent secondary care attendance was non-significant, F (1,166)=0.1, p=0.752, η 2=0.001.
Health and economic outcomes
The health outcomes of service users were measured using the EQ-5D-3L, with 108 service users having complete data on this measure. Fig. 3 presents the proportion of service users indicating a problem (i.e. some/moderate problem or severe problem) in each of the five categories of the EQ-5D-3L at pre- and post-therapy. Using UK normative data for the EQ-5D-3L (Szende et al. Reference Szende, Oppe and Devlin2007), the health utility scores of service users were calculated and compared at pre- and post-therapy using a repeated measures t-test. A statistically significant increase in service users’ mean scores were observed from pre-therapy (M=0.61, s.d.=0.20) to post-therapy (M=0.80, s.d.=0.19), t (107)=9.32, p<0.001, d=0.97.
Proportion of service users indicating a problem in each category of the EQ-5D-3L.
Complete data on the Eco-Psy were available for 123 service users. Comparing the number of healthcare appointments attended at assessment (Median=2) and 3-month follow-up (Median=2), a Wilcoxon signed-rank test showed no significant difference, Z=0.082, p=0.935. A McNemar test indicated a statistically significant reduction in the proportion of service users using psychotropic medication from assessment (54.5%, n=67) to follow-up (42.3%, n=52), p<0.01. With regard to employment outcomes, a McNemar test showed no statistically significant difference in the proportion of service users employed at assessment (46.3%, n=57) and follow-up (46.3%, n=57), p=1. In comparing employed service users’ ratings of how their mental health difficulties affected their work productivity, a statistically significant reduction was observed from assessment (M=2.1, s.d.=0.98) to follow-up (M=1.5, s.d.=0.95), t (40)=3.2, p<0.01, d=0.62.
Discussion
The accessible service model of APSI was found to be clinically effective in treating mild-to-moderate presentations in primary care. Reductions in both anxiety and depressive symptomatology were observed, suggesting the interventions were successful in treating the associated disorders. Although the recovery rate and effect sizes were high relative to those observed for similar primary care mental health services (Stiles et al. Reference Stiles, Barkham, Mellor-Clark and Connell2008; Gyani et al. Reference Gyani, Shafran, Layard and Clark2013; Delgadillo et al. Reference Delgadillo, McMillan, Leach, Lucock, Gilbody and Wood2014), the absence of outcome data on the substantial proportion of non-completers may have inflated the effects (Cahill et al. Reference Cahill, Barkham, Hardy, Rees, Shapiro, Stiles and Macaskill2003). Furthermore, treatment completers may not have been representative of non-completers. However, as the literature has not reliably linked dropout with a primary care diagnosis or mental illness severity (O’Brien et al. Reference O’Brien, Fahmy and Singh2009), it is unlikely that treatment completers were a clinically distinct sample of service users. Thus, the positive outcomes observed for completers would still suggest that the interventions had an acceptable level of effectiveness in treating primary care presentations. The level of dropout was nonetheless unacceptable and it needs to be considered whether it was attributable to a modifiable element of service delivery, or whether dropout was an inherent side effect of the interventions?
The high dropout rate within APSI was largely due to the poor completion of guided self-help treatment. As the treatment effect was large for this intervention, it is possible that aspects of service delivery acted as a barrier to completion, rather than service users being dissatisfied with the content of treatment itself. For example, completing guided self-help required service users to attend APSI for a final appointment. Some service users may have completed the self-help materials much earlier and felt that this appointment had limited value. Greater flexibility is therefore needed in how service users complete this treatment, including facilitating a telephone contact for the final appointment. Furthermore, it may be valuable to educate service users on the dose-effect relationship (i.e. completing the full course of treatment leads to better outcomes; Swift & Callahan, Reference Swift, Greenberg, Whipple and Kominiak2011) as the less directive nature of guided self-help may be more susceptible to partial completion. A high dropout rate was also observed after the initial assessment appointment. Strategies that may support treatment engagement at this time include instilling hope, role induction and using appointment reminders (Oldham et al. Reference Oldham, Kellett, Miles and Sheeran2012; Swift et al. Reference Swift and Callahan2012).
The completion rate for brief CBT was in the typical range, with the effectiveness of this intervention consistent with previous evaluations of the Roscommon primary care psychology service (Bourke & Byrne, Reference Bourke and Byrne2012; McHugh et al. Reference McHugh, Brennan, Galligan, McGonagle and Byrne2013a ; McHugh et al. Reference McHugh, Gordon and Byrne2014). Although brief CBT does not have the same level of throughput as lower intensity interventions, it serves an important role for managing the more complex difficulties presenting to primary care. It should be noted that the highest recovery rate was observed for those completing multiple interventions, which was typically guided self-help followed by brief CBT. This suggests that the system of ‘stepping-up’ service users to higher intensity interventions was effective, as has been observed within IAPT services (Gyani et al. Reference Gyani, Shafran, Layard and Clark2013).
APSI was shown to be effective in enhancing the self-reported health status of service users, consistent with the process of mental health supporting physical health (Prince et al. Reference Prince, Patel, Saxena, Maj, Maselko, Phillips and Rahman2007). Although this may contribute to medical cost offset in the long term, no reductions in healthcare appointments were observed in the 3-month follow-up period. A reduction in psychotropic medication use was observed. However, without further information, it is difficult to determine how much of this reduction was due to the routine completion of short courses of medication. There were also no apparent cost savings through employment facilitation, with identical employment rates for service users at assessment and follow-up. Those who were employed did indicate that their mental health difficulties had less of an impact on their work productivity at follow-up. This suggests potential for APSI to provide more direct support for service users in managing their workplace difficulties, such as proving basic screening and guidance for such difficulties (e.g. workplace stress, bullying, etc.). The success of the Employment Advisory Service associated with IAPT would support the value of such an initiative (Hogarth et al. Reference Hogarth, Hasluck, Gambin, Behle, Li and Lyonette2013).
There are a number of ways to further maximise the effectiveness, efficiency and acceptability of the APSI service model. First, effective group and computerised CBT (cCBT) interventions need to be readily available, consistent with having a spectrum of interventions within the stepped care model (NICE, 2011b ). A manualised stress management group intervention and a cCBT intervention have been developed for APSI and are currently being evaluated. Second, the service could become more involved in providing mental health training and advice to professionals within the primary care network, thereby enhancing the overall capacity of the community to manage mental health difficulties. Lastly, greater planning is needed to determine how APSI can work with other primary care mental health services (e.g. CIPC, Jigsaw) to provide an integrated continuum of care to service users (e.g. agreed referral and risk management protocols, shared assessment process).
Conclusions
By providing rapid access to brief psychological therapies, APSI was found to be clinically effective in managing mild-to-moderate mental health difficulties. The service model therefore needs to be considered as an alternative to the current overreliance on psychotropic medication in primary care. Although the service had a number of limitations, there is much potential to adapt the model to further maximise clinical effectiveness, service user choice and value for money.
Acknowledgements
The authors would like to thank the staff at APSI for their work in collecting the service data on which this evaluation is based.
Financial Support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of Interest
None.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committee on human experimentation with the Helsinki Declaration of 1975, as revised in 2008. The authors assert that ethical approval for publication of this service evaluation has been provided by their local REC.
