Introduction
Children and young people (C/YP) in out-of-home care (OOHC) represent one of the most vulnerable and disadvantaged groups in Western societies, including Australia (Osborn & Bromfield, Reference Osborn and Bromfield2007; Tarren-Sweeney, Reference Tarren-Sweeney2008). There is significant empirical evidence that indicates C/YP in OOHC are at an increased risk for developmental and mental health disorders across all facets of their life and lifespan. Negative outcomes include attachment difficulties (a core feature of many in care), anxiety, depression, post-traumatic stress, conduct problems (including defiance, anger and aggression), sexual reactive problems, inattention/hyperactivity and suicidal behaviour (Briere et al., Reference Briere, Johnson, Bissada, Damon, Crouch, Gil and Ernst2001; Leve et al., Reference Leve, Harold, Chamberlain, Landsverk, Fisher and Vostanis2012; Osborn, Delfabbro, & Barber, Reference Osborn, Delfabbro and Barber2008; Oswald, Heil, & Goldbeck, Reference Oswald, Heil and Goldbeck2010; Sawyer, Carbone, Searle, & Robinson, Reference Sawyer, Carbone, Searle and Robinson2007; Tilbury, Osmond, Wilson, & Clark, Reference Tilbury, Osmond, Wilson and Clark2007). Negative long-term outcomes include drug and alcohol use/misuse, poor physical health, homelessness, criminality and incarceration (Nathanson & Tzioumi, Reference Nathanson and Tzioumi2007; Norman et al., Reference Norman, Byambaa, De, Butchart, Scott and Vos2012; Richardson, Reference Richardson2005) highlighting the devastating costs that a history of complex trauma can have across an individual's entire lifespan.
The mental health needs for C/YP in OOHC is qualitatively and quantitatively different from those in the general population, resulting in experts stressing the need for highly specialised trauma and attachment informed, multi-agency approaches (Golding, Reference Golding2010; Schneider, Grilli, & Schneider, Reference Schneider, Grilli and Schneider2013; Tarren-Sweeney, Reference Tarren-Sweeney2010). Further, due to the complex needs of those in OOHC it has been suggested that mainstream mental health services struggle to meet the needs of this population (Bellamy, Gopalan, & Traube, Reference Bellamy, Gopalan and Traube2010; Raghavan, Inoue, Ettner, Hamilton, & Landsverk, Reference Raghavan, Inoue, Ettner, Hamilton and Landsverk2010; Tarren-Sweeney, Reference Tarren-Sweeney2010). While research has been conducted on the effectiveness of tertiary level mental health intervention and/or program models for young people in OOHC internationally (Arvidson et al., Reference Arvidson, Kinniburgh, Howard, Spinazzola, Strothers, Evans and Blaustein2011; Cantos & Gries, Reference Cantos and Gries2010; Courtney, Reference Courtney2000; Swenson, Schaeffer, Henggeler, Faldowski, & Mayhew, Reference Swenson, Schaeffer, Henggeler, Faldowski and Mayhew2010; Westermark, Hansson, & Olsson, Reference Westermark, Hansson and Olsson2011), there have only been a few studies conducted within Australia (Frederico, Jackson, & Black, Reference Frederico, Jackson and Black2010; Klag et al., Reference Klag, Fox, Martin, Eadie, Bergh, Keegan and Raeburn2016).
In 2004, concerns regarding a clear unmet need for therapeutic services within the OOHC population was identified within the Queensland, Australia Crime and Misconduct Commission (CMC) report “Protecting Children: An Inquiry into Abuse of Children in Foster Care”. At the time, it was identified that up to 17% of those in the care system had particularly high levels of complex and extreme psychological and behavioural problems requiring urgent intervention. As such, the CMC report recommended that ‘more therapeutic treatment services are made available to children with severe psychological and behavioural problems’ and that the implemented service should be evaluated (CMC, 2004, p.194). Following the report recommendations, Evolve Therapeutic Services (ETS) was established as a tertiary level mental health intervention service.
The key focus of ETS is to provide planned and coordinated therapeutic and behaviour supports to C/YP in OOHC, aimed at improving their emotional wellbeing and the development of skills to enhance participation in school and in the community. Eligibility criteria includes: the child is under 18 years of age, presents with severe and/or complex psychological and/or behavioural problems (i.e., a chronic trauma history, extreme behavioural problems across multiple settings, at risk of harming self/others and multiple placement breakdowns), and is in OOHC on interim or finalised Child Protection Orders. ETS is a trauma-informed program grounded in well-established theoretical perspectives (child development, systemic theory, trauma, attachment, psychodynamic theory and grief and loss) that encapsulates a collaborative “wrap-around” model of service. Provision of service is achieved through a flexible use of appropriate evidence-informed individual and systemic therapeutic interventions and a coordinated and sustainable partnership with key government, non-government and private sector agencies. Clinical mental health interventions include an initial and ongoing comprehensive assessment of the bio/psycho/social/cultural aspects of the C/YP and their significant others; risk management and safety planning; attachment and/or trauma focused therapies, which may include dyadic work, individual therapy, family-based intervention or the use of other treatment modalities; and, stakeholder coordination and support. Different interventions were utilised by different clinicians. Some of these interventions included Dyadic Developmental Psychotherapy, Theraplay, Trauma-focussed Cognitive Behaviour Therapy, Narrative and Expressive Therapies, Systemic approaches and Life Story Work.
Currently, there are 10 ETS teams located throughout Queensland. The vast majority of funded staff are frontline qualified and registered allied health workers (e.g., Psychiatry, Psychology, Social Work, Nursing, Occupational Therapy). All teams have a Professional Development Coordinator and a number of teams have an Indigenous Program Coordinator.
The present study evaluates the ETS program across Queensland and its impact on functioning and wellbeing of C/YP in OOHC with severe and complex behavioural and mental health problems. Demographic and clinical symptom profiles, functioning and mental health problems are described.
Method
Ethics approval was obtained from the Children's Health Queensland Health and Hospital Services Human Research Ethics Committee (HREC/09/QRCH/48). The consumer's Guardian, usually Child Safety, gave consent for treatment and research on entry to the service.
Data was collected during treatment, stored in the Queensland Health Mental Health Clinical Information Application (CIMHA) and made available by the Queensland Health Mental Health Information and Performance Unit (MHIPU). Mental health diagnoses occurred in accordance with the International Statistical Classification of Diseases and Related Health Problems (ICD) and confirmed by the treating team's Consultant Psychiatrist. Access to confidential C/YP data was approved under delegation of the Director General, Queensland Health, and in accordance with Section 284 of the Public Health Act 2005. The evaluation was conducted by the ETS evaluation coordinator.
Measures
The Health of the Nations Outcome Scale for Children and Adolescents (HoNOSCA) (Gowers et al., Reference Gowers, Harrington, Whitton, Lelliott, Wing, Beevor and Jezzard1999), is a 15-item clinician-rated measure designed specifically for assessment of child and adolescent outcomes in mental health services. It includes 13 clinical/psychosocial items (disruptive/aggressive behaviour, overactivity and attentional difficulties, non-accidental self-injury, alcohol or substance/solvent misuse, scholastic and language skills, physical illness/disability problems, hallucinations and delusions, non-organic somatic symptoms, emotional and related symptoms, peer relationships, self-care and independence, family life and relationships and poor school attendance) and two items relating to knowledge about the child and/or young person's difficulties, management and services available. Each item is scored on a five-point scale from 0 (no problems) to 4 (severe problems) based on the previous two weeks, with a detailed glossary for each point of the scale and item (Gower et al., Reference Gowers, Harrington, Whitton, Lelliott, Wing, Beevor and Jezzard1999). Pre/post HoNOSCA items were completed by clients’ clinicians. A rating of 2, 3 or 4 indicates clinically significant problems requiring active monitoring or intervention. The scale is a valid measure of global psychiatric outcomes in C/YP, and is sensitive to change (Bilenberg, Reference Bilenberg2003; Garralda, Yates, & Higginson, Reference Garralda, Yates and Higginson2000).
The Children's Global Assessment Scale (C-GAS) (Shaffer et al., Reference Shaffer, Gould, Brasic, Ambrosini, Fisher, Bird and Aluwahlia1983) is clinician-rated and provides a global level of adjustment and functioning on a scale of 1–100. Scores greater than 70 indicate no clinically significant functional impairment, scores less than 70 are associated with increasingly severe dysfunction. C/YP referred to clinical services generally have scores of less than 61 (Bird et al., Reference Bird, Yager, Staghezza, Gould, Canino and Rubio Stipec1990). The CGAS has good psychometric properties and is sensitive to change (Steinhausen, Reference Steinhausen1987).
The Strengths and Difficulties Questionnaire (SDQ) (Goodman, Reference Goodman1999) is a brief behavioural screening questionnaire about C/YP aged 4–17 years. The SDQ includes 25 items relating to attributes of the target C/YP, some positive and some negative. The respondent is asked to rate each of these 25 items on a 3-point scale (0 = Not True; 1 = Somewhat True; 2 = Certainly True). The 25 items are divided between 5 sub-scales: Emotional symptoms, Conduct problems, Hyperactivity/Inattention, Peer relationship problems and Pro-social behaviours. The first four scales are also summed to yield a Total Difficulties score (maximum score = 40 (Australian Mental Health Outcomes and Classification Network (AMHOCN), 2005).
The SDQ has been shown to have good validity and reliability (Goodman, Reference Goodman2001). It has been used successfully with children in OOHC (Goodman, Ford, Corbin et al., Reference Goodman, Ford, Corbin and Meltzer2004; Whyte & Campbell, Reference Whyte and Campbell2008), and norms have been verified in the Australian population (Hawes & Dadds, Reference Hawes and Dadds2004), including an Aboriginal population (Zubrick, Lawrence, DeMaio, & Biddle, Reference Zubrick, Lawrence, De Maio and Biddle2004).
The carer and stakeholder surveys assessed the quality of the C/YP's relationships with both the carer and with peers, the carer's wellbeing, the C/YP's engagement in their education, the safety of the C/YP, the C/YP's ability to maintain their placement, and the C/YP's behaviour and wellbeing. Seventy-one carer surveys were completed with a 43% response rate. The respondents comprised of Foster Carers, Kinship Carers, Residential Care Workers and Biological Parents. Carer survey response rates from child and youth mental health and/or child protection studies ranged from 19.4% to 56% (Department of Human Services, 2005; Paxman, Tully, Burke, & Watson, Reference Paxman, Tully, Burke and Watson2014; Queensland Government, 2009; Riley, Stromberg, & Clark, Reference Riley, Stromberg and Clark2005). Two hundred stakeholder surveys were completed with a 24% response rate. Stakeholders comprised mainly Child Safety staff, Education personnel, Foster Care Support Agency staff and Residential Care managers. The response rate is comparative if not better to a previous study collecting surveys in a similar context (Darlington, Feeney, & Rixon, Reference Darlington, Feeney and Rixon2005).
Procedure
All ETS clinicians received formal and regular training in administration and interpretation of HoNOSCA, CGAS and SDQ in order to maintain inter-rater reliability. Baseline data was included if it had been collected within the first 4 months of allocation to ETS. This timeframe was chosen as it coincided with the final date a comprehensive mental health assessment could be completed. Completion data (the last data collection occasion for each C/YP) was included only if it had been collected within a 4-month period prior to the official closure of the C/YP's service episode. In most cases the same clinician completed the HoNOSCA and CGAS at pre and post, but this could not be guaranteed due to staff movement.
Data Analysis
Data was analysed using SPSS 23. Frequency analyses created a clinical profile of ETS C/YP complexity, severity and mental health issues at admission. Outcomes were assessed by comparing pre and post treatment CGAS, HoNOSCA and SDQ mean scores using repeated-measures t-tests. For estimates of differences in the proportion of C/YP in the clinical range between pre and post-treatment the Related – Samples McNemar test was used.
Results
Demographic Profile
Throughout 2015, ETS provided clinical services to 768 C/YP, 467 males (61%) and 301 females (39%). The mean age of C/YP at time of acceptance was 9.38 (range 1–17 years). Almost 10% of C/YP (n = 72) receiving clinical services were four or under. Thirty-one percent of the sample identified as being of Aboriginal background, 2% identified as being of Torres Strait Islander background, while 3% identified as being of both Aboriginal and Torres Strait Islander background. Thus, overall, 36% of the sample are of Aboriginal and/or Torres Strait Islander background.
The ETS model of service (Department of Communities, 2012) recommends a treatment duration of up to 18 months. ETS closed 264 cases in 2015. The overall mean treatment duration was 18.56 months (standard deviation = 14.51 months), with 59% completing treatment within the recommended 18 months. Treatment duration range was 3 weeks to 6.5 years.
Of the 264 cases closed from the service, 62% were discharged successfully because they had met their treatment goals and only 11% disengaged from treatment. In addition, 9% were closed due to the C/YP moving out of the ETS teams catchment area, 10% due to C/YP no longer meeting ETS eligibility criteria and another 5% were referred on to more suitable services. The closure rates obtained by ETS are favourable when compared to those found in a review of Child and Adolescent Mental Health Services in Melbourne which showed 29% successful completion, 18% referral on, 4% clinician initiated termination and 49% consumer disengagement (Johnson, Mellor, & Brann, Reference Johnson, Mellor and Brann2009).
Clinical Profile
In order to determine if the 2015 cohort of ETS C/YP was consistent with the program's target group, a profile of their clinical symptoms and level of functioning was compiled. Virtually all ETS C/YP (97.5%) had a CGAS score at pre-treatment in the clinical range (score of 70 or below). CGAS scores of 50 or below were recorded for 63.9% of consumers, indicating moderate to severe impairment in functioning for approximately two-thirds of ETS consumers before the commencement of treatment. The mean score of ETS consumers on the CGAS at pre-treatment was 48.41 (n = 677), indicative of “a moderate degree of interference in functioning in most social areas or severe impairment of functioning in one area”.
The sample size for the individual HoNOSCA items varied from n = 652 to n = 675 due to missing values. If any of the 13 Problem areas rated in the clinical range they are deemed to warrant inclusion on a C/YP's treatment plan, requiring at least active monitoring and most probably targeted goal-oriented treatment by the treating clinician.
A high percentage of C/YP scored in the clinical range (i.e., scores of 2 – mild, 3 – moderate or 4 – severe) on six of the 13 HoNOSCA items at their first outcomes measure collection occasion. These were problems with disruptive, antisocial or aggressive behaviour (83%); overactivity, attention and concentration (77%); scholastic or language skills (68%); emotional and related symptoms (92%); peer relationships (83%); and family relationships (91%). Ninety-two percent of C/YP had four or more HoNOSCA items rated as being in the clinical range at their first outcome measures collection occasion, with the modal number of items in the clinical range being seven.
Taken together, the proportions of C/YP scoring in the clinical range on the CGAS and each HoNOSCA item, as well as the number with multiple problem areas in the clinical range, indicate that the C/YP receiving treatment from ETS have severe and highly complex psychological and behavioural problems.
Treatment Outcomes
To report on the outcomes of ETS treatment in terms of change in the symptoms and the functioning of C/YP, and other important mediating variables, outcome data will be reported against the following areas: child/young person wellbeing, carer wellbeing, placement stability, engagement in education, relationships with peers and carers, and stakeholder collaboration.
Child/young person wellbeing
A repeated-measures t-test was conducted to measure changes from pre to post-treatment in consumers’ general level of functioning as measured by the CGAS. There was a statistically significant change in functioning as reflected in CGAS scores across treatment. The mean CGAS score increased from 48.9 (SD = 10.3) to 58.3 (SD = 12.7) (t(222) = -10.595, p < .001), indicating an improved level of general functioning for ETS consumers coinciding with the period of ETS treatment. A statistically significant decrease in the proportion of consumers in the clinical range from pre to post-treatment scores was measured by the Related-Samples McNemar Test; p < .001, n = 236.
The McNemar test was used to measure the significant proportion of consumers that moved from “clinical” to “non-clinical” from pre to post treatment on the HoNOSCA. Individual HoNOSCA scores were categorised as “clinical” (score between 2 and 4) or “non-clinical” (score of 0 or 1), consistent with National reporting processes. Nine HoNOSCA items are identified as relevant to the overall wellbeing of the C/YP. Statistically significant decreases in the proportion of consumers in the clinical range from pre to post-treatment scores were observed for seven of the nine relevant HoNOSCA scales associated with C/YP wellbeing, with Related-Samples McNemar Test statistics obtained (Table 1).
HoNOSCA scales - Related-Samples McNemar Test statistics
A repeated-measures t-test was conducted to measure the impact of ETS treatment on consumers’ general psychosocial functioning as measured by scores on the Total Problem Index of the carer-rated SDQ. There was a statistically significant decrease in the Total Problem Index score from pre (M = 18.90, SD = 4.87) to post (M = 16.95, SD = 4.44), t(116) = 3.085, p < .05.
The SDQ has four problem subscales: Emotional Symptoms; Conduct Problems; Hyperactivity/Inattention and Peer Problems. Scores on each subscale range from 0 to 10, with higher scores indicating greater problems. Repeated-measures t-tests were conducted to measure the impact of ETS treatment on consumers’ difficulties as measured by scores on each of the four problem sub-scales of the SDQ.
Statistically significant decreases (Table 2) in severity of problem scores from pre and post-treatment scores were observed for two of the four carer-rated SDQ problem subscales, emotional symptoms and hyperactivity/inattention. There was a statistically significant decrease in the teacher-rated emotional symptoms subscale and Total Problem Index. There was a statistically significant decrease in the youth-rated emotional symptoms subscale, conduct problems subscale, hyperactivity/inattention subscale and total problem index.
SDQ
One item from the carer and the stakeholder surveys elicited a rating of the C/YP's change in behaviour since commencing with ETS. Seventy-six percent of stakeholders reported that the C/YP's behaviour had improved compared to 68% of carers. Another item elicited a rating of the C/YP's change in overall wellbeing/mental health since commencing with ETS. Seventy-nine percent of stakeholders reported that the C/YP's wellbeing had improved compared to 73% of carers.
Carer wellbeing
Carer wellbeing was assessed via items on the carer survey. Seventy-nine percent (79%) of carers reported that their ability to cope with the C/YP's difficulties had improved. Sixty-eight percent (68%) of carers reported that their ability to maintain the care for the C/YP had also improved since they engaged therapeutically with ETS.
Placement stability
Stakeholders and carers’ survey data indicated that the C/YP's ability to maintain their placement had improved in approximately 75% of cases. Data also demonstrated that since being engaged with ETS 79% of carers reported that C/YP felt more a part of their home, 94% reported that C/YP were more accepted at home and 74% reported that C/YP felt more safe in their placement.
Engagement in education
Statistically significant decreases in the proportion of consumers in the clinical range from pre to post-treatment scores were observed for problems with scholastic and language skills (Related-Sample McNemar Test p < .05, n = 227) and poor school attendance (Related-Sample McNemar Test p < .05, n = 226).
Stakeholders and carers gave a similar survey rating indicating that the C/YP's engagement in education had improved for approximately two-thirds of the sample. Fifty-one percent of carers reported that C/YP's attendance at school had improved since they had been engaged in ETS, 62% reported an improvement in the C/YP feeling safe at school and 61% reported that the C/YP's relationship with their teacher/s had improved.
Relationships with peers and carers
Statistically significant decreases in the proportion of C/YP in the clinical range from pre to post-treatment HoNOSCA scores were observed for problems with peer relationships (Related-Sample McNemar Test p < .001, n = 227) and problems with family life and relationships (Related-Sample McNemar Test p < .001, n = 227).
The carer-rated prosocial behaviour subscale provides a measure of the perceived social skills and social competence of the child or young person in their care. A repeated-measures t-test found a significant improvement in prosocial behaviour (t(116) = −2.175, p < .05) as rated by the carer and a significant deterioration (t(61) = 2.151, p < .05)) as rated by the teacher.
Seventy-six percent of carers reported an improvement in their relationship with the C/YP, and no carer reported a deterioration in their relationship since commencing involvement with ETS.
Results suggest that stakeholders and carers differed in their perception of changes in peer relationships. Sixty-two percent of stakeholders reported these relationships had improved, 29% reported no change and 9% reported a deterioration. Of the carers surveyed, 53% reported an improvement in the peer relationships of the child/young person in their care, while 41% reported no change and 6% reported a deterioration.
Stakeholder collaboration
Stakeholder processes were evaluated using the carer and stakeholder surveys. High proportions of both stakeholders and carers either agreed or strongly agreed that stakeholder processes improved collaboration and communication between stakeholders, and that the stakeholder processes improved the stakeholders understanding of the C/YP's needs.
Discussion
Clinically, these findings reinforce the presenting complexity in terms of mental health, functionality and individually tailored trauma-informed mental health intervention response required for C/YP in OOHC (Golding, Reference Golding2010; Schneider et al., Reference Schneider, Grilli and Schneider2013; Tarren-Sweeney, Reference Tarren-Sweeney2010). Further, the results obtained demonstrate the effectiveness of collaborative, inter-departmental and systems approaches (Frederico et al., Reference Frederico, Jackson and Black2010; Klag et al., Reference Klag, Fox, Martin, Eadie, Bergh, Keegan and Raeburn2016) with a strong tertiary level mental health focus for improving and sustaining health and wellbeing for a very vulnerable population.
Reports from clinician rated measures and carer surveys indicated that consumer wellbeing, carer wellbeing, placement stability, school attendance and family and peer relationships had improved since the C/YP had commenced ETS intervention. The findings support the program's ability to minimise negative outcomes for this vulnerable population (Leve et al., Reference Leve, Harold, Chamberlain, Landsverk, Fisher and Vostanis2012; Oswald et al., Reference Oswald, Heil and Goldbeck2010; Sawyer et al., Reference Sawyer, Carbone, Searle and Robinson2007; Tilbury et al., Reference Tilbury, Osmond, Wilson and Clark2007).
The study had some limitations. The main limitation, as with many intervention studies with this population, is the unethical nature of withholding treatment or providing potentially less effective interventions by way of a control group. It is therefore not possible to attribute the observed treatment effects to the ETS intervention alone, as other confounding variables may, at least in part, explain the observed effects.
Second, missing data raises the question as to whether the data reported actually represents the total population or whether there could be systematic differences between those C/YP for whom data has been reported and those for whom data is missing. Third, children under the age of four years are out of scope for outcomes collection (Australian Mental Health Outcomes and Classification Network, 2015), which means that this cohort of consumers is automatically excluded from the current data set used for the clinical profile or pre-post analyses.
Fourth, the clinician ratings were provided by one rater only. To improve consistency, it would make sense to plan for two raters independently scoring young people. Fifth, consumers had different service episode lengths which meant that outcome measurement at time one and two were different for each consumer. Lastly, while the results demonstrate the effectiveness of a collaborative tertiary level mental health service, there is currently no follow-up assessment completed. A follow-up at 12 months would provide information on whether gains made during treatment have been maintained.
Despite the fact that the structure of OOHC differs across all of Australia's jurisdictions, including the way in which services are provided, the findings of this article highlight important implications for policy and service delivery. All C/YP in OOHC require a mental health therapeutic response that meets their individual identified needs, at times this being a tertiary level response not just a primary/secondary response. This would provide C/YP in OOHC with the best chance to minimise negative outcomes in the short and long term. For an effective mental health tertiary response for C/YP in OOHC, services require to not just be trauma-informed but trauma-integrated and have required resources (e.g., adequate staffing to C/YP ratio and high level trauma and attachment training) that are willing and able to provide a truly collaborative response with stakeholders.
Acknowledgements
The author wishes to acknowledge the staff of Evolve Therapeutic Services for their contribution to service evaluation.
