Introduction
The measurement of consumer satisfaction with therapeutic services and perceptions of their effectiveness is a recognised priority in Australian-based child welfare organisations and child protection research (Trotter, Reference Trotter2008). A small number of studies have examined service satisfaction within the context of out-of-home care (Alexander, Reference Alexander2015; Barker & Place, Reference Barker and Place2005; Harris, Poertner, & Joe, Reference Harris, Poertner and Joe2000), while one study focused specifically on psychometric evaluation of the Client Satisfaction Inventory, which aims to measure general satisfaction with services among clients of human service agencies (McMurtry & Hudson, Reference McMurtry and Hudson2000). The results of that study indicated excellent internal consistency and satisfactory initial validation. Similarly, the 24-item Parents with Children in Foster Care Satisfaction Scale, designed to measure service satisfaction, was found to have good reliability and moderate evidence for validity (Harris et al., Reference Harris, Poertner and Joe2000). Harris et al. reported that the development of the scale had considerable input from the parents (consumers) and this maximised the “possibility that the instrument reflected clients’ actual concerns instead of professionals’ assumptions of what aspects of services are important to clients” (Reference Harris, Poertner and Joe2000, p. 23). All the aforementioned studies are, however, international rather than Australian-based research studies. In Australia itself, there are relatively few well-designed scales to measure Australian consumers’ experiences with therapeutic programs designed to ameliorate the adverse impacts of child abuse and neglect.
Listening to consumers, in particular to the children in out-of-home care, is a well-established priority for the Australian child welfare system. However, only a few systematic attempts have been made to obtain information about children’s experience and satisfaction with care (Barber & Delfabbro, Reference Barber and Delfabbro2005; Holland, Reference Holland2009). Consumer experience surveys, when designed and administered appropriately, are one means of providing robust measures of service satisfaction and experience of care quality. However, within Australia, there is a poor evidence base for various issues related to children in out-of-home care and significant research gaps in listening to consumers, in particular the children (Bromfield, Higgins, Osborn, Panozzo, & Richardson, Reference Bromfield, Higgins, Osborn, Panozzo and Richardson2005; Bromfield & Osborn, Reference Bromfield and Osborn2007; Cashmore & Ainsworth, Reference Cashmore and Ainsworth2004).
Although a review article encompassing 44 studies from 10 different countries (including 4 from Australia) elucidated a number of methodological and theoretical approaches to listening to children in care (Holland, Reference Holland2009), the evidence is still limited. This review identified very few studies that triangulated the qualitative and quantitative designs and data collection for researching children’s perspectives, views and experiences in care. In addition, there was a wide variation in research designs, data collection and methodological choices impeding comparability and generalisation of the findings. For example, within the context of Australian-based foster care research, various obstacles and barriers have been identified that impact the inclusion of children and young people in research designed to measure consumer experiences and feedback (Gilbertson & Barber, Reference Gilbertson and Barber2002). These include recruitment difficulties, high non-response rates, participants of programs not being available because their details are missing or they are transient and cannot be contacted for feedback, lack of cooperation from social and other workers, reluctance to participate, placement instability, lack of follow-up by social workers or other staff, assessments of participants as too distressed to participate in research and/or too dangerous to interview, as well as the inability of participants to keep research-related appointments (Gilbertson & Barber, Reference Gilbertson and Barber2002). Despite an increasing emphasis on the importance of consumers’ experiences of therapeutic services and programs for children affected by abuse and neglect, only a few Australian-based studies reporting on users’ experiences of satisfaction and quality of out-of-home care in Australia were considered to be rigorous (Barber & Delfabbro, Reference Barber and Delfabbro2005).
This article aims to contribute to existing literature by focusing on the consumer experience of a therapeutic program, Take Two, offered by Berry Street in Victoria, Australia. The Berry Street Take Two program is a statewide service funded by the Department of Health and Human Services, conducted in partnership with La Trobe University, the Mindful Centre for Training and Research in Developmental Health, and the Victorian Aboriginal Child Care Agency (VACCA). It is a therapeutic service primarily for clients of Child Protection for whom abuse and neglect have been substantiated. In addition to facilitating direct practice, Take Two also has training and research components. Children from infancy to 17 years of age, who have been severely traumatised by abuse and neglect, are referred to Take Two by Child Protection. The median length of engagement with Take Two is 14 months. Take Two interventions usually involve direct clinical work with the referred child and their family, frequently on an outreach basis. Take Two also engages with families and relevant systems in the child’s life to address the impact of the child’s experiences of trauma on their everyday functioning (Jackson, Frederico, Cox, & Black, Reference Jackson, Frederico, Cox, Black and Huppert2019).
This article integrates the quantitative and qualitative components of the Take Two Stakeholder surveys by exploring the (a) reliability and factorial structure of the survey (b) consumers’ experience with the Take Two service, and (c) the research and clinical implications of the findings in terms of clinical practice improvements and enhanced consumer engagement.
Methodology development of the Take Two Stakeholder survey
The development of the Take Two Stakeholder Survey has been thoroughly described elsewhere (Frederico et al., Reference Frederico, Jackson and Black2005; Frederico, Jackson, & Black, Reference Frederico, Jackson and Black2010). In brief, the survey was developed in partnership between the research and clinical staff of La Trobe University and Take Two. The development of the survey was informed by previous surveys such as the Youth Services Survey (Brunk & Koch, Reference Brunk and Koch1999), Client Satisfaction Questionnaire (Larsen, Attkisson, Hargreaves, & Nguyen, Reference Larsen, Attkisson, Hargreaves and Nguyen1979), Experience of Service Questionnaire (Commission for Health Improvement, 2002), and Youth Satisfaction Questionnaire (Stuntzner-Gibson, Koren, & DeChillo, Reference Stuntzner-Gibson, Koren and DeChillo1995). The survey was developed to measure stakeholders’ experience and satisfaction with the Take Two service in order to inform the work of Take Two. It included obtaining stakeholder perceptions about improvements in children’s wellbeing and changes that occurred as a result of participation in the service. This built on the use of the survey previously to “elicit client and stakeholder perceptions of the program and of whether changes had occurred through participation [with Take Two]” (Frederico, Jackson, & Black, Reference Frederico, Jackson and Black2006, p. 13). The survey findings were expected to contribute to continuous improvement of the program by providing some evaluation of the intended program design.
Description of the Stakeholder Survey
The survey was developed, with minor variations to its earlier format, to cater for parents, children, carers and professional workers. The survey had four sections. Firstly, there were basic socio-demographic questions about the child client such as age, gender and Aboriginal and Torres Strait Islander status. The second section included six fixed-choice statements relating to the experience with the Take Two program, followed by five statements regarding child functional outcomes since receiving the Take Two service. These included statements such as I liked the service given by Take Two; Take Two respected my culture and I get along better with family members. These statements were rated on a 5-point Likert scale to determine the level of agreement (strongly agree to strongly disagree). The fourth section included a question related to the stakeholder’s overall experience with Take Two using a 6-point Likert scale, and two questions which invited free text responses: “What has been the most helpful thing about the service from Take Two?” and “What could improve the Take Two service?”
The Ethics Committee at La Trobe University and the Department of Health and Human Services (DHHS) granted approval for the Take Two stakeholder surveys. The Ethics Approval number is 04-131.
Survey administration
The surveys were distributed by the Take Two clinicians and/or administrative staff at the review and closure phases of program delivery to the children referred to Take Two. Each survey was accompanied by a covering letter and Reply-Paid return envelope. These were either given to the respondent by the clinician or mailed to them. The surveys were given to members of the care team including professional workers, parents and carers, other involved professionals and the children themselves. Professional workers included those working with and for the referred child including teachers, Child Protection workers, community service organisations workers, case managers and youth and education support workers. Carers included a combination of foster carers, kinship carers and residential carers. The completion of surveys was voluntary. The surveys were returned to the Take Two research and evaluation team at La Trobe University.
Data manipulation and analysis
The survey collected both qualitative and quantitative data. Survey data were entered into an ACCESS database and filed electronically at La Trobe University. Data were exported to SPSS V. 24 for quantitative analysis. Descriptive statistics, including frequency distribution of the items, were examined in order to describe stakeholders’ responses, in addition to assessing deviation from normality. The reliability of the survey was assessed using Cronbach’s α (alpha) as an indicator of internal consistency. A value for α greater than 0.70 is considered to be satisfactory (Tavakol & Dennick, Reference Tavakol and Dennick2011). Further, given the exploratory nature of this study, an exploratory factor analysis (EFA) was conducted in order to explore the dataset rather than test specific hypotheses and/or theories. The purpose of EFA was to explore the underlying structure and dimensions of the Take Two stakeholder survey. Kaiser–Meyer–Olkin (KMO) and Bartlett’s test of Sphericity were used to determine data suitability for EFA. Further, only factor loadings greater than 0.3 were retained. Correlational analysis was used to examine the association between perceived satisfaction with the Take Two service and perception of positive clinical outcomes for the children.
In terms of the qualitative data (open-ended questions), the responses were reviewed and coded using content and thematic analysis (Braun & Clarke, Reference Braun and Clarke2006; Schreier, Reference Schreier and Flick2014). Major headings and sub-headings were generated that described as much of the content as possible. The headings with similar content were merged into broader headings; this reduced the number of categories. The data were read and re-read, and coded again according to the emerging themes. Each coded section of the qualitative data was placed under the relevant theme/category. The categorisation process involved two main processes: (a) convergence, that is, deciding which part of data fit together under a category, and (b) divergence, that is, separating categories and themes, once they had been generated and labelled (Lutz & Hill, Reference Lutz and Hill2009). This process was continued until most of the data had been included under the relevant themes.
Results sample characteristics
Between January 2007 and December 2016, the Take Two program worked with 2724 children in 3513 episodes of care. During this period of time, the program received 1545 completed stakeholder surveys related to 675 children (this equates to 25% of the referred children having at least one stakeholder survey). The gender distribution of the children were males (n = 378, 56%) and females (n = 297, 44%). Ages ranged from 0 to 18 years with a mean of 8.04 years (Standard Deviation [SD] = 4.22). The overall average length of service involvement and treatment was 22.37 months (SD = 18.70). An affiliation with Aboriginal and Torres Strait Islander heritage was indicated for 19% of the children (n = 129). The majority of respondents (n = 579, 85.8%) gave consent for clinicians to see the feedback provided. These findings and sample characteristics were slightly different to the program-wide client profile (females 48.3%, average age 9.71 years, length of treatment 11.17 months).
Over the period of this study (2007–2016), the survey respondents were parents (n = 81,5.2%), children (n = 217, 14.0%), carers (n = 344, 22.3%) and professional workers (n = 903, 58.4%). There was no statistical difference in the number of returned surveys across this period by respondent type (χ2 (27) = 21.169, p = 0.778). Specifically, the average age of child respondents was 12.43 years (SD = 2.54 years) and ranged from 8 to 18 years of age. Further, it is to be noted that only 17.59% of all referred children during the period under consideration were living with one or both parents. Table 1 shows the number of returned surveys across the years and the percentage of returns in each stakeholder category.
Number of surveys returned per year by respondent type
No significant differences in the number of returned surveys across the years (χ2 (27) = 19.445, p = 0.853)
Measuring consumers’ overall experience
The results showed that children, parents, carers and workers reported high satisfaction with the Take Two service. Over 80% of respondents reported satisfaction with their experience of Take Two (consistent findings across the years) compared to less than 10% undecided and less than 5% who were not satisfied with the service. There was no pattern in terms of which stakeholders were more or less likely to be satisfied with the service (χ2 (6) = 9.62, p = 0.142). More specifically, a higher percentage of respondents strongly agreed or agreed on items related to quality service provision (I liked the service), understanding the client (I found the people at Take Two understanding), client inclusion in goal and intervention planning, access to service and respect for culture (Take Two respected my culture). In terms of perception of positive outcome for the child, 78% of stakeholders reported positive functional outcomes for the child (improvement in handling daily life stressors, getting along better with family members). Noticeably, significant patterns emerged in terms of children being more likely to endorse positive outcome (86%) compared to parents (80%), professional workers (77%), and carers (75%). These differences were significant (χ2 (6) = 21.86, p = 0.001), suggesting that workers, carers and parents were less likely to report positive changes for the child than the children themselves.
Reliability assessment and factor analysis
The Take Two stakeholder survey was found to have good internal consistency, with a Cronbach’s α (alpha) coefficient of 0.86. In addition, all survey items, with one exception, had a moderate to strong corrected item-total correlation score greater than 0.5. The exception was the question about how easy it was to get to the service (r = 0.410). Overall, these findings indicated satisfactory item reliability, implying that the survey items were consistent and measuring coherent construct. Table 2 shows the inter-correlations of survey items. The item inter-correlations indicated that perceived satisfaction with the service was highly correlated with perceptions of positive change. Higher levels of satisfaction with Take Two correlated with higher levels of perceived positive impact (for example, a child was perceived to be better handling daily life). Further, several outcome-related statements were moderately or strongly correlated with each other. For example, the perception that the child was more able to handle daily life was strongly correlated with outcomes such as improvement in the child’s interaction with family members (r = 0.635, p < 0.001). Overall, these moderate to strong positive item inter-correlations or correlation matrix (often termed Factorability of R (Tabachnick, Reference Tabachnick2013)) indicate that exploratory factor analysis (EFA) is an appropriate statistical method to use with this dataset.
Inter-correlational analysis of core survey items
** Correlation is significant at the 0.01 level (2-tailed); Survey Items 1 = I liked the service given by Take Two; 2 = I found the people at Take Two understanding; 3 = I felt included by Take Two; 4 = It was easy to get to the service; 5 = Take Two staff spoke with me in a way I understood; 6 = Take Two respected my culture; 7 = I am better at handling daily life; 8 = I get along better with family members; 9 = I get along better with friends; 10 = I am doing better in school/training/work; 11 = I can cope better when things go wrong.
In addition, prior to running the factor analysis, examination of the data indicated that not every variable was normally distributed. Given the robust nature of exploratory factor analysis, these deviations were not considered problematic. In addition, Kaiser–Meyer–Olkin (0.889) and a significant Bartlett’s Test of Sphericity (p = 0.000) reflected data suitability for EFA.
The EFA used principal axis factoring with promax rotation. Consistent with Scree plot, the results identified two factors (with eigenvalues exceeding 1) as underlying the 11 core statements of the stakeholder survey. In total, these factors accounted for 63% of the variance in the questionnaire data. Table 3 shows that Factor 1 was categorised as ‘clinical outcomes’ for the child and Factor 2 labelled as ‘satisfaction’ with service. Overall, these findings indicate that the survey was a useful measure of service satisfaction and perceived outcomes for the child.
Factor loadings for core survey items
Extraction Method: Principal Component Analysis; Rotation Method: Varimax with Kaiser Normalization; Survey Items 1 = I liked the service given by Take Two; 2 = I found the people at Take Two understanding; 3 = I felt included by Take Two; 4 = It was easy to get to the service; 5 = Take Two staff spoke with me in a way I understood; 6 = Take Two respected my culture; 7 = I am better at handling daily life; 8 = I get along better with family members; 9 = I get along better with friends; 10 = I am doing better in school/training/work; 11 = I can cope better when things go wrong.
Qualitative analyses
Analysis of stakeholder qualitative comments revealed various themes related to the most helpful aspects of the Take Two service. These themes included the ability of Take Two staff to establish positive therapeutic relationships, commit to quality of care, provide clinical consultation and demonstrate therapeutic expertise (see Table 4).
A selection of stakeholders’ qualitative comments related to the most helpful thing about the Take Two service (Total word count = 21,569)
The comments of the child clients, which are presented in Table 4, show that many respondents experienced an improvement in emotional regulation – an important step in the pathway to improvement. Table 4 shows that stakeholders responded positively to the relationship with the service. This feedback is important as Take Two depends on relationships to engage with clients and other stakeholders.
On the other hand, themes relating to what could be improved included continuity of the Take Two service, stability of the therapeutic worker relationship and early engagement with immediate and extended families of the children. These themes are presented in Table 5. Overall, the nature and quality of the relationship with the Take Two workers and clinicians appear to be critical to the reported positive experiences with the service.
A selection of stakeholders’ qualitative comments related to what could improve Take Two service (Total word count = 10,116)
Discussion
The Take Two Stakeholder Survey was developed to measure stakeholders’ experience and satisfaction with service, as well as perceptions of positive outcomes for the child. Seeking the views of children, parents and stakeholders is an element of improving service effectiveness. Moreover, an opportunity to provide feedback for service development is seen as empowering for clients (Tilbury, Osmond, & Crawford, Reference Tilbury, Osmond and Crawford2010). Overall, this study found stakeholders in general, and the children in particular, were highly satisfied, and positive changes for children were reported in relation to their experience with Take Two. The quantitative survey analysis found good internal consistency (reliability) and satisfactorily identified two factors as underlying the core items of the survey, namely service satisfaction and perception of clinical change.
The findings indicated that reported satisfaction with the Take Two service was positively correlated with perceptions of positive clinical outcomes for the child. One possible explanation of this correlation is that consumers base their satisfaction rating on their perception of positive client outcomes. Further, the correlation between service satisfaction and perception of clinical change may reflect interpersonal care experiences such as consumer–clinician experiences (therapeutic relationship) which represent a unique dimension of quality of care. In fact, in this study, the overarching theme derived from the qualitative responses related to the nature and quality of the relationship with the Take Two worker, which appears to be critical to positive experience with the service. The positive responses to the specific questions in relation to outcomes provide more confidence in the interpretation of the findings of satisfaction. A measure of client satisfaction on its own is not a good measure of service effectiveness (Gain & Young Reference Gain and Young1998).
In addition to informing the work of Take Two, this preliminary study contributes to ongoing efforts to develop, implement and analyse reliable and valid measures for collecting data related to service satisfaction, and perception of positive change for children affected by abuse and neglect in general, and clients of Australian Child Protection authorities in particular. Comparison of the findings to previous research is limited, since there is a paucity of Australian-based studies with systematic attempts to obtain information regarding children’s experience and satisfaction with out-of-home care related services (Barber & Delfabbro, Reference Barber and Delfabbro2005; Holland, Reference Holland2009).
Nevertheless, the basic survey analysis of this study is consistent with the Care Satisfaction Scale designed to assess service satisfaction of parents who have children in out-of-home placement (foster care) (Harris et al., Reference Harris, Poertner and Joe2000). However, unlike Harris’s study, the Take Two Stakeholder Survey was developed without substantial direct input from consumers (parents and children). Hence, replication of the scale development process may be necessary to determine if the items reflect the “clients’ actual concerns instead of professionals’ assumptions of what aspects of services are important to clients” (Harris et al., Reference Harris, Poertner and Joe2000, p. 23).
Within the context of the Take Two program, a number of approaches are used to ensure there is a focus on translating the study findings into service and program improvement. For example, the administration of the Stakeholder Survey to members of the care team and the clients (children) ensures that stakeholders’ feedback is routinely collected and used for service and practice improvement. At a program level, stakeholders’ feedback is discussed at research executive meetings as well as clinical leadership group meetings. Further, individual feedback is sent out to senior clinical team leaders for discussion at team meetings and individual supervision sessions. As the survey collects data from all stakeholders (including teachers and child protection workers), it provides an opportunity for them to provide immediate feedback and for this to be reviewed by senior management and clinicians. These activities and endeavours are likely to contribute to an ongoing modification and refinement of aspects of existing practices, as well as contributing to the enhancement of methods for collecting consumer feedback. For example, a perceived limitation of the current strategy of mail, reply-paid envelope, is that it is limited in gaining more in-depth data or deeper understandings of which aspects of service satisfaction and/or clinical change matter most to consumers. In addition, from a translational point of view, further research is warranted to explore how the collected data are contributing to positive impact for the children and families.
From a review of the current processes and possible gaps between research and practice, an enhanced strategy is proposed to proactively engage consumers and gain feedback through face-to-face semi-structured interviews – facilitated by ex-clients of the Take Two program – in addition to the survey. This strategy of direct interviews is likely to elicit more in-depth data related to both positive and negative experiences of care, such as placement instability and staff changes which may impact consumers’ experience regardless of the care they have received. The engagement of ex-clients to contribute to the interview schedules and undertake the interviews will bring an additional perspective to the gathering of feedback and measuring consumers’ experiences.
Overall, this study, while preliminary and promising, has a number of limitations.
Respondents represent a convenience sample of self-selected stakeholders involved with Berry Street Take Two, hence the findings related to survey feedback may not generalise to other populations. Further, of critical importance, is the substantially low response rate of parents and children. This also needs to be interpreted cautiously, as only 17.59% of all children referred during study period were living with one or both parents. Further, Take Two clients are often involved with statutory services, which may significantly impact on parental and child engagement.
This leaves open the possibility that stakeholders may not have accurately reported their experiences or satisfaction with the Take Two program due to recall and/or social desirability biases. The survey was also administered by mail and/or in person, adding to the methodological limitations of quality data collection. At this stage, it is not possible to estimate the extent of each data collection method. Despite these limitations, the Take Two Stakeholder Survey appears to be a useful measure for collecting consumer feedback, experiences, service satisfaction and perception of clinical change for the child.
Consumer feedback and satisfaction with services is critical to the overall success of social welfare organisations and child-related programs (Kapp & Propp, Reference Kapp and Propp2002). However, Gilbertson and Barber (Reference Gilbertson and Barber2002) identified various obstacles and barriers impeding the systematic collection of consumer feedback within the context of foster care research. Hence, research is urgently required to accurately identify what aids and limits the collection of feedback from care systems, organisations, programs and individuals, while attending to the ethical considerations of interviewing children affected by abuse and neglect. These considerations may inform enhanced methods and approaches for listening to stakeholders, in particular children in care. Client feedback is also important to evidence-based and evidence-informed practice (Tilbury et al., Reference Tilbury, Osmond and Crawford2010). In addition, actively engaging the voices of child clients and other stakeholders through a feedback mechanism is consistent with Take Two’s relational model of intervention and provides a message of empowerment to clients.
In conclusion, listening to the voices of children, their families, carers and members of the service and care system contributes to the rigorous evaluation of any program within the social and child welfare system in Australia. Asking stakeholders, parents and children, what they think of the services and activities in which they are involved is both meaningful and useful as an approach to assess the quality of care and evaluating service effectiveness and impact. Strengthening the capacity to listen carefully to the voices of children involved with the Take Two therapeutic program is a major area for further development in the research design and service delivery. Future directions for research include building upon the current Take Two Stakeholder Survey to explore additional avenues for children to voice to their opinions regarding a range of questions, such as what makes therapy easier to participate in, what they have found welcoming or distancing and what outcomes are important to them that may positively contribute to enhanced quality of life.
Role of funding source
Berry Street Take Two is funded by the Department of Health and Human Services, Victoria.
Author ORCIDs
Margarita Frederico, 0000-0001-9028-7823; Allison Cox 0000-0002-4543-3913
Acknowledgements
We thank the consumers of the Take Two service for their time in providing valuable feedback and the support of Take Two clinicians in assisting with survey distribution. We also thank Jenny Bartlett for overseeing research administrative tasks related to the Take Two Stakeholder Survey.
