Consumer perceptions of care can be used for quality assurance and improvement (Smith et al., Reference Smith, Roche, O'Loughlin, Brennan, Madigan, Lyne and O'Donoghue2014). Assurance that programs are being appropriately delivered provides a basis for evaluating outcomes, and therefore understanding a consumer's perception of their care is an important component of evaluating the way a service delivers treatment. However, consumers’ perception of care is multifaceted. It is therefore important to understand the different components of perceptions of care and how these components relate to one another so that mental health service providers can respond to patient concerns effectively. However, at present, it is unclear as to whether consumer perceptions of treatment outcomes in mental health are primarily related to factors external to individuals (such as judgments about the quality and appropriateness of care received) or internal factors (such as a judgment that treatment has increased a sense of empowerment). A greater understanding of the factors underlying perceptions of care and perceived treatment outcomes can facilitate targeted responses to negative and positive service evaluations (Newbold, Hardy, & Byng, Reference Newbold, Hardy and Byng2013).
Measuring Consumer Perceptions of Care
The Consumer Perceptions of Care Survey (Morris-Yates, Reference Morris-Yates2009; Queensland Health, 2009) is based on the Consumer Surveys developed by the Mental Health and Statistics Improvement Program (MHSIP; Eisen et al., Reference Eisen, Shaul, Leff, Stringfellow, Clarridge and Cleary2001; Jerrell, Reference Jerrell2006). It was developed by Australian mental health facilities for the purposes of quality assurance and improvement. The Consumer Surveys demonstrated good reliability for the three factors of Access (α = .73), Quality and Appropriateness (α = .81), and Perceived Outcomes (α = .79; Jerrell, Reference Jerrell2006), but they did not address patient empowerment, which has since arisen as an important factor in mental health recovery models (Bonney & Stickley, Reference Stickley and Bonney2008). It was also apparent that not all questions were appropriate for the Australian cultural context and private-based psychiatric care. More recently, the Consumer Surveys were reviewed by consumers and carers in Australia and were subsequently adapted into the Consumer Perceptions of Care (CPoC) survey. The CPoC was found to be acceptable to users and appropriate for rating clinical services. The content validity of the CPoC was also found to be acceptable (Morris-Yates, Reference Morris-Yates2009; Queensland Health, 2009). However, despite describing general acceptability (Morris-Yates, Reference Morris-Yates2009), the report did not ascertain the reliability of the instrument, nor the degree to which the items clustered into the three expected domains of Perceived Treatment Outcomes, Quality and Appropriateness, and Empowerment). Hence, one key aim of the current study was to examine the factor structure of the CPoC using structural equation modelling. In the present study, ‘perceived treatment outcomes’ will be used to refer to client self-reported changes in symptoms, daily function and crisis-management, wellbeing, and social function as measured by the CPoC. The ‘quality and appropriateness’ of care refers to perceptions that services were delivered in a professional manner. Finally, ‘empowerment’ will be used to refer to the extent to which individuals feel empowered to take control of their mental health problems.
Empowerment is a concept that has been receiving increased attention within a mental health recovery model (Anderson et al., Reference Anderson, Funnell, Butler, Arnold, Fitzgerald and Feste1995; Arnetz, Almin, Bergström, Franzen, & Nilsson, Reference Arnetz, Almin, Bergström, Franzen and Nilsson2004; Stack & Martin, Reference Stack and Martin2005). According to Berry, Allott, Emsley, Ennion, and Barrowclough (Reference Berry, Allott, Emsley, Ennion and Barrowclough2014), empowerment refers to taking personal responsibility, exerting control over life, and focusing on strengths (see also Berry, Allott, Emsley, Ennion, & Barrowclough, Reference Berry, Allott, Emsley, Ennion and Barrowclough2014), Consumers have been shown to regard being empowered as the most important factor for determining their own recovery (Grealish, Tai, Hunter, & Morrison, Reference Grealish, Tai, Hunter and Morrison2013), and factors such as daily routine, structure, and avoidance of inactivity are seen by consumers as means of increasing empowerment. Mental health providers often aim to facilitate routine, structure, and minimisation of activity, and therefore, the extent to which these are seen as empowering by patients may be associated with improved outcomes in both mental and physical health (Anderson et al., Reference Anderson, Funnell, Butler, Arnold, Fitzgerald and Feste1995). Given the absence of research investigating the measurement properties of the CPoC, it was first necessary to establish the factor structure of the instrument, which in turn would allow examination of the relationships between its expected domains (i.e., quality and appropriateness, empowerment, and perceived treatment outcomes).
Relationship of Perceived Outcomes to Symptom Change
Existing research (Eisen et al., Reference Eisen, Shaul, Leff, Stringfellow, Clarridge and Cleary2001; Jerrell, Reference Jerrell2006) has revealed a strong correlation between quality and appropriateness and perceived treatment outcomes. However, since quality and appropriateness and perceived treatment outcomes are often measured at the same time, an individual's current affective state may disproportionately influence their perceptions of the extent to which their symptoms have actually changed. As such, it is possible that there may be a weak relationship between perceived and actual treatment outcomes. The present study therefore sought to examine the relationship between retrospective patient ratings of treatment outcome on the CPoC and changes in symptoms between admission and discharge.
One barrier to examining the relationship between symptom change and patient perceptions of care is the anonymous manner in which surveys are typically completed. To this end, patients in the present study were given the option of self-identification so that their clinical outcomes could be linked with the CPoC scores, which in turn permitted examination of the construct validity of the CPoC. However, a lack of anonymity in satisfaction surveys can also produce a response bias where patients respond with ‘desirable’ responses if they believe that treating staff might access the data (LeVois, Nguyen, & Attkisson, Reference LeVois, Nguyen and Attkisson1981). Thus, the current study also examined the effects of allowing patients to self-identify on the CPoC on their ratings of perceptions of care.
In summary, the current study had several broad aims. The first aim was to determine the factor structure of the CPoC. It was expected that the CPoC would have three factors: quality and appropriateness of care, perceived treatment outcomes, and empowerment. The second aim of the current study was to understand the relationships between quality and appropriateness of care, perceived treatment outcomes, and empowerment. In addition, the current study examined the relationships between these three constructs (measured at the end of treatment) and actual symptom change, operationalised as the pre-post treatment difference in symptoms. Finally, the current study aimed to determine whether there was a relationship between self-identification on survey responses, such that individuals who self-identified on the CFoC survey would exhibit more favourable perceptions of care than those who remained anonymous.
Method
Participants were patients at an Australian private psychiatric hospital who were invited to complete a CPoC questionnaire upon discharge between July 2010 and August 2012. In the first phase of the study, one group of patients was randomly selected from the total sample and their data was used for an exploratory factor analysis (EFA). The data belonging to the remaining group of patients were used to perform a confirmatory factor analysis (CFA). In the second phase, a subset of the total sample was extracted to examine the relationship between the perceptions of care data and symptom change from admission to discharge. Since these analyses required matching the perceptions of care with the symptom data, it was only possible to use the subgroup of patients who consented to have CPoC responses matched with their existing data.
All participants provided consent for the data to be used for evaluation purposes. The CPoC was offered to all patients upon discharge from hospital. Patients could complete it anonymously or in a manner that allowed the CPoC to be linked to existing clinical data (via a unique record number). If patients chose to self-identify, the CPoC data were linked with the clinical data by hospital staff and any identifying information was removed by the hospital before the database was made available to the research team. This procedure ensured patient confidentiality and was consistent with approval by the university's ethics committee.
Analyses
Exploratory factor analysis (EFA; using SPSS) examined the ways in which items on the CPoC loaded onto the predicted factors. Parallel analysis, which helped to identify the number of factors to interpret in the EFA, was conducted using a syntax script (https://people.ok.ubc.ca/brioconn/nfactors/nfactors.html). Using MPlus Version 7, a CFA was then performed to test the stability of the CPoC factor structure that emerged from the EFA. Finally, to examine the relationships between consumer perceptions of care and symptom change, the relationship of each of the three latent variables comprising the CPoC and symptom change was added to the preceding model.
Participants
Of 3,426 inpatients attending a 100-bed private mental health facility, 2,125 (65%) accepted the invitation to complete the CPoC measure.
Measures
The CPoC consists of 35 items assessing perceptions of care for in-patients in a mental health facility. Each item consists of a declarative statement (e.g., ‘I was treated with dignity and respect’), to which respondents choose a rating from a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).
The 21-item Depression, Anxiety and Stress Scales (S.H. Lovibond, Reference Lovibond1983) is a self-report measure of depression, physical arousal and psychological tension and agitation. The DASS has been found to have excellent reliability and validity in community and clinical samples (Antony, Bieling, Cox, Enns, & Swinson, Reference Antony, Bieling, Cox, Enns and Swinson1998; Page, Hooke, & Morrison, Reference Page, Hooke and Morrison2007). The internal consistency of the Depression (r = .81), Anxiety (r = .73) and Stress (r = .81) scales is high (Lovibond, Reference Lovibond1983), and the stability across time is similarly high for Depression (r = .70), Anxiety (r = .73), and Stress (r = .71; Page et al., Reference Page, Hooke and Morrison2007; Ronk, Korman, Hooke, & Page, Reference Ronk, Korman, Hooke and Page2013).
Procedure
Administration staff invited patients to complete CPoC at discharge. Informed consent was obtained from patients by providing them with the option of completing the survey anonymously, or in a manner that identified them so that their ratings could be matched with their clinical data. Completed CPoC surveys were handed in to a collection box in the clinic's lobby or posted back in a reply-paid envelope. The dataset was the de-identified before analyses began.
Results
Phase 1
Participants were 2,125 inpatients of a 100-bed private mental health facility. Of these participants, 1,063 were randomly selected for the EFA and 1,062 were randomly selected for the CFA. Female patients represented 72.9% of the sample used for EFA and 70.6% of the sample used for CFA. There were no significant differences in the samples used on gender, U(2,016) = 496,620.00, p = .26; age, U(2,050) = 500,021.50, p = .05; and length of stay, U(1,954) = 472,289.50, p = .67. Table 1 displays the descriptive statistics for the EFA and CFA.
Descriptive Statistics for Participants in the EFA, CFA and Self-Identified Datasets
Data screening
The data were screened for univariate outliers. All values were within the stipulated response options.
Reliability
The internal consistency reliability of the CPoC items was found to be excellent (α = .96).
Exploratory Factor Analysis
Factor analysis
The potential for a factor solution was assessed using several criteria. First, 34 of 35 items correlated at least .3 with at least one other item, suggesting reasonable factorability. Second, the Kaiser-Meyer-Olkin measure of sampling adequacy was .94, well above the recommended value of .6, and Bartlett's test of sphericity was significant, χ2(595) = 7,847.22, p < .01. The communalities were all above .3, further confirming that each item shared common variance with other items.
Principal components analysis (PCA) with oblimin rotation was conducted because the goal was to identify factors underlying the CPoC. A three-factor solution, which explained 56.9% of the variance, was selected because parallel analysis (O'Connor, Reference O'Connor2000) indicated this as the optimal solution. A total of nine items were then eliminated because they did not contribute to a simple factor structure and failed to meet a minimum criteria of having a primary factor loading of .4 or above, and no cross-loading of .3 or above. PCA with oblimin rotation was conducted on the remaining 26 items. The three-factor solution represented 58.4% of variance, with factors indicating Quality and Appropriateness, Perceived Treatment Outcomes, and Empowerment. All items except two (‘Opportunity to talk with staff prior to discharge’ and ‘My contact with my Doctor was helpful’) had primary loadings over .40 (‘Both I and staff were actively involved in the treatment plan’ had a primary loading on the fourth factor, which was not interpreted). Only two items had cross-loadings over .30: ‘Staff believed I could grow, change and recover’ and ‘Both I and staff were actively involved in the treatment plan’. The CPoC items, factor loadings and communalities of the final solution for the EFA is shown in Table 2.
Factor Loadings of a Principle Components Analysis With Oblimin Rotation Conducted on the Consumer Perceptions of Care Survey
Note: Q & A = Quality and Appropriateness, Out = Perceived Outcomes, Emp = Empowerment. Factor loadings >.3 are bolded. Numbers in parentheses indicate CPoC item numbers.
Confirmatory Factor Analysis
To examine the replicability of the factor solution that emerged from the EFA, a CFA was performed on the second subsample of inpatients. The chi-square test of the model was statistically significant, with a value of 1,056.81, p < .001, but this value divided by the degrees of freedom resulted in a value of 4.66, indicating an acceptable fit (Marsh & Hocevar, Reference Marsh and Hocevar1985). Furthermore, the model yielded good CFI (.93; Hu & Bentler, Reference Hu and Bentler1999) and acceptable RMSEA (.06; Hu & Bentler, Reference Hu and Bentler1999). A review of the modification indices indicated no necessary adjustments to the model.
The Relationship Between CPoC Domains
There were significant correlations (p < .001) between all CPoC domains. Perceived Treatment Outcomes were positively correlated with both Quality and Appropriateness (r = .52) and Empowerment (r = .64). The correlation between Quality and Appropriateness and Empowerment was the largest (r = .86).
Phase 2
Of the 2,125 participants from Phase 1, 720 provided informed consent to have their CPoC responses linked with other clinical data. Female patients represented 65.1% of self-identified participants. Participants who self-identified were more likely to be older, U(1,003) = 92,267.00, p = .003, and to have a longer stay, U(957) = 88,736.50, p = .04. The primary diagnoses of participants were as follows: 57.1% affective disorder, 19.5% neurotic disorder, 13.0% substance disorder, 4.9% schizophrenia, 4.1% adult personality disorder, and 1.4% other.
The Effect of Self-Identification on CPoC Responses
Independent-sample t tests were conducted to compare the perceptions of care of self-identifiers (n = 310) with non-identifiers (n = 797) across the three scores on the CPoC. Self-identifiers had consistently higher scores than non-identifiers for Quality and Appropriateness (t = 5.30, p < .001), Perceived Treatment Outcomes (t = 4.47, p < .001), and Empowerment (t = 3.64, p < .001). However, the effect size of self-identifying was small (d = .23 to .28; Figure 1).
Comparison of assessments of Quality and Appropriateness, Perceived Outcomes, and Empowerment by self-identifiers and non-identifiers.
The Relationship Between Quality and Appropriateness, Perceived Treatment Outcomes, and Empowerment
The relationships between Quality and Appropriateness, Perceived Treatment Outcomes, and Empowerment were examined in a structural equation model. As displayed in Figure 2, the model exhibited a good fit to the data. The value for the chi-square test of model fit divided by the degrees of freedom was 2.63, and CFI of .91 and RMSEA of .06 were acceptable (Hu & Bentler, Reference Hu and Bentler1999; Marsh & Hocevar, Reference Marsh and Hocevar1985). An examination of the standardised path coefficients from Quality and Appropriateness to Empowerment, r = .82, p < .05, and from Empowerment to Perceived Treatment Outcomes, r = .42, p < .05, were significant, while the path from Quality and Appropriateness (not shown) to Perceived Treatment Outcomes was non-significant, r = 16, p > .05. Thus, perceived quality of care did not have a direct effect on perceived treatment outcomes at the end of treatment, but quality and appropriateness had an effect on outcomes via empowerment.
The structural diagram of the CFI with Quality and Appropriateness (quality), Perceived Outcomes (outcomes) and Empowerment (empower) as latent variables, and their standarised coefficients. Only significant paths are shown.
A final CFA was conducted to evaluate the effect of change in symptomology on the three CPoC domains (Figure 3). The value for the chi-square test of model fit divided by the degrees of freedom was 2.33, and CFI of .97 and RMSEA of .04 were acceptable (Hu & Bentler, Reference Hu and Bentler1999; Marsh & Hocevar, Reference Marsh and Hocevar1985), indicating good model fit. The key points resulting from the model are that the (1) relationships in Figure 2 were largely unchanged and (2) change in depression scores from admission to discharge was correlated with Perceived Quality of Care and Perceived Treatment Outcomes, but not Empowerment. That is, ‘symptom change’ was operationalised as the difference between the sum of DASS Scales (P.F. Lovibond & Lovibond, Reference Lovibond and Lovibond1995) scores at admission and discharge. Perceived Treatment Outcomes (r= -.20; p < .001) and Empowerment (r = -.15; p < .05) were correlated with the change in DASS scores from admission to discharge to a small extent, indicating that smaller symptom improvement on the DASS was associated with lower levels of Empowerment and lower ratings of Perceived Treatment Outcomes.
The structural diagram of the CFI with Quality and Appropriateness (quality), Perceived Outcomes (outcomes), Empowerment (empower) and Change in Symptomology (DASS) as latent variables, and their standardised coefficients. Only significant paths are shown.
Discussion
The first aim of the current study was to understand the structure of the CPoC survey. A confirmatory factor analysis revealed a three-factor structure consisting of Quality and Appropriateness, Perceived Treatment Outcomes, and Empowerment. While the CPoC factor structure identified in the current study was broadly similar to that proposed by the MHSIP Consumer Surveys, the key difference is that the MHSIP identified an access factor (Eisen et al., Reference Eisen, Shaul, Leff, Stringfellow, Clarridge and Cleary2001; Jerrell, Reference Jerrell2006), rather than empowerment. One reason for this difference could be that the current study involved private psychiatric patients in Australia who were able to access and choose between both private and public mental health services. It is therefore possible that access may be more of an issue for patients in the public healthcare system and thus, access could arise as a factor accordingly. Given the nature of the current sample, future research could examine the extent to which the factor structure of the CPoC identified in the current study can be generalised to those who are unable to access private mental health facilities.
The second aim of the study was to investigate the relationship between quality and appropriateness of care, perceived treatment outcomes, and empowerment. Empowerment was more strongly related to perceived treatment outcomes than quality and appropriateness. Furthermore, there was no significant relationship between quality and appropriateness and perceived treatment outcomes. However, the paths from quality and appropriateness to empowerment, and empowerment to perceived treatment outcomes were both significant. These results are consistent with a growing literature that supports the value of addressing patient empowerment as a means to increase perceived outcomes. The finding that patient empowerment is positively associated with perceptions of care provides support for mental health recovery models that point to empowerment as a key factor in recovery from mental health difficulties (Anderson et al., Reference Anderson, Funnell, Butler, Arnold, Fitzgerald and Feste1995; Arnetz et al., Reference Arnetz, Almin, Bergström, Franzen and Nilsson2004; Stack & Martin, Reference Stack and Martin2005). Furthermore, Yanos, Primavera, and Knight (Reference Yanos, Primavera and Knight2014) note that treatments that increase feelings of self-efficacy not only improve outcomes, but may also confer other benefits such as improving social functioning (Yanos et al., Reference Yanos, Primavera and Knight2014). In addition, mental health services aiming to foster empowerment among mental health patients through their programs have observed reductions in self-stigma, which in turn may also increase self-efficacy (Segal, Silverman, & Temkin, Reference Segal, Silverman and Temkin2013). Given the benefits of fostering patient empowerment on treatment outcomes, mental health services should aim to develop initiatives targeted at improving individuals’ feelings of empowerment.
The current study also examined whether ratings of quality and appropriateness, perceived treatment outcomes, and empowerment were related to symptom change from admission to discharge. Although perceived treatment outcomes and empowerment were both found to be positively correlated with patients’ actual symptom change, these correlations were small. This finding suggests that patients’ retroactive assessments of treatment outcomes are not necessarily the same as their actual symptom changes over the course of treatment, and that examining patients’ perceived and actual treatment outcomes may provide mental health service providers with a more nuanced perspective of the hospital experiences of their patients.
The current study also examined the influence of self-identification on the CPoC on individuals’ ratings of perceptions of care. Consistent with previous findings, the present study found that patients who did not self-identify had significantly lower satisfaction scores (LeVois et al., Reference LeVois, Nguyen and Attkisson1981). However, the effect of self-identification was marginal. Given the small effect of self-identification on patient ratings of perceptions of care, mental health facilities may find it worthwhile to give patients the option to self-identify in surveys so that relationships between key patient variables and perceptions of care can be investigated. For example, a facility may wish to investigate which treatment programs or patient diagnoses are associated with higher satisfaction.
A limitation of the current study was that the sample consisted solely of patients at a single-site private mental health facility. Future studies should look at incorporating patients at government facilities and across several sites to determine whether the factorial structure of the CPoC that was identified in this study can be generalised to individuals who are not patients at private mental health facilities. Furthermore, although the ratings of perceived treatment outcomes reflected symptoms as well as several other domains (e.g., Wellbeing), the only assessment of actual ‘outcomes’ against which perceived treatment outcomes was compared was change in self-reported symptoms. Thus, relying on symptoms as the only outcome measure limits the conclusions that can be supported by the present study. Since the measure of perceived treatment outcomes in the CPoC encompassed not only changes in symptoms, but also daily function and crisis-management, wellbeing, and social functioning, it is possible that the observed correlation between perceived treatment outcomes and symptom change may have underestimated the strength of the relationship between perceived and actual treatment outcomes. Future research could use a broader assessment of functioning (e.g., the 36-item Short Form Healthy Survey; Brazier et al., Reference Brazier, Harper, Jones, O'cathain, Thomas, Usherwood and Westlake1992) to examine the pre-post treatment change to explore this relationship more fully. The inclusion of staff-rated measures (e.g., HoNOS; Newnham, Harwood, & Page, Reference Newnham, Harwood and Page2009) may also clarify the extent to which the relationships observed between self-report indices generalise to observations of treatment teams. A further limitation of the present study relates to the cross-sectional nature of the data. Since the consumer perceptions were all obtained post treatment, it was not possible to determine the causal direction of the relationships between quality and appropriateness, perceived treatment outcomes and empowerment. Future research could measure perceptions of constructs such as empowerment during treatment and examine the effects on outcomes. Such analyses would better address the causal issue than the present data, which can only point to the relationships between the consumers’ perceptions.
In conclusion, the current study examined the factor structure of the CPoC and the relationships between its three domains (quality and appropriateness, perceived treatment outcomes, and empowerment). Empowerment was found to influence perceived treatment outcomes to a greater extent than quality and appropriateness, which indicates the potential importance of enhancing patients’ feelings of empowerment in services and programs provided by mental health facilities. Furthermore, the low correlation between perceived treatment outcomes and symptom change suggests that examining both patients’ perceived and actual treatment outcomes may be a useful way for mental health service providers to accurately capture the hospital experiences of their patients. Finally, the small effect of self-identification on ratings of consumer perceptions of care indicates that while non-identification may provide a greater likelihood of survey completion and more honest appraisals, allowing patients to self-identify on surveys should be considered in the context of greater potential for data interrogation.
