Introduction
Patient-reported outcomes (PROs) have become increasingly important in the evaluation of treatment for patients with psychosis. A PRO can be defined as ‘any report coming directly from patients (i.e. study subjects) about a health condition and its treatment’ (FDA, 2006). PRO measures can be used to assess the impact of an intervention on one or more aspects of patients' health status, hereafter referred to as PRO concepts. The term ‘PRO’ has been used in an increasingly inclusive way, referring not only to purely symptomatic outcomes but also to more complex multidomain concepts such as subjective quality of life (SQOL), needs for care, treatment satisfaction, or the quality of the therapeutic relationship. For measures of multidomain concepts, a conceptual framework is generally used, in which items (e.g. satisfaction with physical health) are grouped within domains (e.g. health), and domains within more general PRO concepts (e.g. SQOL). Research evaluating treatment benefits for patients with psychosis has drawn extensively on PROs (McCabe et al. Reference McCabe, Saidi and Priebe2007). Regulatory agencies have also proposed including well-validated PROs as effectiveness end-points in randomized controlled trials (EMEA, 2005; FDA, 2006). At present, in the UK, service providers are expected to use PROs for assessing the quality of routine care (DoH, 2008, 2009).
When assessing treatment benefits for patients through patient-reported measures several distinct outcomes often seem to be relevant. However, using several measures at the same time raises the problem of multiple statistical testing and is associated with an increased burden to respondents and higher study costs. An explicit and theoretically informed choice of which outcome measures are most appropriate to the evaluation of a specific intervention is therefore required (Altman et al. Reference Altman, Schulz, Moher, Egger, Davidoff, Elbourne, Gøtzsche and Lang2001; Moher et al. Reference Moher, Schulz and Altman2001). Empirically, previous studies have reported a considerable overlap of measures designed to assess different outcomes. In these reports, PROs were highly correlated and a single general factor explained more than half of the variance in SQOL, needs for care, treatment satisfaction, and self-rated symptom scores (Priebe et al. Reference Priebe, Kaiser, Huxley, Roder-Wanner and Rudolf1998; Fakhoury et al. Reference Fakhoury, Kaiser, Roeder-Wanner and Priebe2002; Hansson et al. Reference Hansson, Bjorkman and Priebe2007). This general factor has been interpreted as a ‘general appraisal tendency’ (hereafter ‘GAT’) of patients for positive or negative ratings across measures. Findings may, however, also reflect an overlap in specific life or care domains such as the patients' health, living situation, or the accessibility of services (hereafter ‘shared domain content’) at a lower level of generality than established PRO concepts (Floyd & Widaman, Reference Floyd and Widaman1995; Salvi et al. Reference Salvi, Leese and Slade2005). A high degree of overlap in responses to items may considerably impair the ability of each measure to capture distinct information, which in psychometrics is referred to as ‘discriminant validity’ (Campbell & Fiske, Reference Campbell and Fiske1959). It may also affect the extent to which PRO scores are an adequate reflection of the dimensionality of the concept to be measured, commonly referred to as ‘structural validity’ (Mokkink et al. Reference Mokkink, Terwee, Knol, Stratford, Alonso, Patrick, Bouter and de Vet2006). More generally, this overlap makes an explicit and theoretically informed choice of the most appropriate measure difficult when evaluating specific interventions for patients with psychosis.
Previous research into the overlap of PROs has been methodologically limited, for example by examining the overlap as accounting for covariance among sum scores rather than item responses, making it difficult to draw accurate conclusions. Previous reports have also failed to assess the extent to which established measures still may capture specific variance independent from this overlap. A better understanding of the distinct drivers of covariance among item responses would help to increase the discriminant and structural validity of established PROs and justify their inclusion in treatment evaluations. Indeed, it is only recently that increasing attention has been paid to the role of a bifactor model in resolving dimensionality issues in health outcome measurement (Gibbons & Hedeker, Reference Gibbons and Hedeker1992; Gibbons et al. Reference Gibbons, Bock, Hedeker, Weiss, Segawa, Bhaumik, Kupfer, Frank, Grochocinski and Stover2007, Reference Gibbons, Weiss, Kupfer, Frank, Fagiolini, Grochocinski, Bhaumik, Stover, Bock and Immekus2008; Reise et al. Reference Reise, Morizot and Hays2007; Yang et al. Reference Yang, Tommet and Jones2009). This bifactor model recognizes that patients' responses to an item depend on a single general factor that explains covariance among all item responses and also, independently, on specific factors that only account for responses to items of particular life or care domains. This statistical property seems to be particularly relevant in complex measurement situations when ‘broad’ concepts with content heterogeneous items are to be measured (Reise et al. Reference Reise, Morizot and Hays2007). In the context of assessing multiple correlated PROs in psychosis, the bifactor structure provides an opportunity to disentangle concept-specific variance from overlap due to a GAT and/or shared domain content.
Against this background, we set out to investigate the extent to which four widely established PROs (SQOL, needs for care, treatment satisfaction, and the therapeutic relationship) provide distinct information in patients with psychosis independent from overlap across measures. Specifically, we aimed to examine whether the overlap in the information provided by different measures: (i) allows for formation of distinct PRO scales that discriminate precisely between patients from lower and higher levels of each PRO (discriminant validity); and (ii) affects the extent to which previously proposed PRO scores are an adequate reflection of the dimensionality of the concept to be measured (structural validity).
Method
Participants
The samples were taken from two multicentre randomized controlled trials, the UK700 (Burns et al. Reference Burns, Creed, Fahy, Thompson, Tyrer and White1999) and DIALOG (Priebe et al. Reference Priebe, McCabe, Bullenkamp, Hansson, Lauber, Martinez-Leal, Rossler, Salize, Svensson, Torres-Gonzales, van den Brink, Wiersma and Wright2007) studies. Patients in the UK700 sample (n=708) were between 18 and 65 years old (mean=38.3, s.d.=11.6), predominantly male (n=404, 57.1%) and mostly unemployed (n=629, 88.8%). They were recruited between February 1994 and April 1996 from four UK inner-city mental health services in London and Manchester. Most patients had a diagnosis of schizophrenia (n=270, 38.1%) or schizo-affective disorders (n=345, 48.7%). As in the UK700 sample, patients in DIALOG (n=507) were between 18 and 65 years old (mean=42.2, s.d.=11.5), predominantly male (n=336, 66.3%) and mostly unemployed (n=427, 84.2%). The DIALOG sample was recruited between December 2002 and May 2005 from community psychiatric services in Granada (Spain), Groningen (The Netherlands), London (UK), Lund (Sweden), Mannheim (Germany) and Zurich (Switzerland) covering urban and mixed urban–rural areas. DIALOG patients were mostly diagnosed with schizophrenia (n=354, 69.8%). The median duration of illness in years was slightly higher in the DIALOG [median 14, interquartile range (IQR) 7–23] than the UK700 sample (median 10, IQR 5–18). The data presented here are the assessments made at baseline in both intervention and control arms. More detailed information on the UK700 and DIALOG studies is available in Burns et al. (Reference Burns, Creed, Fahy, Thompson, Tyrer and White1999) and Priebe et al. (Reference Priebe, McCabe, Bullenkamp, Hansson, Lauber, Martinez-Leal, Rossler, Salize, Svensson, Torres-Gonzales, van den Brink, Wiersma and Wright2007).
PRO measures
SQOL was measured using the Lancashire Quality of Life Profile (LQOLP; Oliver et al. Reference Oliver, Huxley, Priebe and Kaiser1997) in the UK700 sample, and its short version, the Manchester Short Assessment of Quality of Life (MANSA; Priebe et al. Reference Priebe, Huxley, Knight and Evans1999), in the DIALOG sample. The LQOLP was based on Lehman's approach, operationalizing SQOL as satisfaction with life in general and in major life domains (Lehman, Reference Lehman1996). LQOLP and MANSA contain 24 and 12 items respectively, asking patients to rate their satisfaction with life in general and several life domains on a Likert-type scale from ‘couldn't be worse’ (rating of 1) to ‘couldn't be better’ (rating of 7). Priebe et al. (Reference Priebe, Huxley, Knight and Evans1999) reported good convergent validity for the LQOLP and MANSA.
The number of unmet needs for care was assessed using the Camberwell Assessment of Need (CAN), patient-rated version (Phelan et al. Reference Phelan, Slade, Thornicroft, Dunn, Holloway, Wykes, Strathdee, Loftus, McCrone and Hayward1995), in both samples. The CAN assesses health and social needs across 22 domains. Each domain is rated on a three-point scale distinguishing between ‘no need’ (rating of 0), ‘met need’ (rating of 1) and ‘unmet need’ (rating of 2). Unmet needs for care ratings were reverse coded to achieve consistency in the coding direction across all PROs.
In the UK700 sample, treatment satisfaction was measured using the Patient Satisfaction Questionnaire (PSQ; Tyrer & Remington, Reference Tyrer and Remington1979). The PSQ asks patients to rate nine care domains of satisfaction with services each on a four-point scale (ranging from 1 to 4). The Client Satisfaction Questionnaire (CSQ; Nguyen et al. Reference Nguyen, Attkisson and Stegner1983) was used for assessing treatment satisfaction in the DIALOG sample. The CSQ consists of eight items rated from 1 to 4 (with higher scores indicating greater treatment satisfaction).
A measure of the therapeutic relationship, the Helping Alliance Scale (HAS; Priebe & Gruyters, Reference Priebe and Gruyters1993), was included only in the DIALOG sample. The HAS comprises five items rated on a visual analogue scale ranging from 0 (‘not at all’) to 10 (‘extremely well’).
Statistical analysis
Parameter estimation and model fit
To examine the dimensionality of the four PROs, analyses based on item response modelling were performed using statistical methods appropriate for ordinal item responses. Model estimation used the robust weighted least squares means and variance adjusted (WLSMV) estimator in MPlus, version 5.2 (Muthén & Muthén, 1998–Reference Muthén and Muthén2009). The WLSMV estimator has been found to be robust to violations of the assumption of underlying normality and to provide asymptotically unbiased modified standard errors for examining model fit (Flora & Curran, Reference Flora and Curran2004). It returns coefficients from a probit-probit item factor model equivalent to the two-parameter normal item response theory (IRT) model extended to polytomous items.
The overall model fit of the latent variable models was assessed by computing the root mean square error of approximation (RMSEA; Steiger, Reference Steiger1990), the Comparative Fit Index (CFI; Bentler, Reference Bentler1990) and the Tucker Lewis Index (TLI; Tucker & Lewis, Reference Tucker and Lewis1973). A good model fit is generally indicated by a low RMSEA (<0.10 for acceptable and <0.05 for very good fit; Browne & Cudeck, Reference Browne, Cudeck, Bollen and Long1993) and a high CFI and TLI (>0.90 for acceptable and >0.95 for very good fit; Muthén, Reference Muthén1989; Bentler, Reference Bentler1990).
Model building
Path diagrams of the five alternative latent variable models that were estimated to examine the extent to which different PROs provide distinct information are shown in Fig. 1. Model 1 denotes a unidimensional model with the general factor explaining covariance among all item responses, which can be interpreted as a GAT of patients for positive or negative ratings across measures. Model 2 is a multidimensional model with distinct but correlated concept factors for each PRO scale, that is, SQOL, needs for care, treatment satisfaction and the therapeutic relationship. Model 3 refers to a bifactor model with a general factor independent from uncorrelated concept factors. Model 4 represents a bifactor model with one general factor and several uncorrelated domain factors to account for shared domain content across measures. Model 5 denotes a bifactor model with a general factor, uncorrelated concept factors, and several uncorrelated domain factors (Reise et al. Reference Reise, Morizot and Hays2007). Factors in the bifactor models were specified as uncorrelated to assess the independence of distinct concept versus general and domain factors. Domain factors included into models 4 and 5 were specified as equivalent as possible across the two study samples for LQOLP, MANSA and CAN. More specifically, domain factors on health (D1), socio-economic status (D2), leisure (D3), living situation (D4), friends and intimate relationships (D5) and safety (D6) were included for LQOLP, MANSA and CAN in both samples. Domain factors on religion (D8) and family (D9) were included in models 4 and 5 in the UK700 sample only to account for domains specific to the LQOLP that were not covered by more than one item in the MANSA. Additionally, a domain factor on accessibility of services (D7) was specified for shared domain content of PSQ and LQOLP in the UK700 sample and, similarly, for shared domain content of CSQ and HAS in the DIALOG sample.
Path diagrams of five alternative latent variable models, compared to examine the extent to which different patient-reported outcomes (PROs) provide distinct information. Notation: □, items (observed variables); ○, latent factors (unobserved variables); →, loadings of items onto latent factors; G, general factor; C, concept factor; D, domain factor; C1, subjective quality of life (SQOL); C2, unmet needs for care (reversed); C3, treatment satisfaction; C4, therapeutic relationship; Dx, domain factor (example) accounting for shared domain content across measures; model 1, unidimensional model with one general factor; model 2, multimensional model with correlated concept factors; model 3, bifactor model with general and concept factors; model 4, bifactor model with general and domain factors; model 5, bifactor model with general, concept and domain factors.
Model comparison tests
The five alternative latent variable models were compared on the basis of model fit of each model to the sample data, magnitude of factor loadings and scale information functions. Comparison of model fit indices across models was aimed at testing: first, whether there was any overlap in the information across measures as represented by general and domain factors (model 2 versus models 1, 3, 4 or 5); second, whether the overlap was only due to a GAT or additionally accounted for by shared domain content (models 1 and 3 versus 4 and 5); and third, whether there were concept factors independent from the overlap to allow for formation of distinct PRO scales (models 1 and 4 versus 2, 3 and 5). These comparisons were probed further in a sensitivity analysis to investigate whether each PRO had a sufficiently strong concept factor by comparing models 2, 3 and 5 to reduced models (models 2r, 3r and 5r respectively), in which each PRO concept factor was omitted in turn. We used ▵χ2 tests to assess whether models 2, 3 and 5 better matched the sample data than the reduced models.
Factor loadings were computed to investigate the ability of items to discriminate between patients from lower and higher PRO levels (Reise et al. Reference Reise, Morizot and Hays2007). Total scale information functions, defined as the inverse of measurement error, were calculated based on standard item response Fisher information formulae to assess measurement precision across the full range of each PRO scale (Embretson & Reise, Reference Embretson and Reise2000).
Finally, total scores were computed according to the published version of each PRO measure. These were then regressed on the respective latent concept factors adjusted for the general factor in the best-fitting model using structural equation modelling. This step in the analysis aimed to assess the structural validity of previously proposed scoring methods, that is, the extent to which these simpler scores are an adequate reflection of the dimensionality of the concept to be measured (Mokkink et al. Reference Mokkink, Terwee, Knol, Stratford, Alonso, Patrick, Bouter and de Vet2006).
Results
Descriptive statistics and correlations
Descriptive statistics and correlations of total PRO scores in the UK700 and DIALOG sample are summarized in Table 1. The mean, standard deviation and range of total PRO scores were largely similar across samples. There were highly significant correlations of weak to moderate magnitude among total PRO scores.
Descriptive statistics and mutual correlations of total PRO scores in the UK700 and DIALOG samples
PRO, Patient-reported outcome; LQOLP, Lancashire Quality of Life Profile; CAN, Camberwell Assessment of Needs; PSQ, Patient Satisfaction Questionnaire; MANSA, Manchester Short Assessment of Quality of Life; CSQ, Client Satisfaction Questionnaire; HAS, Helping Alliance Scale; s.d., standard deviation; CI, confidence interval.
Formation of PRO scales independent from overlap
Table 2 shows that a poor model fit was found for model 1 with one general factor in the UK700 sample. Fit was also poor for models 2, 3 and 4. By comparison, a bifactor model with one general factor, three concept and nine domain factors provided a good model fit (model 5). A similar pattern was evident in the DIALOG sample, in which the best-fitting model was also model 5. This model matched the sample data better than a unidimensional model (model 1), a multidimensional model with four correlated concept factors (model 2), a bifactor model with one general and four concept factors (model 3), and a bifactor model with one general and seven domain factors (model 4). Sensitivity analyses showed that inclusion of each PRO concept factor significantly improved model fit.
Model fit statistics for unidimensional, multidimensional and bifactor models in the UK700 and DIALOG samples
CFI, Comparative Fit Index; TLI, Tucker Lewis Index; RMSEA, root mean squared error of approximation; model 1, unidimensional model with one general factor; model 2, multimensional model with correlated concept factors; model 3, bifactor model with general and concept factors; model 4, bifactor model with general domain factors; model 5, bifactor model with general, concept and domain factors.
a Sensitivity analysis using ▵χ2 tests to assess whether model 5 improved model fit in comparison with models 5r, that is reduced models with each patient-reported outcome (PRO) concept factor omitted in turn: UK700 sample: subjective quality of life (SQOL) (▵χ2=77.63, p<0.001), unmet needs for care (▵χ2=48.90, p <.001), and treatment satisfaction (▵χ2=109.82, p<0.001); DIALOG sample: SQOL (▵χ2=69.79, p<0.001), unmet needs for care (▵χ2=138.34, p<0.001), treatment satisfaction (▵χ2=26.29, p<0.001), and therapeutic relationship (▵χ2=104.68, p<0.001).
In both samples, there were common features to the model results. There was overlap in the information provided by the different measures (model 2 versus 5), and there were sufficiently strong concept factors to allow for formation of distinct PRO scales (models 1 and 4 versus 5). In addition, we found that the overlap was due not only to a GAT but also to shared domain content (models 1 and 3 versus 5).
Impact of overlap on discriminative ability of items
Factor loadings of model 5 in the UK700 and DIALOG sample are summarized in Tables 3 and 4, respectively. For most PSQ, CSQ and HAS items, factor loadings of λ ⩾0.35 were observed for the concept factors. For a large number of SQOL and needs for care items, we found factor loadings of λ ⩽0.35, indicating that the ability to discriminate between patients from lower and higher PRO levels was markedly impaired by the overlap.
Standardized factor loadings of LQOLP, CAN and PSQ items in the bifactor model with uncorrelated general, concept and domain factors (model 5) in the UK700 sample
LQOLP, Lancashire Quality of Life Profile; CAN, Camberwell Assessment of Needs; PSQ, Patient Satisfaction Questionnaire; model 5, bifactor model with general (G), concept (C) and domain (D) factors; C1, subjective quality of life (SQOL); C2, unmet needs for care (reversed); C3, treatment satisfaction; D1, health; D2, socio-economic status; D3, leisure; D4, living situation; D5, friends and intimate relationships; D6, safety; D7, accessibility of services; D8, religion; D9, family.
Standardized factor loadings of MANSA, CAN, CSQ and HAS items in the bifactor model with uncorrelated general, concept and domain factors (model 5) in the DIALOG sample
MANSA, Manchester Short Assessment of Quality of Life; CAN, Camberwell Assessment of Needs; CSQ, Client Satisfaction Questionnaire; HAS, Helping Alliance Scale; model 5, bifactor model with general (G), concept (C) and domain (D) factors; C1, subjective quality of life (SQOL); C2, unmet needs for care (reversed); C3, treatment satisfaction; C4, therapeutic relationship; D1, health; D2, socio-economic status; D3, leisure; D4, living situation; D5, friends and intimate relationships; D6, safety; D7, accessibility of services.
In both samples, factor loadings of λ ⩾0.35 were found for more than half of the SQOL and needs for care items on the domain factors of leisure (D3), living situation (D4) and friends and intimate relationships (D5). This was also observed for the domain factors of religion (D8) and family (D9) in the UK700 sample and socio-economic status (D2), safety (D3) and accessibility of services (D7) in the DIALOG sample. In the UK700 sample, less than half of the items loaded ⩾0.35 on the general factor. However, most items loaded ⩾0.35 on this factor in the DIALOG sample. Those items loading ⩾0.35 on the general and/or domain factors in addition to the concept factor were largely as discriminating on the concept as on the general or domain factors, with only a few items being more than twice as informative on general, concept or domain factors (i.e. UK700: LQOLP13, LQOLP15, CAN20; DIALOG: CAN6).
Measurement precision after adjustment for overlap
Scale information functions for concept factors in model 5 are shown in Fig. 2. In both samples, there was a concentration of information coverage at particular points of PRO scales. Patients in the UK700 sample could be scaled precisely around the mean of the SQOL scale only. Information coverage was largely low for needs for care and treatment satisfaction factors and concentrated around the mean in this sample. By comparison, information coverage was mostly higher for concept factors in the DIALOG than the UK700 sample. There was a concentration of information coverage in the more positive range of PRO scales in the DIALOG sample.
Scale information functions of concept factors in model 5. The line charts represent the scale information function (i.e. the inverse of measurement error, y axis) across the range of measurement scale from −6 s.d. below the mean to +6 s.d. above the mean (x axis). Model 5, bifactor model with general, concept and domain factors; SQOL, subjective quality of life.
Associations between latent factors and sum scores
Findings on the relationship of latent factors and sum scores are presented in Table 5. Although significant associations were observed for distinct concept factors and sum scores while controlling for the general factor, only the effect of the treatment satisfaction factor on PSQ sum scores in the UK700 sample was >0.90. Sum scores were markedly affected by the GAT, as indicated by strong associations of sum scores with the general factor.
Regression of sum scores on latent factors in best-fitting model (model 5)
LQOLP, Lancashire Quality of Life Profile; CAN, Camberwell Assessment of Needs; PSQ, Patient Satisfaction Questionnaire; MANSA, Manchester Short Assessment of Quality of Life; CSQ, Client Satisfaction Questionnaire; HAS, Helping Alliance Scale; CI, confidence interval; model 5, bifactor model with general (G), concept (C) and domain factors; C1, subjective quality of life (SQOL); C2, unmet needs for care (reversed); C3, treatment satisfaction; C4, therapeutic relationship.
Discussion
Main findings
This is the first study to report the extent to which four widely established PROs can provide distinct information in patients with psychosis. Analyses, based on item response modelling, yielded consistent findings in two large samples. First, a bifactor model best matched the data in both samples, suggesting that PROs can be assessed independently from overlap across measures. This overlap was found to be due to both a GAT and shared domain content. Second, the ability of items to discriminate between patients across PRO levels was largely unaffected by the overlap for measures of treatment satisfaction and the therapeutic relationship. By comparison, SQOL and needs for care items were markedly more impaired in their discriminative ability. Third, findings were complemented by evidence that, after accounting for the GAT and shared domain content, individuals could not be scaled precisely through the full range of each PRO. Fourth, findings on the relationship of latent factors and sum scores suggested that, taking the overlap into account, sum scores did not adequately reflect the dimensionality of the concept to be measured.
Methodological considerations
Patients included in each study were not randomly selected to represent all patients in the given service and were recruited in the context of a randomized controlled trial. Selection biases might have influenced PRO ratings and findings may not be readily generalizable to all patients with severe and enduring psychosis in routine mental health care.
We sought to determine the replicability of findings by fitting alternative latent variable models in two independent samples (Cudeck & Browne, Reference Cudeck and Browne1983). However, factor loadings and scale information functions varied to some extent across samples. For the only fully identical measure in both samples (i.e. the CAN), differences in item difficulties, large standard errors and a better match of item difficulties and latent PRO level may have accounted for the differences in scale information functions. However, 95% confidence intervals (CIs) still overlapped across the two samples, and therefore, in larger samples, estimates of these parameters may have converged (Tsutakawa & Johnson, Reference Tsutakawa and Johnson1990; Bjorner et al. Reference Bjorner, Chang, Thissen and Reeve2007). Differences in findings may also represent actual differences in the psychometric qualities of the not fully identical measures. For instance, the concentration of information around the mean for the PSQ as compared to high information coverage in the upper range for the CSQ may reflect differences in the precision of these measures (Embretson & Reise, Reference Embretson and Reise2000). Based on these findings, calibrating PSQ and CSQ items into a single scale might increase information coverage across the range of treatment satisfaction levels.
When considered in the context of parsimony, the model that best matched the sample data included more freely estimated parameters than the alternative models considered in this study. In psychometrics, more parsimonious models are commonly considered preferable (James et al. Reference James, Mulai and Brett1982). However, according to the RMSEA, an index sensitive to the number of freely estimated parameters (Steiger, Reference Steiger1990), the bifactor model still matched the sample data best. We would also conclude that it is the conceptual breadth and heterogeneity currently seen in PRO measurement that inevitably require less parsimonious models. Whether or not this heterogeneity is conceptually justified remains to be established (i.e. whether definitions of established PROs are sufficiently distinct to warrant them being measured in a single study).
Comparisons with previous research
There is a wealth of research into PROs in psychiatry. Numerous PRO measures have been developed since the late 1970s when they became increasingly relevant to capture the impact of deinstitutionalization and new psychopharmacological treatments (Kilian & Angermeyer, Reference Kilian and Angermeyer1999). The psychometric qualities of PROs in patients with psychosis have, however, rarely been studied considering more than one outcome at a time and using rigorous psychometric methodology. Those studies examining several PROs have consistently found a considerable overlap of measures. These studies emphasized the role of a GAT of patients for positive or negative ratings (Priebe et al. Reference Priebe, Kaiser, Huxley, Roder-Wanner and Rudolf1998; Fakhoury et al. Reference Fakhoury, Kaiser, Roeder-Wanner and Priebe2002; Hansson et al. Reference Hansson, Bjorkman and Priebe2007). They were, however, methodologically limited and did not account for half of the, potentially concept-specific, variance that remained unexplained. In addition, they identified the problem of the overarching impact of the GAT on different measures without showing a way to advance the methodology of PROs to overcome the problem. Our study has gone a step further. Drawing on recent advances in psychometrics, the bifactor model has provided an approach to consider the GAT and also identify the distinct information provided by four widely established PROs.
Echoing previous reports, we found evidence that PROs are influenced by the GAT. Although this tendency needs to be accounted for when assessing distinct outcomes in psychosis, this finding can also be interpreted in the context of recent efforts to reduce multiple outcomes into one overall measure (Leese et al. Reference Leese, Schene, Koeter, Meijer, Bindman, Mazzi, Puschner, Burti, Becker, Moreno, Celani, White and Thornicroft2008; Speechley et al. Reference Speechley, Forchuk, Hoch, Jensen and Wagg2009). Based on our findings, there is an argument for using the general factor as an aggregate PRO, for example as a surrogate outcome in the modelling or exploratory phase of evaluating interventions. Our study adds to previous work by showing that, over and above the GAT, there was overlap due to shared domain content (Floyd & Widaman, Reference Floyd and Widaman1995) that needs to be taken into account when assessing distinct PROs. However, a case can be made that domain factors may provide clinically actionable information at a low level of generality in the evaluation of routine care. Most importantly, however, our findings suggest that established PROs can provide mutually distinct information in patients with psychosis. They conflict with the idea that PROs may solely capture the same underlying concept (Hansson et al. Reference Hansson, Bjorkman and Priebe2007).
There are only a few studies that have examined the discriminative ability of the PRO measures used in the current study. None of the reports that we are aware of has simultaneously accounted for overlap across measures. Although we found measures of treatment satisfaction and the therapeutic relationship to be largely unaffected in their discriminative ability, those of SQOL and needs for care were markedly more impaired by the overlap. This may reflect limitations in the conceptual distinctiveness of these concepts. That is, some concepts have never been conceptually examined as to whether they are sufficiently distinct from already established concepts so that they should be measured independently (Campbell & Fiske, Reference Campbell and Fiske1959).
There has been even less research into information coverage of PROs in psychosis. Previous studies have almost exclusively reported psychometric properties based on classical test theory (e.g. Oliver et al. Reference Oliver, Huxley, Priebe and Kaiser1997; Gaite et al. Reference Gaite, Vazquez-Barquero, Arrizabalaga, Schene, Welcher, Thornicroft, Ruggeri, Vazquez-Bourgon, Retuerto and Leese2000). We found high information coverage for more favourable PRO levels in one of the psychosis samples. This suggests that evaluation using current PRO measures may provide a more precise picture of positive than of negative patient views (Williams, Reference Williams1994; Crow et al. Reference Crow, Gage, Hampson, Hart, Kimber, Storey and Thomas2002; Elwyn et al. Reference Elwyn, Buetow, Hibbard and Wensing2007; Priebe, Reference Priebe2007). Low information coverage, more generally, makes it difficult to shorten scales while maintaining measurement precision through the full range of each PRO (Rodebaugh et al. Reference Rodebaugh, Woods, Thissen, Heimberg, Chambless and Rapee2004; Uher et al. Reference Uher, Farmer, Maier, Rietschel, Hauser, Marusic, Mors, Elkin, Williamson, Schmael, Henigsberg, Perez, Mendlewicz, Janzing, Zobel, Skibinska, Kozel, Stamp, Bajs, Placentino, Barreto, McGuffin and Aitchison2008). Without shortening scales it is difficult to reduce the assessment burden on patients, which seems to be particularly important in vulnerable patients with psychosis (Gilbody et al. Reference Gilbody, House and Sheldon2002; Gibbons et al. Reference Gibbons, Weiss, Kupfer, Frank, Fagiolini, Grochocinski, Bhaumik, Stover, Bock and Immekus2008).
Measuring distinct PROs in psychosis
Discriminant and structural validity are highly relevant for determining the value of established measures when assessing PROs in psychosis (Altman et al. Reference Altman, Schulz, Moher, Egger, Davidoff, Elbourne, Gøtzsche and Lang2001; Moher et al. Reference Moher, Schulz and Altman2001; FDA, 2006; Mokkink et al. Reference Mokkink, Terwee, Knol, Stratford, Alonso, Patrick, Bouter and de Vet2006). Our finding, that using the bifactor model for important PROs can provide distinct information, represents an essential step for establishing discriminant validity of PROs. The measures examined in the current study contain items with high discriminative ability that can be used in psychosis outcome evaluations. They address different levels of generality (i.e. domains, concepts and aggregate outcomes). Which of these levels is most useful depends on the purpose of the given evaluation. At present, the structural validity of established measures remains limited, as simple sum scores do not adequately reflect the dimensionality of PROs.
Future research faces the challenge to implement model-based approaches to scoring into outcome evaluations, which can be achieved through item banking and computer-based assessments. Item banks are developed from large pools of items from many available instruments applying item response modelling combined with qualitative methods in an iterative process. This approach has been used recently for developing national item banks for use in research and routine care to improve the measurement of PROs in populations other than psychosis (e.g. Fries et al. Reference Fries, Bruce and Cella2005). Psychometric research using rigorous methods such as item banking based on a clearly defined conceptual framework of PROs may now be required in psychosis studies.
Overall, findings suggest that advanced analytic methods can help to disentangle the complex overlap of PROs. The bifactor model provides a reasonable explanation of existing data and future studies measuring more than one PRO may adjust results for the overlap. Different PROs seem to contain distinct, and also shared, information that should be considered in the use and further development of PROs.
Acknowledgements
This work was supported by a Research Training Fellowship funded by the National Institute of Health Research, UK, to U.R. The report is independent research and the views expressed in this publication are those of the authors and not necessarily those of the National Health Service (NHS), the National Institute for Health Research or the Department of Health.
Declaration of Interest
None.
