How best to measure outcomes?

Galileo provided the science community with simple instructions on how to advance knowledge – ‘Measure what is measurable, and make measurable what is not so’. The investment in early psychosis research has helped us better define the onset of psychosis. Many incidence studies routinely estimate duration of untreated psychosis and the field is exploring ways to measure the transition from ultra-high risk to clinical psychosis (Yung & McGorry, Reference Yung and McGorry1996). With respect to measuring recovery from schizophrenia (the offset), and measuring the course of schizophrenia in between onset and offset, the technology is still far from optimal.

In recent years, operationalizable criteria for the assessment of remission of core symptoms of psychosis have become available (Andreasen et al. Reference Andreasen, Carpenter, Kane, Lasser, Marder and Weinberger2005; van Os et al. Reference van Os, Burns, Cavallaro, Leucht, Peuskens, Helldin, Bernardo, Arango, Fleischhacker, Lachaux and Kane2006). These criteria have recommended threshold scores on a range of standard symptom domains (e.g. score of mild, or less than three, for certain items on the Positive and Negative Syndrome Scale) and required a persistence/duration criterion (e.g. subthreshold scores must persist for at least 6 months). The criteria allow for persisting mild symptoms, but do not capture wider aspects of social and functional domains. Individuals may remain relatively symptom free, but not make gains across these wider domains.

However, while symptoms can remit for certain periods of time, recovery assumes a more enduring improvement across several domains. In addition to symptom profiles, social and functional outcomes are also highly important and valid targets to measure (Liberman & Kopelowicz, Reference Liberman and Kopelowicz2005; Harvey & Bellack, Reference Harvey and Bellack2009; Emsley et al. Reference Emsley, Chiliza, Asmal and Lehloenya2011). Here the research community is presented with a much more complex task. The data are less coherent and there is no clear way forward to simplify the research questions. Recovery is a subjectively experienced journey. How could we capture this ‘personalized’ facet of recovery? Can we agree on what outcomes to include in a comprehensive set of recovery-related measures? The challenge for the research community is to develop a panel of outcome measures that are valid, operationalizable and pragmatic (Harvey, Reference Harvey2014).

After choosing the panel of recovery-related measures, the researcher then needs to decide how often these should be assessed. How vigilant should we be in order to prospectively measure the time-dependent nuances of the course of psychosis? On the one hand, psychosis is consistently associated with premature death (as a reflection of the increased risk of co-morbid physical illnesses and increased risk of suicide). On the other hand, people with psychosis can make complete recoveries (i.e. sustained symptom-free periods with good social/occupational functioning). In between these two extremes, how can we capture fluctuations in the course of illness? Earlier studies adopted a taxonomy of course of psychosis that captured features such as phase of illness (onset, mid-course, end-stage), type of course (continuous, fluctuating) and general trajectory (worsening, static, improving) (Harrison et al. Reference Harrison, Hopper, Craig, Laska, Siegel, Wanderling, Dube, Ganev, Giel, an der Heiden, Holmberg, Janca, Lee, Leon, Malhotra, Marsella, Nakane, Sartorius, Shen, Skoda, Thara, Tsirkin, Varma, Walsh and Wiersma2001). Perhaps we could extend this taxonomy to include social, functional and cognitive domains. Simplifying assumptions will need to be made, as no panel of recovery measures will be able to capture all the related (but often orthogonal) facets of recovery.

However, our field has demonstrated the capacity to develop useful criteria to guide other aspects of schizophrenia research. For example, standardized measures for the assessment of cognitive outcomes have been developed, and perhaps those interested in standardized functional outcomes might benefit from lessons learnt from initiatives such as MATRICS (Management and Treatment Research to Improve Cognition in Schizophrenia) (Marder, Reference Marder2011; Velligan et al. Reference Velligan, Fredrick, Mintz, Li, Rubin, Dube, Deshpande, Trivedi, Gautam, Avasthi, Kern and Marder2013; Harvey, Reference Harvey2014).

Recovery from schizophrenia: results from a recent systematic review

We recently completed a systematic review of recovery from schizophrenia which was based on estimates from 50 discrete samples, drawn from 20 different countries with samples dating back to 1921 (Jaaskelainen et al. Reference Jaaskelainen, Juola, Hirvonen, McGrath, Saha, Isohanni, Veijola and Miettunen2013). Following the recommendation of recent reviews (Menezes et al. Reference Menezes, Arenovich and Zipursky2006; Faerden et al. Reference Faerden, Nesvag and Marder2008), our definition of recovery required improvements in at least two domains – symptoms, and social/functional outcomes. In addition, we required that at least one of these two domains should have persisted for at least 2 years. We found that the median proportion that met recovery criteria was 13.5% (interquartile range 8.1–20.0%). As expected, the estimates included in the meta-analyses were highly heterogeneous [as were those in the target paper (Morgan et al. Reference Morgan, Lappin, Heslin, Donoghue, Lomas, Reininghaus, Onyejiaka, Croudace, Jones, Murray, Fearon, Doody and Dazzan2014)]. Unexpectedly, we did not find any significant male versus female difference in recovery from schizophrenia.

In light of the introduction of a wider range of antipsychotic medications, and innovations in psychosocial interventions, and in light of major innovations in service delivery (e.g. de-institutionalization, early psychosis services), one might predict that the proportion of patients with psychosis who meet recovery criteria should have improved over time. Alas, this is not the case (Hegarty et al. Reference Hegarty, Baldessarini, Tohen, Waternaux and Oepen1994). Our systematic review found no evidence that the proportion who met recovery criteria had increased over time (indeed, there was a slight decline over time) (Jaaskelainen et al. Reference Jaaskelainen, Juola, Hirvonen, McGrath, Saha, Isohanni, Veijola and Miettunen2013). This is sobering news.

One of the problems we faced when trying to pool recovery proportions from different studies related to the variable duration of follow-up. For example, it may not be appropriate to pool estimates from studies that have follow-up for only 2 years with estimates from studies that have extended over two decades. Longer studies have more attrition, which can erode the validity of the recovery estimates (Cohen et al. Reference Cohen, Patel, Thara and Gureje2008; McGrath, Reference McGrath2008). One crude parameter that may facilitate this type of comparison is the annualized recovery rate (i.e. the proportion of individuals who meet recovery criteria for a given period, divided by the duration of follow-up in years). For our review, the pooled annualized recovery rate was 1.4% per annum (i.e. over a 10-year period, 14% would be expected to recover). This parameter has some strengths – it optimizes between-study comparisons and the meta-analysis of studies of different durations. However, it has important weaknesses. It is based on the endpoint (cross-sectional) recovery proportion, thus the annualized recovery rate assumes that the chance of recovery is evenly distributed and linear over time (which it is not), and it masks the nuances of recovery and relapses within the time period.

Findings from the AESOP 10-year follow-up study

The AESOP group has provided important new insights into the recovery time course. First, 7% of the sample had died during the 10-year follow-up. This proportion is consistent with the increased mortality associated with schizophrenia (Saha et al. Reference Saha, Chant and McGrath2007; Beary et al. Reference Beary, Hodgson and Wildgust2012). Within broadly defined psychotic disorders, those with affective psychosis did better than non-affective disorders (e.g. schizophrenia). These results are consistent with the general literature (Lang et al. Reference Lang, Kosters, Lang, Becker and Jager2013). Men did worse when compared with women. When the period of observation was the entire 10-year follow-up window, nearly one-quarter (23%) of the cohort had an unrelenting course of illness without a remission of psychotic symptoms for 6 months or more. Only 13% had a remission within the first 6 months of first contact and then remained symptom free for the rest of the follow-up. However, when assessed cross-sectionally at 10-year follow-up, 65% were free of psychotic symptoms, and (based on those with complete data), 46% had been free of psychotic symptoms for the previous 2 years. These estimates highlight the importance of capturing both cross-sectional and longitudinal data in order to describe the fine-grained nuances of recovery.

The AESOP study also confirmed that symptomatic recovery and social recovery are not necessarily congruent (Karow et al. Reference Karow, Moritz, Lambert, Schottle and Naber2012). Despite sizable proportions having favourable symptomatic recovery, social outcomes were less favourable. Most of the cohort (68%) were single and/or not in a relationship at the 10-year follow-up. Nearly three-quarters of the cohort (72%) had been employed for less than 25% of the follow-up period. These social outcome proportions are comparable with those reported in large, multicentre studies (Haro et al. Reference Haro, Novick, Bertsch, Karagianis, Dossenbach and Jones2011). While functional outcomes can be influenced by broader socio-economic factors (Yang et al. Reference Yang, Phillips, Li, Yu, Zhang, Shi, Song, Ding, Pang and Susser2013), the AESOP results foreground how poorly our patients do on basic measures of social inclusion and functional recovery.

Making recovery measurable

In an ideal world, the optimal target for treatment is clear – we want treatments that result in immediate, complete and sustained recovery for everyone. Ideally, these interventions should also be safe, cheap and publicly acceptable. In the absence of this goal, what should we accept as reasonable surrogate targets? It is feasible that an effective intervention may have little impact on symptoms but improves functional outcomes. With respect to social function, the AESOP study suggests that social exclusion (as measured by employment status and relationship status) predated the first contact for psychosis (Morgan et al. Reference Morgan, Kirkbride, Hutchinson, Craig, Morgan, Dazzan, Boydell, Doody, Jones, Murray, Leff and Fearon2008). Cognitive phenotypes also emerge prior to symptom onset (Khandaker et al. Reference Khandaker, Barnett, White and Jones2011). The links between cognitive capacity and functional outcomes are complex (Wykes et al. Reference Wykes, Reeder, Huddy, Taylor, Wood, Ghirasim, Kontis and Landau2012), and are influenced by a range of factors such as motivation and meta-cognitive ability (Medalia & Saperstein, Reference Medalia and Saperstein2013). As we place more emphasis on cognitive remediation and the search for pro-cognitive medications, should cognitive outcomes be routinely included in outcome assessment?

We have seen how clinical interventions related to supporting people in employment regardless of remission of symptoms can optimize functional outcomes [International First Episode Vocational Recovery (iFEVR) Group, 2010; Harvey et al. Reference Harvey, Heaton, Carpenter, Green, Gold and Schoenbaum2012]. Thus, we can expect that the different outcome domains (symptom, social and functional outcomes, cognitive and patient-selected recovery outcomes) to diverge and follow different trajectories. Early findings from multicentre studies have suggested that socio-economic factors may influence outcome (Jablensky et al. Reference Jablensky, Sartorius, Ernberg, Anker, Korten, Cooper, Day and Bertelsen1992); however, these research questions are clouded by important methodological issues (Cohen et al. Reference Cohen, Patel, Thara and Gureje2008; McGrath, Reference McGrath2008). On first principles, based on the underlying heterogeneity in psychosis, and on the variability of services between sites/nations, we would predict appreciable variation in remission and recovery proportions.

It is a given that we need to close the differential mortality gap (Bushe et al. Reference Bushe, Taylor and Haukka2010). Thus, there is a compelling case to include routine physical health outcomes within the matrix of recovery outcome measures. If interventions result in better functional outcomes but worse mortality, would this be ethical? In fact, while not widely appreciated, the disability weight allocated to acute schizophrenia by the recent Global Burden of Disease study was the highest for all 220 disease states assessed (Salomon et al. Reference Salomon, Vos, Hogan, Gagnon, Naghavi, Mokdad, Begum, Shah, Karyana, Kosen, Farje, Moncada, Dutta, Sazawal, Dyer, Seiler, Aboyans, Baker, Baxter, Benjamin, Bhalla, Bin Abdulhak, Blyth, Bourne, Braithwaite, Brooks, Brugha, Bryan-Hancock, Buchbinder, Burney, Calabria, Chen, Chugh, Cooley, Criqui, Cross, Dabhadkar, Dahodwala, Davis, Degenhardt, Diaz-Torne, Dorsey, Driscoll, Edmond, Elbaz, Ezzati, Feigin, Ferri, Flaxman, Flood, Fransen, Fuse, Gabbe, Gillum, Haagsma, Harrison, Havmoeller, Hay, Hel-Baqui, Hoek, Hoffman, Hogeland, Hoy, Jarvis, Karthikeyan, Knowlton, Lathlean, Leasher, Lim, Lipshultz, Lopez, Lozano, Lyons, Malekzadeh, Marcenes, March, Margolis, McGill, McGrath, Mensah, Meyer, Michaud, Moran, Mori, Murdoch, Naldi, Newton, Norman, Omer, Osborne, Pearce, Perez-Ruiz, Perico, Pesudovs, Phillips, Pourmalek, Prince, Rehm, Remuzzi, Richardson, Room, Saha, Sampson, Sanchez-Riera, Segui-Gomez, Shahraz, Shibuya, Singh, Sliwa, Smith, Soerjomataram, Steiner, Stolk, Stovner, Sudfeld, Taylor, Tleyjeh, van der Werf, Watson, Weatherall, Weintraub, Weisskopf, Whiteford, Wilkinson, Woolf, Zheng, Murray and Jonas2012). From a disability-adjusted life years (DALYs) perspective (Ferrari et al. Reference Ferrari, Saha, McGrath, Norman, Baxter, Vos and Whiteford2012; Murray et al. Reference Murray, Vos, Lozano, Naghavi, Flaxman, Michaud, Ezzati, Shibuya, Salomon, Abdalla, Aboyans, Abraham, Ackerman, Aggarwal, Ahn, Ali, Alvarado, Anderson, Anderson, Andrews, Atkinson, Baddour, Bahalim, Barker-Collo, Barrero, Bartels, Basanez, Baxter, Bell, Benjamin, Bennett, Bernabe, Bhalla, Bhandari, Bikbov, Bin Abdulhak, Birbeck, Black, Blencowe, Blore, Blyth, Bolliger, Bonaventure, Boufous, Bourne, Boussinesq, Braithwaite, Brayne, Bridgett, Brooker, Brooks, Brugha, Bryan-Hancock, Bucello, Buchbinder, Buckle, Budke, Burch, Burney, Burstein, Calabria, Campbell, Canter, Carabin, Carapetis, Carmona, Cella, Charlson, Chen, Cheng, Chou, Chugh, Coffeng, Colan, Colquhoun, Colson, Condon, Connor, Cooper, Corriere, Cortinovis, de Vaccaro, Couser, Cowie, Criqui, Cross, Dabhadkar, Dahiya, Dahodwala, Damsere-Derry, Danaei, Davis, De Leo, Degenhardt, Dellavalle, Delossantos, Denenberg, Derrett, Des Jarlais, Dharmaratne, Dherani, Diaz-Torne, Dolk, Dorsey, Driscoll, Duber, Ebel, Edmond, Elbaz, Ali, Erskine, Erwin, Espindola, Ewoigbokhan, Farzadfar, Feigin, Felson, Ferrari, Ferri, Fevre, Finucane, Flaxman, Flood, Foreman, Forouzanfar, Fowkes, Fransen, Freeman, Gabbe, Gabriel, Gakidou, Ganatra, Garcia, Gaspari, Gillum, Gmel, Gonzalez-Medina, Gosselin, Grainger, Grant, Groeger, Guillemin, Gunnell, Gupta, Haagsma, Hagan, Halasa, Hall, Haring, Haro, Harrison, Havmoeller, Hay, Higashi, Hill, Hoen, Hoffman, Hotez, Hoy, Huang, Ibeanusi, Jacobsen, James, Jarvis, Jasrasaria, Jayaraman, Johns, Jonas, Karthikeyan, Kassebaum, Kawakami, Keren, Khoo, King, Knowlton, Kobusingye, Koranteng, Krishnamurthi, Laden, Lalloo, Laslett, Lathlean, Leasher, Lee, Leigh, Levinson, Lim, Limb, Lin, Lipnick, Lipshultz, Liu, Loane, Ohno, Lyons, Mabweijano, MacIntyre, Malekzadeh, Mallinger, Manivannan, Marcenes, March, Margolis, Marks, Marks, Matsumori, Matzopoulos, Mayosi, McAnulty, McDermott, McGill, McGrath, Medina-Mora, Meltzer, Mensah, Merriman, Meyer, Miglioli, Miller, Miller, Mitchell, Mock, Mocumbi, Moffitt, Mokdad, Monasta, Montico, Moradi-Lakeh, Moran, Morawska, Mori, Murdoch, Mwaniki, Naidoo, Nair, Naldi, Narayan, Nelson, Nelson, Nevitt, Newton, Nolte, Norman, Norman, O'Donnell, O'Hanlon, Olives, Omer, Ortblad, Osborne, Ozgediz, Page, Pahari, Pandian, Rivero, Patten, Pearce, Padilla, Perez-Ruiz, Perico, Pesudovs, Phillips, Phillips, Pierce, Pion, Polanczyk, Polinder, Pope, Popova, Porrini, Pourmalek, Prince, Pullan, Ramaiah, Ranganathan, Razavi, Regan, Rehm, Rein, Remuzzi, Richardson, Rivara, Roberts, Robinson, De Leon, Ronfani, Room, Rosenfeld, Rushton, Sacco, Saha, Sampson, Sanchez-Riera, Sanman, Schwebel, Scott, Segui-Gomez, Shahraz, Shepard, Shin, Shivakoti, Singh, Singh, Singh, Singleton, Sleet, Sliwa, Smith, Smith, Stapelberg, Steer, Steiner, Stolk, Stovner, Sudfeld, Syed, Tamburlini, Tavakkoli, Taylor, Taylor, Taylor, Thomas, Thomson, Thurston, Tleyjeh, Tonelli, Towbin, Truelsen, Tsilimbaris, Ubeda, Undurraga, van der Werf, van Os, Vavilala, Venketasubramanian, Wang, Wang, Watt, Weatherall, Weinstock, Weintraub, Weisskopf, Weissman, White, Whiteford, Wiebe, Wiersma, Wilkinson, Williams, Williams, Witt, Wolfe, Woolf, Wulf, Yeh, Zaidi, Zheng, Zonies, Lopez, AlMazroa and Memish2012), how would consumers and caregivers ‘trade off’ lifespan for improvements across different types of outcome measures (e.g. symptoms, social/functional)? If we do not measure these outcomes, it will be difficult to explore these research questions from an empirical perspective.

This discussion raises interesting thought experiments. Would it be possible, as has happened for some cardiovascular outcomes, that we could ‘compress’ or otherwise shift the timing of morbidity and mortality (Fries, Reference Fries2005)? Swapping chronic persistent course types to undulating course might result in an overall reduction in cumulative burden. It is not clear how we could achieve this goal, but careful observational studies like the AESOP 10-year follow-up may provide predictors and correlates of different outcome phenotypes. The genetic architecture of psychosis has now become clearer (Mowry & Gratten, Reference Mowry and Gratten2013), and it is feasible that polygene scores or other neurobiologically informed biomarkers may not only predict risk of a disorder, but predict outcomes related to onset and offset (Bergen et al. Reference Bergen, O'Dushlaine, Lee, Fanous, Ruderfer, Ripke, Sullivan, Smoller, Purcell and Corvin2014). Access to e-health records or linked registers related to mental health, employment and income status may provide more fine-grained cumulative measures (Morgan & Jablensky, Reference Morgan and Jablensky2010; Kohane et al. Reference Kohane, Drazen and Campion2012), just as access to new Internet and mobile technologies may supplement our ability to measure recovery in ‘real time’ (Ben-Zeev et al. Reference Ben-Zeev, Brenner, Begale, Duffecy, Mohr and Mueser2014).

Good progress has been made developing consensus criteria for symptomatic remission (Andreasen et al. Reference Andreasen, Carpenter, Kane, Lasser, Marder and Weinberger2005; Leucht & Lasser, Reference Leucht and Lasser2006; van Os et al. Reference van Os, Burns, Cavallaro, Leucht, Peuskens, Helldin, Bernardo, Arango, Fleischhacker, Lachaux and Kane2006). This work now needs to be extended into functional and social domains (Barak et al. Reference Barak, Bleich and Aizenberg2010). We need to think more about how to personalize recovery measures in order to incorporate subjective aspects of recovery. The need for a standardized panel of recovery-related measures is more important than ever, particularly as we are confronted by evidence that challenges our cherished assumptions about the effectiveness of our current treatment strategies (Wunderink et al. Reference Wunderink, Nieboer, Wiersma, Sytema and Nienhuis2013; Jauhar et al. Reference Jauhar, McKenna, Radua, Fung, Salvador and Laws2014). The new study by Morgan et al. (Reference Morgan, Lappin, Heslin, Donoghue, Lomas, Reininghaus, Onyejiaka, Croudace, Jones, Murray, Fearon, Doody and Dazzan2014) provides a more nuanced and fine-grained assessment of the course of schizophrenia. In the spirit of Galileo (‘make measurable what is not so’), we need to leverage the discoveries from the AESOP 10-year follow-up and develop ways to measure both the time-dependent and multifaceted nuances of recovery. This will not be an easy task, but we believe this is required to take outcome research to the next level.