Criticisms of preference-based utility measures

Dolan provides a critique of the use of preference-based assessments of health utility for informing policy, and recommends a shift to an experience-based, or subjective well-being approach. Specifically, he notes that common utility methods such as the time trade off require people to imagine what their life would be like in health states that they have never experienced. We agree that there are substantial theoretical and empirical reasons to doubt the validity of these types of judgments – indeed, an entire subfield within psychology deals with the errors and biases that occur when people attempt to ‘forecast’ what their quality of life would be like in different circumstances (Loewenstein and Schkade, Reference Loewenstein, Schkade, Kahneman, Diener and Schwarz1999; Wilson and Gilbert, Reference Wilson, Gilbert and Zanna2003).

The problem of asking people about a condition they have not experienced could be sidestepped by using utility values provided by patients themselves. We know that utility valuations from patients are typically higher than those given by the public; it could be that these are higher precisely because patients do not suffer from ‘overfocusing’ on their health condition (De Wit et al., Reference De Wit2000; Ubel et al., Reference Ubel2001). But on closer inspection, it does not seem that asking patients to provide utility estimates addresses the central problem with these methods, which is that they require respondents to think about a health state that they are not actually experiencing. Dolan makes a similar point in his article; that is, patients who respond to utility measures must imagine what life would be like without their current health condition, opening the door for the same sorts of biased predictions that affect utility valuations provided by non-patients. Of course, for some conditions, such as a spinal cord injury, patients could use their memories of quality of life before the condition, but research also casts doubt on the accuracy of these sorts of memory-based judgments (Ross, Reference Ross1989).

Indeed, in two studies we have found evidence that predictions and recollections from patients appear to be biased in ways similar to those of non-patients. In a prospective, longitudinal study of renal transplant patients, we found that patients who were currently on dialysis and awaiting a transplant predicted significantly larger improvements in quality of life than what was actually observed one year after a successful transplant (Smith et al. in submission). In another study, we surveyed current colostomy patients, patients who had recently had their colostomies reversed, and members of the public with no experience with colostomy. We found that former patients gave much lower utility valuations for colostomy than did current patients. In fact, despite having direct experience with the condition, the former patients gave average valuations that were nearly identical to those provided by non-experienced members of the public (Smith et al., Reference Smith2006b).

But are subjective well-being measures any better?

A substantial literature indicates that overall life satisfaction scores are strongly influenced by information that is brought to mind at the time of the assessment, as well as by current and recent emotional experiences (Schwarz and Strack, Reference Schwarz, Strack, Kahneman, Diener and Schwarz1999). So while we agree that SWB measures have the desirable property of not asking respondents to imagine a health state they are not experiencing, we do not agree that they necessarily substantially reduce problems with focusing, or problems associated with biased memory.

In our research, we have observed quite substantial focusing effects, on precisely the sort of 1–10 satisfaction scale Dolan advocates. For example, we surveyed Parkinson's disease (PD) patients, and asked them to indicate how satisfied they were with their life as a whole, and then asked for a separate rating of their health. We randomly assigned the respondents to either receive standard introductions used in health surveys (to paraphrase, ‘We are surveying Parkinson's Disease patients to study their quality of life), or modified instructions in which we made no mention of their illness (‘We are surveying people in your area to study their quality of life’). We found that the study introduction strongly affected how people responded to the life satisfaction item. When the participants knew we were surveying PD patients, their evaluation of their health status correlated much more strongly with life satisfaction; it accounted for 40% of the variability in life satisfaction, compared to only 12% in the condition where we made no mention of PD. These results suggest that the survey introduction focused PD patients’ attention on their health status, making it seem like a much more important aspect of their life satisfaction judgment (Smith et al., Reference Smith2006a).

In addition to focusing problems, the ambiguous nature of the response scales used to assess SWB makes them subject to rescaling, opening up the possibility of systematic differences in how people in different health states interpret the scales. In one study, we found evidence that older participants interpreted the common scale anchor ‘perfect health’ to mean ‘perfect health for someone my age’. This recalibration of the scale resulted in respondents around the age of 75 reporting a mean health value that was similar to that of respondents around the age of 55 (Ubel et al., Reference Ubel2005a). While we agree with Dolan that this sort of rescaling probably does not completely account for adaptation effects – that is, we also think that real psychological adaptation occurs – group differences in interpretation of these scales complicate their use in quantifying the effects of illness on quality of life. More research is needed to determine the pervasiveness of strength of rescaling effects, and potentially to create new scale anchors that might be more resistant to these effects.

Perhaps a more vexing problem for SWB measures lies in the nature of what they measure. SWB measures such as life satisfaction show considerable variation across individuals, but, as Dolan points out, they are highly stable across time. This can be viewed as a strength, but it also raises the troubling possibility that SWB scores are, to a substantial degree, dispositional – reflecting mostly the internal personality traits of respondents, rather than an evaluation of one's objective circumstances (Diener and Lucas, Reference Diener, Lucas, Kahneman, Diener and Schwarz1999). If this is the case, we might expect SWB measures to be relatively insensitive to differences in objective circumstances – such as the presence of a disability – which indeed it seems they are (Brickman et al., Reference Brickman1978; Albrecht and Devlieger, Reference Albrecht and Devlieger1999; Ubel et al., Reference Ubel2005b).

One could argue that, while flawed, life satisfaction judgments are still the most promising tool we have, and that we should simply accept that they are insensitive, and interpret apparently small differences as nonetheless important. This is an attractive argument, but then we must be careful when considering health conditions that are subjectively assessed. The correlation between life satisfaction and depression can be very high (r = 0.72, Pavot and Diener, 1993); thus, reliance on this measure will inevitably lead to much greater prioritization of treatments for depression than for physical conditions. If we are going to adopt these measures despite their insensitivity to objective circumstances, then it is at least important not to compare apples and oranges – you can't use these tools to compare a subjective state – depression – to an objective health condition – diabetes. This sort of comparison awaits better, less biased, more sensitive tools.

Ecological Momentary Assessment and the Day Reconstruction Method

Partially in response to concerns about SWB assessments, methodologists have developed new tools, such as Ecological Momentary Assessment (EMA), and the Day Reconstruction Method (Shiffman and Stone, Reference Shiffman, Stone and Krants1998; Kahneman et al., Reference Kahneman2004). We use these methods in our own research, and share Dolan's enthusiasm for their potential (Riis et al., Reference Riis2005; Smith et al., Reference Smithin press). However, we do not think they should be used as a primary criterion for evaluating health policy, at least not as currently formulated. To date, these measures have primarily been used to measure mood – that is, general positive and negative emotions experienced during ordinary daily events. We think there are two problems with using mood as an overall indicator of quality of life, to quantify the impact of health policies or treatments.

The first problem is that reliance on average mood quality will tend to underestimate the impact of objective circumstances (such as illnesses and their treatments), especially when the circumstances are relatively stable, and are perceived as unchangeable. We argue that mood is an underestimate of objective circumstances because leading theories of emotion hold that emotions are designed to alert people to changes in their circumstances or environment, in order to motivate appropriate action (Lazarus, Reference Lazarus1991). Once it is apparent that a negative circumstance is unalterable, and that no action is required, there would be no adaptive value in continuing to experience highly negative, punishing emotions. Thus, mood states may not accurately reflect the severity of a person's life circumstances. Consistent with this possibility, one study using EMA methods has shown that end-stage renal disease patients on dialysis report similar mood quality to healthy matched controls, and that both groups’ mood quality was mostly positive (Riis et al., Reference Riis2005).

Our second concern with basing policy on mood-based measures is that we think people value certain types of experiences independently from their effects on mood quality. In one study using the DRM, researchers observed that among various daily activities, time spent with one's children was associated with relatively poor mood quality, comparable to moods experienced when performing chores such as grocery shopping (Kahneman et al., Reference Kahneman2004). One interpretation of this data is that people are making a foolish choice when they decide to have children, because they do not realize that this will negatively affect their average mood quality. A contrary interpretation might suggest that people do not have children in the expectation of improving their mood, but do so in search of meaningful experiences and the sense of satisfaction that arises from caring for another. A measure of mood cannot capture these types of valued experiences.

In contrast to momentary mood measures, life satisfaction measures involve cognitive judgments about one's circumstances, and therefore could presumably capture and incorporate things people value beyond immediate mood (e.g., contribution to others, meaning and purpose, etc). But we have already discussed potential problems with these measures. There may be a third alternative: expanding EMA and DRM measures to capture other aspects of experience that people value. Doing so could leverage the methodological advantages of these methods, without an over reliance on momentary mood. In one study, we used EMA measures to assess caregivers’ perceptions of their relationship to the care recipient over time, in addition to the nature of their caregiving activities and their subjective mood state. Results of this study indicated that active caregiving behaviors were associated with improved mood for caregivers, but only to the extent that caregivers reported feeling loved and cared for by the care recipient (Poulin et al., in submission). These results challenge traditional assumptions that caregiving behavior generally causes stress and burnout (Rainer, 2000). Future research efforts could also supplement EMA and DRM methods to include physiological assessments of well-being (e.g., a diurnal cortisol pattern, Goodyer et al., Reference Goodyer and Herbert1996).

Conclusion

We think this is a timely debate, and we agree with many arguments made by Dolan, especially his criticisms of decision utility measures. But we think it is premature to conclude that SWB measures represent a viable alternative, and we do not think there is definitive evidence that they represent a substantial improvement over decision-based measures. We recommend an expansion of research efforts along at least three fronts: (a) to expand EMA and DRM methods to capture other important aspects of experience that go beyond mood; (b) to develop an overarching, integrative theory of well-being that could be used to give some guidance on what should be important to most people; and (c) to develop what we think is tremendous potential in creating physiological measures of emotion and well-being.

There is no doubt that the currently available tools for quantifying the effects of health conditions and their treatments are wanting. We need to invest in the development of more valid, more sensitive, and less biased measures of quality of life.