The non-comprehensiveness of single-question wellbeing indicators

As Frijters et al. eloquently articulate, the core motivation for using survey-based wellbeing indicators is that they hold out the promise of capturing all of the dimensions of subjective experience that people care about – and therefore, according to some thinkers, everything that matters for policy. Traditional economic indicators measure only some of what people care about (e.g., gross domestic product (GDP) aims to capture the consumption of market goods, investment and government expenditures), and therefore they can at best serve as non-comprehensive proxies for overall wellbeing. To the extent that a policy has important effects on unaccounted-for dimensions of wellbeing, such as health, relationships, emotions and other quality-of-life dimensions, reliance on traditional indicators alone can yield bad policy prescriptions.

Do single-question wellbeing indicators, such as questions about happiness or life satisfaction as advocated by Frijters et al., fulfill the promise of being comprehensive measures of wellbeing? Before we turn to research that addresses this question directly, we note that some of the work that has studied wellbeing indicators may raise doubts (that go beyond the conflicting income coefficients mentioned by Frijters et al.). For example, Stevenson and Wolfers (Reference Stevenson and Wolfers2009) report a ‘paradox’: despite improvement in their lives on many objective measures, women's happiness in the USA declined between 1972 and 2006. There are plausible explanations for this decline – for example, women may be held to higher expectations that are more difficult to meet – and understanding the reasons for the decline may yield important insights for policy. Nonetheless, for those of us with strong priors that women are better off today than half a century ago, the paradox suggests skepticism about treating happiness self-reports as comprehensive measures of wellbeing.

Evidence from several papers more directly addresses the question of whether wellbeing indicators are comprehensive measures of wellbeing, including two of ours that we briefly describe below (see also Adler & Dolan, Reference Adler and Dolan2008; Fleurbaey & Schwandt, Reference Fleurbaey and Schwandt2016; Adler et al., Reference Adler, Dolan and Kavetsos2017; Szabó & Ujhelyi, Reference Szabó and Ujhelyi2017). Like Adler (Reference Adler2011) and others, our benchmark for assessing whether a measure comprehensively captures wellbeing is idealized choice. This benchmark is widely – albeit not universally – accepted in modern moral philosophy. One classic definition is: ‘[A]n individual's good consists in what he would want himself to want, or to pursue, were he to contemplate his present situation from a standpoint fully and vividly informed about himself and his circumstances, and entirely free of cognitive error or lapses of instrumental rationality’ (Railton, Reference Railton1986, p. 16). Since idealized choice is unobservable, in our empirical work we attempt to find situations – hypothetical and real – where observed choices and stated choices would come as close as possible to such idealized choices.Footnote ¹

Our first paper (Benjamin et al., Reference Benjamin, Heffetz, Kimball and Rees-Jones2012) is nicely summarized by Frijters et al. We present survey respondents with hypothetical choices between two options – for example, a job with higher income versus a job allowing for more hours of sleep – and ask them two questions: Which of the two options do you think would make you happier (or more satisfied with your life, etc.)? Which of the two do you think you would choose? If people were seeking to maximize their happiness or life satisfaction, then, to the extent that stated choice is a reasonable approximation of idealized choice in the context of the scenarios, our respondents would answer the two questions similarly. We find that they often – but far from always – do, and we identify additional factors – such as predicted sense of purpose, control over one's life, family happiness and social status – that help explain stated choice.

In our second paper (Benjamin et al., Reference Benjamin, Heffetz, Kimball and Rees-Jones2014a), we replace the hypothetical choices from our first paper with real choice data, collected by surveying 561 students from 23 US medical schools shortly after they submitted their choice rankings over residencies to the National Resident Matching Program. We elicit a list of predictions regarding happiness, life satisfaction, and residency features (such as prestige and future career prospects) for respondents’ top-ranking programs and compare reported program choice rankings with happiness- or life-satisfaction-prediction rankings. Our evidence from this context, which is real world and high stakes and in which the medical students deliberate about their choices, is consistent with our hypothetical choice evidence from the earlier paper.

In this second paper, we also provide evidence on the wellbeing indicator-based approach to cost–effectiveness analysis, advocated by Frijters et al. in their ‘Cost–effectiveness’ section. With our data, we can directly estimate the medical students’ willingness to trade off between perceived features of the residency programs by regressing residents’ preference rankings – as revealed by the actual residency choices they report to us – on their beliefs about these features and calculating the ratio of estimated coefficients. We treat these ratios as choice-based estimates of the students’ ‘marginal rates of substitution’. We then compare them with estimates obtained by a wellbeing-indicator-based approach: we use the exact same method as above, but instead of regressing actual choice, we regress predicted happiness (or life satisfaction) for a residency program on beliefs about program features and calculate the ratio of estimated coefficients. We find that these wellbeing-indicator-based ratios are typically different from the choice-based marginal rates of substitution by 60% or more. This evidence – that the coefficients from wellbeing regressions may be rather different from those from (the informed and deliberated) choice regressions – provides an additional reason to be concerned about the use of coefficients from wellbeing regressions for policy analysis (in addition to Frijters et al.'s concern about conflicting income coefficient estimates). We therefore caution against a blanket recommendation to use them in cost–effectiveness analysis.

The findings from these two papers suggest that people care about more than just what is measured by standard, single-question survey measures of ‘happiness’ or even ‘life satisfaction’. Thus, these measures do not solve the basic problem that motivates their use as an alternative to traditional economic indicators. Using such measures to replace more traditional methods could amount to merely replacing one non-comprehensive proxy for wellbeing with another.

One might conjecture that since responses to happiness or life satisfaction questions plausibly capture more of what people care about (despite not being fully comprehensive), they must at least improve on traditional economic indicators. Whether this conjecture is true depends on whether people's responses to these survey questions accurately weight the dimensions of wellbeing relative to each other. This remains an open question, but some of the evidence is negative. For example, in our hypothetical choice paper mentioned above (Benjamin et al., Reference Benjamin, Heffetz, Kimball and Rees-Jones2012), we find that people are more likely to choose options involving higher income than to think that these options will increase their life satisfaction or happiness. This finding suggests that the survey measures underweight the opportunities afforded by a higher income relative to other dimensions of wellbeing – again, providing evidence that adds to Frijters et al.'s conflicting coefficients concern. In other words, while traditional indicators likely overvalue income and consumption, single-question wellbeing indicators may swing too far in the opposite direction, undervaluing them.

An approach to more comprehensive survey-based wellbeing indicators

While effort is ongoing to find a single survey question that would more comprehensively capture wellbeing, another idea – which to us seems more promising at present – is to ask a set of survey questions that (to a reasonable approximation) are: (1) more comprehensive when taken all together; and (2) non-overlapping (or, if overlap is unavoidable, with the weights used to aggregate the questions adjusted to correct for ‘double-counting’).Footnote ² The set of questions may include single-question measures of emotions (e.g., happiness, anxiety) and broad evaluations (e.g., life satisfaction, Cantril ladder), but also questions about other dimensions of wellbeing that people care about, such as health, sense of purpose or supporting their families financially. Responses to the set of questions are then aggregated into an overall wellbeing index that can be used as a comprehensive survey-based measure of wellbeing. To inform that aggregation, other survey questions are used to elicit relative local preferences over changes in the different dimensions of wellbeing.

We are aware of two approaches to wellbeing indices grounded in economic theory that have been explored in the literature to date. For concreteness, here we briefly describe one of them (the other is described in, e.g., Fleurbaey & Blanchet, Reference Fleurbaey and Blanchet2013; Decancq et al., Reference Decancq, Fleurbaey and Schokkaert2015). The approach we sketch here is detailed in Benjamin et al. (Reference Benjamin, Heffetz, Kimball and Szembrot2014b) and is summarized and updated in Benjamin et al. (Reference Benjamin, Cooper, Heffetz and Kimball2017), from which we borrow much of the text below. It aims to derive an individual-level wellbeing index that can be tracked over time or used as an outcome in policy evaluation. To identify the dimensions of wellbeing to measure, we began by constructing a list that includes over 100 potential survey questions, drawing on proposals from psychologists, philosophers and economists, from Maslow (Reference Maslow and Harriman1946) to Stiglitz et al. (Reference Stiglitz, Sen and Fitoussi2009) and beyond. We then expanded this list to over 2000 potential dimensions and proposed ways to eventually shorten them to a manageable list that retains what the data tell us are, empirically, the most important dimensions.

The theory for measuring wellbeing is analogous to that underlying the measurement of aggregate consumption. In the case of aggregate consumption, a utility function u(c) is defined over a consumption vector for M goods. A traditional aggregate consumption index, $\mathop \sum {\limits_{m = 1}^M}\, \overline {p_m} c_m$, weights each good's consumption, c _m, by its price held fixed at a baseline level, $\overline {p_m} $. In the face of small changes in consumption, changes in the index approximate changes in utility (up to a multiplicative constant): $\mathop \sum {\limits_{m = 1}^M} \overline {p_m} \Delta c_m\propto \mathop \sum {\limits_{m = 1}^M} {{\partial u\lpar {\bi c} \rpar } \over {\partial c_m}}\Delta c_m\approx \Delta u$.

In the case of a survey-based wellbeing index, the vector of M consumption goods, c, is replaced with a vector of J dimensions of wellbeing, w. These dimensions are the final goods that people ultimately care about (market goods are now treated as intermediate goods in the production of dimensions of wellbeing, à la Becker, Reference Becker1965). The levels of w are assumed to be what are measured by responses to the survey questions.

Because the dimensions are not traded in markets, price data are unavailable, and different individuals may have different marginal rates of substitution across the dimensions. Nevertheless, a wellbeing index can be constructed using each individual's marginal rates of substitution for the dimensions as weights. Specifically, the index is given by$\;\mathop \sum {\limits_{j = 1}^J} \overline {{{\partial u\lpar {\bi w} \rpar } \over {\partial w_j}}} w_j$, where the marginal utilities are defined relative to an arbitrary numeraire dimension. Small changes in this index provide a first-order approximation of changes in the individual's (ordinal) utility (even if the individual's preferences are non-linear).

In practical terms, then, this approach requires: (1) identifying a set of dimensions of wellbeing to measure; (2) asking survey questions that measure each dimension; and (3) asking additional, stated preference survey questions that measure how respondents would be willing to make trade-offs between the dimensions. The marginal rates of substitution estimated from responses to these stated preference questions are then used as weights to construct an index that aggregates across the dimensions of wellbeing.

Of course, such an approach to constructing a wellbeing index is data hungry, both in absolute terms and, especially, compared with a single wellbeing question. We certainly acknowledge the practical challenges of implementing the index approaches – the higher costs of data collection requirements that they place on governments, the effort they impose on respondents given the survey length and format, the difficulty of explaining the wellbeing measurement methodology to constituents, etc. – all of which may make it tempting to take the single-question data we already have at present and run with them. However, these practical challenges should not be confused with arguments about what is the appropriate methodology for measuring wellbeing in a way that is normatively appealing and theoretically sound.

Other issues for ongoing research

We have highlighted non-comprehensiveness because we think it is a major reason to be cautious about basing policy on single-question wellbeing measures – and because the search for comprehensiveness is an important reason for economists to look ‘beyond GDP’ in the first place. However, there are also a number of other not-yet-resolved challenges to the use of survey-based wellbeing for policy, regardless of whether these are single-question measures or indices (see also Adler, Reference Adler2013). Some examples follow.