2.1 Health system performance indicators

The measurement of health system performance has been a big topic of debate particularly following the publication of the World Health Organisation (WHO) health system performance index (Abelson et al., Reference Abelson, Forest, Eyles, Smith, Martin and Gauvin2007). However, in designing performance measures it is unclear how dimensions of value coincide with peoples’ values. Some studies, such as Murray and Frenk (Reference Murray, Lauer, Tandon and Frenk2000), provide a specification of health system goals, which is a first step to operationalise a measure of performance. In this paper, we offer an alternative methodology. First, we do not distinguish between ‘instrumental’ and ‘intrinsic’ sources of value, insofar as they both interrelated. Second, an important difference lies in that we validate what ‘health system value’ is perceived to be by the population. Different beneficiaries might place varying weight on different dimensions of value. Hence, such measures of value ought to be related in some way to measures elicited by the population itself. There have been some attempts of this both in the context of developing and developed countries (Costa-Font et al., Reference Costa-Font, Forns and Sato2015; Costa-Font et al., 2016; Costa-Font and Rovira, 2005). However, previous research focusses on elicitation methodologies rather than on the identification of value dimensions.

As a natural response of western systems governed by elitism, there is growing demand for involving citizens in policy development to ensure that decisions are legitimate, and reflect broad social values. This entails including patients, service users (or ‘consumers’) and the general public when making decisions, and aims to ensure that all decisions are informed, transparent and legitimate (Handler et al., Reference Handler, Issel and Turnock2001; Abelson et al., Reference Abelson, Giacomini, Lehoux and Gauvin2003). However, typical elicitation experiments lack realism, and often reduce the decision-making question to a handful of programmes which do not capture the set of value trade-offs that health systems regularly make. Indeed, some decisions (e.g., investment in prevention rather than medicines) are grounded on specific social values. Hence, the elicitation of health system values could help towards making normative decisions less problematic.

Limited available research has been devoted to designing elicitation instruments and methodologies that do not conflict with prevailing collective decision-making nature of European health systems (Abelson et al., Reference Abelson, Forest, Eyles, Smith, Martin and Gauvin2007). On a macro-level, elicitation instruments ought to ask communities to express their preferences between alternative health system goals. The challenge, however, is how to assess health systems that are extremely complex, especially when they are collectively funded and pursue multi-dimensional goals.

Franken and Koolman (Reference Franken and Koolman2013) developed a multi-attribute choice technique to elicit goal valuations using a ‘choice experiment’, which allows the indirect elicitation of health system dimensions. One downside of these methods alongside other stated preference techniques is the so-called ‘hypothetical bias’ (where participants respond differently in hypothetical scenarios as opposed to realistic decision-making contexts), given that choice experiment conditions are generally hypothetical, and often not perceived as ‘realistic’. Another potential problem is that individuals often construct their preferences ‘on the spot’, when requested, rather than understanding the consequences of different courses of actions, and hence observed values are likely to be sensitive to the elicitation methods employed to identify values dimensions. The latter can be overcome using a deliberation method (Handler et al., Reference Handler, Issel and Turnock2001), or alternatively, natural experiments where individuals make judgements on a real decision and are asked to make trade-offs.

2.2 Classifying health system values

In analysing dimensions of latent expectations of value from health systems, one can use different definitions such as Murray and Frenk (Reference Murray, Lauer, Tandon and Frenk2000). From a value standpoint, it is possible to identify sources of value that refer to the process of health care delivery (‘process utility dimensions’). Hence, it is possible to distinguish health care utility derived from interactions between people and the health system (‘how people are treated by the health system’), and utility (or values) obtained from certain health system institutions (‘how the health system allocates resources and how redistributive decisions are taken’) (Costa-Font et al., Reference Costa-Font, Forns and Sato2015).

In addition to process-related utility values, equity or distributional values consider that the health system can influence the distribution of health care use, and more widely, health outcomes. Income-related equity dimensions incorporate the idea that individuals might care about how health care reforms influence access to health by those with lower income, or how they affect the economic well-being of individuals (e.g., whether individuals become impoverished as a result of co-payments, etc.). Equity in health care financing has previously been used to rate health systems (Murray et al., Reference Murray and Frenk2000). However, in defining equity values, one must make implicit assumptions of a common ideology (Richardson et al., Reference Richardson, Wildman and Robertson2003).

3.1 Research instruments

The goal of this project is to reveal and validate a number of dimensions of (national) health system values to inform policy. This is a difficult endeavour because individuals do not necessarily have a broad and well-defined idea of what they can expect from the health system. Hence, it appears that deliberative and participatory techniques are required to make sure that individuals understand the meaning of ‘health system value’ reasoning. In participatory exercises, participants can focus on the specific questions under discussion, and identify ‘reasoned values’ (Costa-Font et al., Reference Costa-Font, Forns and Sato2015). Alternative techniques, including survey methods, were considered, but were thought to produce even less reasoned responses (and ratings); however, they can be used to indirectly reveal health system values. Another important advantage of deliberative methods for this particular exercise lies in that health system values can be described as ‘holistic’ and not ‘specific to programmes’. Hence, as mentioned before, often preference elicitation techniques suffer from a ‘hypothetical bias’ (Aadland and Caplan, Reference Aadland and Caplan2006), where individuals value a standalone programme rather than considering the wider knock-on effects on health systems. Among the potential alternative methods considered, focus groups are suitable for identifying a wider list of benefit dimensions. As we describe below, focus groups are not absent from issues themselves, but allow for a considered identification of the different sources of health system value.

3.2 Methods and design

We describe an experiment to identify and validate a narrow list of health system value dimensions. The elicitation mechanisms employed were individual based, but mediated by the group, and were followed, by a questionnaire.

In the first step, we divided participants into eight groups (as explained above) which were purposefully heterogeneous but homogenous enough within to allow for a discussion with equal chance of participation and viewpoints. The goal was to elicit very precise questions about the health system from a focus group representing a small-scale social group. Participants in different groups were selected following pre-defined socio-demographic characteristics (as defined in Figure A1). In the facilitator’s manual, ethical considerations were discussed in a specific section, and adequate ethical clearance was obtained. Participants signed a consent form and Oral Interventions were transcribed after the session, and later, results were coded based on such transcriptions. Importantly, participants were asked not to take a simple user/consumer perspective in participating in the different exercises, but to pretend to be a member of a specialised ‘citizens’ board’ with the aim of voicing opinions to the moderator who followed experiment guidelines. We are aware that data focus group participants generate are unique and require ample transcription and processing, and hence, are not unbiased.Footnote ¹ Hence, in designing the experiment we attempted to minimise such biases by specifying the moderator should not intervene or take position in the discussion, and focus on steering the discussion including all participants. Given that discussion took place in groups that share some common features, participants were equipped with information-processing information to balance potential differences between groups due to social background and/or education levels.

In the second step, we attempted to validate the value dimensions elicited in the focus group exercise using a questionnaire and the design of a realistic decision-making reform at the time of the study. We first gathered expert advice from health care planners of the Catalan health service (although the experiment could be based on any collectively funded health system), to design a number of new potential programmes that attained such values. Subsequently, we designed and implemented a ‘validation exercise’ to review whether participants were able to reveal previously elicited values from the programmes designed and identify its dimensions. To examine the results of such exercise, we employ data reduction statistical techniques which reduce the long list of health programmes to a manageable number. The latter qualifies as a ‘validation exercise’ insofar as it allows us to test whether the dimensions elicited in the focus group coincide with those individuals identified in the realistic reform exercise.

3.3 Data and participants’ characteristics

The experiments took place in Barcelona between 2000 and 2004, in the context of an expansive health reform. We expected to undertake a follow-up exercise but this did not happen due to limited funding. Participants were selected from a wide range of pre-established backgrounds (although grouped later according to similar group characteristics), given that the purpose here was to extract as much information as possible without priming considering external validity beyond the country-specific setting. The choice of the different groups was to ensure some degree of heterogeneity such as high-/low-skilled young participants, middle-aged individuals with and without children, older age participants, both high- and low-income individuals, and one representative from an interest group, namely a group of trade unionists. Groups were both urban and non-urban as well as gender balanced. As mentioned, the variety in participant characteristics means we obtain more dynamic and varied responses, whilst containing potential ‘apprehension biases’ resulting from discussing experiences or thoughts that are perceived to be at the other extreme of that of other participants (Smithson, 2000). Before the meeting, participants were given a briefing on the nature of the exercise to provide the necessary context.

Groups were composed of 8–15 participants, and participants were recruited after a careful segmentation of the population so that each group represented a different socio-economic group in the specific area of the study (Catalonia). Details are found in Table A1 of Online Appendix II. Recruitment was carried out by experienced social scientists that identified a group that either was already meeting regularly for some other purpose, or that could be called upon from an existing group. As is standard practice, a reference guide was developed and a facilitator’s manual was drawn up. The focus groups discussed general questions which were part of a template the moderator had to follow. These questions include some information the basic organisation of the Catalan health service and health system. We then proceeded to ask at least the three following questions: ‘How do you judge the health system (in general)?’, ‘What in your view does the health systems do very well?’ and ‘What are the main problems of the health system in your view?’ These questions led to others depending on the dynamic of the focus group.

We expected every participant to contribute to the discussion with both views and arguments and the moderator guidelines explicitly required the involvement of all participants. Simultaneously, a basic questionnaire of the experiment was prepared and mailed to a single contact person in each group which was filled in by group members (before and during the group), after which all information was then made anonymous and processed. The outcome of this step was the identification of a long list of both positive and negative values. Importantly, for the discussion session, the concept of a ‘health system value’ was defined in a broad sense as ‘any increase in value resulting from a new health care programme’. The outcomes that every participant revealed were defined as ‘positive’, or ‘negative’, represented by a positive (+) and negative (−) mark, respectively. We did not define the value dimensions ourselves but instead recoded the reported information. The raw output of this exercise was a two hour (typed) recorded conversation, which was consequently converted into text and interpreted as a list of health system values.

3.4 Validation exercise

Once health system values were identified and categorised, we proceeded to validate value dimensions using a validation exercise. We first drew upon advice from health system policy makers (working for the Catalan government at the time), to define a number of health system programmes, which were expected to attain those values identified in Table A2. Three policy makers (acting as advisors) were consulted, and asked to independently define programmes that would attain at least one of the desired health system values, and that would be ‘realistic enough to consider implementing’ based on their experience in health programme design. Advisors were asked to define a precise description of the programme, but with the understanding that the list would not be exhaustive although the final list of programmes was expected to cover all value dimensions. Policy makers were asked to provide as much information on health care programmes as possible, including its cost (see Figure A1), and were asked to validate the final programme list upon collection of the input of all three advisors.

Once the programme list was designed, the first exercise of the validation process consisted of a direct rating of health system value dimensions each programme would attain. The rating was explained in a visual analogue scale format where the highest and lowest rate could be identified to ensure rating comparability. Sessions lasted about two hours and participants were invited for refreshments and feedback from the first exercise. Before every session, participants were requested to fill in a personal questionnaire, such that socio-demographic information is available for each participant. Participants rated the programmes based on the attainment of five value dimensions: coverage, accessibility, quality, public health and social concerns.

A second exercise was to validate that the public is able to identify a list of value dimensions in programmes designed to attain those values, which was a more limited exercise than the one carried out before (as per Figures A2 and A3 in the Online Appendix). Hence, participants were informed of values identified in the original experiment (in Figure A1 in the Online Appendix), and were asked to associate a number of values to each programme. This was done using a participant survey (in order to quantify the qualitative results) drawing on pre-specified cards where every programme and values were briefly described. Every programme (conveying a health care system improvement) was briefly described to each respondent in separate information cards, randomly.Footnote ² Examples of the description and the cards can be found in Figures A1 and A2 in the Online Appendix. Subsequently, four random groups were also asked to rate on a 1–7 scale the degree of values from each programme. The exercise drew some initial difficulties. To identify each category with their value, we were required to explain what we meant by ‘health system value’. At the same time, we gave examples of possible situations and experiences and allowed participants to comment in order to ensure that they had understood correctly. We then asked to list, in order of preference, the most important values within a health system.

Subsequently, we undertook descriptive statistical analysis of the ratings obtained, this allowed to reduce the number of health system value dimensions identified in the questionnaire using data reduction techniques. Specifically, we used principal components analysis (PCA) (a data reduction methodology) for the data obtained in the validation stage where each participant associated a health system value to each health programme. PCA allows for the identification of the principal directions in which the data varies which are linear combinations of the original variables. The main limitation is that since values are computed in terms of rating rather than cardinally, results do not show large sensitivity to value variations, and therefore tends to limit the explanatory power of the regression results (thus results should be interpreted with caution). We therefore identified the values significantly associated to the overall value without computing its implicit value.