No CrossRef data available.
Article contents
DO PARTICIPANTS UNDERSTAND A STATED PREFERENCE HEALTH SURVEY? A QUALITATIVE APPROACH TO ASSESSING VALIDITY
Published online by Cambridge University Press: 21 April 2004
Abstract
Objectives: Examine the validity of using a self-completed Stated Preference Discrete Choice Modeling (SPDCM) questionnaire to measure parents' preferences for vaccinating their children against varicella.
Methods: A qualitative approach was used to assess the way parents understood the technical information in the questionnaire and the factors they considered to be important to the immunization decision. After completion of the SPDCM questionnaire, thirty-four participants completed a semistructured interview by telephone. Interview transcripts were analyzed by using content analysis. Comparisons were then made with the SPDCM questionnaire results.
Results: The technical information used to describe the program attributes appeared to be used appropriately by participants, although their explanations indicated that their understanding did not always come from the questionnaire information. Only one participant appeared to misunderstand the stated preference task, and a small number thought that the complexity and length should be reduced. The results of analysis of the questionnaire data were supported by the qualitative study, with the notable features of the model being reflected in the views commonly expressed about the immunization decision. Several additional factors were identified as important to the choice, including beliefs about vaccination generally and perceptions of the seriousness of varicella.
Conclusions: Although more research is required to investigate the validity of SPDCM for the measurement of preferences in health care, this study supports the validity of its use in childhood immunization where parents are familiar with the decision context.
Information
- Type
- GENERAL ESSAYS
- Information
- International Journal of Technology Assessment in Health Care , Volume 19 , Issue 4 , December 2003 , pp. 664 - 681
- Copyright
- © 2004 Cambridge University Press
There has been a growing interest in the measurement of patient preferences for health care, and the development of a range of methods for this purpose by health researchers from different disciplines (6). Stated Preference Discrete Choice Modeling (SPDCM), is one such method that is being used increasingly in health economics (see e.g., 5;7;17;21). SPDCM involves the use of “choice experiments” to elicit consumers' preferences, and has been widely used in marketing, transport economics, and environmental economics to estimate demand for new products and nonmarket goods, when market (revealed preference) data are not available. One of the potential applications of this method in health is in areas of health policy where predicting program acceptance and utilization is a critical aspect of the policy decision.
Like many public health programs, the success of an immunization program in reducing mortality and morbidity is dependent on the program uptake rate. The importance of uptake to program success is particularly relevant to the case of the varicella (chickenpox) vaccination, for which the greatest rates of disease mortality and morbidity are seen in adults (10). With a vaccination program in place, unvaccinated children will come into less contact with the disease, and, as the cohort ages, the proportion of susceptible adults may increase. If a high coverage rate is not achieved, a shift in the age distribution of cases could increase overall morbidity and mortality, even though the total number of cases was reduced. Thus accurately predicting the program uptake is essential for evaluation of the potential costs and consequences before making the policy decision.
The SPDCM approach involves the identification of the important attributes that might underlie preferences for the new program, and the levels of each attribute relevant to the research question. Hypothetical scenarios are then constructed using different combinations of the attribute levels. Because the number of possible combinations is usually too large for practical purposes, an experimental design is used to select the combinations such that attribute levels appear with equal frequency and there is no correlation between attributes across scenarios (12). Participants are then asked about their preferences with regard to each hypothetical scenario (stated preference). Statistical modeling of participants' evaluations can provide estimates of the extent to which components of programs (attributes) contribute to preference ordering and, therefore, choices. These can be used to forecast choices or option values.
Thus, SPDCM enables the estimation of the relative importance consumers place on the different components of a health care program and the trade-offs they are willing to make between these components. In the case of varicella vaccination, for which the rate of program acceptance is crucial for the success of the program, it allows for the prediction of program utilization and the estimation of the effects of individual program components on the predictions. The quantification of the effect of program components on utilization allows policy makers to predict program participation under different policy options and enables the development of programs that maximize uptake (22). However, this is dependent on the correct specification of program attributes and levels to address the policy issue and the development of a discrete choice experiment that can be readily administered to and understood by the target population.
To minimize measurement error and maximize the precision of the SPDCM model estimates, it is essential that participants interpret the task and the attributes accurately, and that the survey instrument include the appropriate attributes. Misinterpretations or omission of important attributes are likely to result in an increase in the size of the error component of variance and less precise parameter estimates (random error). However, they may also result in biased estimates (systematic error), for example, if several participants misinterpret the task or an attribute in the same way. Published studies that have examined measurement error in health applications of SPDCM have reported test re-test reliability (8), the proportion of respondents who appear consistent in rationality tests (11;16;18), and scenario ordering effects (18).
SPDCM methods are still new to health economics, and the decision-making context for health care and disease prevention is more complex than for many other goods. Thus, there is still skepticism about their validity and usefulness in studying health care decisions. Research is necessary to assess the extent to which this complexity can be incorporated in a choice experiment that is well understood by the target population. Qualitative research methods permit the detailed examination of a small number of cases and are useful for exploring the way individuals perceive their experiences. They provide a means of investigating participants' opinions of a choice experiment and their understanding of the exercise. Although a qualitative approach is sometimes used at the questionnaire development phase of survey research, it has rarely been reported as an approach to evaluating questionnaire validity. Mallinson (13) analyzed respondents' comments during interviewer administration of the SF-36 health status questionnaire to provide insights into the misinterpretation of questions and the limitations of response options. Paterson and Britten (14) compared qualitative and quantitative data to assess the validity of the MYMOP, a self-completed questionnaire to measure individualized patient outcomes.
This study reports a study that examined the validity of a self-completed SPDCM questionnaire. The questionnaire was designed to assess the value parents would attach to various aspects of a childhood immunization program against varicella, which could then be used to predict program utilization. At the time of the study, a varicella vaccine had not been licensed for use in Australia. A qualitative approach was used to explore the extent to which parents understood the questionnaire and the SPDCM task. The study also examined the factors parents considered to be important to their decisions about the immunization of their children and the extent to which these findings were consistent with their questionnaire responses.
The specific aims of the study were to assess:
- the extent to which parents would understand the technical information used in the SPDCM questionnaire to describe attributes of a new immunization program,
- the way parents used that information to make choices in each scenario, and
- if the models produced by analysis of the questionnaire data were consistent with the most frequently expressed views about what attributes were important to the immunization decision.
METHODS
Recruitment and Data Collection
Participants were recruited from within the Sydney (Australia) metropolitan area, by using two approaches: (i) random telephone recruitment from the electronic White Pages, and (ii) recruitment through schools and child care centers. The recruitment period was from September to November 1999. Eligibility criteria were parent or guardian of at least one child under twelve years of age who has never had chickenpox and able to read English. The SPDCM questionnaires were mailed to recruits, and data were collected by a follow-up telephone interview. The telephone interview comprised two parts: the first recording the participant's responses to the mailed SPDCM questionnaire and the second asking open-ended questions about the participant's understanding of the questionnaire.
The SPDCM questionnaire was mailed to sixty-two eligible recruits and completed by fifty respondents (81%). Of the twelve nonrespondents, seven told the interviewer at every phone call that they had not yet had time to complete the questionnaire, four were not able to be contacted after many attempts (either not available, no answer, or phone disconnected), and one refused to participate. A subsample of questionnaire respondents was invited to participate in the semistructured interview, which was completed by the first thirty-four respondents (no questionnaire respondent refused the semistructured interview). Sample size for this part of the study was based on the need to keep the transcript data to a manageable volume, while ensuring coverage of the range of opinions and interpretations. In addition, it was important to sample a sufficient proportion of questionnaire respondents to allow comparisons with quantitative data.
Questionnaire
The SPDCM questionnaire comprised 16 immunization scenarios, each followed by a question asking parents if they would have their child immunized under the conditions described in the scenario. Each immunization scenario is a combination of seven attributes, selecting one level for each attribute. The selection of attributes and levels and the procedure for creating scenarios have been described elsewhere (9). An example of a scenario and the accompanying question is given in Table 1. Parents were asked to complete the SPDCM questionnaire with regard to a particular child, named on the questionnaire. The questionnaire also included introductory information about chickenpox and the vaccination (including an explanation of mild and severe vaccine side effects). In addition to the SPDCM questions, six closed-format, multiple-choice questions were included at the end of the questionnaire, asking participants how concerned they were about particular levels of mild and severe vaccine side effects and vaccine effectiveness. There were two questions about each attribute, covering different levels, to identify inconsistencies in interpreting the direction of probabilistic information. The questionnaire was mailed to the study recruits approximately one week before the interview. Participants were asked to complete the questionnaire and to have it available to refer to when the interviewer telephoned. Instead of returning the questionnaire by mail, the first part of the telephone interview asked participants to read their answers to the questionnaire items.
Semistructured Interview
The second part of the telephone interview was a semistructured interview that was recorded on audiotape, with the participant's consent. Questions from the interview guide were supplemented with additional questions when required for clarification or to get more detail on a particular issue. Although recruits were informed of the purpose of the interview, they were not provided with a copy of the interview schedule in advance. The interview guide comprised four open-ended questions asking participants about (i) any difficulties they had with completing the questionnaire, (ii) the strategy they used to make their decisions, (iii) the main factors influencing their choices, and (iv) any omitted factors that might have influenced their choices. The fifth and final part of the interview was intended to assess the participant's understanding of three of the questionnaire attributes: mild side effects, severe side effects, and vaccine effectiveness. Participants were asked to explain what they thought was meant by each of these attributes. To minimize burden, participants were not asked to explain the remaining four attributes (cost, location, support from health authorities, and the number of children vaccinated), which were expected to pose the least difficulty for interpretation.
The interviews took between ten and thirty minutes to complete, with the exception of one interview that took more than one hour. Two interviewers each conducted half of the interviews.
Analysis
The taped interviews were transcribed and analyzed using content analysis, described by Patton (15) as identifying, coding, and categorizing primary patterns in the data. A coding system was developed by using Atlas/ti computer software (19). Transcripts were initially coded and sorted under three main headings (from the interview guide): (i) Strategy, (ii) Influential factors, and (iii) Interpretation of attributes. A detailed system of subcodes was then developed from within the data. All transcripts were separately coded by two coders (the interviewers) and then compared. Coding differences between coders were discussed, and agreement was reached on a solution. Another researcher, not directly involved in the study, read all transcripts to provide a check on the interpretation of the data.
The patterns in the detailed codes were then examined, and categories were devised to illustrate similarities and differences between participants. The results were presented in two sections: (i) understanding the questionnaire, and (ii) important decision influences. The first section has two parts covering the interpretation of technical information and understanding the task, whereas the second section covers the range of factors influencing participants' immunization choices and comparison of the two data sources. Category frequency counts were presented in tables, and the categories were explained and illustrated by using verbatim quotation.
Questionnaire data, both SPDCM data and items asking about level of concern regarding attribute probabilities, were also incorporated into the analysis. The additional items about concern regarding various levels of attributes were used (in addition to the qualitative interview data) to assess the participants' ability to interpret the technical information. The six closed-format items (i.e., two questions for each of three attributes) were analyzed by comparing responses between the two questions about the same attribute, for each participant. Questionnaire participants were judged to be consistent in interpreting the attribute probabilities if they reported a higher level of concern about a worse probability on the same attribute.
The SPDCM questionnaire responses were compared with the results from the qualitative interviews, at the group level. Consistency between the two types of data, for the most important or influential attributes, would support questionnaire validity. It was expected that the attributes having the most influence in the statistical model, judged according to both estimate of effect and significance, would also be most frequently identified as important attributes at interview.
RESULTS
Sample
The sociodemographic characteristics of the interview participants were similar to those of the questionnaire respondents overall (Table 2). Half of the interview participants were recruited by random dialing telephone recruitment, with the remainder recruited through schools and day care centers. They were predominantly Australian-born, female, between 30 and 40 years of age, and educated beyond secondary school level (Table 2). Most had one or two children, were in full- or part-time employment, and had a partner who was employed. For all participants, the nominated child had received other vaccinations, but none had received varicella vaccination.
Understanding the SPDCM Questionnaire
Generally, participants said that they found the questionnaire reasonably easy to understand. Almost half said that it was an unfamiliar format to which they became accustomed, after a few scenarios for some and after reading the whole questionnaire for others. Participants' understanding of the questionnaire is described in terms of their ability to interpret the technical information and their understanding of the task.
Interpretation of Attributes
Interpretation of the technical information was assessed for three attributes by using responses to closed-ended questions in the questionnaire and open-ended questions at the interview. For each of the three attributes, the majority of questionnaire responses to the closed-ended items showed consistent interpretation in terms of a higher level of concern about the worst probability level (see Table 3). A substantial minority reported the same level of concern about the high and low probabilities (mild side effects, 34%; severe side effects, 28%; and guarantee of effectiveness, 18%). There was one inconsistent response (i.e., reported a higher level of concern about a better probability for the same attribute) for each of the three attributes but from different participants.
Most interview participants gave an accurate interpretation for each of the three attributes. A small number of participants appeared to have difficulty in expressing their interpretation of some of the attributes; this finding was particularly the case for participants from a non-English speaking background.
The questionnaire information page described mild side effects as follows: “Mild side effects may include an episode of fever, a mild chickenpox like rash, or pain, swelling or redness at the injection site.” Almost all interview participants (29 of 34) gave an example of mild side effects, which was accurate in terms of the questionnaire information page (Table 3). Many of these participants (16) also gave an example of a mild effect that was not on the information page, for example, nausea or headache. More participants referred to their experience with other vaccinations (13) than to the questionnaire information page (7) as the source of their information on mild side effects. In addition, some participants gave an example to indicate their interpretation of the severity of mild side effects. The most frequent of these was that the problem could be fixed with Panadol (paracetamol), mentioned by four participants. Only one participant gave no interpretation of mild side effects, saying she did not know what they could be. The following quotations are some examples of what participants understood by mild side effects.
Well I just took it from what you had written…. that is what I've done it from, I can't see where it's written, but just yes it hurts, you get a sore arm, you might get a temperature and you might be sick for the day or something, that is what I'd assume a mild reaction is. [Participant 21]
Well the things you talked about on the sheets, the nausea, and the headache and I can't remember what the others were, but I suppose it's like the other immunizations. They might have a temperature or a sore leg or something that can make the child feel slightly ill, there might be a bit of nausea, nothing like a fit or fainting spell or anything like that. [Participant 34]
Headache, slight swelling around where you had the injection, maybe slight fever or something like that. I am relating it to other vaccinations that my sons had. And I think maybe I gave them some Panadol, or something like that, just to get them through it. That to me would be mild. [Participant 4]
Mild side effects, I thought maybe a little bit of discomfort, slight temperature, something that Panadol could handle. And that could be a little bit of skin irritation or a little lump or something, that would be mild to me. And a temperature that would go away after Panadol, not continuing or anything like that. [Participant 28]
The questionnaire information page described severe side effects as follows: “Severe side effects are extremely rare and may include sudden collapse or seizure, encephalitis, pneumonia, or death.” Although more than half of the interview participants (21) cited examples of severe side effects, which were accurate in terms of the questionnaire information page, this was a smaller proportion than for mild side effects (Table 3). Again, some participants (12) also gave examples of severe effects that were not on the information page such as brain damage. Only five participants identified their source of information about severe side effects as the questionnaire information page, the media was given by three participants, and experience with other vaccinations by one. Participants were much more likely to give an example to illustrate level of severity for severe side effects than for mild side effects. These examples included something that warranted hospitalization (n=12), was long-lasting (n=6), resulted in function loss (n=3), or was life-threatening (n=2). Again, only one participant gave no interpretation of severe side effects, instead giving his opinion of the probability (a different participant from the person giving no interpretation of mild side effects). The following quotations illustrate some of the ways the interview participants interpreted severe side effects.
I assumed that you meant things like encephalitis or, things that you had in that first page, um pneumonia, that sort of major thing that could have lasting effects. That's how I'd interpret it. [Participant 31]
Brain damage or something like that, that is probably things I have heard in the media that I have heard about other vaccinations that have had severe side effects, children have had brain damage, or death. [Participant 4]
If the child needed hospitalization, if it was life-threatening. [Participant 2]
Severe side effects – something that I'd have to take my child to hospital and be admitted to hospital, that would be a severe side effect. [Participant 20]
The questionnaire information page did not include information about vaccine effectiveness. To assess interpretation of this attribute, participants were asked to explain what was meant by “Vaccination will guarantee children don't get chickenpox for 69 out of every 100 vaccinated.” Most participants (n=30) indicated that they understood that some, but not all, children vaccinated would be protected from the disease. There was some variation as to the range of aspects described (Table 3). Seven participants appropriately quantified the guarantee in terms of those protected and not protected from chickenpox, whereas only four of these specified that it was the vaccine that made the difference (probably assumed by the others).
We have 100 kids in front of us. They have all been immunized, sixty-nine of them won't get chickenpox, but it may well happen that thirty-one of them will. [Participant 6]
I suppose that it was fully effective in 69 cases out of 100, so that they were actually immunized against the disease, and the other 31 may get it still if they come into contact with it. [Participant 29]
A further eighteen participants appropriately quantified one aspect of the guarantee (either those protected or those not protected), whereas fourteen of these participants specified that the vaccine made the difference.
I would think that would mean that at least 69 children out of every 100 are not going to get chickenpox if they are vaccinated. [Participant 1]
That the probability is that 31 out of 100 would get chickenpox. [Participant 23]
Another five participants explained the guarantee, without quantifying it, in terms of the vaccine conferring protection for some of those vaccinated but not all.
Well, I suppose that was getting to whether the vaccination would be effective or not. That even if you had the vaccination, you still had a chance of contracting the disease. [Participant 34]
The responses of three of the remaining four participants indicated only that a high number was preferred, whereas one participant gave no interpretation.
The higher the percentage, the more chance you are in taking it. Depending on the side effects that is. [Participant 9]
Understanding the Task
The participants' understanding of the SPDCM task was revealed from the way they described their strategies for evaluating scenarios and deciding whether or not to immunize. The strategies were classified into three categories: (i) would usually vaccinate anyway, (ii) focused on one or two attributes, and (iii) considered multiple attributes (three or more). Only four participants said that they would usually vaccinate under most circumstances. Three of these participants were able to identify important attributes, which did not affect the choice because all levels were acceptable. Two also identified attributes used to decide which of the three “Yes” options to choose (see Table 1, responses 2, 3, and 4). The following quotation illustrates the different aspects of this approach.
… with the guarantee percentages so high, and me agreeing with the whole process of vaccination, it was pretty straightforward for me… In all the situations I would get the child vaccinated, it is just a matter of how I would do it, whereas if it cost too much, then I would like it to have been an all in one injection, therefore saving me some money. Otherwise, overall, I would have the child vaccinated no matter what. [Participant 10]
This participant found that all the levels of the guarantee of vaccine effectiveness were high enough to warrant vaccination in every scenario, but if the cost was high would only vaccinate with a multivalent injection on the assumption that this would save costs at the next scheduled immunization.
The second approach to the immunization choice was to focus on one or two attributes, reported by thirteen participants. Some of these participants described their consideration of other attributes, but decided that most were not relevant for their choices. Others simply described the decision in terms of the one or two attributes used. One participant explicitly simplified the task because of difficulties evaluating probabilities. The following quotations illustrate all these strategies where a single attribute determined the choice in each case.
The main thing that I was looking at was the severe side effects where it got down to one out of every 100,000 I would answer “No.” The milder side effects I didn't worry about. If the severe side effects went up to the next level, I was answering “Yes.” And as for the rest, the costs didn't bother me, whether I had it done, at a doctor or at school didn't worry me. And even to some degree, whether or not the guarantee they didn't get it, I would still have it done because I would rather have a good chance of not getting it with a vaccine than just taking my chance. [Participant 1]
Well basically, the ones I said “No” to, were because of the cost, they were $75, a lot of people can't afford $75 for one injection, but that is why I put “No” for there, and the other ones, that is why I put “Yes”. [Participant 33]
Look, I suppose it's pretty obvious when you look at my answers, what criteria I looked at, I pretty much relied on whether my local doctor would have recommended the vaccination in those situations, some of them weren't relevant to me at all, like I didn't look at the cost thing at all, because I mean if it's a good thing to do, then it's something you would spend money on. I did look at the statistics to see what the risk factors were, but it's pretty hard to get your head around those, like I mean, that sounds a slight risk if it's 1 in 100,000. So yes I mainly was governed more by if my doctor who knows more about medical things than me recommended the vaccination then I'd probably go with it. [Participant 34]
Although able to eliminate some attributes as irrelevant, Participant 34 still had difficulty assessing the probabilities associated with the remaining attributes. Consequently, she based her choices on the support from health authorities attribute, which includes “Your doctor recommends the vaccine” in three of the four levels.
The third approach to the immunization choice involved assessment of multiple attributes. This approach was reported by sixteen participants, most of whom reported considering between three and five attributes. One participant reported considering the changes for all attributes between scenarios, but was then able to identify three main attributes that were important to the decision process. The following quotations illustrate this approach.
I started with the chance of not getting the chickenpox, and then I worked it out whether it was severe, the chance is less or more of being severe and then whether it was recommended by the doctor and on the standard schedule or not. Finally the cost. [Participant 15]
The severe side effects were a concern, especially when it got down to around 1 in 100,000. With the vaccination guarantee, that you wouldn't get any chickenpox, once it got under 79% I wasn't very keen on it, so 79 to 100 that was good…. The level of support for the vaccine, I tended to take it more seriously, when it was part of a standard schedule, rather than having to remember it myself. [Participant 22]
I looked at every variable, and just compared the changes…. The main thing influencing my choices were the vaccinations that would guarantee the children don't get chickenpox compared, like if it was low, like if more children would get it than wouldn't, then it wasn't worthwhile…. If the guarantee wasn't good – and the side effects, the mild side effects and the severe side effects made a difference as well. [Participant 20]
Many of these participants prioritized the attributes they selected (for example Participant 15), whereas others identified the relevant attributes and levels without a clear priority (Participant 22).
The degree of difficulty in evaluating the scenarios varied, with some participants reading the entire questionnaire several times and going back and changing answers, whereas others worked out the criteria they would use after reading a couple of scenarios. Ten participants, who were spread across all three approaches to the choice, reported some difficulty assessing the differences between scenarios. The following quotations illustrate the range of difficulty (or ease) reported.
The questions were so similar that I did get confused in the end and I actually went back over it after I had filled it out the first time and changed my mind on just about every question that I had answered. The numbers just changed mildly and I also changed my mind as to which bits were more important to me than others. [Participant 24]
Sometimes it was a bit confusing, like you had to read it a couple of times … but then when I started reading all the questions, like I took it on, like it was fine, I sort of understood. [Participant 27]
At first there was a lot of stuff to read, but once I got the pattern it was easy. What I did was I went through a few questions, because at first I had a few problems deciding, and then I thought what is important to me out of the points. [Participant 32]
Six participants thought there were too many scenarios and/or attributes, making the questionnaire either monotonous (five participants) or confusing (one participant). Four reported difficulty with evaluating probabilities, particularly deciding on the importance of the risk of rare events (for example, Participant 34 quoted above). Three thought it would help to provide the same vaccine and disease information about other vaccines already on the schedule (for comparison). There was only one participant who appeared to not fully understand the task, as the following quotation illustrates.
I guess one of the confusions was that you needed to have the questions answered on their individual merit, whereas my inclination was to read all the questions and choose the best option and it was hard for me to kind of make the leap into the imagination and look at them all on their individual merit. [Participant 13]
Perceived Influential Factors
Interview participants were asked to identify the main factors influencing their immunization choices in the questionnaire and then to identify any other factors that might influence their decisions about immunizing children. Most participants saw the questionnaire attributes as important, with a small number of additional factors identified. Of the questionnaire attributes, severe side effects, support from health authorities and guarantee of vaccine effectiveness were most commonly seen as important; whereas the location where the vaccine is available was not seen as important at all. A small number of additional factors were identified; the most frequently reported of these was a belief that vaccination generally is a good thing.
Most participants identified two or three attributes that were important to their immunization decision. Only one participant did not identify an important attribute. The majority of participants (twenty-six of thirty-four) reported severe side effects as being important to the decision (Table 4). Half of the participants also thought that support from health authorities was important. Within this attribute, doctor recommends or school requirement were mentioned more frequently than having the vaccine on the immunization schedule. Vaccine effectiveness, mild side effects, price, and number of children vaccinated were mentioned as important factors by fewer than half of the participants. No participant identified location as important, and those who mentioned this attribute were happy to obtain the vaccine from their local doctor (the levels for this attribute were (i) local doctor only, and (2) local doctor or school/child health clinic).
Several other factors (apart from the SPDCM attributes) were also mentioned as important to the immunization decision. The most frequently mentioned of these was a belief in the benefits of immunization generally, which was reported by eight participants. The following quotations illustrate this attitude.
… because I tend to have a reasonably strong viewpoint on the good factor of getting immunizations, maybe that's also a factor that came into answering the survey because I would tend to do it rather than not. [Participant 31]
Well I think that a child has to be vaccinated, if you really want a child to be vaccinated and not get this um chickenpox. I mean my mind is already made up…. I've always wanted to you know, get my children vaccinated like for every illness. [Participant 30]
Because I don't object to vaccinations, I actually think they are very important, so when I went through the questionnaire it didn't bother me what the percentage was of the vaccination being effective. Provided of course it was a high percentage, if the vaccination was only say 10% effective, then maybe I would have rethought. However, with the guarantee percentages so high, and me agreeing with the whole process of vaccination, it was pretty straightforward for me. [Participant 10]
Mainly what I was more concerned about was the severe side effects. But I mean that it's such a big probability, so that was fine…. I am agreeable with immunization so, I didn't have a problem with that 1 in 100,000 children get a severe side effect, that is what it is with any other vaccination, so it didn't upset me. [Participant 27]
Although this attitude meant that the decision was already made for some of these participants (for example, Participant 30), many of these participants believed that this was simply an attitude underlying their evaluation of the scenarios. Six of the eight participants who mentioned this issue had answered “Yes” to every scenario in the questionnaire. Most of these participants were still able to identify attributes that were important to their decisions, but found all the levels acceptable (for example, Participants 10 and 27). Only one participant did not identify any attributes as important (Participant 30).
The participants' perceptions of the severity of chickenpox were also identified as influencing the decision. Five participants raised concerns about whether or not chickenpox was sufficiently serious to warrant vaccination. Another four participants described chickenpox as unpleasant and something to be avoided. Both viewpoints are illustrated below.
I'm not that much of an advocate for it because I don't think it's that bad a disease from what I've seen, provided they have it young. [Participant 3]
I had always thought that chickenpox was a fairly minor sort of thing. So again that is why I thought I'd decide on my doctor's recommendation, because I suppose I thought “Why would you vaccinate if it was a minor childhood thing?” [Participant 34]
I would vaccinate my children, just because it's [chickenpox] such a horrible experience to go through, not painful or anything, but just not a very good experience to go through, getting it all through your hair, and that is what I remember about it. [Participant 27]
You see chickenpox itself, if you get a nasty dose of it, it is a fairly unpleasant thing to have even as a child and it can scar you quite badly if you get it all over your face and things. I still carry some so I would prefer them not to have it. [Participant 1]
Two participants said that they would be more likely to immunize an older child. Three participants thought difficulties with child care arrangements were important to the decision, increasing the desire to avoid infectious childhood illnesses. A small number of participants identified additional vaccine information, not included in the questionnaire, as potentially important to the decision had it been available. This included the availability of an oral vaccine, the effect of the vaccine on disease severity if contracted after vaccination, duration of immunity after vaccination, and the effect of the vaccination on the risk of shingles as an adult.
Consistency Between Interview and Questionnaire Data
The results of the quantitative analysis were consistent with the qualitative results. They showed a significant effect on the immunization decision by all attributes except location of vaccination. The size of the effect was greatest for the severe side effects attribute. The main effects model (9) predicted that a reduction in the risk of severe side effects (from 1 in 100,000 to 1 in 10,000,000) would increase the probability of immunizing by 30% (from 63% to 93%). The support from health authorities attribute showed the next largest effect size. Making immunization a school requirement, including it on the schedule, and having a general practitioner recommend immunization would increase the probability of immunizing by 20%, to 88% (compared with 68% if the vaccination was available but patients had to ask their doctors for it). The four remaining attributes (mild side effects, vaccine effectiveness, price, and number of children vaccinated) had similar effect sizes, changing the probability of immunizing by between 8% and 13%. These effect sizes predicted by the model are reflected in the frequency with which participants reported attributes as important to their decisions (Table 4).
DISCUSSION
To draw valid inferences from a population survey, it is essential that most members of the target population interpret the survey instrument as the researchers intended. This study used a qualitative approach to assess parents' understanding of a self-completed SPDCM questionnaire for assessing the uptake of a new childhood vaccination against chickenpox. The majority of participants appeared to understand the questionnaire, and the qualitative study generally supports the results of the quantitative analysis. The qualitative approach provides the advantage of assessing participants' views in greater detail and from a different perspective to the survey methods. Thus, it has the capacity to identify misunderstandings of either the survey information or the task. It also permits the identification of any major inconsistencies between participants' views and the inferences drawn from the quantitative analysis.
The sampling frame for the study was English-speaking parents of children six months to twelve years of age who had not yet had chickenpox. Consequently, elderly participants were not recruited as the survey content would have been irrelevant. Although the use of telephone interviews for data collection may have excluded some potential participants, this bias is expected to be minimal as most Australian households have a telephone (97.5% of households in 1996) (2). The survey response rate was 81%. Although it is possible that nonresponse was related to difficulties with the questionnaire, we think that a busy lifestyle is the more likely explanation. The logistics of arranging interviews was generally difficult in this target group because of work and family commitments; the nonresponders we were able to contact said they had not had time to do the survey, and the level of education was higher among the 12 nonresponders. The participants were predominantly females with a post-school qualification. The higher response rate from females probably reflects gender roles where, in many households, mothers take more responsibility for the health care of children. This finding is not a problem for the study, if these mothers normally take responsibility for such decisions. The proportion of participants with a post-school qualification (60%) slightly overrepresents the proportion for 25- to 54-year-olds in the Australian population (nearly 52%) (1). The study does include participants with a wide range of educational levels, which is more important to this study than the proportions represented. Although the results did not generally differ by education level, the participants who had most difficulty in expressing their interpretation of attributes did not have a post-school qualification.
The technical information used to describe the program attributes appeared to be used appropriately by participants, although their explanations indicated that their understanding did not always come from the questionnaire information. In the case of mild side effects, many participants tended to rely on their experience with other vaccinations, whereas the media was sometimes mentioned as a source of information about severe side effects. Most participants were able to accurately quantify the effectiveness attribute, and almost all participants demonstrated an understanding of the direction of the probabilities. The difficulties evaluating probabilities (reported by a small number of participants) were generally related to deciding on the importance of a rare but severe effect, rather than understanding the probability per se. Several participants believed that providing similar information about other vaccinations currently on the schedule (for comparison) would make this assessment easier.
Parents have to make immunization decisions, regardless of their degree of understanding or preparedness to read information. However, the actual decision will probably be made in consultation with a doctor where the degree of understanding will depend on the doctor's explanation and response to questions. The nature of the medical encounter will also influence the extent to which the decision strategy adopted in the survey is similar to that of the actual immunization decision. Participants drew on their prior knowledge and experience when completing the survey, which would also be expected when making actual immunization decisions. The high rate of reliance on experience, rather than the information provided, suggests that SPDCM may be readily applied to areas of health where participants are familiar with the decision context and generally well-informed about the attributes (which is often not the case in health care decisions).
The group results for the questionnaire data were supported by the qualitative study, with the notable features of the model being reflected in the views commonly expressed about the immunization decision. Most participants were concerned about the risk of severe side effects, and half said they were influenced by the level of support for the vaccine from health authorities. These were also the attributes that the model predicted would have the greatest effect on immunization uptake. This result provides some evidence for the validity of SPDCM in this context. However, it is possible that these results would not be achieved in another setting where participants are less familiar with the decision-making context.
As a potential source of error, attribute selection can be as important as interpretation (3). In the current study, although participants identified some additional factors that might influence the decision, important omitted attributes were not identified. The most frequently identified additional factors related to individual beliefs and perceptions about vaccinations or the disease, which are attributes of the individual rather than the program. These can be accounted for in SPDCM by including them as covariates. One attribute, location at which the vaccination is available, appeared to be redundant in this sample. However, this finding may not have been the case had the study recruited rural residents.
Only one participant appeared to misunderstand the SPDCM task; this participant was educated to tertiary level. Although this finding suggests that interviewer-administration would be less prone to measurement error than the self-administered method of the current study, it is possible that (given the expense of interviews for large studies) the problem could be avoided with more carefully worded survey instructions which clarify the task. A small number thought that the complexity (seven attributes) and length (sixteen scenarios) should be reduced for ease of completion. Although we have found no published evidence about the number of scenarios an individual can cope with, only one participant in the current study reported finding the number of scenarios confusing. Bennett and Blamey (4) identified cognitive burden as a potential problem for SPDCM, where task complexity or responder fatigue could lead to the increased use of simplifying decision strategies.
The strategies used to make choices ranged from the apparently simple, for example always agreeing to immunize or basing the choice on one attribute, to a more complex approach using multiple attributes. Only one participant reported considering the changes in all attributes between scenarios. The participants who agreed to immunize in every scenario identified themselves as favoring immunization or found all attribute levels acceptable (or both). Although only one of the participants who based the decision on a single attribute explained this as simplifying a task she found difficult, it is possible that this was also the case for others. However, the use of a simplifying decision strategy does not necessarily indicate a problem with the SPDCM task per se, but may well reflect the approach taken when making actual complex decisions under conditions of uncertainty. There is considerable evidence that individuals use simplifying strategies when making judgments about uncertain events; see, for example, Tversky and Kahneman (20). The challenge here is to decide if the level of use of such strategies is appropriate and, therefore, likely to produce reliable estimates of value (4).
The lack of trading across attributes by a substantial proportion of participants may indicate the need for a wider range of attribute levels. However, the levels were chosen to be within a realistic range. For example, we would not expect health authorities to license extremely ineffective vaccinations, and to include such levels in the experiment may have compromised it overall. In addition, if there are individuals who are unwilling to trade across the range of attribute levels appropriate to the policy context, the inclusion of this information is essential for the accurate prediction of uptake and the accurate estimation of marginal effects.
The study used semistructured interview data to provide insights into the way respondents understood the questionnaire and used the attributes to make choices. Although this approach has been used previously to assess questionnaire validity (13;14), we found no published studies of its use to examine the validity of a SPDCM questionnaire. Although more research is required to investigate the validity of SPDCM for the measurement of preferences in health care, this study supports the validity of its use in childhood immunization where parents are familiar with the decision context. Although revealed preference is the true test of validity, this can only be assessed after the introduction of the program. The evidence provided by this study suggests that SPDCM could be a useful tool for predicting parents' choices in childhood immunization, before deciding whether or not to introduce a new vaccination. It also contributes to the as yet sparse literature examining the validity of these methods in health.
POLICY IMPLICATIONS
The study shows that a Stated Preference Discrete Choice Modeling questionnaire can be a valid instrument for collecting data on parents' preferences in childhood immunization and lends support for the application of this approach for predicting the uptake of new vaccines.
The authors are grateful to Professor Jordan Louviere and Ms. Madeleine King for comments on an earlier draft of this manuscript and to Ms. Angela Yeoh who assisted with data collection. This work was supported by a Research Program Grant from the Medical Foundation of the University of Sydney.
References
Australian Bureau of Statistics.
2000
Transition from education to work.
Report No.: 6227.0.
Canberra:
Australian Bureau of Statistics
Australian Bureau of Statistics.
2001
Australian social trends.
Report No.: 4102.0.
Canberra:
Australian Bureau of Statistics
Bennett J,
Adamowicz V.
2001
Some fundamentals of environmental choice modelling.
In: Bennett J,
Blamey R, eds.
The choice modelling approach to environmental valuation.
Cheltenham:
Edward Elgar
Bennett J,
Blamey R.
2001
The strengths and weaknesses of environmental choice modelling.
In: Bennett J,
Blamey R, eds.
The choice modelling approach to environmental valuation.
Cheltenham:
Edward Elgar
Booske BC,
Sainfort F,
Hundt AS.
1999
Eliciting consumer preferences for health plans.
Health Serv Res.
34:
839-
854.Google Scholar
Bowling A,
Ebrahim S.
2001
Measuring patients' preferences for treatment and perceptions of risk.
Qual Health Care.
10:
i2-
i8.Google Scholar
Bryan S,
Buxton M,
Sheldon R,
Grant A.
1998
Magnetic resonance imaging for the investigation of knee injuries: An investigation of preferences.
Health Econ.
7:
595-
603.Google Scholar
Bryan S,
Gold L,
Sheldon R,
Buxton M.
2000
Preference measurement using conjoint methods: An empirical investigation of reliability.
Health Econ.
9:
385-
395.Google Scholar
Hall J,
Kenny P,
King M et al.
2002
Using stated preference discrete choice modelling to evaluate the introduction of varicella vaccination.
Health Econ.
11:
457-
465.Google Scholar
Halloran ME,
Cochi SL,
Lieu TA,
Wharton M,
Fehrs L.
1994
Theoretical epidemiologic and morbidity effects of routine varicella immunization of preschool children in the United States.
Am J Epidemiol.
140:
81-
104.Google Scholar
Jan S,
Mooney G,
Ryan M,
Bruggemenn K,
Alexander K.
2000
The use of conjoint analysis to elicit community preferences in public health research: A case study of hospital services in south Australia.
Aust N Z J Public Health.
24:
64-
70.Google Scholar
Louviere J,
Hensher D,
Swait J.
2000
Stated choice methods: Theory methods and applications.
Melbourne:
Cambridge University Press
Mallinson S.
2002
Listening to participants: A qualitative assessment of the short-form 36 health status questionnaire.
Soc Sci Med.
54:
11-
21.Google Scholar
Paterson C,
Britten N.
2000
In pursuit of patient-centred outcomes: A qualitative evaluation of the “measure yourself medical outcome profile.”
J Health Serv Res Policy.
5:
27-
36.Google Scholar
Patton MQ.
1990
Qualitative evaluation and research methods.
2nd ed.
Newbury Park, California:
Sage
Ryan M,
Bate A,
Eastmond CJ,
Ludbrook A.
2001
Use of discrete choice experiments to elicit preferences.
Qual Health Care.
10:
i55-
i60.Google Scholar
Ryan M,
Hughes J.
1997
Using conjoint analysis to assess women's preferences for miscarriage management.
Health Econ.
6:
261-
273.Google Scholar
Ryan M,
McIntosh E,
Shackley P.
1998
Methodological issues in the application of conjoint analysis in health care.
Health Econ.
7:
373-
378.Google Scholar
Scientific Software Development.
1997
Atlas/ti visual qualitative data analysis management theory building: User's guide and reference.
Berlin:
Scientific Software Development
Tversky A,
Kahneman D.
1982
Judgment under uncertainty: Heuristics and biases.
In: Kahneman D,
Slovic P,
Tversky A, eds.
Judgment under uncertainty: Heuristics and biases.
Cambridge:
Cambridge University Press
Van der Pol M,
Cairns J.
1998
Establishing patient preferences for blood transfusion support: An application of conjoint analysis.
J Health Serv Res Policy.
3:
70-
76.Google Scholar
Viney R,
Lancsar E,
Louviere J.
2002
Discrete choice experiments to measure consumer preferences for health and healthcare.
Expert Rev Pharmacoeconomics Outcomes Res.
2:
319-
326.Google Scholar
Example of a Scenario and Question from the DCM Questionnaire
Percentage of Interview and Questionnaire Participants with Various Sociodemographic Characteristics
Frequency of Participants' Ability to Interpret Attributes and Probabilities for Mild Side Effects, Severe Side Effects, and Vaccine Effectiveness
Frequency of Factors Reported in Interview as Influencing the Immunization Decision and Effect of Attribute in SPDCM
You have
Access
- 15
- Cited by