Decision making in health care is a process that moves from evidence generation to deliberation and communication of the decision made. Health technology assessment (HTA) is only a part of this process whereby the best available evidence is assessed with the aim of informing decision makers about the most efficient use of resources under conditions of uncertainty. Reimbursement decision making also involves appraising the available evidence, while bearing in mind societal values and ethical considerations (Reference Miot, Wagner, Khoury, Rindress and Goetghebeur1).
While multiple studies and publications have examined the role of HTA through the collection of data, there is still limited knowledge of the perceived needs and expectations of decision makers regarding its use as a source of evidence, as well as the challenge of incorporating other broader criteria in an explicit manner. Drummond and Sorenson (Reference Drummond and Sorenson2) suggested a “divorce” between the evidence produced and decision-making processes, since many published economic evaluations are performed with no specific decision maker in mind. Indeed, Zwart-van Rijkom et al. (Reference Zwart-van Rijkom, Leufkens, Busschbach, Broekmans and Rutten3) found that most decision makers do not want to base their decisions strictly on cost-effectiveness rankings. According to Miot et al. (Reference Miot, Wagner, Khoury, Rindress and Goetghebeur1), systematic and transparent approaches to priority setting are needed to produce decisions that are sound and acceptable to stakeholders.
Country-specific HTA organizations and processes for priority setting have emerged in recent decades and principles such as transparency, robust and appropriate methods for combining costs and benefits, explicit characterization of uncertainty, and active engagement with stakeholders have been associated with the robust operation of HTA programs and institutions. Nonetheless, according to Pichon-Riviere et al. (Reference Pichon-Riviere, Augustovski and Rubinstein4) in Latin America, “the current level of application [of these principles] is considered uniformly poor”.
All health systems face the challenge of managing finite resources to address unlimited demand for services (Reference Glassman, Chalkidou and Giedion5). Of worth noting that even after robust HTA is conducted, transparent and systematic decision making cannot be guaranteed. Rational decision making requires an efficient and explicit process to ensure transparency and consistency of factors considered (Reference Goetghebeur, Wagner and Khoury6). Indeed, even in countries where formal HTA activities are ongoing, and in most LMICs, rationing still occurs as an ad-hoc, haphazard series of nontransparent choices that reflect the competing interests of governments, donors, and other stakeholders (Reference Glassman, Chalkidou and Giedion5).
Multi-criteria decision analysis (MCDA) has been widely used in supporting decisions in transport, agriculture, marketing, and environmental engineering, and more recently it has emerged as a tool to support decision making in health care at many levels (Reference Miot, Wagner, Khoury, Rindress and Goetghebeur1;Reference Baltussen, Ten Asbroek and Koolman7–Reference Radaelli, Lettieri and Masella9). MCDA approaches are “especially helpful when there is a need to combine ‘hard data’ with subjective preferences or to make trade-offs that involve multiple decision makers” (Reference Dolan8). In MCDA, the decision problem (e.g., the evaluation/choice of an intervention) is analyzed to identify all the factors (i.e., criteria), bearing in mind MCDA principles of completeness, nonredundancy, mutual independence, and operationality that contribute to its evaluation (or value), and thus develop ex-ante, a full set of decision criteria.
The next step of the process is to ask decision makers to make their values and objectives explicit by assigning weights to each stated criterion. Next, the performance of each intervention of interest is evaluated using predefined scoring scales for each relevant criterion. This process forces decision makers to think hard about what they value, why they value it, and how they value it, thus promoting discussion across stakeholders (Reference Goetghebeur, Wagner and Khoury6). By combining weights and scores, an aggregated measure of each intervention of interest can be produced.
Building on ethical, MCDA, and HTA principles, a framework known as Evidence and Value: Impact on Decision-Making (EVIDEM) was developed in 2008. The framework has been: adapted, tested, and implemented for clinical and resource-allocation decision making in developed and developing countries such as Canada (Reference Tony, Wagner and Khoury10), South Africa (Reference Miot, Wagner, Khoury, Rindress and Goetghebeur1), and Italy (Reference Radaelli, Lettieri and Masella9). EVIDEM is updated through a not-for-profit open source international collaborative network.
In 2012, Colombia experienced an important institutional transformation after the establishment (in September) of the Health Technology Assessment Institute (IETS) aimed at better informing coverage decision making, later that year (December) the Regulatory Commission for Health (CRES) was disbanded and the Ministry of Health and Social Protection (MoHSP) regained reimbursement decision-making powers. This environment provided the opportunity to test an MCDA approach for systematic and more transparent resource-allocation decision making in this setting and serve the interest of the MoHSP of Colombia on establishing a more robust resource-allocation process.
METHODS
Overview
The general methodological approach undertaken and described is similar to that previously followed in South Africa and Canada (Reference Miot, Wagner, Khoury, Rindress and Goetghebeur1;Reference Goetghebeur, Wagner and Khoury6) to merge HTA and MCDA for coverage decision making (Figure 1). In a first step developed by the former CRES in October 2012, a panel of senior decision makers adapted the EVIDEM framework and defined fifteen relevant criteria; 201 citizens around the country then weighted these criteria.
Scheme of work for piloting the MCDA approach.
Following on this after CRES was disbanded, the principal investigator conducted literature searches for four selected competing healthcare technologies and produced MCDA evidence matrices using the predefined criteria by CRES. During the appraisal session, a group of panelists were asked to score each criterion for these four technologies of interest. Weights and scores were combined to obtain an MCDA value estimate per-technology to rank all of them, following the EVIDEM approach.
To explore their preferences regarding methods, panelists were also presented with narrative HTA reports alongside comprehensive budget impact analyses (BIAs) for each technology (as it was the approach used in Colombia for updating the publicly financed benefits package, POS). When using this second approach participants were asked to directly rank technologies based on the narrative report content. Structured discussion to reflect on appraisal methods followed the two prioritizing methods.
Adaptation of EVIDEM and Weighting of Criteria
During three workshops organized by CRES in 2012, twelve local stakeholders (three academics, five researchers, three civil servants, and one senior hospital manager), were presented with the list of criteria of the EVIDEM framework (listed in Table 1) and asked to nominate a list of additional contextual aspects they considered as relevant for resource-allocation decision making in the country. Once the panel had agreed on the final criteria and their definitions, participants were asked to weight each criterion irrespective of any healthcare intervention, meaning that weights were assigned ex-ante of scoring any intervention of interest.
Final List of Criteria and Weights for the MCDA Approach
CRES used a ranking voting system to assign weights (Borda count) that departed from the 1 to 5 range used by the EVIDEM collaboration; because there were fifteen criteria, each time a criterion was ranked first it obtained fifteen points, subsequently the one that was ranked 15th was assigned one point. The calculation of weights was performed by aggregating points per-participant and dividing total points by the number of participants. The weights were normalized by distributing them across the number of criteria to sum up to 1.
CRES delivered further consensus meetings with different stakeholders around the country to disseminate the selected list of criteria and ask further participants to weight them. In total of 201 individuals including patients’ associations, citizen's councils, and representatives from the medical societies around the country and randomly chosen by CRES voted on their level of agreement and preferences regarding each of the fifteen criteria (11). Weights obtained from all three groups of participants (experts, patients’ associations, academics) were averaged. For the final list of criteria and weights see Table 1.
MCDA Evidence Matrices and Narrative HTA Reports for Selected Technologies
Primary prophylaxis (PP) with exogenous clotting Factor VIII for severe hemophilia A (SHA), zinc supply for diarrhea prevention (ADI), anastrozole as first line therapy for hormone-receptor-positive postmenopausal women with metastatic breast cancer (BC), and ticagrelor + acetylsalicylic acid for patients with acute coronary syndrome without ST elevation and moderate to high cardiovascular risk (CVD) were chosen for the pilot. Technology selection was based on convenience because over the past 2 years local HTA summaries following good HTA practice for these interventions had been published. In addition, all three non–hemophilia-related technologies were considered as potentially cost-effective whilst prophylaxis was not. At the time of running the pilot, no reimbursement decision had been made as to whether they would be publicly reimbursed.
The clinical practice guidelines for BC by Perry et al. in 2012 (Reference Perry, Garcia and Diaz12), ADI by Florez et al. (Reference Florez, Lugo and Tamayo13), and CVD by Senior et al. (Reference Senior, Lugo and Acosta14) in 2013 were used to produce the HTA reports. In the case of PP, the HTA report made use of publicly available data, a literature review and a cost-utility analysis (CUA) by Castro et al. (Reference Castro15–17). The EVIDEM by-criterion MCDA matrix (modified version) was used to assemble the HTA information of the four technologies in Spanish. All reports contained the relevant information organized as per each of the fifteen weighted criteria by CRES in 2012.
The pilot study was also intended to inform the MoHSP on how to design a process to update POS in Colombia; therefore, the standard approach to inform coverage decision making in use in this context was also tested. This consists of narrative HTA synthesis of evidence not summarized by criteria, supplemented by a comprehensive BIAs. BIAs were prepared presenting the result of subtracting the average annual cost of a new technology (new scenario) from the annual cost of the current technology (current scenario). Only average cost per-year (year one) of interventions if they were incorporated into POS were considered.
Appraisal of Technologies and Promoting Discussion
A focus group approach was considered to be the most suitable to explore participants’ reasoning, the feasibility of the proposed approach and policy implications (Reference Elliott, Heesterbeek, Lukensmeyer and Slocum18;Reference Britten19). Eligibility characteristics of participants for the focus group were established before sampling started, these differed form CRES’ approach because this was aimed at emulating a decision-making committee instead of conducting an academic exercise to assign weights. Twelve organizations were identified as containing potential sources of participants (government, insurers, providers, patients groups, academics, healthcare professionals, people's advocates, and lay members). Individuals representing these relevant organizations were invited to participate; however, to the focus group, only seven members were able to attend, all sectors were fully represented by this mix of participants (Supplementary Table 1 presents a summary of their profile). It is of worth noting that, because eligibility criteria differed from CRES’ work, only one participant had previously participated during the weighting stage in 2012 also participated during the scoring session.
A 2-hour focus group was held in the conference room of IETS to evaluate the usefulness of the EVIDEM framework and test the current method of appraisal in Colombia. This was performed within the Chatham house approach whereby comments remain anonymous; the discussion was recorded upon consent of all participants, they were also asked to declare any potential conflicts of interest with the group of technologies to be appraised.
The feasibility of using and incorporating HTA to inform resource-allocation decision making, their perception of MCDA, and their policy implications were explored through a set of open-ended questions to allow participants to guide the discussion, and diverge if necessary to address ideas and concepts not anticipated by the moderator (Reference Green and Thorogood20).
To appraise the four technologies with the MCDA approach, participants were presented with an MCDA evidence matrix with synthesized evidence for each criteria as per CRES approach. Because they seemed to agree on the relevant criteria were then asked to score each intervention on a 4-point scale (0–3), with predefined anchors where 3 represents the highest level of fulfilment of the decision criterion and 0 the lowest.
For testing the standard narrative HTA approach, participants were presented with four mini-HTA narrative reports supplemented by detailed BIAs and asked to rank each intervention on a scale from 1 to 4 based on the merits of the narrative description and BIAs, where 1 represented the highest priority intervention and 4 the lowest. Average times of assessment per-technology and per-participant were also measured.
The last part of the focus group aimed at discussing the results and their policy implications. The following questions were posed to participants to gather their inputs, concerns, and expectations about the MCDA approach. For example: was there enough information to make resource-allocation decisions in Colombia? Participants were also asked about which of the methods of presenting HTA information they preferred and what changes or improvements could be added to the processes and methods presented in the pilot for future implementation. At the conclusion of the meeting, the main points of view of participants were summarized and validated for accuracy.
Analyses
The calculation of the MCDA value estimates was performed using a linear model, in which the value of a parameter for a given value of a factor is assumed to be equal to a+ bx, where a is a constant, and b is the coefficient on variable x. In this case b is the weight of each criterion (already assigned by CRES) and x the score by-criterion, and a is the aggregated weighted score of any previous criterion. The scores were normalized by dividing them by the maximum possible score of 3.
Hence, the MCDA value estimates would range between 0 and 1 as a sum of combined weights and scores for all decision criteria (with 1 as the highest value of an intervention, perceived as an “ideal” intervention according to criteria, and 0 the lowest). Once weights and scores were obtained, they were tabulated using Excel software and descriptive statistics were used to calculate the mean values. Results were presented to participants after testing the alternative approach requested by the MoHSP to assist decision making at the end of the second part of the session.
For the ranking scheme, the number of points given to each technology depended on the total number of technologies being considered; because there were four competing technologies, each time a technology was ranked as first it obtained 4 points and the one that was ranked fourth assigned 1 point (Borda count). The aggregated mean value was obtained by dividing the individual ranking score over total number of participants; this according to the original authors seems to be more sensitive than estimating ordinal average scores. According to this system, ranking first would mean the technology with the highest value and thus higher priority of reimbursement; the opposite would be true for ranking fourth. Using this second format, an aggregated score per-person per-technology, rather than by-criterion, was assigned.
RESULTS
Value of Interventions: MCDA Estimates
Table 2 reports the overall MCDA value estimate for each intervention, as well as the contribution of each criterion to this value estimate. MCDA value estimate was highest for zinc supplementation for acute diarrhea (0.904), followed by anastrozole for metastatic breast cancer (0.822), primary prophylaxis (PP) of severe hemophilia (0.792), and ticagrelor for acute coronary syndrome (0.708). The three most important contributors to value for the four technologies of interest were disease severity, size of population, and improvement of efficacy.
Results of the Comparative Value of Interventions and Primary Prophylaxis by Criterion: MCDA approach
* Primary prophylaxis with exogenous clotting Factor VIII
MCDA estimates of perceived value for all technologies varied across participants ranging 0.782 to 0.986 for zinc, anastrozole (0.698–0.934), PP (0.595–0.977), and ticagrelor (0.449–0.945) reflecting the diverse perspective of participants. Details of these variations at the criteria level are illustrated also in Table 2.
Value of Interventions: Direct Ranking
Using the direct ranking method and after presenting narrative reports and comprehensive BIAs, ticagrelor ranked first with an overall score of 24 points followed by zinc (19 points), anastrozole (17 points), and PP (10 points) (Table 3). It is noteworthy that ticagrelor, which was at the bottom of the list with the MCDA approach, climbed to the top with the direct ranking system, of worth noting that it was the only technology considered as cost-saving.
Results of the Direct Ranking of Interventions
a Primary prophylaxis with exogenous clotting Factor VIII
Participants’ Input, Concerns, and Expectations
According to participants’ remarks certain criteria emerged as creating more difficulties for interpretation and evaluation than others; these included the criteria Clinical guidelines (which at the end was not considered), the Financial impact of interventions (created potential confusion in scoring less costly interventions versus costly ones), and Vulnerability, which was perceived as ambiguous and should be better defined.
To the question, “Was there enough information to make resource-allocation decisions?” participants’ perceptions diverged. Some considered the information insufficient, because of gaps or heterogeneous quality of reporting; others thought it was sufficient. There was extensive discussion among participants to reach an agreement on their preferences about methods. Participants found the MCDA evidence matrix useful and comprehensive to inform their preferences, but also considered a detailed budget impact analysis a useful tool to assist coverage decisions. Participants came to a final agreement that a mixed methods approach including an MCDA evidence matrix for estimation of the holistic value of the interventions, supplemented by a detailed budget impact analysis, would be ideal.
To the question, “What changes or improvements could be added to the processes and methods presented in the pilot for future implementation?” participants answered that more training is needed to use these tools appropriately. Participants also considered that decision making in Colombia should go beyond the use of incremental cost-effectiveness ratios (ICERs). According to them MCDA served as a means to incorporate HTA into decision making, but also to prioritize different health interventions for decision making. According to participants’ responses systematic priority setting should take place in Colombia.
DISCUSSION
This study sought to preserve key principles of qualitative research. Data were collected through a focus group with relevant stakeholders resembling a decision-making body. It was able to capture the nature and intensity of stakeholders’ concerns and values, and also to obtain a snapshot of their opinions and reactions about the methods presented, also to collect details of their needs relating to information (Reference Britten19).
The results should be considered in light of the study limitations. The limited number of participants who finally attended the focus group may well represent a limitation of the study. Of special interest is the fact that the group of participants who chose and weighted criteria were not the same that score technologies of interest potentially having an impact on the consistency of reports. This latter group was not previously trained to familiarize with MCDA methods.
Also among the limitations, the language differences between the original EVIDEM tools used (matrix and by-criterion) published in English and the nonvalidated Spanish versions used. Also no strategy was considered to consistently synthesize or standardize HTA evidence or avoid double counting (consideration of the same evidence in multiple criteria).
The research processes aimed to minimize the values and assumptions of the moderator, but as noted by Green and Thorogood (Reference Green and Thorogood20), whenever conducting qualitative enquiry “it is impossible to have a field for study that is untainted by values and impossible for the researcher to stand outside those values and subjectivities”. The Delphi approach used to select the contextual criteria was previously used in MCDA applications (Reference Baltussen, Ten Asbroek and Koolman7), while the Borda count used for weighting provided an innovative approach, may have had an impact on final weights and MCDA value estimates by making respondents to shy away from extremes as it happens with other rating scales (Reference Robinson, Loomes and Jones-Lee21).
This study showed benefits of MCDA approaches adaptable to specific contexts, which provide the means to reveal the perspectives of decision makers and facilitate discussion and consensus seeking on recommendations and decisions. Consistent application of an MCDA model by a standing decision-making committee would produce a robust ranking of interventions, bearing in mind that it should not be used as a formulaic prescription but more as a supporting tool for decision making and prioritizing.
MCDA appeared superior to direct ranking technologies based on standard HTA reports as it makes explicit consideration of relevant aspects into decision making by encouraging decision makers to think hard about what they value, why they value it, and in what context they value it. These findings are in line with those of Tanios et al. (Reference Tanios, Wagner and Tony22) on decision-makers’ perceptions of the relevance of a core set of criteria and on the need to consider a wider range of criteria to assist decision making. Also with Guindo et al. (Reference Guindo, Wagner and Baltussen23) on the perceived importance of considering both normative and feasibility criteria for fair allocation of resources and optimized decision making.
Participant perceptions differed as to the adequacy of information presented in the MCDA evidence matrix. Some specific criteria represented more challenges for interpretation and valuation by participants than others did. Some concerns and considerations emerged from participants of the methods used to conduct HTA in Colombia, but also on how to incorporate its results into decision making; for instance of validity of data used for modelling, the use of quality-adjusted life-years (QALYs) when conducting CUA, or reliance on ICERs alone to inform decision making.
The valuation and ranking of technologies was sensitive to the methods and criteria used to assist decision making. For instance, ticagrelor ranked last when using the MCDA approach but climbed to the top when the direct ranking was applied. Possibly, the fact that this was the only technology considered as cost-saving may have impacted participants’ preferences after the budget impact was presented in a second step when testing the direct ranking approach.
Concerns for opportunity costs following reimbursement of new interventions has raised in many countries. Such concerns were recently highlighted in the United Kingdom (Reference Claxton, Sculpher, Palmer and Culyer24), claiming that decision-making processes failing to consider opportunity costs may be quite detrimental. Indeed, the application of the EVIDEM framework v2.4 proposes, in a first step, to estimate quantitatively the holistic value of healthcare interventions using a wide range of criteria, and in a second step, to explore contextual aspects, including opportunity costs using a financial exercise, providing an integrated process to select interventions to invest or to disinvest (25). The proposition of participants in this study in Colombia to establish a two-step process, which includes MCDA valuation followed by financial analyses based on a budget impact, is in line with most recent reflection on the necessity to establish decision processes and methodologies that allows systematic consideration of opportunity costs.
CONCLUSIONS
Participants agreed that systematic priority setting should take place in Colombia, regardless of the number of competing technologies (up to 2012 there were more than 200 competing technologies awaiting for reimbursement decisions by CRES). Based on structured discussion to reflect on appraisal methods explored for this study, participants considered that a two-step approach including first MCDA to value interventions with a transparent and systematic method, complemented in a second step by a comprehensive budget analysis would be the best approach to assist decision making in this context.
Structured and objective consideration of the factors that are both measurable and value-based in an open and transparent manner is feasible in Colombia by using MCDA approaches combined with financial analyses for the explicit consideration of opportunity costs. This research provided useful insights to optimize decision-making processes, and provides a basis to serve the MoHSP of Colombia on establishing a systematic and more transparent process for resource-allocation.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462316000350
CONFLICTS OF INTEREST
The authors declare no conflicts of interest.
