The continuous growth in healthcare spending can be mainly explained by two main reasons: the ongoing inclusion of new medications in hospital formularies (Reference Hartman, Martin, Benson and Catlin1) and the appreciable increase in the usage of medicines for noncommunicable diseases across Europe and many other countries. However, the impact has been moderated by increasing use of low cost generics in countries with demand side measures to encourage their use (Reference Godman, Wettermark and van Woerkom2). However, new medications do not always result in better health outcomes, with very few offering significant advantages over existing therapies in terms of efficacy, safety, and cost-effectiveness (Reference Zhang and Soumerai3). Prescrire believes only 2 percent+ of new medicines are truly innovative, with the vast majority being similar or offering only marginal benefits over available alternatives (4). Hospitals should, therefore, put considerable effort into the selection process when incorporating new drugs and have adopted diverse ways to establish a drug policy aimed to promote its safe, effective and efficient use in the hospital setting. This evaluation and selection process of drugs leads to the decision whether to include or not the drug into the hospital formulary. The decision usually involves a multidisciplinary pharmacy and therapeutics committees (P&TCs), in operation for decades in hospitals worldwide, as well as in Spain (Reference Durán-García, Santos-Ramos, Puigventos-Latorre and Ortega5;Reference Tyler, Cole and May6).
P&TCs might be formed by independent pharmacotherapeutic drug experts, including hospital pharmacists, hospital clinicians, surgeons, nursing staff, and community pharmacists. Even though very little is known about the factors that influence the decision-making process in a P&TC setting, it has been reported that clinical trial results and drug costs are the most influential factors rather than a pharmacoeconomic evaluation, due to the lack of specific training of committee members and the difficulty of extrapolating pharmacoeconomic studies to current hospital practice (Reference Durán-García, Santos-Ramos, Puigventos-Latorre and Ortega5;Reference Jenkings and Barber7).
In Spain, health competences are transferred to the governments of the Autonomous Communities, that have recently published legal regulations to enhance the evaluation of drugs with centralized rational criteria, but no shared guidelines to rationalize the use of medicines in the Spanish National Health Service are set out (Reference Durán-García, Santos-Ramos, Puigventos-Latorre and Ortega5;Reference Lozano-Blázquez, Calvo-Pita and Carbajales-Álvarez8). In this context, several collaborative initiatives between P&TCs and scientific societies have recently developed common methods and systems for evaluating new drugs, like the Group for Innovation, Assessment, Standardization and Research in the Selection of Drugs (GENESIS) of the Spanish Society of Hospital Pharmacy (SEFH) that has developed an evaluation report for new drugs, a procedures manual, a program for drafting of reports, and a standard request form to incorporate a new drug into the hospital formulary (Reference Ortega Eslava, Puigventós Latorre and Santos-Ramos9).
To our knowledge, no approach including relevant criteria to assess the evaluation of new drugs, other than efficacy, safety, cost, budget impact, or cost-effectiveness analyses has been applied in the P&TC setting. Multi-criteria decision analysis (MCDA) is a method that structures complex problems into a comprehensive set of criteria that are relevant for establishing the value of healthcare interventions in different contexts and that may impact in the healthcare decision making, under a systematic and transparent process and incorporating a wide range of stakeholder views (Reference Angelis and Kanavos10). This methodology has been widely used in drug context for evaluating several types of drugs and treatments (Reference Marsh, Zaiser and Orfanos11;Reference Wagner, Khoury and Bennetts12).
In addition, the EVIDEM (Evidence and Value: Impact on DEcisionMaking) framework is currently the only reflective MCDA approach, in which the participants of the MCDA evaluation process share their reflections behind their criteria weighting and scoring and creates an open reflective discussion among participants. The EVIDEM framework has been created as an adaptable and pragmatic open-source MCDA-based framework for accountable and reasonable healthcare decision making and priority setting (Reference Goetghebeur, Wagner and Khoury13).
The present pilot study aims to develop and to assess a specific reflective value framework to evaluate new drugs in a hospital P&TC setting by defining which criteria would be important for the P&TC's members and testing the utility of this defined framework in drug evaluation process.
METHODS
Study Design
This pilot study included two different phases for the development and assessment of a value framework in drug evaluation process. The first part of the study consisted on identification and definition of the framework criteria to be evaluated and the assessment of the criteria relative importance by the P&TC's members. Following this first part of the study, two designated P&TC's members (U.B.R. and J.A.M.R.) chose two drugs for evaluation, based on a list of future drugs to be evaluated by the P&TC shortly. The selection of the two drugs aimed to cover an orphan drug for which there is usually less clinical information available (Eliglustat, for the treatment of Gaucher disease) and a nonorphan drug with more clinical information available (Vedolizumab, a monoclonal antibody for the treatment of inflammatory bowel disease). After choosing the drugs, the same two P&TC's members developed the evidence matrices needed for the second part of the study. The second part of the study was undertaken 2 months after the first one and a panel of P&TC's members evaluated the two chosen drugs using the framework defined in the first part of the study and the matrices previously developed. All P&TC's members participated in this study on a voluntary basis and as a nonpaid contribution.
Identification and Definition of Criteria
To identify and to define a set of criteria suitable for drug evaluation from a P&TC's perspective, the EVIDEM methodology (14) was selected considering that this tool is the only reflective MCDA framework already applied and validated in several studies about drug value evaluation and decision making in Spain (Reference Wagner, Khoury and Bennetts12;Reference Hernández, Blázquez, Gil and Xzzz15;Reference Gilabert-Perramon, Torrent-Farnell and Catalan16)
The EVIDEM framework has been designed to evaluate the value of interventions and facilitate their prioritization using a comprehensive group of generic decision criteria organized into a pragmatic tool. It is composed of a standard set of criteria structured into two distinct sections: MCDA Core Model, which is formed by thirteen quantitative criteria focused on the assessment of the drug (disease severity, size of affected population, unmet needs, comparative effectiveness, comparative safety/tolerability, comparative patient-reported outcomes, type of preventive benefit, type of therapeutic benefit, comparative cost consequences-cost of intervention, comparative cost consequences-other medical costs, comparative cost consequences-nonmedical costs, quality of evidence, and expert consensus/clinical practice guidelines) and MCDA Contextual Tool, composed by seven qualitative criteria focused on the consideration of the context surrounding the decision making (mandate and scope of healthcare system, population priorities and access, common goal and specific interests, environmental impact, system capacity and appropriate use of intervention, political/historical/cultural context, and opportunity costs and affordability) (14)
In addition to the EVIDEM criteria framework, other relevant criteria were also considered and extracted based on previous applications of MCDA methodology in other settings, such as evaluation and decision making in drug evaluation regional committees (Reference Hernández, Blázquez, Gil and Xzzz15;Reference Gilabert-Perramon, Torrent-Farnell and Catalan16) and involvement of patients in healthcare decision making (Reference Aguarón, Fernández and Gimón17). Hence, the EVIDEM criteria framework and their definitions together with the suitable extracted criteria from previous evaluations intended to cover all criteria that would be useful by a P&TC in drug evaluation.
Criteria Assessment
The criteria were assessed by a panel of P&TC's members through a first face-to-face workshop to define a pilot value assessment framework. In preparation for the workshop, each participant was trained on reflective MCDA methodology, receiving a guideline including the basis for conducting the workshop exercises, and including the EVIDEM criteria definitions and the weighting tools to be used during the workshop. This training was conducted previously to validate the criteria. To consider the inclusion, exclusion or adaptation of the criteria in the framework, the expert panel was asked individually for each criterion, whether the criteria were suitable for assessing the evaluation on new drugs from a P&TC's perspective. Each individual criterion was considered excluded if more than 50 percent of respondents answered “no,” included if more than 50 percent answered “yes,” and adaptable if there was a tie or other combinations of answers.
Criteria Weighting
Following EVIDEM methodology (14), the weighting of quantitative criteria and contextual criteria included in the pilot criteria framework developed for drug evaluation in a P&TC setting was done using a five-point weighting technique. According to this technique, each participant gave a relative weight per criterion using a nonhierarchical simple 5-point scale (1 = lowest relative importance; 5 = highest relative importance).
Matrix Development and Scoring
To score each criterion of the initial pilot framework, an independent evidence matrix was developed for each of the two drugs chosen by two designed P&TC's members: the first drug, Eliglustat, and the second one, Vedolizumab, as well as for their respective comparators. A related literature review was carried out to find the required information to complete the matrices. Data were collected, summarized, and presented as a comparison between the tested drug and its comparator (14). The value contribution of the two tested drugs for each criterion was scored by a panel of P&TC's members and the results were analyzed.
Value Framework Agreement
To assess the suitability of the criteria included in the matrix, a reflective discussion based on the score assigned to each criterion by the P&TC's members in the evaluation of the two specific drugs was carried out. The matrices scoring and the potential discussion from values assigned by each member was considered the main part of the study to assess the utility of the MCDA methodology. An agreed final pilot reflective MCDA framework was obtained through this discussion.
Data Analysis
Data were collected individually, transferred to a common database, and analyzed with Microsoft Excel software. Data obtained from criteria weighting were analyzed including mean, standard deviation, and minimum and maximum value. Criteria weights were normalized to sum up to 1 for each participant: each weight was divided by the sum of weights across all criteria. Scoring of quantitative criteria and contextual criteria was performed on a scale of −5 (worst score) or 0 to +5 (best score). The mean, standard deviation (SD) and range of minimum and maximum scores were calculated. The value contribution (VCx) of each quantitative criterion and contextual criterion was calculated as the product of its normalized weight (Wx, ∑Wx = 1) and standardized score (Sx = score/5). The overall MCDA value estimate (VE) is the sum of all criteria value contributions:
The evaluation of contextual criteria was performed on a qualitative scale with 3 options (positive +1, neutral 0, or negative impact −1). Scores were calculated as the percentage of members considering the contextual criteria positive, neutral, or negatively influencing the drug value. A descriptive analysis of the value of each criterion was conducted separately.
RESULTS
Identification and Definition of Criteria
Considering a P&TC setting, we designed an initial pilot framework including twenty-one criteria covering seven different domains: disease impact, comparative results of intervention, type of health benefit of intervention, economic consequences of intervention, knowledge about intervention, normative criteria, and viability. These twenty-one criteria were divided in fifteen quantitative criteria: thirteen of those criteria were extracted from the EVIDEM framework plus two criteria extracted from other previous MCDA applications the authors considered as relevant (therapeutic innovation degree and comparative cost-effectiveness) and six contextual criteria extracted from the EVIDEM framework (Figure 1A).
List of criteria composing the initial pilot framework (A) and the final pilot framework (B).
Criteria Assessment
The criteria were assessed by the P&TC from public Hospital Universitario Virgen Macarena in Seville, composed by four hospital pharmacists, three clinicians, three unit director clinicians, and one medical director. As a result of the assessment, some criteria were excluded, and others adapted, and nine of twenty-one initial criteria were retained in the final criteria framework, showed in Figure 1B, in five domains: disease impact, comparative results of intervention, economic consequences of intervention, knowledge about intervention, and viability. The definitions of two criteria were adapted, based on the comments of P&TC's members, to include them in the framework: the “unmet needs” criterion was considered as relevant but a suggestion for a more representative assessment was proposed by the P&TC's members. Unmet needs related to efficacy, safety, and results focused on the patient were classified as very relevant, while needs required by the patient and needs in convenience/ability of use/ type of administration were considered as less relevant and already included in the criterion “patient reported outcomes”.
The other criterion that was redefined is “quality of evidence” as “quality of evidence related to evaluator,” which considers that evaluating the quality of evidence regarding a drug is based on the experience of evaluator instead of based on GRADE or Cochrane methodology. Thus, the final pilot framework included eight quantitative criteria and one contextual criterion, showed in Figure 1B.
Criteria Weighting
The P&TC participants were invited to do the weighting of the initial pilot criteria framework (quantitative criteria). According to the mean results collected from the five-point weighting scale, most important criteria were (mean ± SD): “comparative safety/tolerability (4.5 points ± 0.5), followed by “disease severity” (4.4 points ± 0.8) and “comparative efficacy/effectiveness” (4.4 points ± 0.8). Criteria least important were “comparative cost consequences, other medical costs” (2.2 points ± 0.9), followed by “comparative patient-reported outcomes” (2.9 points ± 1.1). Criteria ranked with medium importance were “unmet needs” (3.7 points ± 1.1), followed by “quality of evidence” (3.6 points ± 1.4) and “comparative cost consequences, cost of treatment” (3.3 points ± 0.6).
The lowest agreement criteria were “quality of evidence” (SD 1.4), “comparative patient-reported outcomes” (SD 1.1), and “unmet needs” (SD 1.1). The criteria with the highest agreement were “comparative safety/tolerability” (SD 0.5), and “comparative cost consequences, cost of treatment” (0.6).
MCDA Matrix Development and Scoring
To validate the final pilot criteria framework two drugs were evaluated in a second face-to-face meeting. For evaluating the orphan drug Eliglustat, an evidence matrix was developed using Imiglucerase as a comparator. The criteria weighting of the 8 quantitative criteria and the contextual criterion of Eliglustat versus Imiglucerase is shown in Figure 2A. The most valued criteria by the P&TC's members were “disease severity,” “unmet needs,” and “quality of evidence,” as they were evaluating a severe disease with important unmet needs and where the evidence came from clinical trials head to head with Imiglucerase. The less valued criteria were “comparative safety/tolerability” and “comparative efficacy/effectiveness” because Eliglustat had a similar efficacy and safety profile compared with Imiglucerase. Eliglustat showed a negative impact when considering “opportunity costs and affordability” in comparison to Imiglucerase, because P&TC's members pointed out that some patients already treated with lower costs drugs would be treated with Eliglustat and this treatment would result in a greater disease cost.
Criteria weighting (quantitative criteria, mean) for the two evaluated drugs. (A) Eliglustat versus Imiglucerase. (B) Vedolizumab versus Adalimumab. (C) Vedolizumab versus Ustekinumab.
To evaluate the second drug, Vedolizumab, two different comparators were used: Adalimumab and Ustekinumab, and a matrix of evidence was developed for each of the two comparators (Figures 2B and 2C). In the first evaluation, Vedolizumab versus Adalimumab, the most valued criteria were “disease severity,” “unmet needs,” and “quality of evidence,” as all P&TC's members classified inflammatory bowel disease as a severe disease with important unmet needs. The “quality of evidence” was positively valued due to the design of clinical trials of Vedolizumab. The less valued criteria were “comparative cost consequences, cost of treatment,” because the P&TC's members considered that Vedolizumab would suggest and additional cost for the healthcare system due to its higher cost compared with Adalimumab, and “comparative efficacy/effectiveness” as Vedolizumab was ranked slightly less effective in comparison to Adalimumab. The negative impact on “opportunity costs and affordability” reached 100 percent, because all the participants recognized that Vedolizumab would have an incremental hospital budget impact versus Adalimumab (Figure 2B).
In the second comparison, Vedolizumab versus Ustekinumab, the two drugs were defined as a very similar option in almost all criteria, recognizing that Vedolizumab would add an additional value in criteria related to disease characteristics: “disease severity,” “unmet needs,” and “quality of evidence” (Figure 2C).
Numerical results of value contribution of the two evaluated drugs are showed in Figure 3. The overall value contribution of Eliglustat was +0.32 of the potential maximum value (+1), while Vedolizumab showed a value contribution of +0.10 and +0.12 compared with Adalimumab and Ustekinumab, respectively. Disease severity of the two indications was considered high, reflecting the perception of their impact on mortality and morbidity and which the current available treatments remain as unmet needs with limited therapeutic options. Eliglustat scored with a very moderate benefit based on the worse adverse effects profile and the similar efficacy characteristics compared with Imiglucerase. Its value contribution was mainly based in the criteria related to disease characteristics, “disease severity” and “unmet needs” (Figure 3A). Regarding Vedolizumab, the drug showed to add an additional value in “disease severity,” “unmet needs,” and “quality of evidence,” but it showed a very marginal value related to efficacy, safety, and patient-reported outcomes versus its comparators, having a higher cost of treatment and hospital budget impact compared with Adalimumab (Figures 3B and 3C).
Value of contribution of each criterion and overall by the two evaluated drugs versus comparators. (A) Value contribution of Eliglustat. (B) Value contribution of Vedolizumab. (C) Value contribution of Vedolizumab.
Value Framework Agreement
After testing the final criteria framework, a reflective discussion was held to agree on a final set of criteria potentially used by P&TC’ s members to assess drug evaluation in the future. This discussion led an agreement in eight quantitative criteria (including the two criteria whose definitions were adapted: “unmet needs” and “quality of evidence”) and the “opportunity costs and affordability” contextual criterion remaining in the final framework. All the P&TC's members considered the procedure as useful or very useful to assess the value of new drugs.
DISCUSSION
This pilot study sought to test the feasibility of MCDA methodology for appraising the overall value of new drugs being evaluated for inclusion in hospital formulary by a P&TC. In a first attempt to develop and to validate a framework, committee's members showed a high agreement on most relevant criteria and a final framework was defined including eight quantitative criteria: “disease severity,” “unmet needs,” “comparative efficacy/effectiveness,” “comparative safety/tolerability,” “comparative patient-reported outcomes,” “comparative cost consequences-cost of treatment,” “comparative cost consequences-other medical costs,” and “quality of evidence”; and one contextual criterion: “opportunity costs and affordability.” Second, this validated framework was used to assess an orphan drug for Gaucher disease and a nonorphan drug for inflammatory bowel disease and P&TC's members evidenced the utility of this methodological tool for developing matrices of evidence and drug evaluation.
The reflective MCDA methodology applied in the present study provided most of the information required for drug evaluation in a structured and transparent way and made possible the comparison between different drugs in an objective and systematic way. Furthermore, MCDA allowed the integration of points of views form different stakeholders from a multidisciplinary P&TC. MCDA is increasingly being applied to support healthcare decision making and has been tested to develop specific criteria framework in the context of several diseases and drugs evaluation, based on the specific objectives of evaluators and decision-making committees (Reference Friedmann, Levy, Hensel and Hiligsmann18). For example, in oncology field, several clinical organizations have developed criteria frameworks to systematically assess the value of new drugs (Reference Bentley, Cohen and Elkin19), and the criteria used have provided a common interpretive tool to support individual reflection and share perspectives between patients and clinicians (Reference Wagner, Samaha and Khoury20).
Previous studies evaluating the value of orphan drugs have demonstrated that an MCDA approach is a viable and useful tool for decision-making process and could be used by payers and health technology assessment bodies (Reference Gilabert-Perramon, Torrent-Farnell and Catalan16;Reference Schey, Krabbe, Postma and Connolly21). When evaluating treatment options for COPD, MCDA methodology has been used to design a criteria framework to compare different treatment options prioritizing clinical benefits and risks which are the relevant criteria for clinicians (Reference Marsh, Zaiser and Orfanos11). In Spain, Catalonia has been pioneered in exploring and identifying MCDA as a tool that could improve these procedures. In this context, the Catalan Health Service (CatSalut) has already begun to incorporate the reflective MCDA EVIDEM framework, including the whole tool, to enhance evidence-based discussions among all pharmacotherapeutic committees’ members involved in evaluation and decision making of new drugs (Reference Gilabert-Perramon, Torrent-Farnell and Catalan16). Further research on differences between P&TCs from different countries is needed to identify and compare the different value criteria that P&TCs use in their evaluation and decision-making processes.
In the present study, the criteria framework was adapted to cover all the relevant criteria in a specific P&TC setting, considering that the objective of a P&TC is to evaluate and value new drugs for being incorporated into hospital formulary. After validation of final framework, the P&TC's members agreed on higher values for criteria related to disease severity, efficacy and safety, cost of treatment, and patient-reported outcomes as they were considered as very relevant from a P&TC's perspective when evaluating new drugs. The P&TC's members ranked some criteria as not relevant in their decision making because these criteria did not fit with a hospital context, like indirect costs or normative contextual criteria such as political, historical, and contextual context. Other criteria the P&TC's members suggested to exclude were the criteria related to type of benefit of intervention because they considered these criteria already included in comparatives of efficacy and safety. Quality of evidence among P&TC members is very important, but the fact that company sponsored studies can be biased versus. independent sponsored studies (Reference Marra, Araújo and Silva22) made the P&TC members weigh this criterion with an intermediate value score (3.6 of 5).
The selection of the two drugs to be tested in this study intended to show differences in criteria weighting and value contribution according to the evaluated drug (an orphan drug versus a nonorphan drug). P&TC's members considered that criteria related to severity of disease, unmet needs, efficacy, and safety as well as economic impact were the most valued, reflecting the objective of a hospital committee. Moreover, most valued was the reflective discussion that is allowed with this methodology based on the different points of view of participants which leads to share information through individual preferences and exchange of views. In this study, the discussion carried out after scoring among the P&TC's members evidenced the need of a change in the way of evaluation drugs for hospital formulary that does not fit with the traditional way assessing only efficacy, safety, and/or cost-effectiveness (Reference Badia, Poveda-Andrés and Merino-Montero23).
This pilot study had some limitations mainly due to the inherent nature of the study focused only in a P&TC setting. Recent studies have suggested that, in healthcare decision making, the stakeholders would give different values to different criteria and that values assigned in drug evaluation are subjective and depending on their objectives and their points of view (Reference Terkola, Antoñanzas and Postma24). In this context, it has been reported that when evaluating the contribution of a new medical device the relative importance of the criteria framework was in accordance with points of view of evaluators and their objectives (25). Differences in points of view when valuing criteria related to decision making has been shown when comparing evaluator's perspective with patient's perspective as the relative importance of valued criteria might be different (Reference Aguarón, Fernández and Gimón17) as well as when the evaluation is made by local committees responsible for drug evaluation and decision making (Reference Friedmann, Levy, Hensel and Hiligsmann18).
These differences between different stakeholders involved in valuing criteria related to health decision making evidences the importance of MCDA as a tool that allows to consider all points of view and to make decisions without excluding any collective of the process.
The reflective MCDA methodology used in this study would integrate quantitative and qualitative criteria relevant for P&TC's members and it would allow reflective discussions and individual comments based on the criteria weighting score. Thus, this approach would be scaled in other P&TC settings to understand its utility in evaluation of new drugs.
Due to the positive experience in the present study, our idea in the near future is to conduct this same experiment with a wider number of drugs and involving more P&TCs from the Spanish region of Andalusia to compare the obtained results and investigate how this methodology correlates with real-world practice in decision making from a PC&T's perspective.
