In 2004, the global market for pharmaceuticals was approximately $518 billion (18), whereas the estimated world gross domestic product (GDP) was $55.9 trillion (24). Pharmaceuticals thus account for approximately 1 percent of world GDP. This high proportion underscores the need for rigorous assessments of their risks and benefits.
Pharmaceuticals undergo two assessments. The first focuses on drug efficacy, determined in most cases in randomized controlled trials (RCTs). The assessment is performed before the granting of marketing approval using well-established, standardized registration procedures which owe much to the ICH project (International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use). Regular meetings now take place between pharmaceutical firms and regulatory agencies (European, Japanese, and North American). The second assessment determines the usefulness of the drug to decide on coverage by national (or private) health insurance funds. The usefulness of a drug is usually given by its efficacy (“does the drug work under optimal circumstances?”), effectiveness (“does the drug work in routine care?”), and efficiency (“is the drug used to maximize value for money?”) (Reference Bombardier and Maetzel3): The determination of effectiveness and efficiency is thus part of the post-listing assessment process. They depend on national priorities, taxation rules, and culture.
In England, post-listing assessment is the remit of the National Institute for Health and Clinical Excellence (NICE). Prices are fixed by pharmaceutical firms. The effectiveness of a drug class (multiple health technology assessments [HTA]) is assessed once results under real-life conditions have become available (negative list). An appraisal committee comprising National Health Service (NHS) clinicians and managers, academic experts, industry, and lay representatives discuss and amend a systematic review of the clinical and economic evidence on the drug—the so-called Technology Assessment Report (TAR) (Reference Hill, Garattini, van Loenhout, O'Brien and de Joncheere11). A key item in the TAR is the cost/utility ratio, that is, the cost per quality-adjusted life-year (QALY) gained by an average patient from the use of the drug (16). QALYs are estimated from the revealed preference of patients. The TAR integrates data on effectiveness and especially efficiency. The cost per QALY has been subject to criticism both from conceptual and ethical viewpoints (Reference Harris10;Reference Schlander20).
In Germany, medical and economic assessment of health technologies is conducted by IQWIG (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, Institute for Quality and Efficiency in Health Care). Reimbursement and pricing decisions are taken some time after listing (delisting, negative list). IQWIG sets a ceiling price for drugs in a given therapeutic area. The price set by the pharmaceutical company is reimbursed until the HTA report becomes available. Once the report is available, the German Federal Joint Committee (G-BA, Gemeinsame Bundesausschuss), which includes all stakeholders, takes the decision on reimbursement level.
In France, unlike in England and in Germany, the effectiveness of each and every drug is assessed immediately after listing (positive list) to help set price and ensure early reimbursement. The assessment of effectiveness is performed by the Transparency Committee of the Haute Autorité de Santé (HAS) (French National Authority for Health) with the help of a large body of experts. The twenty appointed voting members of the Committee rate the clinical value of the drug (SMR for Service Médical Rendu—Actual Benefit—which can be important, moderate, minor, or insufficient) and its added value compared with existing treatments (ASMR for Amélioration du Service Médical Rendu, Improvement in Actual Benefit) from 1 (major improvement) to 5 (no improvement). The French Ministry of Health then immediately uses these ratings to determine reimbursement rate (SMR) and price (ASMR). This process thus clearly separates the assessment of effectiveness (HAS) and of efficiency (Ministry of Health). It has, however, a drawback. There is no time to conduct post-approval studies to translate efficacy data into effectiveness and efficiency. The following question thus arises: “How to obtain an early assessment of effectiveness?”
This article will present a new formal method for comparing the effectiveness of medical treatments based on expert opinion. The method combines RCT efficacy data and their generalizability and transposability to an unselected real-life population of patients in a specific national context, and includes indirect comparisons when head-to-head comparisons are not available.
METHOD
The three-step REAL procedure was used to compare the effectiveness of n treatments.
First Step: Collection of RCT Data
Pharmaceutical firms must submit a standard report of all their efficacy and safety data on the drug to the Transparency Committee. Should any major trials be missing from the report (e.g., because they were not conducted by the firm in question), the Committee may add them to the critical appraisal. However, a thorough formal review of all published and unpublished trials on the drug will not be done systematically. An analysis based for instance on the QUOROM statement (Quality of Reporting Meta-Analyses) would be more reliable, but it is not feasible because of the number of drugs the Committee has to assess (or re-assess) each year (485 single drugs in 2004). The selection of RCTs will thus depend on the importance of the issues raised by the introduction of the new drug.
The first step is thus effectively whittled down to: (i) Selecting the list of drugs to be compared, that is, the drug being assessed and a set of comparators proposed by the firm and by the experts and members of the Transparency Committee. (ii) Selecting trials comparing at least two drugs in the list. All selected trials must include a binary primary or secondary end point (e.g., success/failure). (iii) Selecting the reference arm for each trial (the choice will not impact on the results). An odds ratio (OR) giving the relative efficacy with respect to the reference arm is estimated for all the other trial arms. Thus, if A is the reference and B a comparator, and if pA and pB are the success rates and qA and qB the failure rates for A and B, then the relative efficacy of B versus A is given by log(ORRCT) = log[(pB/qB)/(pA/qA)].
Second Step: Collection of Data on the Generalizability and Transposability of the Results of the RCTs Selected in Step 1
Much work and thought in the literature have gone into the translation of clinical trial data into real-life conditions (Reference Boissel, Collet, Lievre and Girard2;Reference Davis6;Reference Thompson and Higgins23). The effect model, in particular, describes “a function that defines the relationship between the clinical efficacy of a treatment and specific covariates” (Reference Boissel, Collet, Lievre and Girard2). Here, we shall postulate that there is a linear relationship between the relative efficacy of A versus B as given by an RCT (i.e., log(ORRCT)), and the expected relative effectiveness of A versus B as might be estimated in a real-life study (i.e., log(ORRL)). Thus let
![\begin{equation}
{\rm E}[\log ({\rm OR}_{{\rm RL}})] = {\rm a} + {\rm E}[\log ({\rm OR}_{{\rm RCT}})]\end{equation}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054934689-0407:S0266462309990821:S0266462309990821_eqn1.gif?pub-status=live){kind=small}
and
![\begin{equation}
{\rm Var}[\log ({\rm OR}_{{\rm RL}})] = {\rm b} \times {\rm Var}[\log ({\rm OR}_{{\rm RCT}})]({\rm b} \ge 1)\end{equation}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054934689-0407:S0266462309990821:S0266462309990821_eqn2.gif?pub-status=live){kind=small}
If a < 0, then the effectiveness will be less than the efficacy observed in the trial. If a > 0, the converse is true. The constant b reflects a lack of confidence in the estimation of log(ORRL) as compared to the confidence available for the estimation of log(ORRCT).
Ideally, a and b should be estimated from observational studies or from clinical trials in a real-life setting but, because such studies are the exception rather than the rule, we need surrogate estimates. These estimates are given by the answers of experts to 4 questions for each arm of each trial. The exact formulation of these questions is available from the authors upon request. They correspond approximately to:
Question 1: Give a number between 0 and 100 percent that reflects best your confidence in the relative efficacy (Re) of the drug because of the potential methodological flaws of the trial.
Question 2: Give a number between 0 and 100 percent that reflects best your confidence in the relative effectiveness (Rr) of the drug under real-life conditions because of the problems of transposing trial results into real life.
Question 3: Give a number between +5 and +1 or between −1 and −5 (with one decimal) that reflects best your perception of the relationship between the relative efficacy (Re) of the drug observed in the trial and the relative effectiveness (Rr) that might be observed under real-life conditions after short-term administration (a few months, a few years).
Question 4: Give a number between +5 and +1 or between −1 and −5 (with one decimal) that corresponds best to the change you anticipate in the relative effectiveness Rr of the two drugs in the long term (patient's life expectancy) in real life.
If b1 and b2 denote the answers to questions 1 and 2, and a1 and a2 denote the answers to question 3 and 4, then a = log(a1 × a2) and b = 1/(b1 × b2), and the relative effectiveness log(ORRL) and its confidence can be calculated from equations [1] and [2] (if a1 or a2 are negative then a1 or a2 is changed in −1/a1 or −1/a2).
In practice, the twenty voting members of the Transparency Committee answer the four questions in real time, during the session, just after the critical appraisal of the clinical data presented by the external experts. An electronic device collects the answers and computes pooled real-life ORs.
Third Step: Pooling log(ORRL), Sensitivity Analysis
See the technical appendix available in the electronic version of the paper (www.journals.cambridge.org/thc2010009).
RESULTS
We shall illustrate the application of the REAL procedure using an example. Rotigotine is a new drug for the treatment of patients with advanced Parkinson's disease and wearing-off type motor fluctuations. We compared the effectiveness of rotigotine with that of the drugs most often used by French neurologists in this indication.
First Step: Selecting Comparators and RCTs
A member of the Transparency Committee selected the comparators: placebo, pramipexole, entacapone, and rasagiline. Trials were eligible for inclusion if they were part of the firm's application, if the expert considered them to be indispensable, or if they were published in one of the following journals (Lancet, New England Journal of Medicine, JAMA, British Medical Journal, Neurology, Archives of Neurology). Three trials were identified (Reference LeWitt, Lyons and Pahwa13;Reference Poewe, Rascol and Quinn17;Reference Rascol, Brooks and Melamed19): one compared two titrations of rotigotine with placebo, one compared rotigotine and pramipexole with placebo, and one compared rasagiline and entacapone with placebo. The binary end point was reduction in “off” time and was fairly similar in all trials.
Second Step: Experts’ Rating of Generalizability and Transposability
The questionnaire was administered to the members of the Transparency Committee (m = 17) who had to answer to questions 1–4 for the six active arms proposed in the three trials.
Third Step: Pooling Log(ORRL), Sensitivity Analysis
Results are shown in Figures 1 (placebo as reference) and 2 (rotigotine as reference). In terms of effectiveness, rotigotine appears to be superior to placebo, with an effect that is intermediate between rasagiline and pramipexole. Relative efficacy is also shown (for estimates obtained with a = 0 and b = 1) to visualize the influence of the experts’ ratings on the distributions. Rotigotine does not therefore appear to be a substantial breakthrough in the treatment of patients with advanced Parkinson's disease. This is consistent with the Transparency Committee's official opinion (April 30, 2008) that it has no added value over pramipexole.
Relative effectiveness (solid lines) and relative efficacy (dashed lines) of four treatments for advanced Parkinson's disease. Placebo was taken as the reference.
Relative effectiveness (solid lines) and relative efficacy (dashed lines) of three treatments and of placebo for advanced Parkinson's disease. Rotigotine was taken as the reference.
DISCUSSION
The methodology of post-listing assessments is an active area of research at the cross-roads of many disciplines [human and social sciences (ethics and economics), statistics, epidemiology, clinical and biological research (pharmacology, controlled trials, etc.)]. The technical issues are complex, and the overall issue is particularly sensitive because of its potential implications for the general public.
Currently, the evaluation of QALYs is the most widely used method of post-listing assessment. It has several advantages. The elegant framework on which QALYs are based (von Neumann & Morgenstern theorem) enables the computation of a cost per QALY. Different types of treatment (psychological, surgical, drugs, etc.) with a common outcome can be compared. A solid body of experimental work is available. However, the use of QALYs also raises concerns (Reference La Puma12): (i) implementation can result in much controversy as in the Oregon Medicaid health-rationing experiment (Reference Brannigan4), (ii) there are methodological problems (Reference La Puma12): it is difficult to obtain reliable estimates of the revealed patient preferences on which QALYs are based (Reference Danis, Patrick, Southerland and Green5); (iii) there are ethical reservations (Reference Harris10). QALYs postulate that “utilitarianism (the greatest good for the greatest number) is the appropriate ethical theory for resolving resources allocation dilemmas” (Reference La Puma12). This is debatable especially in France, a country known for its long-standing emphasis on existentialism and the essential singularity of each human being. Subjectivity is inbuilt in both the QALY process used by NICE and the REAL method under development by HAS. In the QALY process, it is present at the patient or citizen level, with revealed preferences about quality of life (QoL) being averaged secondarily to computing a cost/effectiveness or a cost/efficiency ratio. In the HAS procedure, it is present in the Transparency Committee members’ vote when they rate the added value of the drug (ASMR). The decision to reimburse the drug, however, belongs to the French Ministry of Health, which is accountable to the French parliament. The REAL method is thus a statistical tool designed to help experts and decision makers determine, on the basis of their personal judgment, whether a drug qualifies for reimbursement, at a given price, in a given population of patients.
Of course, the REAL method is not a true alternative to a post-listing assessment based on a cost per QALY. REAL focuses on effectiveness, whereas a cost per QALY deals also, by definition, with economic issues. In France, the effectiveness of a drug and its price are assessed one after the other as if they belonged to different epistemological frameworks. This is not to say, however, that REAL considers only hard clinical outcomes and overlooks patients’ preferences (e.g., QoL measurements). Whenever possible, members of the Transparency Committee take account of patient reported outcomes such as QoL, alongside more objective outcomes such as survival, when rating the added value (ASMR) of a drug (see question 3 of the REAL method).
The REAL method has several limitations. The first and most important is the reliability of the Committee members’ answers to the four questions. These apparently trivial questions are in fact complex and, if the drug being assessed is novel, experience of its use is limited. However, it should be noted that (i) pricing is even more arbitrary if questions 1 to 4 are not answered; (ii) clinicians are accustomed to balancing trial results against real-life observations in their everyday practice; and (iii) all drugs are not novel but all are reassessed every 5 years. Inter-rater agreement will be estimated when sufficient data become available.
A second important issue is who should be called upon to answer questions 1 to 4. We chose the twenty members of the Transparency Committee because (i) most are practicing physicians; (ii) all are experts in the evaluation of clinical trials; (iii) they will have a cross-disciplinary perspective; (iv) a sample size of twenty is compatible with the bootstrap procedure used for the sensitivity analysis and would be difficult to achieve by recruiting specialists for each drug.
A third point to be considered is the unconventional statistical approach of the REAL method. It is different from customary approaches, that is, from variants of meta-analysis used to analyze data derived from a systematic review of the literature. The current favorites are linear mixed models (Reference Lumley15) and Bayesian analyses (Reference Lu and Ades14), which can handle both direct and indirect comparisons. The so-called “adjusted indirect comparisons” method has given results consistent with direct comparisons (Reference Song, Altman, Glenny and Deeks22). The limitations of these statistical methods are the number of trials needed for robust comparisons (minimum of four suggested in Glenny et al.) (Reference Glenny, Altman and Song9) and the arbitrary a priori distributions in Bayesian models which can impact on results (Reference Gelman8).
A fourth limitation of the REAL procedure is the need for a binary primary or secondary end point. If this is too restrictive in the long run, quantitative end points may be used instead because relative effectiveness is estimated from Log-ORs, which have an asymptotic normal distribution (Reference Agresti1). Trials must also have similar outcomes and comparable baseline severity of illness (differences in baseline severity could hamper the estimation of an unbiased pooled effect) (Reference Schmid, Lau, McIntosh and Cappelleri21). The experts can, however, take these differences into account in their answers to question 3, although truly comparable end points are always preferable to an expert's ability to make them comparable.
Other potential limitations are as follows: Direct account is not taken of possible trial heterogeneity—as by a random effect in a linear mixed or Bayesian model—although the answer to question 1 does cover a potential loss of confidence due to methodological issues. Treatment comparisons cannot be adjusted on potential confounders as in a meta-regression although question 3 does address a potential shift in efficacy. Estimating a differential long-term effect may be an impossible challenge although guessing a constant to include in the model may be preferable to ignoring the question.
In conclusion, the REAL method has several advantages over previous methods of assessment of relative effectiveness: (i) Its transparency: there is a clear divide between efficacy data and expert opinion on generalizability and transposability to real life. (ii) Its potential: expert opinion could be replaced by data from observational studies (evidence-based answers to questions 1 to 4). (iii) Its statistical strengths: it deals with direct, indirect, or mixed comparisons, proposes a sensitivity analysis that accounts for a maximum number of sources of uncertainty, and provides user-friendly results for the nonspecialist. (iv) Its specific evaluation of effectiveness in clearly defined medical indications. (v) Its feasibility and gain in time: the effectiveness of a new drug can be compared objectively with that of others before price negotiations begin. The average time from submission of a drug application to the Transparency Committee to informing the French Ministry of Health of the Committee's opinion is under 4 months. The system used by NICE in the United Kingdom should optimally take 54 weeks but may take very much longer (Reference Hill, Garattini, van Loenhout, O'Brien and de Joncheere11).
SUPPLEMENTARY MATERIALS
Supplementary Technical Appendix www.journals.cambridge.org/thc2010009
CONTACT INFORMATION
Bruno Falissard, MD, PhD (falissard_b@wanadoo.fr), Professor, Department of Public Health, South Paris University, 63, rue Gabriel Péri, Le kremlin Bicêtre 94276, France
Valérie Izard, MD (v.izard@has-sante.fr), Evaluator, Bertrand Xerri, MD, PhD (b.xerri@has-sante.fr), Associate Director, l'Evaluation Médicale, Economique et de Santé Publique, Haute autorité de santé, 2, avenue du Stade-de-France, Saint Denis la Plaine 93218, France
Gilles Bouvenot, MD (gilles.bouvenot@ap-hm.fr), Professor, Department of Therapeutics, Marseille Faculty of Medicine, Hôpital Sainte-Marguerite, Marseille, 13000, France; Head (president), Transparency Commission, Haute autorité de santé, 2, avenue du Stade-de-France, Saint Denis la Plaine 93218, France
François Meyer, MD (f.meyer@has-sante.fr), Director, Evaluation Médicale, Economique et de Santé Publique, Haute autorité de santé, 2, avenue du Stade-de-France, Saint Denis la Plaine 93218, France
Laurent Degos, MD, PhD (l.degos@has-sante.fr), Professor, Department of Hematology, Paris-VII university, Hôpital Saint Louis, Paris, 75010, France; Head (president), Collège, Haute autorité de santé, 2, avenue du Stade-de-France, Saint Denis la Plaine 93218, France
