Healthcare expenditure adjusted for inflation has increased by more than 40 percent since 1984 in Canada. Health spending, as a percentage of Gross Domestic Product, has increased by 1 percent every 10 years since 1970 (http://secure.cihi.ca/cihiweb/dispPage.jsp?cw_page=home_e). These trends have been observed in most developed countries. One of the major drivers of healthcare costs has been the discovery and diffusion of health technologies (www.oecd.org) (Reference Aaron1;Reference Chernew, Hirth, Sonnad, Rrmann and Fendrick4;Reference Fuchs and Sox9;Reference Gelijins and Rosenberg11;Reference Newhouse19). Payers are therefore starting to take account of costs, in addition to clinical efficacy (Reference Sackett, Rosenberg, Gray, Haynes and Richardson24) and patient safety when making decisions about the introduction of new technologies. Decisions about new technology, however, involve considerations beyond effectiveness and costs. Payers must now also take account of social and ethical values associated with new technology. In addition, payers are faced with demands that the process by which decisions are made about new technology be fair and transparent. Designing a process that weighs scientific evidence appropriately, takes cost and values into consideration, and makes decisions fairly is not straightforward. In this report, we describe how such a process was developed in Ontario, Canada.
In 2003, the Ontario Ministry of Health and Long Term Care recognized the need for a system to address the uptake and diffusion of health technologies. The Ontario Health Technology Advisory Committee (OHTAC) was created to serve as a single provincial portal for recommendations on the introduction of nondrug health technologies (Reference Levin, Goeree and Sikich16).
The health technology assessment cycle at OHTAC begins when OHTAC receives a request for a review of a health technology from potential purchasers—mostly hospitals—or from the Ministry of Health and Long-Term Care. Only those health technologies that have been approved by Health Canada are taken into consideration. In essence, OHTAC's health technology assessment cycle comprises six phases: (i) phase 1: vignette preparation phase—the health technology and preliminary background information are presented to the committee (the committee then decides whether to proceed to a full review of the health technology); (ii) phase 2: review period phase—the Medical Advisory Secretariat engages in a 16-week systematic review of the literature for presentation to OHTAC; (iii) phase 3: recommendation development phase—the committee engages in deliberations until a recommendation is made; and (iv) phases 4 to 6: recommendation is made public, and feedback is sought. The recommendations may be appealed through a presentation of evidence-based information to OHTAC. OHTAC undertakes approximately sixteen health technology policy assessments yearly (Reference Levin, Goeree and Sikich16).
In October 2006, OHTAC requested an evaluation of the process used to develop Ontario specific recommendations for the uptake and diffusion of health technologies. OHTAC believed that to consider economic, social, and ethical aspects within the recommendation process more fully, revision of the existing decision process was required. To meet this need, a “Decision Determinants Sub-Committee” was convened in January 2007 to provide guidance to OHTAC on the development of a framework for decision making. Committee members comprised primarily a subset of OHTAC members and additional nonmembers: decision-making experts, clinical epidemiologists, clinicians, health economists, policy makers, and health services researchers.
METHODS
The work of the “Decision Determinants Sub-Committee” was guided by a review of the literature and discussions with key informants in the areas of evidence-based medicine, health economics, decision analysis, bioethics, and health policy. The purpose of the literature review was to examine methods and models that would be relevant in our setting. The main research questions were the following.
• What criteria are used to make health technology recommendations?
• What methods are used to evaluate health technologies using these criteria (e.g., assigned weights, ranking, rating)?
• What methodology is used to synthesize these criteria (e.g., process/rules/frameworks)?
A computerized literature search was conducted (on February 14, 2007) in the following databases: OVID MEDLINE, MEDLINE in-Process & Other Non-Indexed Citations, EMBASE, ETX on the Web, HTA (INAHTA), and Cochrane Library. The World Wide Web was searched for any published guideline documents pertaining to decision making. Bibliographies of references of relevant papers were also searched for additional references that may have been missed in the computerized database search. The inclusion criteria for the literature review were the following: (i) articles with human subjects published between the years 2000 to 2007; (ii) published descriptive reports, guidance documents, and surveys; (iii) criteria/processes from international agencies; (iv) information regarding sources of evidence used for making recommendations; (v) published research describing recommendation processes; and (f) English language documents. The following were exclusion criteria: (i) primary research evaluating the effectiveness of specific aspects of the recommendations; and (ii) information regarding health technology assessment agency operations (e.g., number of days to complete the health technology assessment, number of committee members, appeal processes).
Five key informants were invited to present their views and perspectives: a representative from the Ontario Committee to Evaluate Drugs; one of the lead investigators from the Grading Recommendations Assessment, Development, and Evaluation (GRADE); a representative from the Ontario Ministry of Health; a bioethics researcher; and a researcher and policy expert from the National Institute for Health and Clinical Excellence. These presentations occurred over a span of five months between February and May 2007. The expertise of these five individuals encompassed health research methodology, health policy, health economics, decision making, bioethics, and clinical medicine.
RESULTS
Literature Review and Key Informants
Results from our literature review and key informants are presented in Table 1 (see previous section for purpose, questions, and criteria). Certain pertinent “methods/models” were not included in Table 1 because they did not have sufficient information about all three categories (decision criteria, evaluation of criteria, methods to synthesize criteria). However, the partial information they did provide was still considered in the overall framework development. Key informants provided their views on processes used for recommendations as they related to each representative setting. Based on the review of the literature and information from key informants, an explicit and transparent recommendation process was developed.
Literature Review and Key Informant Interviews
OCED, Ontario Committee to Evaluate Drugs (formerly Drug Quality and Therapeutics Committee); NICE, National Institutes for Clinical Excellence; GRADE, Grading Recommendations, Assessment, Development, and Evaluation; MOH – ON, Ontario Ministry of Health and Long-Term Care; A4R, Accountability for Reasonableness; ICER, incremental cost-effectiveness ratio; QALY, quality-adjusted life-year.
Conceptual Foundations
The subcommittee drew on several key ideas: evidence based medicine to reflect the primacy of scientific evidence; cost-effectiveness analysis to reflect the importance of using society's resources wisely; “Accountability for Reasonableness” (Reference Daniels, Sabin, Daniels and Sabin6;Reference Daniels and Sabin7) (“Accountability for Reasonableness” is an ethical decision-making framework for fair policy setting) to describe the elements of a fair decision-making process; and a deliberative process to describe how the separate attributes of health technology should be evaluated and weighed. These ideas formed the conceptual foundation of our health technology recommendation process.
The “Decision Determinants Sub-Committee” believed that scientific or context-free evidence must play a key role in the decision making for a health technology (Reference Culyer5). Context-free evidence is defined as evidence gathered through systematic and replicable methods: a randomized controlled trial is an example of context-free evidence. The “Decision Determinants Sub-Committee” also recognized the importance of other forms of evidence including context-sensitive evidence (evidence collected to test hypotheses in systematic ways that are relevant to the context in which the health technology is used—for example, cost-effectiveness and social surveys), expert opinion and experiential or colloquial evidence (Reference Culyer5;Reference Lehoux and Williams-Jones15).
The subcommittee believed that cost-effectiveness analysis would assist in attaining an overall value for society's resources for the healthcare system, would offer an explicit framework for considering resource allocation decisions (Reference Daniels, Sabin, Daniels and Sabin6;Reference Neumann, Palmer and Daniels18), and was a key ingredient in decision making.
Pertinent concepts from “Accountability for Reasonableness” include the following: (i) transparency of decisions and their rationale; (ii) decisions made on the basis of reasons that fair-minded people would consider; (iii) procedures for revising decisions in light of challenges; and (iv) enforcement (voluntary or public regulation) of the first three elements.
From the key ideas, the “Decision Determinants Sub-Committee” developed guiding principles to serve as the supporting framework for OHTAC's recommendation process: (i) recommendations are to be consistent with principles of fairness and transparency; (ii) the recommendation process itself should be consistent across health technologies; (iii) OHTAC's decisions are to be made after considering evidence pertaining to the following explicit criteria: Overall Clinical Benefit, Consistency with Societal and Ethical Values, Value for Money, and Feasibility of Adoption into the Health System (evidence of Overall Clinical Benefit must be present before evaluating any of the other criteria); (iv) context-free, context-sensitive, and colloquial forms of evidence all have a place in the evaluation of a health technology (in general, this will be subsidiary to high quality studies of context-free and context-sensitive evidence); (v) the recommendation process is to take place within a deliberative process defined as one that involves interaction of OHTAC members; (vi) OHTAC's recommendations for a health technology are to indicate that the use of an intervention is limited to a specific demographic population (e.g., age, gender, and ethnicity) only if there is clear evidence that these demographic differences are important for clinical benefit; and (vii) OHTAC's recommendations may include adoption (with or without conditions or limitations), rejection, or further evaluation (field study) of the health technology.
As described above, the “Decision Determinants Sub-Committee” recommended that OHTAC consider evidence on four explicit criteria when making a draft recommendation regarding a health technology (Table 2): (i) Overall Clinical Benefit, (ii) Consistency with Societal and Ethical Values, (iii) Value for Money, and (iv) Feasibility of Adoption into the Health System. As shown in Table 2, each criterion has two or four subcriteria. Definitions of the criteria are shown in the Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc.
Criteria and Subcriteria
The criteria and subcriteria are each discussed briefly here.
(i) Overall clinical benefit is a measure of the overall health benefit of using a health technology to either diagnose or manage a disease:
a. effectiveness addresses the potential effect of the technology on health;
b. safety refers to the frequency and severity of adverse events or complications arising as a result of using the new technology compared to an alternative;
c. burden of illness relates to incidence, prevalence, or other measurements of impact of an illness on population health; and
d. need is measured with respect to the availability of alternative options, and need may be large if there are no alternative relevant health technologies.
(ii) Consonance with social and ethical values relates to considering
a. societal values regarding appropriate usage and impact of the health technology; and
b. ethical issues inherent in using or not using the health technology.
(iii) Efficiency or value for money is a measure of the appropriateness of using societal resources for a specific technology. It is a measure of the health gain, relative to resources consumed for a particular health technology compared to available alternatives. It is determined after completing appropriate economic evaluations.
(iv) Impact on the healthcare system is a measure of ease with which the health technology can be adopted into the Ontario healthcare system with respect to implementation.
a. Economic feasibility addresses the affordability of implementing the health technology in Ontario.
b. Organizational feasibility refers to system enablers and barriers to diffusion within the healthcare system's infrastructure, including human resources and regulatory aspects.
The “Decision Determinants Sub-Committee” recommended that the evaluation of the four explicit criteria be reported as one of four options (high/large; moderate/medium; low/small; unknown/uncertain) in relation to magnitude of evidence, degree of certainty regarding the evidence, and quality of evidence with respect to data sources. The committee also recommended that a graphical aid be used to reduce the cognitive demands of integrating input from these disparate areas. The aid is in the form of a system of open and filled circles that correspond to the strength of evidence within a given category. Supplementary Table 2 displays the interpretation of the legend meanings with respect to the criteria and subcriteria for magnitude of evidence.
The “Decision Determinants Sub-Committee” found that a deliberative process, which has been used by OHTAC since its inception, remains the best way of integrating the various types of evidence (www.chsrf.ca) (Reference Chernew, Hirth, Sonnad, Rrmann and Fendrick4;Reference Culyer5). A deliberative process is defined as “. . .a tool for producing guidance based on heterogeneous evidence. It is a participatory process that includes representation from experts and stakeholders, face-to-face interaction, criteria for the sources of scientific evidence and their weight. . ..” (Page 7). The “Decision Determinants Sub-Committee” recognized that the relevance of a particular criterion may change on a case by case basis, and because of this, a deliberative process is conducive to the individualized appraisal necessary for each health technology.
Example
The form to be used to guide the recommendation process following the completion of the Medical Advisory's Secretariat's health technology policy analysis/assessment is presented in Table 3 with an example. On the form, “Magnitude of Evidence” refers to the size of the effect measured for the evidence parameters (e.g., the magnitude of effectiveness may be reflected in the odds ratio; the magnitude of the burden of illness may be reflected in the prevalence or incidence rates); and “Evidence Source” refers to the source of the evidence (e.g., published randomized controlled trials, Ministry of Health database, expert opinion). The health technology being evaluated is the Low Density Lipoprotein (LDL) Apheresis with the Heparin-induced, Extracorporeal Low Density Lipoprotein Precipitation (HELP) system for use in heterozygous familial hypercholesterolemia. Apheresis is akin to dialysis and can be used to eliminate cholesterol-containing particles from the bloodstream. It removes the LDL-cholesterol from the blood by passing it through a filtration machine which then returns the plasma and blood back to the body. This procedure could be used when other interventions such as diet or lipid lowering drug therapy have not been successful in lowering high LDL-cholesterol levels sufficiently (http://www.health.gov.on.ca/english/providers/program/ohtac/tech/reviews/sum_ldl_20071122.html). Supplementary Figure 1 illustrates the proposed health technology recommendation process in relation to the interaction between the criteria and the recommendation.
Example: Evaluation of Criteria for LDL Apheresis: Heterozygous (HTZ) Population (
= High, 
= Moderate, 
= Low, 
= Uncertain)
aFor both Magnitude of Evidence and Quality of Evidence, comment on the degree of certainty when appropriate: variability around the evidence effect estimates (e.g., wide confidence intervals will indicate a low degree of certainty in the point estimate).
bQuality of Evidence refers to the likelihood that further research or evidence will change the magnitude and/or the degree of certainty of the evidence effect estimates: “high” quality of evidence indicates further research is unlikely to change confidence in the evidence effect estimates; “moderate” quality of evidence indicates further research is likely to change confidence in evidence effect estimates; “low” quality of evidence indicates that further research is very likely to change confidence in the evidence effect estimates; and “unknown” indicates the evidence effect estimates are very uncertain.
Proposed Health Technology Assessment Cycle Within OHTAC
An overview of the proposed health technology assessment cycle is presented in Figure 1. The use of the four criteria and associated subcriteria are incorporated into five of the six phases of this cycle and are to be considered at several phases (shaded boxes): at the vignette phase (Phase 1), at the review phase (Phase 2), at the recommendation development phase (Phase 3), at the public engagement phase (Phase 5), and at the appeal process phase (if a recommendation should be subject to an appeal) occurring during the post-recommendation phase (Phase 6). The details on these phases are described below.
MAS-OHTAC decision determinants overview. OHTAC, Ontario Health Technology Advisory Committee; MAS, Medical Advisory Secretariat.
Phase 1: Vignette Phase. During this phase, the proposed criteria are to be reviewed to help identify any knowledge gaps and stakeholder engagements that need to be initiated to fully understand the health technology under review.
Phase 2: Review Period Phase. The Medical Advisory Secretariat engages in a 16-week systematic review of the literature for presentation to the Ontario Health Technology Committee. The systematic literature review is presented with respect to the four criteria and subcriteria: forms such as the one presented in Table 3 are filled out by the Medical Advisory Secretariat.
Phase 3: Recommendation Development Phase. During this phase, the evidence is to be reviewed using a deliberative process with emphasis on the proposed criteria.
Phase 5: Public Engagement Phase. Comments on the draft OHTAC recommendation are to be solicited from the public during the public engagement process carried out during this phase. The public engagement comments are to be reviewed with respect to their impact on the evaluation of the four main decision determinants criteria (Overall Clinical Benefit, Consistency with Societal and Ethical values, Value for Money, and Feasibility of Adoption into Health System). The evaluation of the four main criteria may be amended depending on any new or significant information obtained from the public engagement process. OHTAC's recommendation may need to be revised. Once finalized, it is posted on the OHTAC/Medical Advisory Secretariat's Web site.
Phase 6: Post-Recommendation Phase – Appeal Process. During this phase any new evidence introduced during the appeal process is to be evaluated using the proposed criteria.
Conceptual Foundation and Process for Recommendation
OHTAC's process of recommendation described above fits with its conceptual foundation: (i) the process of recommendation is fair because it is transparent, there is an appeal process, and it is multidisciplinary as well as multiprofessional; (ii) the process of recommendation is consistent because the same set of explicit criteria are considered for each health technology; and (iii) it is deliberative because it involves the interaction of OHTAC members and entails consideration of myriad types of evidence (scientific and colloquial).
CONCLUSION
Although technology assessment agencies throughout the world (Reference Garcia-Altes, Ondategui-Parra and Neumann10;Reference Noorani, Husereau, Boudreau and Skidmore20) produce a myriad of high-quality scientific reports, their influence in policy making is dependent on the extent to which payers use findings to make decisions. Many countries have organizations that conduct health technology assessment. This work tends to be shared in the international sphere through the International Network of Agencies for Health Technology Assessment (www.inahta.org). In the United Kingdom, the National Institute of Clinical Excellence conducts appraisals that are expected to be adopted throughout the National Health Service (Reference Rawlins22;Reference Royle and Waugh23). In the United States, there is competition between clinicians, manufacturers of health technologies, and evidence-based technology assessment with respect to influencing technology diffusion (Reference Borowski, Brehaut and Hailey3). Recently, there have been important steps toward coordination of policy on technology assessment including the Health Technology Strategy (http://www.hc-sc.gc.ca/hcs-sss/pubs/ehealth-esante/2004-tech-strateg/index-eng.php) which is a Pan-Canadian Strategy enabling the sharing of policy priorities as well as best practices, thereby creating an “exchange” forum for technology assessors and a “policy” forum for decision makers. Even in the United States, where a globally budgeted system is nonexistent (Reference Bodenheimer2), a panel on Integrating Cost-Effectiveness Considerations into Health Policy Decisions was recently convened in 2006 with a goal to advance the use of evidence-based medicine and in particular cost-effectiveness analyses (Reference Neumann, Palmer and Daniels18;Reference Rawlins22). In general, these health technology assessment agencies throughout the globe aim to make evidence based decisions. However, fairness concepts are inconsistently incorporated, and there is substantial variability across agencies regarding the role and strength of evidence, the role of cost-effectiveness, and the extent to which social and ethical aspects are considered. Our process recognizes the importance not only of scientific evidence, but also includes a formal role for other types of evidence. It also offers an example of a process that integrates a variety of important ideas. The robustness and utility of our approach to decision makers will require ongoing evaluation. During this phase, the evidence is to be reviewed using a deliberative process with emphasis on the proposed criteria.
The Ontario experience represents the newest HTA initiative, while Agence d'evaluation des technologies et des modes d'intervention en santé (AETMIS) is the oldest HTA agency in Canada (http://www.aetmis.gouv.qc.ca/site/en_agence.phtml). Although there appears to be close proximity between the evaluation and policy processes in the Ontario process (given that the Medical Advisory Secretariat conducts a systematic review of the literature within a 16-week period to be presented to the OHTAC), the recommendation process reflects a deliberative process among members of OHTAC, which is independent of the systematic review. To make the evaluation and policy processes truly separate, the systematic review would need to be conducted by researchers who are not part of MAS. The advantages of the subsequent phases of the Ontario process include engagement of the public through the solicitation of input concerning recommendations and the opportunity for appeals. The Ontario process does not yet possess a citizens' council as exists at NICE. However, as with NICE, OHTAC does not set a threshold for the economic evaluations, uses accountability for reasonableness in its deliberation process, and accounts for three levels of evidence (scientific and context-free, scientific but context-sensitive, colloquial). Unlike NICE, it does not consistently use cost per QALY as the metric for recommendations. The Alberta and Ontario HTA processes have some features in common but also some differences: (i) the identification and selection of topics for HTA in Alberta rely on advice from the Committee; in Ontario, OHTAC primarily receives requests for reviews of health technologies from potential purchasers (typically hospitals) or from MOHLTC itself; (ii) OHTAC prioritizes its HTAs through a process based on a MAS vignette whereas, in Alberta, a project charter is prepared specifying questions and issues to be addressed; (iii) HTAs are undertaken by an external partner or partner (universities or IHE) or contracted consultants in Alberta; in Ontario, MAS undertakes the HTA itself; (iv) in Alberta, stakeholders are provided with a synthesis report and asked to provide responses to specific questions; in Ontario, stakeholders are now asked to be involved throughout all phases of the HTA assessment; (v) in Alberta, a recommendation on the health technology is formulated by Ministry staff, while in Ontario, recommendations are formulated by OHTAC and submitted to the Deputy Minister of Health.
The focus of this article has been on the development of an explicit framework for the recommendation of adoption of new health technologies. A “Decision Determinants Sub-Committee” was formed to develop this framework, and specific criteria to be used during the recommendation process. A review of the literature was conducted along with key informant interviews. A proposal for a consistent method of recommendation was developed and presented to OHTAC, which has since agreed to pilot test this framework. This framework is consonant with OHTAC's objective to be consistent in its work. This process represents an important step toward achieving this objective. The framework is currently being used by OHTAC, and the transparency of the process should enable one to have a clear understanding of how OHTAC arrives at its recommendations regarding specific technologies. It is envisaged that this framework will generate future research to validate its effectiveness (Reference Bodenheimer2).
CONTACT INFORMATION
Ana Johnson, PhD (ana.johnson@queensu.ca), Canada Research Chair in Health Policy, Community Health and Epidemiology, Queen's University, Abramsky Hall, 21 Arch Street, Room 311, Kingston, Ontario K7L 3N6, Canada
Nancy Sikich, MSc (Nancy.Sikich@ontario.ca), Clinical Epidemiologist, Medical Advisory Secretariat, Ministry of Health and Long Term Care, 20 Dundas Street West, 10th Floor, Toronto, Ontario M5G 2N6, Canada
Gerald Evans, MD, FRCPC (evansg@queensu.ca), Associate Professor, Department of Medicine, Division of Infectious Diseases, Queen's University, Room 3013 Etherington Hall; Consultant, Department of Medicine, Division of Infectious Diseases, Kingston General Hospital, 76 Stuart Street, Kingston, Ontario K7L 3N6, Canada
William Evans, MD, FRCPC (bill.evans@jcc.hhsc.ca), Professor, Department of Oncology, McMaster University; President, Juravinski Cancer Centre, 699 Concession Street, Hamilton, Ontario L8V 5C2, Canada
Mita Giacomini, PhD (giacomin@mcmaster.ca), Professor, Department of Clinical Epidemiology and Biostatistics, McMaster University, Health Sciences Centre, 1200 Main Street West, Hamilton, Ontario L8N 3Z5, Canada
Murray Glendining (glendining@hhsc.ca), Executive Vice President, Corporate Affairs, Hamilton Health Sciences, 237 Barton Street East, Hamilton, Ontario L8L 2X2, Canada
Murray Krahn, MD, MSc, FRCPC (murray.krahn@uhn.on.ca), Professor, Department of Health Policy, Management and Evaluation, University of Toronto; Toronto General Hospital, 200 Elizabeth Street, ENG 248, Toronto, Ontario M5G 2C4, Canada
Les Levin, MD, FRCP, FRCPC (les.levin@ontario.ca) Head, Senior Medical, Scientific and Health Technology Advisor; Head, Medical Advisory Secretariat, Ministry of Health and Long Term Care, 20 Dundas Street West, 10th Floor, Toronto, Ontario M5G 2N6, Canada
Paul Oh, MD, MSc (oh.pauldr@torontorehab.on.ca), Medical Director, Cardiac Rehabilitation and Secondary Prevention Program, Toronto Rehabilitation Institute; Lecturer, Department of Medicine, University of Toronto, 347 Rumsey Road, Toronto, Ontario M4G 3M5, Canada
Charmaine Perera (charmaine.perera3@ontario.ca), Senior Policy Analyst, Ministry of Education, Policy Program Branch, 15th floor, Mount Block, 900 Bay Street, Toronto, Ontario M7A 1L2, Canada
