Principle 1: The Goal and Scope of the HTA Should Be Explicit and Relevant to Its Use

A detailed scoping document should be developed before initiation of the HTA process, with broad, multidisciplinary, stakeholder involvement. The document should focus on defining the questions to be addressed by the HTA, plus the link between the HTA and any decisions about the use of the technology.

Defining the scope of the appraisal is central to the HTA process. As the objective of HTAs is to inform and guide clinical and policy actions, there should be a scoping document that clearly and explicitly identifies the decisions on which the HTA will be focused. The questions to be addressed should be stated with as much precision as possible, with specific aims clearly stated and testable hypotheses developed, when possible. Question development and definition explicitly should consider the context of the decisions to be made and how the technology will be used.

The draft scoping document should be widely circulated to all stakeholders, with extensive and meaningful opportunities to constructively critique, and potentially influence, the process. Responses should be provided for major questions raised in the scoping development process so that the resulting HTA process is anchored around a common understanding of the intent of the review and the totality of evidence required to answer its questions.

For example, in the U.S. public sector (e.g., the Agency for Healthcare Research and Quality AHRQ), the HTA problem scoping process often is vague and rarely framed around how the results will be used (e.g., in reimbursement). When outside stakeholders are permitted to review the draft scoping document, their ability to critique or influence the scoping frequently is limited to submission of formal written comments. In contrast, in the United Kingdom, NICE outlines very clearly the decision to which the HTA relates and holds scoping workshops where sponsors, HTA researchers and other key stakeholders can discuss the proposed scope and inform the final scoping process. In Germany, the law requires that IQWiG gives predefined individuals and organizations the opportunity to participate in all key steps of the assessment procedure.

In some jurisdictions, the United States being the most prominent, there is resistance to explicitly including considerations of cost in HTAs. In a diverse, decentralized system with multiple payers, insurers, healthcare organizations, and other providers, costs and perspectives may differ widely. (The same is true of many European healthcare systems.) More importantly, inclusion of cost into HTAs raises explicit questions of rationing of care, which is controversial and has limited public support in the United States. For instance, although one of the stated objectives of AHRQ evidence assessments is to help purchase services, such analyses are confined to evaluation of effectiveness evaluations, excluding cost considerations. CMS, which makes coverage decisions and funds care for the elderly, does not have the authority to consider costs in its decisions. It is also true of many private sector payers, which have explicit policies to not consider costs in their HTAs (e.g., Blue Cross and Blue Shield Associations TEC and its associate Medical Advisory Committee). In contrast, the Centers for Disease Control and Prevention Advisory Committee on Immunization Practices (ACIP) does explicitly consider costs and cost-effectiveness in its deliberations and recommendations.

Despite these formal constraints, cost-effectiveness is an implicit consideration in some policies and private plans (Reference Neumann and Sullivan29;Reference Tunis40) and practice guidelines (e.g., USPSTF, National Heart Lung and Blood Institute National Cholesterol Education Project Adult Treatment Panel III). However, without a formal, structured consideration of costs, all relevant economic factors may not be considered, resulting in biased, inaccurate conclusions and recommendations, especially for reimbursement and purchasing decisions. Furthermore, a focus on clinical outcomes alone may exclude important advantages and disadvantages of health technologies, such as impacts on quality of life, patient preferences, and use of other healthcare resources.

Principle 2: HTA Should Be an Unbiased and Transparent Exercise

Given the inherently complicated and controversial nature of HTA-based decisions and their importance to multiple decision makers and stakeholders, the HTA process is best conducted independently of the body that ultimately will be responsible for adopting, paying and implementing the HTA decisions. Furthermore, the HTA process and the detailed basis on which recommendations and decisions are made must be transparent.

Inherent to HTA is that multiple parties (including payers, manufacturers, patients, healthcare professionals, healthcare institutions, and the general public) have an interest in the process and results. Therefore, if HTA is to be widely accepted, it needs to be unbiased and transparent, in perception as well as in fact. In many jurisdictions, organizations undertaking HTAs or EBM reviews are perceived as being too closely affiliated with specific interests, particularly if they are part of the government structure (e.g., the Pharmaceutical Benefits Advisory Committee (PBAC) in Australia, or DAHTA@DIMDI in Germany), linked to a payer (e.g., NICE as part of the NHS in England and Wales, and in the U.S. private sector health insurance companies and insurance company hired pharmacy benefit managers) or part of a professional society (e.g., American College of Cardiology; American College of Gastroenterology; American College of Obstetrics and Gynecology; American Pediatric Association; American Society of Clinical Oncology).

In some countries, mechanisms are used to reduce perceptions of bias. The most common approach is to have recommendations made by an external expert advisory committee, comprised of academics, healthcare professionals, patient representatives, and (sometimes) industry representatives. In Canada (CADTH) and the United Kingdom (NICE), the HTA agency has been established at an “arms length” relationship, with government funding but an independent oversight board. In Sweden, where LFN is an independent public authority set up to evaluate applications for reimbursement from the manufacturers of drugs, the actual decisions on reimbursement are taken by a committee of external members, appointed by the government. SBU is also an independent authority, led by a director general, but final decisions about its reports are taken by its governing board, which has members from different stakeholders. The country councils, the “payers,” have an opportunity to be informed about and influence the process, but cannot dictate the decisions. The AHRQ's U.S. Preventative Services Task Force (USPSTF), the Centers for Disease Control and Prevention Advisory Committee on Immunization Practices (ACIP), and the National Institutes of Health Heart, Lung and Blood Institute's National Cholesterol Education Committee Adult Treatment Panel (NCEP-ATP) are government appointed but independent expert advisory panels. In contrast, the Centers for Medicaid and Medicare Services (CMS) Medical Care Advisory Committee (MCAC) and Food and Drug Administration expert advisory committees are government appointed, independent expert advisory panels whose recommendations are presented to agencies that have direct regulatory authority.

While some groups perform their own HTA evaluations using their own dedicated staff, supplemented by external consultants (Blue Cross and Blue Shield Associations Technology Evaluation Center, SBU and LFN in Sweden), several HTA organizations rely extensively upon commissioned external groups (often academic centers; occasionally health care consulting companies), to conduct HTA reviews. These can be full, independent assessments (e.g., NICE Multiple Technology Appraisals and reports commissioned or produced by IQWiG, AHRQ, the Centers for Medicare and Medicaid Services Medical Clinical Advisory Committee and the American College of Physicians Clinical Efficacy Assessment Project), or independent critiques of company submissions, as occurs in Australia, Canada, Scotland, and in many health insurers in the United States and by most U.S. pharmacy benefit managers.

HTAs, including detailed, specific discussion of the basis for conclusions, should be freely and publicly accessible to stakeholders. In some jurisdictions, the assessments of medical technologies, procedures, and practices and the reasons for the subsequent decisions, are made public (e.g. SBU and LFN in Sweden, NICE in the United Kingdom, American Heart Association, American College of Physicians Clinical Efficacy Assessment Project) or the process itself is open to the public (e.g., Centers for Medicare and Medicaid Services Medical Clinical Advisory Committee), thereby increasing the transparency of the process and the underlying rationale.

However, few jurisdictions have yet taken the step of holding all HTA deliberations and committee meetings in public. More commonly, the HTA processes and rationales of many organizations conducting HTAs (especially U.S. private sector payers, PBMs, and many CMS decisions) are performed in private, with only incomplete and often vague information made public to explain recommendations or decisions. The All Wales Medicines Strategy Group is one example of an organization adopting an open and transparent approach for its deliberations, and there are indications that NICE may hold the discussions of the evidence in public in the future.

HTA organizations also differ widely in the degree to which stakeholders are allowed to participate in the HTA process and interact with the decision makers. While almost all HTA organizations encourage manufacturers and other interested parties to provide relevant information and data for consideration, most do not have a formal mechanism allowing external interested parties to review and critique draft analyses and recommendations before their final determinations. These practices would further increase transparency and perception of independence and objectivity, thereby building acceptance of the process among stakeholders and improving HTA content and accuracy. For example, the NICE technology appraisal process involves extensive interaction between the sponsor (that for new technologies often has all the data, but a specific interest) and agency staff (who are trying to understand the unique attributes and appropriate clinical role of the technology being evaluated), with substantial conduct and provision of supplementary analyses that address questions that arise or are re-specified as the review process progresses. In Germany, IQWiG asks manufacturers for their opinions at different stages of the evaluation process, in hearings or by means of written comments.

It might be argued that the issues of perceived independence and transparency should be different for technology assessments and appraisals undertaken by private payers, where the HTA may be viewed as a vehicle for assisting the discussion and negotiation about the value of the technology and the price to be paid. However, it is difficult to see why the same standards for independence and transparency should not apply for technology assessments undertaken by private payers, albeit with some potential modifications discussed below.

Objective scientific review of the evidence is the foundation for a rigorous assessment. This evaluation should be performed objectively and without regard to the economic interests of the evaluator. (It has often been said that everyone is entitled to his/her own opinion, but not his/her own facts.) We believe that clinical assessment requires full transparency. While economic assessment also demands significant transparency, economic transparency in private markets raises several issues (e.g., protecting proprietary or competitively sensitive information, such as negotiated prices) that may complicate full application of this principle. One potential mechanism for handling this is to have complete transparency surrounding the economic models used, but a more general level of disclosure regarding selected, proprietary data inputs. This is similar to the approach that is followed in the United Kingdom by NICE.

Issues of independence and transparency may be more complicated for private payer technology coverage decisions. Even here, however, plan enrollees and providers should have substantial information on how decisions are made regarding access to and reimbursement for treatments, particularly in the United States, where there are no public insurance options for most nonindigent Americans younger than age 65 and many employees do not have a choice of insurers. In this case private payers have much of the same authority and serve the same coverage decision-making function as governmental agencies do elsewhere and thus should be held to similar standard.

Moreover, transparent information is required for prudent purchasing in a competitive market and informed decisions by physicians and patients. In addition, technology developers will also benefit from greater transparency regarding technology appraisals by private payers, as they will be in a better position to address concerns raised by payers during the appraisal process (e.g., by developing additional evidence to support the value of their product). Thus, while different payers may interpret and implement decisions differently based on contractual and/or market factors, they all should be held to high general standards of independence and transparency. While it may be appropriate to have a different level of transparency for certain specific (clearly proprietary) aspects of the assessment or the appraisal, the general principles should nevertheless apply.

Principle 3: HTA Should Include All Relevant Technologies

Because potential inefficiencies exist in all forms of healthcare, all health technologies should be potential candidates for HTA. Otherwise, decision making concerning the use of resources is likely to be distorted.

Health technologies include drugs, devices, procedures, diagnostics, and treatment strategies. The range of relevant technologies relates to the breadth of the budget that the decision maker is seeking to optimize. Therefore, if HTA is being conducted at the level of a health plan or on the national level, it should include all health technologies, including current standard or commonly used interventions using clearly defined, explicit criteria. Otherwise policies and clinical practices and policies inevitably will be distorted, with investment and practice gravitating toward those interventions that are free of evaluation, for which regulatory barriers are lower.

However, in many of the countries where HTA has been linked to reimbursement or coverage decisions, it has focused on drugs alone. There are several reasons for this. In many countries, there is an established procedure for approving drugs for reimbursement (i.e., a “positive list”), a process that often does not exist for other health technologies. Furthermore, because drugs are subject to a highly structured and rigorous licensing procedure, there usually is a substantial body of clinical literature that provides a strong basis for conducting HTAs. Pharmaceutical price increases also have exceeded that of most other health care components in recent years. Pharmaceuticals also are discrete goods with a purchasing chain that is more easily identified than many other medical goods and services. In addition (or, perhaps because of these factors), some HTA programs only have a remit to consider drugs (e.g., the LFN in Sweden, the SMC in Scotland).

However, none of the reasons given above justifies a focus on drugs alone, as doing so will inappropriately distort medical decision making, resource investment, and care. While in some HTA programs all new technologies are candidates for formal, rigorous assessment (e.g. NICE in the United Kingdom, IQWiG and DAHTA@DIMDI in Germany, CMS and AHRQ in the United States), in practice a disproportionate amount of HTA activity is still directed toward drugs.

The other common inappropriate focus of HTA is concentration on assessing new technologies. This likely occurs because new technologies are easily identifiable, tend to be more expensive than existing technologies, and their emergence requires a policy determination. However, many existing technologies are themselves thought to be inefficient or often used inappropriately. This leads to two problems. First, resources devoted to less effective, less safe, or less efficient medical practices represent wasted resources that otherwise could be allocated to more effective and cost-effective health technologies, thereby reducing public welfare. Second, when new technologies are being assessed they may be compared with a baseline (of “current care”), which itself may be inefficient. Therefore, an assessment that a new technology is cost-effective when compared with an existing treatment, that is itself inefficient, can be misleading. Increasingly, some jurisdictions are beginning to tackle this issue. For example, in Sweden the Pharmaceutical Benefits Board (LFN) is reviewing existing drugs as well as new ones.

Principle 4: A Clear System for Setting Priorities for HTA Should Exist

A clear process for prioritizing and selecting topics needs to be established, because in situations where not all technologies are assessed, there will be distortions in decision making about the investment and use of resources.

As with all other healthcare resources, the resources used in HTA should themselves be used in a cost-effective manner. No jurisdiction assesses all health technologies, although some of those focusing on drugs do assess all new products and formulations before listing for reimbursement (e.g., Australia, Canada, Sweden, PBMs in the United States).

In situations where only some technologies are assessed, selection priorities need to be set. In the United Kingdom, NICE sets priorities based on six criteria: (i) burden of disease; (ii) resource impact; (iii) clinical and policy importance; (iv) presence of inappropriate variation in practice; (v) potential factors affecting the timeliness of guidance; and (vi) likelihood of the guidance having an impact. A similar approach is followed by DAHTA@DIMDI, where a board of trustees uses similar criteria to derive priorities for future HTA projects. In the United States, the AHRQ solicits suggestions for topics for formal review from the general public, as well as stakeholders, and has identified a list of factors considered when selecting topics for review, along with a list of 10 priority health conditions. A recent systematic review found a broad range of criteria being used for priority setting in HTA organizations (Reference Noorani, Husereau, Boudreau and Skidmore30).

The other common approach to determining priority for assessment is essentially a procedural one. For example, in the Netherlands an HTA is conducted if there is an a priori case that a new drug should not be clustered with other, existing drugs in the nation's therapeutic reference price system. The HTA is used to assess the incremental value (if any) a drug provides over existing medicines, so as to determine whether a price premium is justified (and, if so, how much). A similar approach is being followed in Germany, where the G-BA can ask IQWiG to assess whether a new drug has sufficient incremental benefit to be excluded from the reference price system.

Nevertheless, in many HTA programs the process for setting priorities for topics is ill-defined and it is unclear what factors are considered or how topics are chosen. As noted above, in circumstances where only some technologies are assessed, this will distort decision making (as well as undermine perceptions of fairness and transparency).

Principle 5: HTA Should Incorporate Appropriate Methods for Assessing Costs and Benefits

Development and consistent implementation of rigorous, analytical methods is required to engender stakeholder and public trust in the process and its findings. This requires clarity of HTA process and methods, as well as access to experts with appropriate clinical and multidisciplinary methodological training.

Establishing appropriate methods is an essential feature of HTA processes. While most organizations conducting HTA have established their own methods (Reference Tarn and Smith39), existing HTA guidelines differ across organizations, often in important ways, including scoping of the issues to be addressed and the range of evidence accepted and methods used (e.g., the role and acceptability of quasi- and nonexperimental generated evidence; costs and cost-effectiveness; modeling). Some variation in the conduct of HTA is appropriate given differences in objectives (e.g., reimbursement versus clinical guidance) and healthcare system structure. In addition, there is a healthy debate among HTA experts about certain methodological issues.

However, clarity and specificity regarding the use and interpretation of data sources and methods that comprise the HTA process is required to ensure analytic objectivity and consistency, guide interpretation and application of results, and inform developers and proponents of the analytical requirements to which they will be held and thus guide what data need to be collected. Thus, it is important that those conducting HTA develop rigorous, evidence-based, state-of-the-art, clinical and policy relevant processes, and methods appropriate to local needs. Once established, methodological and procedural guidelines periodically should be reviewed so as to ensure that they keep pace with methodological developments (Reference Sculpher and Drummond35) and changing environments.

A common approach in HTAs incorporating an economic evaluation is to propose a baseline analysis, or “reference case,” using a standardized approach with the best available data, with additional analyses conducted that examine the impact of alternative estimates on findings (sensitivity analyses) and the impact in specific clinically relevant subpopulations (Reference Gold, Siegel, Russell and Weinstein18). The goal is to ensure consistency across different studies without stifling methodological developments and to examine the robustness of findings and results within the range of data uncertainty, methodological biases and limitations, and clinical populations.

Given the need to use HTA resources in a cost-effective manner, it is important that the methods used are “fit-for-purpose.” A comprehensive, well-conducted, EBM review is a necessary first step in the HTA appraisal process. EBM methods should be specified in advance of conducting the review and should use the best available evidence for the questions to be addressed. Thus, while well-conducted RCTs provide high quality evidence of efficacy, evidence from rigorous quasi- and nonexperimental studies will usually be required to address issues of effectiveness (benefit under typical clinical conditions), comparative effectiveness (incremental benefit versus the best available alternative interventions) and differential effects in specific clinical populations (e.g., subgroups defined by patient age, gender, ethnic, comorbidity).

However, new analyses or re-analyses of existing data may be required to inform the HTA, considering its intended purposes and resultant decisions. Typically, these analyses will be conducted to populate economic decision-analytic or simulation models, for example to project health and economic consequences over an adequate time horizon. In particular situations, it may be difficult to have meaningful RCT data at all and short-term clinical trial data must be linked with data on patient-relevant health outcomes and long-term costs. Typical examples include the evaluation of health promotion programs, screening programs, diagnostic procedures, or treatment of chronic diseases with a relatively slow disease progression (Reference Siebert37). Modeling techniques include decision tree analysis, Markov models, discrete-event simulation, and others. There is no single first choice of a modeling technique; the optimal modeling technique should rather depend on the particular research question of interest (Reference Brennan, Chick and Davies3;Reference Siebert37).

Sophisticated statistical and methodological techniques often are useful to address known methodological or data shortcomings. However, the resulting analytical complexity may reduce clarity and transparency of results. Thus, these techniques should be used only when justified, appropriate, and necessary. Occasionally, this issue has led to heated debate, such as in the United Kingdom, following NICE's insistence on probabilistic sensitivity analysis in its methods guidelines (Reference Griffin, Claxton, Hawkins and Sculpher19).

Finally, as well as having clear methods, it is important that those undertaking and reviewing HTAs are adequately trained. Given the multidisciplinary approach embodied in HTAs, it is important that research teams include both clinical and methodological expertise. Moreover, those conducting these analyses require some training in both of these areas, Although several relevant clinical epidemiology and health services masters and PhD level training programs exist, there are no formal training programs in HTA in many countries and capacity currently is a limiting factor in establishing and operating high level programs. Therefore, it is important that all stakeholders (e.g., industry, governments, academic institutions, medical specialty societies) invest in capacity building for HTA.

Principle 6: HTAs Should Consider a Wide Range of Evidence and Outcomes

HTAs require use of data from experimental, quasiexperimental, observational, and qualitative studies, integration of both endpoint and validated surrogate data, and assessment of the incremental impact of and trade-offs among multiple clinical, economic and social outcomes in clinically relevant populations.

While rigorously conducted RCTs often are held to be the reference standard for clinical evidence (and the cornerstone of EBM and CER) and are necessary to establish efficacy and causal inferences, they generally are not sufficient for the conduct of HTAs. RCTs have important, well-recognized, practical limitations in terms of number of questions that can be examined, sample size, length of follow-up, inclusion of a broadly representative population, generalizability, subgroup analyses and types of outcomes assessed. Moreover, RCTs are not immune from a variety of common experimental biases related to data specification, selective enrollment and incomplete data collection, recording, coding and follow-up. Thus, important information relevant to HTAs often must be obtained from quasi- or nonexperimental data and studies, including rigorously conducted observational studies, with appropriate control for confounding.

Similarly, surrogate endpoints frequently need to be considered and extrapolated to outcomes and endpoints of interest not examined, or able to be examined adequately in RCTs. While sophisticated analytical frameworks and methods can reduce the potential for bias inherent in nonexperimental studies, such biases cannot be eliminated. Despite this, failure to consider all available relevant information across the full spectrum of study designs, weighing the evidence according to its estimated validity and generalizability, will result in flawed analyses and the potential for incorrect and biased assessment.

Clinical benefits, risks, and costs must be defined broadly to include all relevant outcomes of interest. Thus, in addition to mortality and morbidity, outcomes assessed and integrated into the decision process should include impact on patient functional status and quality of life and economic outcomes (direct and indirect medical costs, productivity effects) not only for patients but also for other relevant parties (e.g., family, employer, and the broader society). Medical interventions are not simply good or bad—rather, their performance, outcomes and value vary across patients and clinical conditions: how good; in which people/patients; under what conditions. Thus, variations in clinical benefit, risk, and cost must be assessed across relevant patient and population subgroups (e.g., sociodemographic groups; clinical subgroups; healthcare systems). This is particularly true when conducting analyses where there are trade-offs in outcomes among the alternative interventions being compared.

Principle 7: A Full Societal Perspective Should Be Considered When Undertaking HTAs

HTAs should adopt a broad societal perspective to optimize efficiency and societal benefit and to avoid and identify potentially distorted clinical decisions and health policies resulting from adoption of narrower perspectives used by various healthcare system stakeholders.

The perspective of HTA assessments often is restricted, a function of a decision-maker's specific mission and perspective. For example, IQWiG considers the perspective of the community of German citizens insured by the statutory health insurance (21). In the United Kingdom, NICE only considers National Health Service (NHS) and Personal Social Services (PSS) costs in its clinical technology appraisals, excluding costs falling on other public sector budgets, on patients and families and on the broader economy (e.g., productivity costs) (27). However, a technology's impacts on other public sector budgets can be considered in NICE's appraisals of public health interventions. Therefore, an evaluation of a public health education campaign to reduce substance abuse can consider the potential savings in the costs of crime, whereas an evaluation of a drug maintenance program for addicts cannot. In the United States, government funded analyses often restrict consideration to patient level direct clinical benefits and risks.

Such narrow perspectives distort HTAs, healthcare decisions and health policies for society and for individual healthcare system components and constituencies. For example, in many health systems, pharmaceuticals are budgeted separately from other medical services. In such situations, the impact of drugs on overall clinical and cost-effectiveness and resulting clinical and cost offsets often are not fully considered and may be underestimated (e.g., compounds that increase drug costs but whose benefits and savings accrue to another health care system component budget).

In addition to the importance of optimizing societal resources, there often are advantages of using a broader perspective for the HTA, even for decision makers with limited budgetary or programmatic perspectives. For example, examination of the costs falling on patients and families might give an indication of the likelihood of individuals participating in and adhering with a prescribed treatment regimen, or an indication of what payers might accept in increased premiums to have additional treatments covered in the plan. Moreover, such analyses will help guide broad management decisions enhancing patient global health and function.

In all countries, an HTA must meet the needs of multiple decision makers. The more decentralized the healthcare system, the greater the challenge. Therefore, it makes sense to keep the assessment broad, disaggregating the portion of the benefits and costs associated with various components, thus facilitating the validity of the overall analysis. Armed with a broad assessment, individual decision makers can then identify those costs and benefits most important to them. In such cases, a broad perspective will highlight significant differences and subsequent distortions between alternative decision makers and is required to avoid rejecting interventions that provide clear societal benefit, even if increasing cost to various system components or constituencies.

Principle 8: HTAs Should Explicitly Characterize Uncertainty Surrounding Estimates

All data are imperfect point estimates of underlying distributions that incorporate a variety of errors. All analytical methods are subject to biases and limitations. Thus, extensive sensitivity analyses are required to determine the robustness of HTA findings and conclusions. The limitations of the analysis should always be acknowledged.

All decisions are subject to uncertainty. All data, regardless of source, represent point estimates of a range and distribution of values from which patients and populations are sampled. All data are subject to errors in ascertainment, measurement and interpretation. In addition, all decisions rely on underlying conceptual and theoretical models, which are subject to structural and mechanistic uncertainty (as are their representations). These uncertainties are important if alternative values within the estimated range of certainty result in different HTA findings and conclusions. Thus, extensive one-way and multiway sensitivity analyses of input variables (input data estimates, transition matrices, outcomes, costs) and alternative structural representations are required to define the thresholds within which decisions are stable and beyond which decisions may change.

Furthermore, sensitivity analyses, in conjunction with sophisticated modeling techniques (e.g., value of perfect information), help characterize the confidence levels of data results, assessments and policy conclusions. In addition to their prescriptive value, sensitivity analyses also characterize the nature as well as the magnitude of decision uncertainty. This, in turn, provides important insights into the clinical or policy problem and its management, identifies key parameters that require more precise and reliable estimation, and helps prioritize future research and targets for future intervention. Similarly, there often is value in determining the consistency of overall results and conclusions across different data sets and analytical methods, each of which is subject to its own limitations and biases. The greater the robustness of results across methods and sensitivity analyses, the more confidence one may have in the ultimate findings and conclusions. However, any analysis has remaining limitations and these should always be acknowledged.

Principle 9: HTAs Should Consider and Address Issues of Generalizability and Transferability

Examination of the generalizability and transferability of HTA findings across clinical populations and policy relevant perspectives is required, given the inherent variability of disease, intervention responses, and outcomes across patients, populations, providers, healthcare delivery sites and healthcare systems.

All data, analyses, and findings have, to varying degrees, limited validity, generalizability, and transferability. Such limitations result from differences in patient selection, geography, sociodemographics, genetics, disease presentation, disease stage, patient behavior, patient and provider preferences, economic factors, and healthcare delivery system differences to the extent that these factors are associated with varying clinical, economic, and social outcomes. Economic analyses are more prone to these problems than clinical analyses, because economic data and study samples have often not been prospectively selected, or designed to address economic issues and significant differences in practice patterns, resource use, and unit costs across healthcare delivery systems. Varying decision perspectives also impact on the validity, generalizability, and transferability of study results, HTA findings, and clinical and policy decisions.

Generalizability and transferability across patients, populations, countries, and systems become more problematic the broader the range of stakeholder and decision-maker perspectives and preferences. Thus, HTAs explicitly should consider the degree to which assessments and decisions are generalizable and transferable across patients, populations and settings of care. As discussed above, use of a broad range of experimental, quasiexperimental and nonexperimental observational data will tend to enhance generalizability and transferability to the extent that results and conclusions are consistent across populations and methods. Also, modeling and sensitivity analyses provide estimates of the confidence levels and robustness of HTA results. When conclusions vary across populations and methods, the differences often provide insight into data and study limitations and thereby inform interpretation of overall data and technology value and use.

Principle 10: Those Conducting HTAs Should Actively Engage All Key Stakeholder Groups

HTA programs should actively engage all key stakeholders in all stages of the HTA process, as this is likely to result in technology assessments of higher quality that are more widely accepted and stand a greater chance of being implemented. Moreover, such an open process will enhance transparency and trust in the process as stakeholders develop a greater understanding of the criteria and standards used.

As noted earlier, HTA, by its nature, involves many interests. Jurisdictions differ greatly in the extent to which the various stakeholders are involved in and influence the process. At one end of the spectrum, the only involvement of stakeholders, particularly manufacturers, is in the submission of data and/or analyses for potential consideration (e.g., Blue Cross and Blue Shield Associations, Centers for Medicare and Medicaid Services). Several HTA programs, such as the AHRQ in the United States and IQWiG in Germany, allow stakeholders to comment on draft reports. However, response times for making comments are often short, limiting opportunity for input by all but the most involved parties. Moreover, while submission of formal written comments is permitted, formal responses to issues raised and disagreements in interpretation of data are not permitted and opportunities for a true constructive dialogue is limited, at best.

At the other end of the spectrum, but much less commonly, some HTA organizations (e.g., NICE) involve stakeholders at several stages in the process, including scoping of the assessment, submission of evidence, commenting on the draft report and, if necessary, appealing the decision. In the case of NICE, approximately 30 percent of decisions have gone to appeal, approximately 50 percent of which have been upheld (28). When appropriately structured, such broad interactions with sponsors as well as other interested stakeholders is to be encouraged, as there are many examples where such interaction identified errors in the analysis performed by the assessment groups (e.g., in the NICE HTAs of temozolomide and ezetimibe) and resulted in more rigorous, valid findings, recommendations and policies. In Germany, IQWiG has introduced a procedure where stakeholders are invited to participate in a formally structured manner, in hearings and by written comments. All comments and discussions are reported in the supplements of the reports (20).

NICE's introduction of a scoping workshop in 2005 has proven very popular, especially with technology manufacturers. The workshops offer the opportunity for meaningful dialogue between the Institute, the independent assessment group, and key stakeholders about (i) the precise questions to be addressed; (ii) alternative technologies to be appraised; (iii) patient populations to be studied; and (iv) important methodological issues, such as aspects of the modeling to be performed.

Technology manufacturers are particularly interested in having early involvement in the process, as it provides them more time and opportunity to plan appropriate studies and analyses that will be most responsive to the assessor's interests and needs. In addition, earlier involvement in and more interaction with the process provides greater opportunity to provide alternative perspectives regarding analysis and interpretation of existing available data. In response to requests from manufacturers, NICE in the United Kingdom has initiated a program of early involvement on an experimental basis. Whether or not this delivers the level of specificity in study designs resulting from similar interactions with licensing authorities is yet to be seen.

Principle 11: Those Undertaking HTAs Should Actively Seek All Available Data

Those conducting HTAs should actively seek all available data, whether confidential or not. In situations where confidential data are used, confidentiality should be defined as narrowly as possible and efforts should be made to make it publicly available as soon as possible, in the interests of maintaining transparency and engendering understanding of and trust in decisions.

At the time of conducting an HTA, some of the most important data may be confidential (e.g., unpublished clinical trial data). Handling of confidential data varies substantially among HTA bodies. NICE considers confidential data its assessments and excludes such information from public versions of the technology assessment reports. In contrast, in the United States both AHRQ and the DERP (in its EBM reviews) consider all data submitted subject to public disclosure to maximize the transparency of the process.

While transparency is a critical aspect of all HTA processes, this objective also needs to be balanced against the desire to make the best possible decisions based on all available data. Thus, it is important that explicit processes be developed to allow confidential data to be used, while protecting its confidentiality. While at times it may be necessary to consider confidential data in arriving at a decision, determinations regarding the validity of confidentiality claims should be carefully considered and defined as narrowly as required to maximize transparency, trust and understanding. The presumption should be that key data upon which HTA decisions are based should be made public (at least in summary form) in the absence of compelling confidentiality claims.

In formulating its guidance, NICE often requests permission from technology manufacturers, to release confidential data central to the decision in an effort to increase transparency. Often these data can be released by the time the guidance is published as, through the passage of time, they may have been published elsewhere. Furthermore, because a major barrier to access to otherwise confidential information is adherence with peer-reviewed journal publication policies, such policies should be re-examined in cases of release of clinical and cost-effectiveness information to HTA agencies (e.g., in summary form as is done for presentation of scientific abstracts), subject to reasonable safeguards (e.g., maintenance of strict limits on further publication and dissemination of information beyond the decision-making body). Indeed, some journals have taken this view (Reference Mauskopf and Drummond25).

Principle 12: The Implementation of HTA Findings Needs to Be Monitored

Implementation of HTA findings need to be monitored, both to ensure that the original investment in conducting HTAs is valuable and to ensure that findings are being implemented in a fair and even-handed manner.

The impact of HTA findings needs to be monitored in all settings, but especially in those jurisdictions where HTAs are performed to help guide a particular decision. Evidence from several jurisdictions indicates that the results of HTAs are not always implemented (Reference Drummond and Weatherly7;Reference Sheldon, Cullum and Dawson36). A recent report in the United Kingdom (1) found that NICE guidance sometimes was not being implemented in the NHS because of inappropriate use of allocated funding, lack of horizon scanning and poor planning. One particular concern of technology manufacturers is that HTA may be used to pursue a cost-containment agenda. That is, negative guidance will always be implemented, whereas positive guidance often may not, because of the resource consequences.

It also will be useful to explore mechanisms for encouraging greater implementation, which are likely to differ greatly among settings. For example, in a healthcare system like the British NHS, which operates with global budgets and salaries for medical practitioners, it may be more difficult to use financial incentives (e.g., fees for particular procedures) to either encourage or discourage the adoption of particular health technologies. Therefore, there is considerable interest in payment for performance, as in the NHS prescribing incentive schemes. On the other hand, financial incentives may be underutilized in many settings.

HTAs also should be evaluated with regard to their validity and clinical impact over time (e.g., how often subsequent data demonstrate errors in initial determinations) as a “quality control’ measure, with modification and revision of the HTA process and methods as required.

Principle 13: HTA Should Be Timely

HTAs should be conducted when they can inform key decisions in the diffusion and use of health technologies, and assessments should be kept up-to-date. To accomplish this goal requires timely conduct of studies by manufacturers and other advocates and, in selected circumstances, requires limited reimbursement conditional upon enrollment in a study to inform safety, effectiveness, and cost-effectiveness.

HTA is not a “one-shot” activity and assessments need to be revised as new data become available. Some organizations have a commitment to review their decisions after a period of time (e.g., the DERP updates its EBM reviews every 2 years; NICE reviews its guidance based on HTAs every 3 years, or more frequently if major, relevant new information becomes available). In addition, some of the data required to assess the true value of a new technology (long-term safety; uncommon adverse events; effectiveness as opposed to efficacy) can be gathered only after the treatment is used in clinical practice for some period of time (Reference Garrison, Neumann, Erickson, Marshall and Mullins14;Reference Mason, Drummond and Towse24).

Therefore, the timing of HTA assessments should be consistent with key steps in the development of new technologies and their introduction into the healthcare system. The goal should be to allow reimbursement decisions to be made as soon as possible after market authorization, while minimizing the risk to the payer that resources will be wasted on treatments that subsequently turn out to provide little or no incremental value, or are unsafe. This requires that manufacturers and other advocates generate relevant required information in anticipation of HTA assessment, which, in turn, requires that manufacturers know what data will be required.

In highly selected situations, several payers and HTA programs have implemented a form of “conditional reimbursement” or “coverage with evidence development” (Reference Eddy8;Reference Garber13;Reference Glasziou16;Reference Reiser33;Reference Steinberg, Tunis and Shapiro38;Reference Tunis and Pearson41) for new technologies with promising evidence of major potential clinical impact but which do not meet current evidentiary standards (e.g., limited population spectrum or longitudinal follow-up). “Conditional reimbursement” is formally linked to requirements for structured, defined data collection and specified further study, usually requiring enrollment in ongoing data collection in order for a service or product to be reimbursed. Examples include CMS's “coverage with evidence development” in the United States (Reference Tunis and Pearson41), similar programs by major U.S. health insurers (e.g., Aetna, United Health Care), the UK Multiple Sclerosis Risk Sharing Scheme, and recent initiatives in Australia and in the Netherlands for drugs and in Ontario (Canada) for devices and procedures (Reference Goeree and Levin17).

Principle 14: HTA Findings Need to Be Communicated Appropriately to Different Decision Makers

Given the multiple audiences for HTA findings, effective communication strategies need to be developed to meet the disparate needs of different users.

There are many potential audiences for HTA findings. In some settings, such as those with centralized, public, healthcare systems, the main audience may be relatively clear. For example, in the United Kingdom the main audience for NICE guidance is decision makers in the National Health Service, but even here there may be slightly differing perspectives between the central and local level, between different regions of the country and between the various professional groups.

In more decentralized healthcare systems, like those existing in the United States and several European countries, there are many decision makers, all with slightly different perspectives on the relevance of various benefits, risks, and costs. Therefore, whereas a broad HTA conducted along the lines set out above would consider most relevant items, analyses frequently are performed incorporating fairly narrow perspectives (e.g., payer, provider) and may exclude important outcomes (e.g., quality-of-life, convenience, adherence, cost, cost-effectiveness), thus limiting the value and generalizability of HTA findings to many key stakeholders and decision makers. For example, while one might argue that an estimate of the incremental cost-effectiveness ratio for a given technology, calculated by a government HTA agency might apply to a national healthcare system, the same will not be true for multiple payers in the United States or some other countries.

One approach for dealing with this issue is to build an interactive model, following the basic structure of the core HTA. Different decision makers then can enter the costs and benefits relevant to their setting, so as to produce a customized cost-effectiveness result. Whereas there are practical and methodological issues surrounding the use of such models, these have become increasingly popular in some settings (e.g., U.S. pharmaceutical benefit managers and formularies following the guidelines of the Academy of Managed Care Pharmacy [AMCP]). Regardless of the method used, information on which the assessment was based must be clearly and fully presented in a manner that is understandable to a wide range of audiences.

Patients and the general public also are major audiences for HTA findings. In recognition of this, several organizations involved in HTA activities issue nontechnical versions of their reports for the lay public (e.g., AHRQ, DAHTA@DIMDI, SBU, NICE). The goal of these efforts is to ensure that patients and the public understand the scientific evidence to guide them in their decision making and behavior. In other settings where patient co-pays are substantial, HTAs might play an important role by helping patients understand why co-payment levels vary among technologies and (possibly) among subgroups of patients, or may serve to reduce inappropriately high co-payment levels.

Principle 15: The Link Between HTA Findings and Decision-Making Processes Needs to Be Transparent and Clearly Defined

A clear distinction needs to be made between the HTA itself and the resulting decisions. The link between the assessment and the decision will be different in various settings, but in all cases it should be transparent.

As noted earlier, in many settings, HTAs are not tied to a particular decision about the diffusion or use of a health technology. However, given the focus in this study on the growing number of jurisdictions where they are, a clear distinction needs to be made between the assessment (i.e., the science) and the decision.

Several agencies involved in HTA been very careful to make this distinction, because while it is possible to be fairly prescriptive about what constitutes a good assessment (i.e., through the development of methods guidance), it is more difficult to be prescriptive about the elements of a good decision, because the latter inherently involves significant subjectivity. This distinction is likely to be even more important in jurisdictions like the United States, where there are multiple decision makers, each with different perspectives, contractual obligations and priorities. In such situations, it may be possible for the various parties to share a common assessment using state-of-the-art methods, but to weight elements differently and thus arrive at different decisions. (In Europe, efforts are currently underway to explore the potential for a “core” HTA process (11).

The first step toward clarity in these matters is to ensure that the results of any assessments used in the decision are published and widely distributed (see the discussion of transparency above). The next step involves transparency in the criteria used in reaching a decision. Although there are likely to be several criteria that influence a decision, most of the discussion surrounding HTAs involves whether there is some implicit, or explicit, threshold of risk–benefit or cost-effectiveness, beyond which the technology will not be approved for funding. Whether or not the threshold is explicit is a moot point—as long as there is reasonable consistency in the process, methods and application of assessment and decision criteria across decisions, it often is possible to infer the threshold by examining past decisions (Reference Devlin and Parkin5).

One of the few public discussions by decision makers of the use of decision thresholds is that by Rawlins and Culyer (Reference Rawlins and Culyer32) in the United Kingdom, who argue that NICE applies a threshold range in its recommendations: interventions with an incremental cost per quality-adjusted life-year (QALY) ratio of less than £20,000 have a high probability of being funded; those with a ratio exceeding £30,000 have a low probability of being funded. Rawlins and Culyer also discuss possible situations, as in allowing access to a new cancer drug, where the upper bound of £30,000 might be exceeded on grounds of equity.

For example, in the appraisal of imatinib for chronic myeloid leukemia, the incremental cost-effectiveness ratio for treatment in the chronic phase of the disease was £37,000 per QALY. This was deemed to be close enough to the Institute's upper bound on cost-effectiveness of £30,000 per QALY to warrant approval. However, The Appraisal Committee also approved treatment with imatinib in the blast crisis phase of the disease, even though the incremental ratio was £49,000 per QALY. The argument was that individuals at this advanced stage reasonably should have been identified and treated at an earlier stage of disease. The fact that they were not, could be because of a failing in the healthcare system. Therefore, on grounds of equity it was considered that they should have access to treatment at the later stage of the disease.

The main arguments for an explicit threshold are that it is transparent, may encourage more consistency in decision making and that a threshold will be inferred even if not explicitly stated. The main argument against an explicit threshold is that it is unlikely that a single threshold will apply in all situations, because other factors enter into decisions (e.g., seriousness of the patient's condition, availability of alternative treatments and overall affordability based on the total size of the patient population in a given setting) (Reference George, Harris and Mitchell15). Moreover, the incremental cost-effectiveness ratio tells us little about the true opportunity cost of adopting a given technology (Reference Birch and Gafni2).

A second problem is that an explicit threshold provides guidance to manufacturers on the maximum amount they can charge. However, even if there were evidence that manufacturers’ estimates of the incremental cost per QALY of their products were clustering around the threshold, it would not be clear whether this was the result of raising, or lowering price expectations.

As with several of the principles discussed earlier, this principle may be applied differently in the context of private payers. Presumably the parties in any price negotiation need to be aware of the criteria to be used in the decision. The negotiation also may be informed by the HTA's findings. Whether or not the implicit or explicit cost-effectiveness thresholds of different private payers need to be revealed is a matter for debate. Certainly, those choosing among health plans will be interested in knowing the availability of various health technologies in relation to their premiums and co-payments.