Health technology assessment (HTA) is an evaluative approach that assesses the impact on society of health technologies and “makes possible the acceptance, modification or rejection of technologies on a rational basis” (5). It does this by asking four questions: does the technology work, for whom, at what cost, and how does it compare with alternatives? (9). The impacts considered in a full HTA include the normal constituents of cost-effectiveness analysis—health benefits and resource use—but go wider to include social, legal, and ethical concerns, and the effects of using technologies on health care budgets and the national economy. The need for evaluative methods going beyond the assessment of clinical effectiveness was identified in the 1970s, as a gap emerged between the volume of available effective technologies and the ability and willingness of taxpayers and health insurers to fund them (6). A more comprehensive approach was needed to help health system decision-makers set priorities and to obtain maximum benefit from limited budgets, without abandoning the ethical and social values underpinning health care systems. Although the potential benefits of HTA were widely recognized, its use by decision-makers was limited (3).
The policies developed in Europe for controlling the pricing and reimbursement of technologies, were more restricted in their scope, and most frequently directed at pharmaceuticals rather than other technologies such as medical devices and surgical procedures. They considered only a limited range of factors, often concentrating on clinical effects and budget impact (e.g., France). Consequently, HTA was not integrated with other mechanisms for resource allocation in health systems.
The requirement to justify the reimbursement of pharmaceuticals by health systems has been labeled “the fourth hurdle”, as it is perceived by manufacturers to be an additional barrier to market access, after demonstration of product quality, efficacy, and safety to obtain a product license. Requirements for the licensing of other health technologies, such as devices and surgical procedures, differ from those for drugs, so the term “fourth hurdle” may not be strictly accurate in this context. However, it has become an internationally recognized descriptive term, understood by all stakeholders in the HTA field, so we shall apply it to all such systems in the remainder of the study.
The work reported here addresses the consequences of the particular use of HTA in health system decision making in Europe, Canada, and Australia. The focus of the paper is on the HTA decision-making systems of Europe, Canada, and Australia, and the use of analytical techniques, rather than on HTA methodologies. In particular, the work provides a basis for understanding the consequences of the under use or misuse of HTA in decision making. More generally, the study is concerned with establishing an analytical framework within which the different fourth hurdle systems in European countries, and their use of HTA, can be described and classified, providing a basis for future empirical and comparative research.
The following sections cover the background on fourth hurdle systems in Europe, Canada, and Australia; the objectives of the study; the methods used to develop the framework; the results of the study, including the framework itself and the influences on its evolution; and discussion and conclusions regarding the health policy implications and the feasibility and potential value of using the framework in further research.
BACKGROUND
Health technologies have always been appraised in terms of safety or effectiveness, and economic evaluation methods have been used in this area for many years (see, for example, Klarman et al.) (7). To facilitate the broader assessment needed for full HTA, methods were adapted from other areas of public policy, such as the environment, where wider impact assessments were more established.
The varying nature and emerging complexity of health technologies called for a formal technology appraisal method that can be applied across the board to all forms of technologies—drugs, medical devices, medical/surgical procedures, and also life-critical computer software and biologics—and whatever they do, for example, prevention, diagnosis, or treatment; and of course, to technologies that cross between these categories.
The first explicit use of an element of HTA in routine decision making regarding the reimbursement of health technologies was in Australia with the introduction of the PBAC system to evaluate pharmaceuticals in 1992 (1), followed by developments in Ontario (11), which were broadened to Canada (2). These systems addressed the cost-effectiveness of technologies, concentrating on the health benefits and the costs to the health care system, and provided guidance on the preparation of submissions. Although they allowed other relevant considerations in the decision-making process, for example, the broader social and economic concerns included in a full HTA, it was not clear how the systems took these factors into account.
Following these leads, many European countries produced guidelines on scientific standards for the economic evaluation of health technologies (4), but, although they often had the support of the government or health system, the institutions and decision-making processes in which they might have been used were not created. For example, the Netherlands produced a comprehensive set of economic evaluation guidelines in 1999, but the submission of cost-effectiveness data in support of drug reimbursement applications was not made mandatory until 2005 (12). The United Kingdom was the first European country to introduce an explicit HTA decision-making process by creating the National Institute for Clinical Excellence (NICE) in 1999. NICE procedures include consideration of wider social impacts (although attention is often focused on the cost-effectiveness data) and encompass devices and procedures as well as drugs.
Since then, many European countries have created formal decision structures in which HTA and economic evaluation can be used, mostly relating to drugs. Some (Sweden and Belgium) have followed the United Kingdom approach, going beyond clinical effectiveness and cost-effectiveness. Others (Portugal, Netherlands, Denmark, Norway, Spain, Hungary, and Poland) now mandate the use of pharmacoeconomic analysis. Italy and France continue to concentrate on budget impact analysis rather than full economic evaluation or HTA. In Germany, the first moves toward the more formal use of HTA methods in decision making are taking place with the creation of the new Institute for Health Care Quality (IQWiG; see Table 1). The differences between countries in reimbursement decision-making systems partly reflect differences in their political systems and the organization of their health sectors, which have influenced the political and health care objectives behind the introduction of HTA.
As a result of these developments, the manufacturers and sponsors of health technologies now face different decision-making procedures, with varying information requirements, in seeking to obtain reimbursement for their products from government funded health systems. It remains unclear whether decision making has been improved by these changes. In fact, the methods to determine this improvement have yet to be defined.
OBJECTIVES
The aim of the current research is to understand better the characteristics of existing fourth hurdle systems and how they endeavor to use HTA. Improved knowledge of their establishment and purpose, what decisions they make, what processes they use, and which interest groups are involved, may help to improve the operation of existing systems and provide guidance for future policy development.
The first objective is to develop an analytical framework to describe and classify fourth hurdle systems. By making this framework available to the stakeholders in fourth hurdle systems and researchers working in the field, we hope to move toward consensus on the important characteristics of such systems. Consensus will provide the basis for comparative, empirical work on fourth hurdle systems among countries.
In the future, we hope to use the framework in gathering information for such comparative analysis and encourage others to do the same. In analyzing the design, operation, and impact of fourth hurdle systems, best practice in particular decision contexts may be identified. This study concentrates on the first step—the creation of the analytical framework—which we hope will allow the analysis of system characteristics in a more systematic way than has hitherto been achieved.
METHODS
A modified Delphi process was used to develop the analytical framework. A systematic review was undertaken of the published health policy literature regarding the use of HTA and economic evaluation in reimbursement decision making. This approach was supplemented by a review of the official documentation and Web sites of the reimbursement decision-making systems in the countries of interest. From the information gathered, a first draft of the proposed framework of descriptors of fourth hurdle systems was produced, which included discussion of the issues and questions needing to be addressed to complete the descriptive framework
In February 2005, a meeting was organized, at which the comprehensiveness and relevance of the framework was assessed by a panel of international experts with expertise in HTA, health policy, and experience in designing or working within the fourth hurdle system within their own country. The comments of the experts were incorporated in a revised draft. This draft was then circulated to the expert panel and approved (with minor amendment).
The next stage was to assess the applicability of the framework to fourth hurdle systems and the feasibility of collecting the information needed to address the issues raised. A pilot study was undertaken in a subset of countries, using published documents and official Web sites as the only sources of information, and information gaps were identified. A system of present/partially present/absent was used to categorize the extent to which the information describing each country system was available and, hence, determine the extent to which a full picture of the country system could be obtained. The results of the pilot study were reported to the expert panel at a second meeting in June 2005. The experts commented on the applicability of the framework to their country and suggested practical ways of addressing the data-collection issues.
RESULTS
Development of the Draft Analytical Framework
The main information sources on fourth hurdle systems identified were the official Web sites and documents from the relevant government departments and agencies, and published articles and reports. The academic literature contains many reviews of economic evaluation guidelines (e.g., ISPOR Web site) but few discussions of fourth hurdle decision-making systems in their entirety.
A two-level framework was defined:
- Policy implementation level: the establishment of the fourth hurdle system as a policy decision of government, the policy objectives of the system, its legal status, and its relationships with the remainder of the health system, with other public sector bodies, and with other stakeholders, such as industry and patient groups.
- Individual technology decision level: the processes by which individual technologies are dealt with by the system, for example, assessment processes, how decisions are made, and how they are implemented.
Policy Implementation Level
At this level, we need to know the following: who set up the system; with what objective; from where does its budget come; whom does it advise; and who assesses its performance? The main issues are summarized in Table 2. Although special institutes and committees are often created to manage some parts of the system, the health ministry is usually the dominant force. The special institutions, such as NICE in England and Wales, receive their budgets from the ministry, which also reviews performance. This is not always the case. In social insurance–funded health systems, such as the Netherlands and Belgium, the fourth hurdle systems are driven by the insurance organizations, which are not public bodies. In Scotland, the Scottish Medicines Consortium (SMC) and its fourth hurdle system came from an initiative of the National Health Service (NHS) health boards, with responsibility for health care provision.
As one part of a broader health system decision-making process, the fourth hurdle system will reflect the overall policy preoccupations of the day. Recently, in Europe, these have been predominantly related to the control of health expenditures (with the notable exception of the United Kingdom). It would not be surprising, therefore, to find many fourth hurdle systems have cost-control high on their list of objectives. The fourth hurdle systems operate alongside a selection of related policies regarding the pricing and utilization of technologies. Appraisal is used in the Netherlands to determine the place of technologies inside or outside the reference pricing system. The legal establishment of the fourth hurdle system usually defines the type of technology it will appraise, and the majority of existing systems deal with all newly registered pharmaceuticals. NICE in England and Wales is an exception to this rule, covering all types of health technology; it only carries out appraisals of technologies identified by the Department of Health and the National Assembly for Wales.
The point of implementation of decisions is another key interface between the fourth hurdle system and the health system. Often, the implementation is within the control of the health ministry or insurance organization, relating to the level at which reimbursement is paid for a drug or other technology. With NICE, the decisions are expressed as a recommendation to NHS service providers on how a technology should be used. Implementation of such recommendations is more difficult to control, requiring collaboration between health care funders, providers, and the fourth hurdle system.
Other aspects of the legal establishment and governance of the system to be taken into account are any requirements to include representatives of stakeholder groups within its organization (e.g., industry, patient groups, general public, medical professional groups) and the existence of multiple levels of reimbursement decision making (e.g., national and regional), with the potential conflict which that might bring. Having identified these attributes, it is possible to conduct detailed analysis of the processes used in the appraisal of individual technologies.
Technology Decision Level
The documentary review identified three distinct stages in the operation of a fourth hurdle system: (i) the assessment of evidence, (ii) actual decision making, and (iii) outputs and their implementation. The documentation showed that systems varied in the configuration of institutions responsible for each stage and in the sophistication and transparency of the processes undertaken. In some systems, the assessment phase was merged with decision making; whereas in others, separate steps were undertaken by different groups.
To clarify these differences, the characteristics of each system and the institutions responsible for its operation at each stage must be identified. These sets were grouped in four areas: (i) constitution and governance, (ii) methods and processes, (iii) use of evidence, and (iv) accountability and transparency.
We used the three stages and the four underlying sets of characteristics to form a matrix for codifying available information on each country system (Table 3). The information needed to fill each cell was identified and formulated as a set of questions. The main questions, for each of which there are many subsidiaries, are discussed below. The full list of questions, referenced to the relevant cell in the matrix in Table 3, is available from the authors on request.
Assessment
The establishment of the assessment process determines the rights of different stakeholders to take part in the process. Can they submit evidence or merely comment on the analysis of others? Can they influence the timing and the method of assessment or is this rigidly determined by the legislative foundation of the system? Another important consideration is whether technologies are assessed individually or whether those for the treatment of similar groups of patients are assessed together.
The assessment links the roles of the stakeholders with the methodology of assessment and the way in which evidence is sourced and used. In most systems, the whole appraisal process, from topic selection to decision and implementation, is overseen by the health ministry, but in many countries, separate institutions have been set up to manage the assessment. These bodies are likely to contract out the review and analysis of evidence to independent organizations such as university research institutes. In Sweden, and other countries, only the evidence submitted by the sponsors of a technology is considered. In England and Wales, NICE encourages independent review groups to carry out their own new analysis using evidence beyond that contained in company submissions.
Systems vary in prescriptiveness regarding the type and quality of evidence to be considered and the analytical framework in which it is to be presented. Australia requires effectiveness data from clinical trials and the use of a standard set of resource costs in cost-effectiveness analyses. In England and Wales, NICE requests the “best available evidence,” meaning that the evidential standard on which assessments are based can vary between technologies.
How transparent is the communication of the results of the assessment that are communicated to the decision-makers? Is the full analysis provided or just a summary of outcomes, for example, a measure such as “cost per QALY” (quality-adjusted life year)? Are conflicting sources of evidence reported? How are qualitative impacts, for example, social and economic, reported? Are the views of all stakeholders communicated separately?
Decision Process
Where does responsibility for the final decision lie? Is there a separate institute or committee established to consider the evidence presented in the assessment? If so, how are its members selected; what are their technical qualifications? Do the decisions reached by the fourth hurdle system have to be approved by the health ministry before implementation? The answers to these questions will reveal a great deal about how the system functions. Another important influence is whether the fourth hurdle system has any responsibility for ensuring implementation of its decisions by the health service?
The processes used by the group making the actual decision are of great interest. Do they meet in public? Do they receive oral as well as written evidence? Is a vote taken when there are disagreements?
A distinction is made here between the collection and analysis of evidence in the assessment phase and the weighing of the relative importance of different pieces of evidence in the actual decision process. Does the decision group rely on the conclusions of an assessment report produced earlier or does it exercise independent judgment? What other information is brought to bear on the decision, apart from that in the assessment report, and how is it presented? Do stakeholders have the opportunity to clarify their submissions when they see how the submissions are being used?
The accountability of the group will depend on their independence from ministerial control. If their decisions have independent force, as with NICE, then they will be politically accountable to the health ministry and legally accountable to society at large for the fairness of their decisions. How decisions are communicated will have an important influence on their acceptance and impact on health care delivery. For example, is an explanation or commentary provided to show the main reasons for a particular decision being reached?
Outputs and Implementation
Once the decision is reached, we need to know whether the system has its own formal appeals procedure. If so, who can appeal and who judges the validity of the appeal? The transparency of both grounds for appeal and justification of appeal judgments is important. The legal status of any recommendations—mandatory or advisory—will have an important bearing on approaches to implementation. An administrative decision as to whether a drug should be reimbursed or not, is easier to implement than a recommendation that a technology should only be used in a limited patient group.
Health care managers and the health ministry have the direct responsibility of implementing decisions, but the institutions of the fourth hurdle system may also play a role, with, for example, technology appraisals informing established programs of clinical guideline development. Other characteristics of the health system may impede the implementation of decisions taken at the national level, for example, devolved management and silo budgeting. Are there processes for measuring the impact of decisions by the fourth hurdle system? In systems where the impact of decisions is monitored, does this monitoring lead to further action? Is there a process for reviewing the continued relevance of past decisions and recommendations?
Results of Expert Consultation on the Framework
The experts agreed that the descriptors were sufficiently comprehensive to cover all aspects of fourth hurdle systems. However, they requested that the boundaries of a given system under analysis be defined, that it is placed in the context of the health system, and that whole decision-making systems be described, not just given institutions or parts of the systems. This request led to modification of the framework to separate the system level characteristics from the more detailed description of the processes used to appraise technologies. The experts suggested a series of minor modifications to some subsidiary questions. The framework in Table 3 includes the panel's main comments.
Pilot Study on Information Collection
Using the detailed list of questions and issues as a guide, we attempted to complete the study framework, at the technology decision level, for France, Germany, England and Wales, Sweden, Hungary, Portugal, Australia, and Ontario. The three-point classification [present/partially present/absent] was converted to a simple quantitative measure by applying scores of 100 percent, 50 percent, and 0 percent, respectively. The average percentage of information available for each cell was then calculated. (Percentages were used because the number of relevant pieces of information differed between cells.) See Table 4 for the aggregated results.
Overall, publicly available sources provided less than half of the necessary information. Within the stages, the “decision” was the best documented, particularly with regard to who was responsible for making it and the processes used. Accountability and transparency were the worst documented. For example, no system provided any information on the measurement of the impact of decisions. England and Wales (64 percent) and Australia (61 percent) were the best documented and Germany (29 percent) and France (29 percent) the worst. (Data not shown in table.)
Expert Consultation on Feasibility of Data Collection
The experts suggested adding a protocol to the descriptors template, and possibly a new data collection tool, to guide and support the process of collecting information on country systems. The descriptors are a framework for analysis and should not be used directly for data collection purposes. Without this approach, the pilot study may have underestimated the availability of information. They also expressed the view that, in some countries, some of the questions could only be answered by interviewing those directly involved in the system.
The experts also highlighted a series of data collection challenges: (i) Describing systems undergoing reform (e.g., Germany); (ii) Discrepancies between different official documents and discrepancy between public information and real practice, as a consequence, objective description without judgment may not be possible; and (iii) simultaneous existence of multiple fourth hurdle systems at different levels within a single health care system (e.g., national and regional in Sweden and Spain), and additional nongovernmental fourth hurdles (hospital formularies, in integrated care and physician networks)—and potential conflicts between these layers.
Expert Consultation on Applicability to Country Systems
Although generally supportive of the use of the framework and aware of its potential value, some concerns were expressed about the ability of the framework to deal satisfactorily with issues, such as implementation, which might involve large parts of the health system beyond those bodies dealing directly with the fourth hurdle system. To help overcome this problem, the experts suggested starting all applications of the framework with a map of the fourth hurdle system, its specific institutions and their interactions with the rest of the health system.
Because of the difficulty in obtaining definitive documented information on many of the important issues raised by the use of the framework, the experts believed that it would be best used by those with a background in HTA and health policy, able to interpret ambiguities and discrepancies in official information sources. This reinforced the view that, although the framework was a major step forward, it probably would never be possible to fill it out completely for a system without exercising judgment on some issues. This finding was the case particularly for elements of the policy implementation level, such as the objectives.
DISCUSSION
Any analytical framework attempting to encompass the many aspects of fourth hurdle systems must be very broad to deal with the legal and political characteristics at the system level and the detailed nuances of varying assessment and decision-making procedures at the decisional level. Nonetheless, attempting to construct one is worthwhile as it clarifies our understanding of how present systems operate. Despite the steady growth of HTA activities over the past 20 years, there is still widespread dissatisfaction with the impact of HTA on health system decision making. Many authors have found that the influence of the results of HTA on policy decisions has been slight, partly because it has not been an integral part of routine decision processes (10). On the other hand, it has been argued that the need to supply varying forms of evidence to decision-makers is an unnecessary burden on the manufacturers of technologies, especially as the utilization of the data is uncertain (9).
The outcome of our expert consultation was agreement that the framework is a useful basis for understanding the nature of existing fourth hurdle systems and that a complete set of information on the systems in Europe, Canada, and Australia would be a valuable resource for researchers and policy-makers. Such a complete data set is unlikely to be available from publicly accessible sources in all countries, but the documentary sources can be supplemented from interviews with experts and decision-makers. This strategy would introduce an element of subjective judgment, which would have to be taken into account if the data were used for comparative analysis.
POLICY IMPLICATIONS
The performance of regulatory bodies is a key concern of governments, industry, and the public (as taxpayers and service users). It is hoped that the dissemination and utilization of this framework will contribute to greater accountability and transparency in this area of health care regulation. It provides a basis for more formal assessment of the performance of fourth hurdle systems, against their declared objectives, and by international benchmarking. Comparisons between countries using the information would enable stakeholders to understand the workings of their own country systems and judge the value of importing ideas from elsewhere. This comparison would provide a firmer evidence base for countries designing new fourth hurdle systems, enabling them to identify the key factors for a successful system in their own health care decision context.
NEXT STEPS
The subsidiary questions schedule is being converted into a data collection instrument to improve the efficiency of the collection of the country-specific information and minimize the need for subjective input to complete the framework. As experience is gained with the application of the framework, more confidence may then be placed in subsequent comparative studies using the information. This work will also be used to identify relationships between the legal constitution of systems and the detailed processes that they develop at the decision-making level. This will help in deciding whether general recommendations can be formulated on the use of HTA in this context. For example, is there an ideal model of the use of full HTA in reimbursement decision making, or will the best system always be determined by the health policy and health system context?
CONTACT INFORMATION
John Hutton, BPhil (john.hutton@unitedbiosource.com), Visiting Professor, Department of Health Sciences, University of York, Heslington, York YO10 5DD, UK; Senior Research Leader, European Office, The MEDTAP Institute at UBC, 20 Bloomsbury Square, London WC1A 2NS, UK
Clare McGrath, BSc (clare.mcgrath@pfizer.com), Area Director, Africa Middle East, Jean-Marc Frybourg, PhD (jean-marc.frybourg@pfizer.com), Senior Manager, Europe, Africa Middle East, Department of Outcomes Research, Pfizer Ltd.,Walton Oaks, Dorking Road, Tadworth, Surrey KT20 7NS, UK
Mike Tremblay, PhD (mike@tremblay-consulting.biz), Principal, Tremblay Consulting International, 454 Mississauga Street, PO Box 8000, Suite 145, Niagara-on-the-Lake, Ontario L0S 1J0, Canada
Edward Bramley-Harker, MA (edward.bramley.harker@nera.com), Associate Director, NERA Economic Consulting, 15 Stratford Place, London, W1C 1BE, UK
Christopher Henshall, PhD (ch523@york.ac.uk), Honorary Professor, Department of Public Health and Policy, The London School of Hygeine and Tropical Medicine, Keppel Street, London WC1 7HT, UK; Pro Vice Chancellor for External Relations, Enterprise and Innovation Office, University of York, Innovation Centre, York Science Park, York, North Yorkshire YO10 5DG, UK
We acknowledge the important contribution of the panel of international experts: Pf. L. Annemans, Ghent University, Belgium; Pf. F. Antoñanzas Villar, University. of la Rioja, Spain; Pf. V. Atella, University of Rome Tor Vergata, Italy; Dr. J. Bridges, University of Heidelberg, Germany; Pf. M. J. Buxton, Brunel University, UK; Pf. Y.-A. Flori, University Victor Segalen Bordeaux 2, France; Pf. W. Greiner, University of Hannover, Germany; Dr. M. Koopmanschap, Erasmus University, Netherlands; Dr. A. Kozierkiewicz, Institute of Public Health, Krakow, Poland; Dr. M. Levine, McMaster University, Canada; Dr B. orland, Norwegian Knowledge Centre for the Health Services, Norway; Dr. U. Persson, Lund University, Sweden; Pf. Dr. O. Schöffski, University of Erlangen-Nuremberg, Germany; A. Towse, Office of Health Economics, UK.
This work was funded by a grant from Pfizer Ltd.
