Decision makers in health care all share the same concern: they do not want to be taken by surprise by unexpected consequences of the introduction of a new health technology. One such example is political embarrassment, when decision makers are confronted with patient lobbies to gain access to a new pharmaceutical without previous notice that this drug was on the brink of being introduced. Another example is a pending financial crisis due to the costs of a newly introduced technology that cannot be dealt with in the current regional or institutional healthcare budget. Seen from a wider perspective, decision makers also want to protect and promote public health but often lack information on what potentially might do more harm than good. Worldwide, there are pressures to speed up decision making on new health technology. In the United States for example, the Food and Drug Administration (FDA) has established a fast-track procedure (14). In Europe, the European Agency for the Evaluation of Medicinal Products (EMEA) is under constant pressure to reduce the duration of the approval process (10). On the one hand, these faster procedures can be considered as an important step toward a new way of protecting public health by ensuring that beneficial drugs are made available as quickly as possible, but it might also be a dangerous shortcut that might jeopardize consumers' health, allowing unsafe and ineffective drugs to be marketed and prescribed (1;4). For healthcare decision makers deciding on purchasing health technologies and being pressurized by patient organizations and other stakeholders, it can be difficult to tell the two apart. The above-mentioned concerns and developments illustrate that decision making on the uptake and use of a health technology needs to be supported by timely and high quality information provision. Whereas the developments at the FDA and EMEA predominantly state a case for timely information on pharmaceuticals, it is perhaps even more important in relation to nonpharmaceutical advances (devices, diagnostic tests, and so on), as there is no regulatory system in place that enables systematic early identification and assessment of these technologies, and because their development may be rapid in comparison to new drugs (22). An important tool for information support to decision makers is health technology assessment (HTA). HTA is a form of policy research that studies the short- and long-term consequences of the introduction of health technologies in a healthcare system, in terms of safety, clinical benefit, costs, effects on organization of health services, and legal, social, and ethical consequences (15). Ninety-five percent of HTA reports build on literature reviews (9). In the early years, HTAs could well take 3 to 4 years from start to publication, a time period considered too long by some, in that there is a risk that policy makers will not have the time to wait for assessment reports to support their decision-making process (5). As HTA as an activity became more common practice around the world, it was increasingly recognized that timeliness of the assessments was key in the support of healthcare decision makers (18).
Banta and Gelijns (2) were the first to conclude that it is not satisfactory to react to technological developments only when confronted with their consequences. Their study for the Dutch government in the 1980s called for a systematic approach to identification and early assessment of new health technologies to provide early notice to decision makers in health care. They called this an Early Warning System, which was subsequently established at the Dutch Health Council (3). The Swedish HTA agency (SBU) also reacted to the problem of time delay expressed by healthcare decision makers by establishing SBU ALERT in 1997, the Swedish Early Warning System (5). The more common term used nowadays is “horizon scanning systems” (HSSs), because “the horizon of the healthcare system” is scanned for future drugs, medical devices, and medical and surgical procedures. In 1999, several HSSs established EuroScan, an information network on new and changing health technologies (11). The developments that led to EuroScan have been reported in a special issue in this journal (16). The network currently consists of thirteen representatives located in twelve countries, which undertake substantial early identification and assessment activities on behalf of their regional or national governments. These countries include Canada, Denmark, Norway, Sweden, Australia, and New Zealand (collaborating in one center), The Netherlands, The United Kingdom, Israel, Spain, France, and Switzerland. Membership is open to any agency that has a substantial program for the early identification and assessment of emerging, new or changing health technologies, and which has an ongoing, officially recognized role in relation to regional or national government, and is a nonprofit organization with at least 50 percent funding from public sources (11).
Most of these horizon scanning systems evolved from the work of HTA agencies, the main difference between the two being that HSSs focus on technologies early in the life cycle. EuroScan defines the focus of horizon scanning as those technologies that are not yet adopted by the healthcare system and those that are in the phase of adoption (12). In general, this comes down to a time horizon of 0–5 years before introduction.
Table 1 shows that the majority of HSSs is part of a national or regional HTA agency. Other HSSs are located at a university, a federal social insurance office, and a hospital group. Country-specific jurisdictions define how horizon-scanning information feeds into the decision-making process. Some HSSs have a well-defined audience. In the United Kingdom for example, the National Horizon Scanning Centre provides information to the Department of Health for England and Wales (19). In France, the Committee for Evaluation & Diffusion of Innovative Technologies informs an association of public hospitals in the Paris area (6). Apart from EuroScan and its members, several other organizations have established Horizon Scanning Systems, for example, Hayes, Inc., and ECRI, both in the United States.
Since the 1998 special issue in this journal, limited publications have appeared on horizon scanning and the value of systems has been described mainly by means of surveys among (potential) customers. The study by Simpson et al. (24) is one of the first to evaluate the accuracy of a system's selection of significant technologies. This aspect is one of the key determinants of the value of HSSs for decision making. However, this study does not provide details on how significant technologies were selected, which is essential to understand how the activities of HSSs can be of value for decision making. We studied how the members of EuroScan select health technologies for assessment and discuss how the findings relate to the aim of informing decision makers on the most significant health technologies they may be confronted with.
METHODS
Data collection was organized in two stages. First, an e-mail was sent to all thirteen contact persons of the member organizations of EuroScan. The e-mail contained general information on the purpose of the study, a short questionnaire with predefined answering options, and a request for a face-to-face interview. Interviews were planned in relation to a meeting of EuroScan and a subsequent HTAi Conference in Krakov (Poland) in the summer of 2004. The themes that were selected for the questionnaire and interview were derived from the author's previous work in the area of priority setting (7;20;21) and included how priorities are set, the use of criteria in the process, the role of individual criteria, and who is involved in the process. The e-mail questionnaire was intended to provide the research team with baseline information and as a source for preparing individual-tailored semistructured interviews. A reminder was sent after 2 weeks. The contact persons were all in charge of the respective HSSs and, therefore, knowledgeable on the operation of the HSSs, including the issue of priority setting. All interviews were conducted by the first author (K.D.) and audio taped.
Data were categorized according to the themes included in the questionnaire and the interviews. The second author (H.V.) went through all the data to check for a correct interpretation and filling in of the categories. Descriptive statistics were used to analyze the data. The results are presented anonymously and on an aggregate level. A draft of a manuscript including all results was sent to all participants for comments, preceding a presentation of the main finding by the first author (K.D.) at a formal meeting of the network in autumn of 2004.
RESULTS
The response rate on the questionnaire was 85 percent (n = 11). The participation rate in the interviews was 92 percent (n = 12). Nine of the interviews were face-to-face; three telephone interviews were organized. In the case of the single contact person who was not interviewed, additional data on the HSS was provided by means of printed documents and e-mail exchange. When interviewing the EuroScan members, it turned out that one member organization did not prioritize health technologies and was for that reason excluded from further analysis. Another organization prioritized health technologies after they had been introduced in the healthcare system. The system reported to make a full HTA, and then prioritized technologies for inclusion in the benefit package. This activity is described in the literature as priority setting in health care, which is distinctively different from prioritizing technologies for early assessment. This HSS has indicated that development of a system for identification of new and emerging health technologies is under way, but the methods are not yet in place. We, therefore, report findings of eleven HSSs.
Customers and the Need for Priority Setting
Seven of the eleven HSSs (64 percent) have customers for their assessments, that is, there is a formal arrangement with a decision maker to make use of their output. In the majority of these cases, the customer is a regional or national Department of Health (see Table 1). In four of the seven HSSs serving a customer, a varying share of the total number of assessments is carried out on request of the customer. Because a part of the assessments are predefined, the systems stated to perceive only a moderate need for priority setting. The remaining three systems serving a customer are fully independent in priority setting. They have formal arrangements with a decision maker to deliver assessments, but the customer does not put in requests. These systems experience the need for priority setting as either high or moderate, depending on resources available. Four HSSs (36 percent) do not have a customer but inform a wide variety of decision makers, without explicitly being asked to do so. These systems are all established at the national level. In this group of systems, a moderate need for priority setting was explained by the fact that there is not a lot of pressure/demand from outside.
Predictive Value of the System
A theme that emerged from the interviews was that all systems share the same concerns, that is, to miss an important technology and to select an unimportant technology. This concern relates to making a type I and type II error in predicting (selecting) which technologies will be significant to assess and refers to the proportion of technologies that have been wrongly sorted out as insignificant (false negatives) or wrongly selected as significant (false positives).
Filtering
Seven systems (64 percent) define a step before the actual selection of technologies. This step is called filtering and is aimed at discarding trivial technologies. Filtering happens on the basis of insight and knowledge on technological change. It is a rather implicit process, described by one of the interviewees as “mainly taking place in someone's head.” Two systems use individual experts or groups of clinical experts for this purpose. These systems use a form with predefined questions to consult experts on the novelty of the technology for the healthcare system, the time horizon of introduction of the new technology in the healthcare system, and the likely impact of the technology on the healthcare system.
Priority Setting Process
Two systems mentioned to have attempted using formal priority-setting methods, that is, a method in which criteria are scored and weighted, with a final ranking achieved on the basis of decision rules. However, both systems stopped using these formal procedures. In one case, the instrument did not fit to the conditions of an HSS, in that a lengthy procedure, including Delphi panels, was used. In the other case, it was judged that the experts involved would need more training and time to get used to score and weight criteria, which is planned for the future. The system's instrument, therefore, currently is used only as a checklist.
Priority Setting Criteria
EuroScan has developed explicit priority-setting criteria, aimed at serving as an example for the individual members (13). Eight HSSs (73 percent) report to use explicit criteria. Explicit criteria at the level of individual HSS were defined as criteria documented in internal documents of the HSS. Two HSSs apply a combination of explicit and implicit criteria, whereas one HSS reported to use implicit criteria. Of those HSS using explicit criteria, one exclusively uses the EuroScan criteria, but all other systems have developed their own specific set, with individual EuroScan criteria usually well-represented (see Table 2). The only exception is the criterion on cost-effectiveness, which is used by a minority of EuroScan members.
The number of criteria used to select technologies ranges between three and thirteen. The median number of priority-setting criteria mentioned by the HSSs is six, that is, half of the HSSs used six criteria or less.
Table 2 shows that most HSSs use costs and health benefits at the population level; organizational consequences of the technology; rate of diffusion of the technology; ethical, legal, and social aspects; and number of patients involved. Eight criteria, not included in the table, were mentioned only once by eight different HSS. Examples include the presence of prompts from one or more highly valued external agencies and the involvement of strategic stakes of customers.
None of the HSSs mentions weighting criteria explicitly, but when asked if any of the criteria was more important than others, seven systems (64 percent) indicated that they would prioritize a technology with a high health benefit even when this would apply to a small number of patients. In contrast, two systems indicated that the number of patients probably was the most important criterion. One system documented a long list of criteria, but clarified that a technology could be a priority with very few criteria applying to it. For example, technologies likely to be very expensive, more than 100,000 US$ per patient per year, would be prioritized right away. Yet another system indicated that the importance of a criterion depends on the technology being considered.
Few HSSs have operationalized individual criteria beyond expressions such as “large,” “major,” “obvious,” and “far-reaching.” Two systems have operationalized “number of patients” (relevant = over 1,000), and one operationalized “costs per patient” (high = over 10,000 euro, low = <500 euro), and “cost impact” (high = over 10 million euro and low = <1 million euro).
Actual Selection of Technologies
The final decision on which technologies to assess is made either by the HSS or the HTA agency that hosts the HSS (45 percent; n = 5), or it is delegated to experts (55 percent; n = 6). In all systems, the decision on the selection of technologies is based on agreement by consensus. All systems indicated that they did not experience any difficulties in arriving at a consensus. The role of the chairman is mentioned by one system to be crucial to arrive at a consensus. In the majority of cases, it is unclear if criteria are applied systematically in this process and no checks are reported for a consistent use of criteria. Only one system indicates that the list of criteria is systematically addressed when discussing the technologies. In all the other systems, the process is described as subjective. Subjective in this respect suggests that it is not clear if all criteria are taken into account and if the same process is followed for each technology. The following statements exemplify this. One respondent states that “criteria are considered as explicit, but the process of deciding on a technology is subjective.” Another calls the application of criteria “an on the-back-of-our-mind exercise.” The decision-making process by experts in committees is described by one respondent as follows: “At the meeting, hearing all the arguments, they will change their mind. The committee members may also change their mind while reading the comments of others on a technology. This shows how subjective it is.” And finally, a respondent concludes: “You wouldn't want to stand up in court and argue why you prioritized something and not something else.” Furthermore, it became apparent from the interviews that the priority-setting process in the advisory committees can be such that one member highlights one aspect of a technology, for example its effectiveness, and this is then taken on by the rest of the committee without discussing other aspects.
The process of deciding on technologies for assessment is undocumented in all but one system. This system documents afterward, for reasons of accountability, which criteria were taken into account and to what extent these have been discussed in a priority-setting meeting.
Use of Clinical Experts in Priority Setting
Experts are used either for filtering or for making the final decision on which health technologies to assess. Some of the HSSs have established links with medical associations, and experts are then contacted through these. One system works together with a nurses association. The background of experts used in committees is diverse. In one system the committee is solely made up of policy makers. In the other HSSs, the experts are either clinicians from relevant clinical specialties or a combination of clinicians, senior policy makers, and other healthcare stakeholders (e.g., representatives of industry, pharmacists).
HSSs and the Link with Full HTA
Five systems' identified that technologies or assessments are used to decide whether a technology is in need of a full HTA. As such, these systems form the first stage in Health Technology Assessment. In one case, the HSS's list of identified technologies feeds directly into a priority-setting process that defines if a technology requires an early assessment or a full HTA. In the other systems, the early assessment that is made of the technology is used to decide whether a full HTA is warranted or development of a (clinical) guideline, or whether clinical research is needed to obtain more information.
CONCLUSION AND DISCUSSION
In an ideal world, the capacity of HSSs to assess new health technologies would match the total number of emerging and new technologies, and it is no surprise, therefore, that some of the HSSs involved in this study expressed the wish for increased resources for assessment, in principle to the extent that priority setting would not be needed at all. Unfortunately, however, the reality of virtually all HSSs dictates otherwise and consequently almost all HSSs have selection procedures in place. This study has documented differences in the nature and number of criteria applied in the priority-setting process. These differences may be interpreted as reflecting differences in values, cultures, and healthcare priorities in different countries. For example, in some countries, cost containment may be the highest priority, whereas in other countries, faster access to modern treatment may be an important policy goal as well. However, given the emphasis on, for example, issues of value for money related to the introduction of new health technologies, it is a surprising finding that this criterion is only used by a few HSSs. Perhaps the high degree of uncertainty surrounding cost-effectiveness estimates of health technology early in its life-cycle precludes systematic application of this criterion.
There are differences between HSSs not only in criteria applied but also with regard to the actors involved in the final decision of which technologies to assess. Notwithstanding these differences, the overall process shows striking similarities. In all systems, the decision-making process on which technologies to assess resembles what Dowie has called TIABIM (“Taking Into Account, and Bearing In Mind”) (8). We conclude that, although decisions are made, the process is not transparent; therefore, it is unclear which criteria in reality influence the final selection of technologies and, if so, to what extent each criterion contributes to a particular decision. The major limitation of our study is that we have not fully opened this black box, for example, by attending priority-setting meetings of HS staff or expert committees. Observational studies, such as those carried out by Singer et al. (23) and Martin et al. (17) would be useful for this purpose. Given these limitations, the high response rate in the surveys and interviews, combined with relevant background data, leads us to believe that the findings are representative for EuroScan members.
In our view, the TIABIM way of making decisions referred to above, reduces the potential value of HSSs for decision making, if the major objective of HSSs is defined as to provide decision makers with information on the potentially most significant new and emerging technologies. We would like to argue that, with current priority-setting procedures in HSSs, that is, one in which criteria are not operationalized and in which criteria are not systematically and consistently applied, decision makers cannot be ensured that HSSs provide them with the information they may be most in need of. This statement coincides with one of the major concerns of all HSSs included in the study, that is, to miss significant technologies. Missing a relevant technology suggests not providing decision makers with information they need. It may be dependent on the context of an individual HSS, however, how big this latter concern needs to be. As our study showed, there are systems with active customers (putting in requests), passive customers, and no customers. Those with active customers do not have much reason to be concerned about missing anything significant, as they assess technologies that decision makers want to have assessed. Although these technologies need not be the most significant technologies, this is not the HSSs' concern. The HSSs in our study that are fully independent in priority setting, that is, do not have to take requests into consideration, have reason to be concerned about missing significant health technologies. Those having the most reason to be concerned are the HSSs with a customer, as the customer expects to be informed on technologies that prove to be significant for the healthcare system. The HSSs that do not have a customer might share the same concern, but there is no specific target group or organization that can legitimately criticize the choices made.
We hypothesize, therefore, that different groups of HSSs may constitute different “risk-groups” of missing relevant technologies, and different “risk-groups” when it comes to being criticized for missing a relevant technology. Notwithstanding these hypothesized differences, it seems obvious that all systems' selection procedures will benefit from explicitly operationalizing criteria, applying these criteria consistently, and securing more transparent procedures when selecting the most important technologies for assessment.
The second concern of HSSs, to select irrelevant technologies, is related to wasting scarce analytical resources. To avoid this waste, the systems' filtering processes could be improved. We recommend in particular to further explore the role of experts in this process, as this approach is virtually uncharted territory.
As HSSs are unlikely ever to be completely accurate (24), we conclude with reiterating the recommendation originally formulated by Banta and Gelijns (3), that HSSs should always act as a first stage of assessment, with full integration in HTA or other types of health technology research programs. This strategy allows mistakes to be corrected and exploits the full potential of horizon scanning.
CONTACT INFORMATION
Karla Douw, MSc (kdo@sam.sdu.dk), Hindrik Vondeling, PhD (hvo@sam.sdu.dk), Department of Health Economics, University of Southern Denmark, Winsloewsvej 9b, DK-5000 Odense, Denmark
