Early awareness and alert systems (EAAS) have been created with the aim of identifying and providing timely information on potentially relevant technologies that are at the stage of being introduced into clinical practice or that have not yet been adopted but are in the implementation stage (Reference Simpson, Packer and Carlsson1). Possessing preliminary information on effectiveness, clinical utility, and costs is considered essential for planning and organizing health services properly, and for disseminating quality information to managers and health professionals on technologies which could prove significant for the healthcare system. Having timely, high-quality information on the possible consequences of a new technology is recognized as being fundamental for improving the quality of health care and could serve the purpose of rationalizing the use of technologies that are potentially ineffective or even cause undesirable effects.
Dating from the creation of the International Information Network on New and Emerging Health Technologies (EuroScan) in 1999 (2), this activity has experienced a notable boom worldwide (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3). Here in Spain, four health technology assessment agencies conduct early awareness and alert activities, namely: AETS-ISCIII, avalia-t, AETSA and OSTEBA (Table 1). Since 2006, these agencies have constituted the Spanish Network for Early Identification, Prioritization and Assessment of Emerging Health Technologies (GENTECS), set up for providing the Spanish National Health System with high quality information on nonpharmaceutical technologies expected to have a high impact on the health care system (Reference Benguria Arrate, Gutiérrez-Ibarluzea and Llanos4).
Information on National and International Early Awareness Alert Systems
The GENTECS collaboration network relies on groups of experts, voluntary notifications, high-impact medical reviews (BMJ, Lancet, JAMA), specific databases, web alerts and contact with manufacturers for identification of technologies (Reference Benguria Arrate, Gutiérrez-Ibarluzea and Llanos4). Although these information sources have been recommended and used since 1998 by different groups of EAA experts (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3;Reference Robert, Gabbay and Stevens5;Reference Smith, Cook and Packer6), it has recently been suggested that they may not be sufficiently exhaustive to cover the entire existing spectrum or identify significant technologies far enough in advance to allow for early assessment (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3;Reference Packer, Fung and Stevens7). Similarly, some papers have highlighted the fact that technologies which are prioritized and assessed by EAAS do not always coincide with those demanded in clinical practice. Hence, when the data systems of NICE were analyzed from 1998 to 2010, the NIHR Horizon Scanning Centre was shown to have identified 92.5 percent of all technologies appraised by NICE, yet only 40 percent of technologies identified were prioritized for assessment, a false positive rate of 60 percent (Reference Packer, Fung and Stevens7). In Australia, O’Malley et al. (Reference O’Malley and Jordan8) analyzed how many of the assessment reports requested across the period 2003–08 as a prerequisite for the funding of medical technologies in the Australian public health system had been the subject of a report on new and emerging technologies by ANZHSN. They found that only 26 percent met this criterion (11/43). These authors attributed these findings to the nonexhaustive nature of the data sources and/or to the lack of specific methods for the selection of the most relevant technologies (Reference Packer, Fung and Stevens7;Reference O’Malley and Jordan8). It has been noted that there are several errors in the data presented by O’Malley et al. and that some of the technologies might not have been assessed by the Australian EAAS because they were already mature when the ANZHSN was founded (Reference Mundy, Hiller and Merlin9).
A survey conducted by Douw and Vondeling (Reference Douw and Vondeling10) suggested that new methods could be used to enhance this process and make it more explicit and systematic. They noted that two EAAS sought to objectify the selection process by using formal priority methods, whereby the criteria were weighted and a final ranking was achieved on the basis of decision rules, but they stopped using these methods due to the great effort entailed. Gallego et al. (Reference Gallego, Bridges, Flynn, Blauvelt and Niessen11) demonstrated the value of best–worst scaling in exploring clinicians’ preferences regarding emerging technologies that would impact on outcomes in hepatocellular carcinoma, yet the fact that it was not clear whether all the important criteria for prioritization had been taken into account was seen as a limitation.
The current document, drawn up within the new collaboration framework of the Red Española de Agencias de Evaluación de Tecnologías y Prestaciones del SNS (Spanish Network of Agencies for Health Technology & Service Assessment), seeks to contribute to the development of an effective national strategy for the early identification, selection and assessment of nonpharmaceutical technologies envisaged as having a high clinical impact. This study explores the use of a simplified method for filtering and prioritizing new and emerging technologies, which is based on clinicians’ experience and uses explicit considerations for ranking priorities. The study presents a prioritized list of new and emerging technologies identified by a previously piloted and validated Medline search.
METHODS
Identification and Filtering
Identification was carried out in Medline from January to June 2012 by applying a search strategy that had been specifically designed to locate new and emerging nonpharmaceutical technologies, which had been previously piloted and validated with respect to general and specialized high-impact journals (Reference Varela-Lema, Punal-Riobóo, Casal Acción, Ruano-Ravina and Garcia12) (Table 2). A potentially new and emerging technology was defined as any technology that had first been reported in 2008 or later. As described in the previous study (Reference Varela-Lema, Punal-Riobóo, Casal Acción, Ruano-Ravina and Garcia12), the technologies were preselected by two highly experienced HTA experts (minimum of 5 years working experience in horizon scanning activities) based on what was published in the abstracts and on information retrieved by means of the Internet (full-text papers and grey literature). Technologies were considered potentially relevant where they were perceived to represent a substantially novel contribution to prevention, diagnosis or treatment which might have a significant impact on patients’ functional capacity, quality of life or clinical outcomes. Discrepancies were resolved by discussion. After the review, these technologies were grouped by medical specialty and mailed to a panel of experts belonging to the different clinical departments or units responsible for their application. Some technologies were grouped into more than one specialty.
Search Strategy for Identification of New and Emerging Technologies in Medline
Participants
For the panel of experts, we recruited all health care professionals involved in the Galician New and Emerging Technology Detection Network (detecta-t; n = 102) and all specialists included in the list of avalia-t’s regular collaborators (external reviewers, clinical guideline development groups, participants in postintroduction studies, etc.; n = 182). To achieve a minimum of three participants per health care department or unit, this group was completed by selectively re-mailing leading specialists belonging to the Servicio Gallego de Salud/SERGAS (Galician Health Service). Specialists were identified through the SERGAS corporate mailing list, and respondents were selected if they belonged to major hospitals and acknowledged having practiced medicine for more than 5 years. The final decision on participants was taken in collaboration with the management team to prioritize the involvement of more active clinicians (heads of departments, members of national and international associations, authors of extensive publications in peer-review papers). Two e-mail invitations were sent per person and, if the recipient failed to respond within two weeks, another candidate was selected. This process continued until the minimum number had been reached for each medical specialty.
When respondents confirmed their willingness to participate, they were sent a self-report questionnaire by e-mail with an accompanying cover letter conveying information about the rationale and purpose of the assessment, as well as explanatory instructions about completing the questionnaire (Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000774). Potential respondents were sent a maximum of two reminders and, if they did not respond within 2 weeks, were contacted by telephone call to schedule an appointment and administer the survey in person.
Classification and Prioritization
To assess the opinion of and degree of agreement among the professionals with respect to the classification of technologies, respondents were asked to classify the technologies into “new and emerging” and “innovative”, taking into account their personal knowledge, key full-text papers provided, additional information on the Internet and colleagues’ opinions. Based on the EuroScan definition (13), we established that a technology was to be classified as “new and emerging” where it was at a stage before its authorization or adoption by the health system, or alternatively, where it was being used in clinical practice but its use was not yet diffused and was restricted to a few health centers. “Innovative” was defined as any technology that was totally new (no other therapeutic or diagnostic option was available), displayed a mechanism of action or indication that it was very different to existing alternatives, or substantially improved treatment or diagnosis with respect to current options.
The foreseeable impact on the health system was assessed using a purpose-designed scale. The individual members of the panel of experts were instructed to rate the different technologies, taking into account explicit prioritization criteria (Supplementary Table 1). Respondents were asked to provide a summary score of the foreseeable impact of each technology, taking into account all the individual criteria. No criteria were considered more important than others. Technologies were scored from 1 to 9 as follows: low impact, 1–3; moderate impact, 4–6; and high impact, 7–9.
The classification and scoring of the technologies took place from December 2012 to February 2013.
Analysis of Results
The data were analyzed overall and by medical specialty. The medians and the ranges of the scores awarded were obtained. The reliability of the measurements was analyzed by calculating the intraclass correlation coefficient (ICC), based on an intra-rater or repeated measures analysis of the variance model (Reference Hallgren14). The ICC was calculated using a two-way random-effects model. This entailed assuming that the assessors represented a random sample of all possible assessors, and that the technologies represented a random sample of all assessable technologies. The type of ICC measure chosen was consistency rather than absolute agreement, because we believed that there might be a certain degree of variability in the interpretation of the scale. Calculations were performed using the SPSS 12.0 computer software package.
We calculated the lower and upper limits of the ICCs together with their 95 percent confidence intervals (Reference Fleiss and Cohen15) and assessed the results using the cutpoints proposed by Cicchetti (Reference Cicchetti16): excellent, ICC: 0.75–1.0; good, ICC: 0.60–0.74; regular, ICC: 0.40–0.59; and poor concordance, ICC <0.4.
Drawing-up of the Prioritized List
For the purposes of compiling the prioritized list, we solely included technologies that had been classified as new or emerging, and were considered innovative by at least 50 percent of the assessors, with the two criteria being regarded as mutually exclusive.
The list shows technologies envisaged as having a high impact, namely, those with a median >6 when ranked by score. Only technologies scored by two or more assessors were included in the final list.
RESULTS
The systematized search generated a total of 6,960 references to potentially new or emerging technologies. After a review of the abstracts and additional information retrieved by means of the Internet, 246 were deemed to be potentially relevant. Based on the application criteria, these were classified into 54 different medical-surgical areas or units, with 61 being included in two or more areas for independent assessment (a total of 341 assessments). The filtering and prioritization process is summarized in Figure 1.
Results of the filtration and prioritization process.
A total of 607 specialists were invited to collaborate: 146 (24 percent) acknowledged their willingness to participate; of these, 99 returned the completed e-mail survey (68 percent) and 47 completed the questionnaire delivered to them in person (32 percent). The questionnaires were completed by a minimum of three experts from each medical specialty (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3–Reference Robert, Gabbay and Stevens5). The results of the assessments, are presented in Supplementary Table 2, showing the number of technologies identified, selected and prioritized in each medical surgical area or unit and the intra-rater ICC. The greatest number of new and innovative technologies corresponded to the cardiology health care area (cardiology, pediatric cardiology, interventional cardiology, cardiac electrophysiology; n = 26), general and digestive surgery (n = 25), gastroenterology (n = 21), angiology and vascular surgery (n = 13), respiratory medicine (n = 12), and radiology (n = 10). After eliminating duplicates (n = 5), these areas accounted for 34 of the 68 technologies considered to be of foreseeable high impact (median score >6).
Intra-rater concordance, as shown by the ICC, was good in the areas of cardiology (0.63 to 0.83), general and digestive surgery (0.68, 95 percent CI: 0.35–0.86), and respiratory medicine (0.63, 95 percent CI: 0.22–0.67), and poor in vascular surgery (0.21, 95 percent CI: −0.24–0.78), gastroenterology (0.03, 95 percent CI: −0.36–0.72) and radiology (0.08, 95 percent CI: −0.41–0.94). The degree of agreement was also fair–good in nuclear medicine, nephrology, neurology, obstetrics and gynaecology, traumatology, and urology (ICC: 0.4–1). The ICC confidence intervals were very wide and in many cases included both “poor” and “good/excellent” ratings.
In the aggregate analysis, the number of assessors per technology ranged from three to seventeen. A total of eighty-six technologies were unanimously classified as new and/or emerging, and seventy-eight as innovative, with fifty-four fulfilling both criteria (22 percent). The assessors agreed unanimously that twenty-one were not new and/or emerging and that thirty-eight were not innovative, and disagreed as to the classification of the remainder (Figure 1). By reference to the preselection criteria, a total of 167 technologies (68 percent of those identified and filtered) were considered new and innovative. Of these, fifty-seven registered a median score >6, with fifty-one being considered for the final list (Table 3) and six being excluded for having been scored by a single assessor.
Final Prioritized List of New and Emerging Technologies
DISCUSSION
The early assessment of technologies likely to be demanded in the near future constitutes a great challenge for HTA bodies. Despite the fact that a great number of agencies worldwide have early awareness and alert systems, it has been established that these are not always effective in identifying and selecting the most significant technologies for subsequent assessment. This study constitutes an innovative contribution to this field, providing a systematic and reproducible basis for the process of identifying and prioritizing new and emerging technologies, based on the views and values of those health professionals who are directly or indirectly involved in their use.
The search strategy used for the identification of new and emerging nonpharmaceutical technologies, which had previously been validated with respect to high-impact journals (general and specific), was able to detect an elevated number of potentially relevant technologies at a very early stage in their development. This was corroborated by a previous study (Reference Varela-Lema, Punal-Riobóo, Casal Acción, Ruano-Ravina and Garcia12;Reference Douw, Vondeling, Eskildsen and Simpson17), which established that the strategy enabled 83 percent of technologies to be identified 4 years before their publication in fourteen high-impact journals. A cross-check against the EuroScan database showed that technologies tend to be referenced in Medline several years before being assessed by existing detection systems, something that indicates the strategy's great potential for identifying them in the early stages. This was confirmed in our study: of the 341 judgments made on the 223 technologies that had been identified and filtered from January 2012 to June 2012, 65 percent classified the technology as new and emerging in 2013 (Figure 1).
Yet, identification is only the first stage of the process. Subsequent assessment of all technologies identified is unfeasible, and so those of greatest impact and with the greatest probability of being incorporated into clinical practice must thus be filtered and prioritized (Reference Douw and Vondeling10;Reference Murphy, Packer, Stevens and Simpson18). According to the results of a recent survey, most of the EuroScan member agencies undertake this selection process internally (60 percent), and only some resort to groups of experts (25 percent), political decision makers (15 percent), or health professionals (10 percent) to identify the technologies that are most relevant (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3). Even though the vast majority (90 percent) apply similar prioritization criteria (number of patients/disease burden, disease severity, potential clinical benefits, speed of adoption, costs, organizational consequences, ethical, social, and legal aspects) (Reference Gutiérrez-Ibarluzea, Simpson and Benguria-Arrate3;Reference Douw and Vondeling10;Reference Noorani, Husereau, Boudreau and Skidmore19), Douw and Vondeling (Reference Douw and Vondeling10) have acknowledged that the process is highly susceptible to subjectivity, because it is not clear which criteria are taken into account or whether the same process is followed for each technology. These authors concluded that the participation of clinicians is fundamental for selecting technologies envisaged as possibly having a high impact in clinical practice. Furthermore, like the EUR-ASSESS (Reference Henshall, Oortwijn, Stevens, Granados and Banta20) and other groups (Reference Gallego, Bridges, Flynn, Blauvelt and Niessen11;Reference Noorani, Husereau, Boudreau and Skidmore19), they also proposed that efficacy could be improved if quantitative techniques were developed that served to reduce the subjectivity of choices.
In line with these considerations, the current study proposes a methodology that incorporates the view of health professionals directly involved in the use of the technologies at all stages of the selection process. These professionals were tasked with defining the degree of innovation and allocating scores of 1 to 9 according to the foreseeable impact of the technologies, by using a formal priority-setting method. Although there is no way of ensuring that all criteria are taken into account or that assessors appraise all criteria equally, we nonetheless believe that subjectivity is very much reduced by this method.
In our opinion, restricting the list to technologies with a foreseeable high impact (median >6), and which are classified as new/emerging and innovative by at least 50 percent of the professionals, minimizes the likelihood of prioritizing irrelevant technologies. The fact that 86 percent (51/59) of the technologies selected by medical specialties were maintained in the joint analysis clearly supports the hypothesis that the technologies selected are indeed highly significant. Nevertheless, when it is taken into account that the classification of the technologies as new or emerging was not uniform in 59 percent of the assessments, one cannot rule out the possibility that some high-impact technologies which are still not widely adopted might have been omitted because of misclassification. However, because agreement was unanimous in seven of the ten technologies of foreseeable high impact (median >6) that were excluded because they were not considered new and/or innovative, we believe that this is highly unlikely in our setting.
As has been acknowledged in the preliminary validation experience (Reference Varela-Lema, Punal-Riobóo, Casal Acción, Ruano-Ravina and Garcia12), the identification and filtering process was complex and time consuming, requiring the analyst to have a certain degree of experience to identify new and innovative technologies. Whether adding filters or including other search terms such as “health”, “health care” or “disease” could serve to reduce the number of references for screening and facilitate the preselection process should be explored. However, special care should be taken when refining the search strategy because an increase in the specificity could lead to the omission of relevant articles.
The difficulty surrounding the recruitment of participants was an important study limitation, and this was also acknowledged in the best–worst scaling experience by Gallego et al. (Reference Gallego, Bridges, Flynn, Blauvelt and Niessen11). Although we could rely on the involvement of members of the detecta-t network and different professional collaborators of avalia-t, and complement the study with a selective re-mailing to Galician Health Service specialists, it was difficult to achieve the collaboration of three experts in each field, which was the agreed minimum. For unknown reasons, the initial response rate from some medical specialties was very poor, so we had to resort to personal delivery of the questionnaire. This not only lengthened the time needed for the study but also influenced its accuracy. Probably because health professionals could not consult additional information, questionnaires completed in situ failed to provide scores for many of the technologies in these medical specialties (which included radiology, gastroenterology, and neuroradiology, among others) and concordance among assessors was poor. To enhance the list's reliability, we decided to include only those technologies that were scored by two or more assessors (n = 6), something that might have partly affected the study's exhaustiveness.
With a view to future updates, we believe that it would be relevant to form a stable group of national/international experts who had indicated their willingness to participate and their interest in collaborating. The NESPECIALIST study undertaken in Spain suggests that professionals with leadership qualities should be identified (unit heads, members of scientific societies, participants in groups of experts, etc.), and their knowledge of and involvement in innovation should be sought (Reference Vidal-Espana, Leiva-Fernandez and Prados-Torres21). Although involving a broad group of experts is recognized to be challenging, there are many experiences around the world in other fields that can suggest that it is achievable (Reference Balabanova, Gilsdorf and Buda22;Reference Eger, Gleichweit, Rieder and Stein23). One necessary step in building international working groups could be identifying Horizon Scanning Organizations that have similar perspectives and priorities and rely on their experts for input.
In the current study, several participants expressed doubts regarding the ranking of the prioritization criteria. We believe that this may also have contributed, in part, to the heterogeneity observed, and that the quantitative analysis could be far more accurate if the prioritization criteria were weighted. To this end, attention should be given to the question of whether developing prioritization-weighting tools such as PritecTools (http:pritectools.es) could facilitate this process. For this purpose, working groups should be set up that include the different sectors involved in the introduction and use of new technologies. In our region, the key stakeholders would be administrators, health care managers and clinicians, but this may vary substantially depending on a given detection system's target audience.
CONCLUSIONS AND POLICY IMPLICATIONS
Despite the fact that there are certain methodological aspects which are open to improvement, we believe that the methodology provided as part of this study can clearly contribute to the development of a systematic and reproducible framework for the effective identification and selection of relevant new and emerging technologies that are likely to be demanded in clinical practice. Although the list drawn up might not include all the high-impact nonpharmaceutical technologies for which assessment might be sought in the near future, there is no doubt about the relevance of those selected, and it is foreseeable that they may constitute an accurate proxy for technologies which, if not yet incorporated into clinical practice, may be relevant for different health care systems worldwide. Early assessment of these technologies by early awareness and alert systems could help to accelerate the incorporation of those technologies expected to yield important clinical benefits, and prevent the inappropriate use of those that show evidence of being less effective or less safe than existing alternatives.
This study proposes that the regular updating of the search strategy (every 3–6 months) could be used in combination with other information sources, which may conceivably be different depending on the context, to serve as the basis for a comprehensive list of new and emerging technologies. After being filtered by expert clinicians, this could then be prioritized by the respective health systems of different countries in line with their national assessment priorities.
It must be acknowledged that, being the first real experience of this work, the development of the current list was delayed due to the various methodological drawbacks that had to be addressed; it is foreseeable that in future these technologies will be identified within a much shorter time frame. Nonetheless, this proposal has been specifically designed for those running early awareness and alert systems who are interested, not so much in long-term planning, but rather in ascertaining the effectiveness, safety and cost-effectiveness of nonpharmaceutical technologies before they are demanded for coverage. As discussed in a previous study (Reference Varela-Lema, Punal-Riobóo, Casal Acción, Ruano-Ravina and Garcia12), it might be necessary for different countries to adapt the scope of the search to the technologies scanned.
CONTACT INFORMATION
Leonor Varela-Lema, PhD, MS (leonor.varela.lema@sergas.es), Ramó De La Fuente-Cid, Marisa, MD, PhD, López-García, MD, MS, Galician Department of Health, Galician Agency for Health Technology Assessment (Axencia de Avaliación de Tecnoloxías Sanitarias de Galicia/avalia-t), Santiago de Compostela, Spain
SUPPLEMENTARY MATERIAL
Supplementary Tables 1 and 2 http://dx.doi.org/10.1017/S0266462314000774
CONFLICTS OF INTEREST
The authors declare that they have no conflicts of interest.
