It is widely acknowledged that early identification and assessment of significant healthcare technologies is crucial for guiding decision-making and research prioritization (Reference Goodman6). To this end, many countries worldwide have established structured horizon scanning systems (HSSs) to identify new or emerging technologies that might have an important impact on their healthcare organizations (Reference Mundy, Merlin and Parrella9;Reference Packer12). Even though the scope (type of technology covered, customers) and timeliness of the different HSSs vary slightly depending on the policy context, the majority follow similar identification processes and share the same concerns regarding the missing of relevant technologies (Reference Douw and Vondeling2).
Within Spain, health technology assessment (HTA) is envisioned by law to have a direct role in supporting decision making regarding the incorporation of new medical devices and procedures into the National Health System's Health Care basket (ROYAL ORDER 1030/2006) (14). The Network for New and Emerging Technologies “GenTecs” was created to provide the Spanish national health system with relevant information on new or emerging technologies that might be subject to coverage application by National or Regional Authorities within the time frame of 1–3 years (Reference Benguria Arrate, Gutiérrez-Ibarluzea and Llanos1). The network integrates five regional HTA agencies, among them the Galician HTA Agency (avalia-t), that are coordinated to cover a wide range of information resources previously recognized to be of value for horizon scanning (Reference Douw, Vondeling, Eskildsen and Simpson3;Reference Glenn, Gabbay and Stevens5;Reference Smith, Cook and Packer18): manufactures, developers, licensing applications, company Web sites, major specialist medical and scientific journals, news Web sites, health organizations, horizon scanning organizations, conference papers and key expert groups.
The collaboration of five regional HSSs allows for a very resourceful intensive scanning process but as yet, it remains unclear if these traditional sources are adequate to identify relevant technologies in the early stages of implementation. Like in the case of Australia, it has been observed that coverage applications made at Regional level are frequently previous to the assessment of these technologies by the horizon scanning network (Reference O'Malley and Jordan11). The evaluation study carried out in Australia showed that some relevant technologies, like “double balloon enteroscopy,” which resulted in 176 publications from 2003 to 2008, were not subject to horizon scanning by any EuroScan member, suggesting that there are important gaps to be covered in the identification process.
In 2003, Dow et al described the importance of Internet as a source of information for identifying new or emerging technologies (Reference Douw, Vondeling, Eskildsen and Simpson3). The authors found that agencies spent up to 2 hours a week and at most 8 hours per week scanning Internet sites but established that there were great variations regarding the sites that were considered important to scan routinely. The authors concluded that horizon scanning agencies might benefit from further investigation into which sites could deliver most output and stated that future activity could include a coordinated effort to develop Internet scanning strategies for different categories of health technologies or different specialties.
Some working groups have addressed this issue in recent years, but investigations have been mainly focused on selecting databases or developing and ranking prioritization criteria for assessing the value of the identification sources (Reference Glenn, Gabbay and Stevens5;Reference Simpson, Hiller and Gutiérrez-Ibarluzea16). Following a recent review of this literature and discussion with English Horizon Scanning Centre analysts, eight prioritization criteria were identified as being the most relevant for assessing the value of scanning sources (Reference Smith, Cook and Packer18): (i) coverage (approximate percentage of relevant information), (ii) quality (reliable, accurate, and objective), (iii) efficiency (estimated time to identify one potentially significant health technology or relevant information), (iv) accessibility of information, (v) frequency of scanning (how often the source is updated), (vi) cost (subscription or registration cost), (vii) memory (news archive), and (viii) contact details for further information.
To our knowledge, there are no previous studies that have approached the design of general bibliographic scanning strategies for horizon scanning. However, some authors did succeed in developing search strategies for specific pathologies. For example, Thygesen et al (Reference Thygesen, Vondeling and Douw19) developed an effective Internet strategy to identify new medical technologies in prostate cancer. The strategy consists of ten sites including Medscape, Medline Plus, and the National Electronic Library for Medicine. Although the use is limited for horizon scanning systems that have a wide focus, it does provide methods that can be useful to develop similar strategies.
Within this framework, the current work aims to present a Medline search strategy (PubMed search engine) that can be used by HSSs to systematically identify new or emerging nonpharmaceutical healthcare technologies. The study describes the development of the search strategy and provides an analysis of the retrieval efficiency in comparison to the perusal of preselected key research medical journals, estimating the sensitivity for early identification of relevant new or emerging technologies.
METHODS
Search Strategy Design
A preliminary Medline search strategy was built up by an experienced information specialist, aided by all authors, combining free-text terms and synonyms that are commonly used in article titles to denote innovation with index terms that make reference to the specific fields of interest. The free text words for innovation were identified by scanning the references of 162 early warning system reports published in the EUROSCAN database (4). The references of the first clinical articles published for new or emerging technologies were reviewed and the text words reflecting the novelty of the technology compiled and combined. The reports reviewed corresponded to reports published during the period 2008–10 that were classified under the heading “procedure,” “device,” and “diagnostic.”
The search strategy was designed to recover only health technologies that are relevant to our HSS, excluding among others, pharmaceutical technologies, vaccines and technologies that are still in the experimental phase (animal studies). Following other examples, the construction of the final search strategy was iterative (Reference Haase, Follmann, Skipka and Kirchner7). The preliminary search strategy, developed using multiple combinations of potentially relevant terms, combined using Boolean operators, was run in multiple simulation trials and tested against the clinical articles referenced in the 162 early warning system reports. In each trial, single terms were removed or filters added to assess their influence on sensitivity and potency. After analyzing and discussing the results, publication type restrictions were applied and several terms (such as first study, first report, first case or preliminary report) removed because they produced a great number of irrelevant hits and made the review inefficiently labor intensive. In addition, it was deemed relevant to restrict the search for terms in the title field, instead of title and/or abstract because this approach very much minimized stray hits without significantly affecting sensitivity.
Efficacy and Precision in Comparison to Handsearching Key Research Scientific Journals
The efficacy of the Medline search strategy was assessed by running the search over a period of one year (2009) and analyzing its retrieval performance (number and characteristics of the new or emerging technologies identified). The search was run from 1 January 2009 to 31 December 2009 and technologies were classified based on the abstracts and additional full text information recovered online. Technologies were classified as new or emerging when the first clinical studies referencing the technologies in Medline dated back 4–5 years (≥ 2005). In addition, technologies were considered new or emerging only when they were truly innovative, that is, the design, material or functioning was totally new or very different to those of proceeding technologies or the indication was totally new. For each of the technologies classified as new or emerging, relevant information was recorded in a database regarding the type of technology, patient indication, medical specialty that is responsible for applying the technology (Spanish classification), type of technology and journal of publication.
For comparison purposes, the abstracts of key research scientific journals recognized by authors to be highly relevant for identification of new or emerging technologies were also scanned. Six prominent research journals and eight high impact surgery journals (impact factor ≥ 4) were selected for assessment: Science, Journal of the American Medical Association, Lancet, Annals of Internal Medicine, Archives of Internal Medicine, British Medical Journal, Annals of Surgery, American Journal of Transplantation, Endoscopy, Journal of Neurology Neurosurgery & Psychology, Archives of Surgery, Annals of Surgical Oncology, British Journal of Surgery, and American Journal of Surgical Pathology. All article abstracts published in these journals during 2009 were reviewed for identification of new or emerging technologies and additional information retrieved to classify technologies.
The precision of each identification process was calculated by the proportion of new or emerging technologies identified among the total number retrieved. The number of retrievals needed to read (NNR) was defined as the number of retrievals that had to be screened to find one new or emerging technology.
Sensitivity of the Medline Search
The sensitivity of the Medline search strategy for early scanning of new and emerging technologies was estimated following the methodology proposed by Haynes et al (Reference Haynes, Wilczynski, McKibbon, Walker and Sinclair8). Sensitivity was defined as the proportion of relevant new or emerging technologies identified by key medical journals that are retrieved with the designed search strategy within the first 2 years of publication in Medline. To establish sensitivity, all the technologies identified through the preselected medical journals were backtracked in Medline to find articles indexed in the first 2 years since first publication and citations cross-checked against the retrievals of the search strategy across that period. When technologies were missed by the Medline search (false negatives) a thorough analysis was made of the key terms of first clinical studies published to establish possible reasons for these losses. The analysis of results was carried out using the statistical software SPSS v.12.
RESULTS
The final Medline search strategy is outlined in Table 1. The application of the search strategy yielded 6,228 abstracts of potentially new or emerging technologies. After a first perusal of the search titles, 5,260 technologies were discarded, either because they were not new or because they were not health technologies covered by our horizon scanning system. The review of the abstracts led to the exclusion of another 269 technologies, leaving a total of 699 potentially new or emerging technologies.
Search strategy proposal for identification of new or emerging non pharmaceutical health technologies in Medline.
Through the retrieval of additional information in Medline or Google, 383 of the 699 technologies were classified as truly new or emerging (55 percent), 76 as new indications (11 percent) and the remaining 240 were considered slight modifications or incremental developments of existing technologies (35 percent). Dismissing these last technologies, 459 technologies were considered relevant, which means that the precision, calculated as the number of technologies identified among the number retrieved was 7 percent (459/6,228). Overall, thirteen articles had to be screened to find one relevant article. By way of example, Table 2 lists some of the new or emerging technologies identified by the search.
Some of the New or Emerging Technologies Identified by the Medline Search
Of the new or emerging health technologies detected, approximately 63 percent were devices, 24 percent procedures, 13 percent diagnostic tests, and 3 percent corresponded to other technology types (rehabilitation processes and contrast materials). The majority of these technologies were used for treatment (n = 263) or disease diagnosis (n = 176). Only 4 percent of the identified technologies corresponded to technologies used for palliative or preventive care. The technologies derived mainly from the medical specialties of Radiology, Cardiology, Cardiac surgery, Oncology, Traumatology and Orthopedic, Angiology and Cardiovascular Surgery, and Urology. These healthcare areas were responsible for applying more than 70 percent of the identified technologies (n = 334).
The new or emerging technologies identified in the Medline search were published in 314 different journals, the vast majority medical specialized journals (96.3 percent). The two journals that provided the greatest hits were Endoscopy and Catheter Cardiovascular Surgery. Table 3 provides the technologies detected broken down by type of intervention, type of technology, medical specialty, publication journal, and type of journal.
Descriptive Analysis of Medline Search Results
aIncludes nuclear medicine, radiodiagnostics and interventional radiology
bIncludes medical and radiotherapeutic oncology
The scanning of 12,061 journal abstracts derived from the fourteen medical journals selected for assessment led to the identification of fifty potentially new or emerging technologies. Of these, eight were disregarded because the first clinical studies were published before 2005, two were excluded because they were still in the animal investigation phase, and five were considered modifications of already existing technologies. Overall, thirty-five technologies were classified as relevant: thirty-one truly new or emerging, and four new indications of existing technologies. The great majority of new or emerging technologies identified were published in specialized medical journals (Endoscopy = 15; Annals of Surgery = 10; others = 4) and only 6 were published in general medical journals (Lancet = 4; JAMA = 2). The calculated precision was 0.29 percent (35/12,061) and the NNR 326. Table 4 provides a description of the technologies identified by the search in key medical journals.
Descriptive Analysis of High Impact General and Surgery Specialized Journals Search Results
When technologies identified in the key scientific journals were backtracked, it was found that thirteen corresponded to technologies that were first referenced in Medline in 2009 and twenty-two were technologies indexed throughout the previous four years of study (2005–08). The designed Medline search strategy could locate twenty-nine of the thirty-five publications. Of the six technologies not identified, three had keyword combinations that were eliminated from the initial search strategy to increase specificity: “first study,” “preliminary report,” “or “first report.” Three did not make reference to the novelty of the technology in the title, being two new indications of already established technologies. The estimated sensitivity of the search strategy, considering all relevant misses was 83 percent (29/35).
DISCUSSION
The present study provides an effective Medline search strategy for the early identification of relevant new or emerging technologies. The search strategy has been validated against key scientific journals and has demonstrated to be valuable for horizon scanning. Even though it can require adaptations depending on the scope of the Horizon Scanning System, we believe that this strategy could clearly serve to simplify the scanning process and standardize existing methodologies in this field.
The development of systematic Internet scanning strategies for different categories of health technologies or different health specialties has been recognized for many years to have a great potential for improving the efficiency and quality of horizon scanning, but few advances have been made in this field (3). To our knowledge, the present study is the first to succeed in designing an effective Medline strategy for automatic search scanning. The reason for choosing Medline, the United States National Library of Medicine's search service, was based on the fact that it is a free search engine that allows for comprehensive searching of more than 25,000 journals (10). It covers all the key peer review biomedical literature, providing access to summaries and some free full text online. It is updated and provides basic and advanced options for searching and tags that can allow for filtering the search.
The value of the Medline search strategy for horizon scanning is demonstrated by the great number of hits that it can provide, but also by the ability to identify relevant technologies in the first 2 years of appearance in Medline, which is frequently 4–5 years ahead of their publication in high impact scientific journals. For the medical specialties assessed, we have found that more than 90 percent of the technologies identified were first published in field specific medical journals different to those high impact journals selected for study purposes. Even though this finding is not completely new, because it has been previously acknowledged that early case series, case reports or uncontrolled trials that influence early adopters are frequently published in many different specialized medical journals (Reference Robert, Stevens and Gabbay15), this is the first time that the search yield of key research medical journals for identification of new or emerging technologies is quantitatively assessed. The findings show that these sources are relatively inefficient for early identification of new or emerging technologies.
Nonetheless, the handsearch of all relevant specialized medical journals requires intensive time resources and is practically impossible for horizon scanning systems that, like ours, intend to cover all nonpharmaceutical healthcare technologies that are likely to be requested for public funding. Furthermore, in many times, search results have to be completed through using Google engine or other quasi-scientific information sources to assess the novelty of the technology or classify it. The Medline search strategy is presumed to be able to partly cover this information gap and facilitates the identification of technologies in the early diffusion period. To partly test this assumption, search findings were correlated with the new coverage applications made to the Galician Health Care System and our own early warning system (detecta-t) notifications received after 2009. It was found that many of the submissions made after 2009 corresponded to technologies that had been identified in the search. For example, technologies like the Restech Dx-pH Measurement System Dx, MitraClip, Amplatzer, Holmium Laser (Ho:YAG) were notified during 2010. When technologies were cross-checked with horizon scanning reports published in EuroScan between 2007 and 2010 (4), it was also found that horizon scanning reports were always subsequent to the first publications in Medline. For example, relevant technologies like the Cervista HPV HR test, Penumbra System; transoral gastroplasty, MERCI Retriever, and autologous Stem Cell transplantation for type 1 diabetics were published in EuroScan during 2010–11.
Limitations of the Study
The main limitation to the validation of results is that efficacy was only assessed against key medical journals and high impact surgery journals (gold standard). To determine the true accuracy, it would have been necessary to run the search against a bigger gold standard and determine if the Medline search strategy could locate all nonpharmaceutical technologies assessed by EuroScan horizon scanning agencies. Ideally a fully effective scanning source would be one that identifies all potentially relevant new or emerging technologies even if many are false-positive (Reference Douw and Vondeling2). Because this has not been ascertained, it cannot be dismissed that some clinical areas or types of technologies might be undercovered. For this reason, the current search strategy must be considered a preliminary proposal that should be further explored and tested in different contexts to check if it covers efficiently all relevant fields.
It must be acknowledged that the current proposal was specifically designed for systems that are not so interested in long-term planning, but on ascertaining effectiveness, safety and cost-effectiveness of nonpharmaceutical technologies before they are demanded for coverage. The value of horizon scanning within Spain is to inform coverage decision makers and petitioners on the relevancy of coverage applications. For this purpose, it is recognized as sufficient to identify technologies within the first 1–3 years of publication. For other horizon scanning applications or types of technologies, this timeframe might be considered insufficient (Reference Simpson, Packer and Carlsson17).
To cover these longer time spans it might be necessary to broaden the search strategy to other medical bibliographic databases (EMBASE or ISI Web of Knowledge) (Reference Topfer, Parada and Menon20) that include more non-English publications and gray literature (case reports, congress abstracts) or modify the strategy to include animal studies and additional terms that were omitted from the current proposal because they added great noise. However, adding these changes or broadening the search to terms in the title and/or abstract might not be efficient because it can provide a great volume of literature to be reviewed.
The absence of specific medical subject headings (MeSH) to index studies on innovative technologies is an important drawback in refining the search strategy. At present, the sensitivity of the Medline search can only be increased at the expense of decreasing specificity, which is estimated to be low. For systems that use the early warning system for broader health problems, long-term planning or anticipate future needs, it might be necessary to complement the automatized Medline search with other sources of primary information. Nonetheless, this does not reduce the value of the search, which could in many cases avoid reviewing the great majority of key medical journals, which are inefficiently labor-intensive to search.
A possible limitation of using automatized scanning for identification of potentially relevant technologies is that the information provided in abstracts is not always sufficient to classify the technologies. Accessing the full article is not always possible due to financial restrictions and in such cases the technology must be classified based on additional information retrieved online. The current experience shows us that in these cases; technology selection can require not only certain experience from the analyst but also explicit criteria for classifying technologies. In the current study, technologies were classified as new or emerging when the first clinical studies referencing the technologies in Medline dated back to 4–5 years. It is recognized that this classification might not be suitable for all contexts and thus might require adaptations to cover the needs of the individual different horizon scanning systems. The great number of relevant technologies that arise from the search demonstrates its efficiency but raises great uncertainties regarding how these technologies are going to be prioritized. The identification of technologies for formal assessment can require an in-depth assessment of each technology (Reference Douw and Vondeling2;Reference Pluddemann, Heneghan and Thompson13), suggesting that a preliminary filtering process might be required to roughly identify the most significant technologies. The next phase of this project will be to run the search at regular intervals (2–3 months) and explore how it can be used in a real-setting of an HTA agency.
CONCLUSIONS AND POLICY IMPLICATIONS
The current study serves to prove that a specifically designed Medline search strategy can be highly efficient for the identification of new or emerging technologies and thus serve to fill an important gap in the horizon scanning process.
The present work is relevant because it clearly identifies opportunities for improvements in horizon scanning activities. Even though the applicability of the proposed search strategy might be limited to countries, such as Spain, that only cover nonpharmaceuticals and are not so interested in very long-term planning it can serve as a starting point for developing and validating context-adapted search strategies. It is acknowledged that search terms used to denote innovation are totally transferable and thus the search strategy can be easily adapted for use in other contexts by incorporating additional terms to retrieve other types of technologies or limiting filters to cover more records. Adapting the proposed strategy to other databases is also possible but in this case, matching thesaurus terms must be identified.
The lack of Medical Subject Headings (MeSH) to define new or emerging technologies is an important limitation for the elaboration of automatized search strategy proposals. However, adding medical Subject headings to define new or emerging technologies would only be possible if these headings were removed from the references when technologies were considered more established and this is very challenging. The definition of a common terminology and the universal acknowledgement of these terms are essential to progress in the refinement of scanning strategies. We recommend that further work must be done in this area to overcome current limitations and make the search much more specific, avoiding unnecessary filtering processes. In the meantime we encourage authors to use more consistent vocabulary in the reference titles. In its current form the search strategy must be considered a preliminary proposal that should be explored within different horizon scanning systems because it seems to have a great potential to avoid reviewing the great majority of medical journals, which are inefficiently labor-intensive to search.
CONTACT INFORMATION
Dr Leonor Varela-Lema, Senior Technician of the Galician Agency for Health Technology Assessment (avalia-t), Galician Department of Health, Santiago de Compostela, Spain
Jeanette Punal-Riobóo, Senior Technician of the Galician Agency for Health Technology Assessment (avalia-t), Galician Department of Health, Santiago de Compostela, Spain
Beatriz Casal Acción, Senior Technician of the Galician Agency for Health Technology Assessment (avalia-t), Galician Department of Health, Santiago de Compostela, Spain
Professor Alberto Ruano-Ravina, Department of Preventive Medicine and Public Health, University of Santiago de Compostela. Santiago de Compostela, Spain; CIBER de Epidemiología y Salud Pública, CIBERESP, Spain
Dr Marisa López-García, Director of the Galician Agency of Health Technology Assessment (avalia-t), Galician Department of Health, Santiago de Compostela, Spain
CONFLICTS OF INTEREST
The authors do not have any conflict of interest.
