Many countries have established systems to support the uptake of new and emerging health technologies. The National Horizon Scanning Centre (NHSC) in England is one such early awareness and alert (EAA) system, and provides advanced notice of new and emerging health technologies and interventions that are likely to have a significant impact on the English National Health Service (NHS) and/or patients, within the next 2 to 3 years (Reference Packer4). The work of the NHSC informs the future work program of the National Institute for Health and Clinical Excellence (NICE), the Health Technology Assessment (HTA) program, and other policy-making bodies.
Horizon scanning involves looking at many different sources of information for scientific developments on the horizon (Reference Wild and Langer11). A comprehensive EAA system requires a wide range of scanning sources to ensure adequate coverage of all types of technologies (Reference Robert, Gabbay and Stevens7). The NHSC scanning program includes over thirty information sources for routine scanning. Scanning sources may be categorized into primary, secondary, and tertiary sources (Reference Douw and Vondeling1;Reference Robert, Gabbay and Stevens7;Reference Stevens, Milne and Lilford9). Primary sources provide information directly from developers or manufacturers and the NHSC routinely contacts all major pharmaceutical and medical technology companies to invite them to discuss their pipeline developments, as well as individual contacts regarding specific technologies. Secondary sources are those where some filtering of topics has already been undertaken, and include key medical journals, consultation with experts in the field, and Internet media sources. Tertiary sources provide information from other horizon scanning organizations such as EuroScan, the Australia and New Zealand Horizon Scanning Network (ANZHSN), and the Canadian Agency for Drugs and Technologies in Health (CADTH). This combination of primary, secondary, and tertiary sources provides corroboration, and increases accuracy and coverage of relevant information (Reference Douw, Vondeling and Eskildsen2).
The initial selection of NHSC scanning sources was, in part, based on the findings of Robert et al. (Reference Robert, Gabbay and Stevens7). This early study aimed to determine which types of information sources would be most useful in identifying new healthcare technologies. A list of potential information sources was compiled from existing or previous EAA systems through a Delphi survey, and included: key medical journals (e.g., British Medical Journal), the pharmaceutical media (e.g., Scrip, www.scripnews.com) and information directly from pharmaceutical companies.
A more recent study by Douw et al. (Reference Douw, Vondeling and Eskildsen2) found variation between individual horizon scanning agencies in their approach to Internet-based sources. The authors found that, whereas horizon scanning agencies remits often differ in scope, on average the same types of information sources and Web sites are used. However, there was little correlation between the individual Web sites that agencies had selected for scanning and therefore judged as important. Currently, new and potentially valuable sources are identified by the NHSC in an ad hoc way from current scanning and information sources and through information exchange with other EAA systems and HTA agencies. The NHSC trials new sources informally and subjectively before a recommendation is made to include or exclude the source from ongoing routine scanning. There is a lack of tools to aid horizon scanners to assess new sources identified more objectively, although several authors have attempted to address this.
Robert et al. (Reference Robert, Gabbay and Stevens7) identified and ranked criteria for assessing the value of potential information sources. These criteria were timeliness, time efficiency, sensitivity of source (the identification of all potential new technologies in the knowledge that many will be false-positives), correlation with other sources, objectivity, depth of source (level of detail), elucidation of likely knock-on effects of the technology (potential impact), specificity of source (the identification of only the most important new technologies), and explicitness of limitations. Sources used in horizon scanning were subsequently rated.
Douw et al. (Reference Douw, Vondeling and Eskildsen2) described the importance of devising efficient procedures for the identification of new health technologies. The authors found that Internet sites most attractive for horizon scanning were easy to scan, free of charge, appeared to provide objective information and an email alert service. In terms of frequency of scanning, most sites were scanned weekly or monthly. Thygesen et al. (Reference Thygesen, Vondeling and Douw10) aimed to develop a list of prioritized Web sites that would be easy to search and provide trustworthy, accurate and valuable information. Inclusion and exclusion criteria were developed based on the literature and applied to a variety of potentially relevant Web sites. The criteria related to relevance to specific clinical indications, user-friendliness, efficiency of searching, and quality/reliability-related criteria. A pilot study undertaken by Perras and Topfer (Reference Perras and Topfer5) aimed to determine which current CADTH sources were most useful in terms of the identification of new drugs, cost and ease of access, and to suggest which sources should no longer be scanned. Key characteristics of sources related to publication frequency, annual cost, accessibility, and number of relevant “hits.” One limitation of this study was that sources were not compared for the quality and quantity of the information included in their coverage.
We aimed to develop and apply a set of evaluation criteria to the current NHSC sources. The objective was to identify a cutoff score below which sources would be recommended for removal from routine horizon scanning. The rationale for the work is that developing a set of evaluation criteria could result in a more objective approach in the selection of sources to scan, thereby increasing the efficiency of scanning time.
METHODS
A review of the literature, using MEDLINE, EMBASE, PubMed, international horizon scanning and HTA agencies, and Google, identified papers that potentially discussed quality criteria and inclusion/exclusion criteria. Keywords used included: “Horizon scanning systems”; “Search strategy”; “Priority setting”; “New and emerging health technologies”; “Systematic scanning” and “Health technology assessment”. Draft evaluation criteria were developed based on the collated literature, and discussions with NHSC horizon analysts (Table 1). A scoring structure was devised to try to ensure the criteria could be applied in a more quantitative way. We used the Modified Portsmouth Scorecard (6), developed to aid the commissioning of healthcare services to prioritize and guide decision making, as the basis of our scoring system. The Modified Portsmouth Scorecard enables scoring and prioritization of health services and interventions according to locally developed preselected criteria often including cost, benefit and the level of evidence of benefit or harm. A pilot study to test the practicality of the criteria and initial scoring system was undertaken in February 2008, using six of the NHSC's thirty-five current sources selected at random. NHSC horizon analysts used the criteria and scoring system to assess the six sources. Feedback from the horizon analysts was used to further develop the categories, scoring, and recording system. The final set of evaluation criteria and scores (Table 2) were used by horizon analysts to assess the value of all thirty-five scanning sources in the NHSC horizon scanning program. As part of the recording system, the horizon analysts were also asked to record their opinion as to whether a source should continue to be scanned or not. The final scores assigned to each source were summed and the sources were ranked. The maximum total score possible for a source was 240. The cutoff score was identified by using the scores and horizon analyst opinion on the value of the source.
Draft Evaluation Criteria
Evaluation Criteria, Scores, and Recording System
Four experienced horizon analysts assessed the relative importance of the eight criteria using the Analytic Hierarchy Process (AHP) (Reference Saaty8). The AHP, developed by Thomas L. Saaty in the 1970s, is a structured technique based on mathematics and psychology for breaking down and evaluating complex decisions. The AHP is a structured framework to systematically evaluate criteria, comparing them in pairs, using judgment (individual or group) of the criteria's relative importance. AHP converts these evaluations to a numerical value or “priority” and this indicates the criteria's relative ability to achieve the “decision goal,” that is, which criteria are the most important in assessing scanning sources.
RESULTS
The AHP “priority” derived for each of the eight factors is shown in Table 3. The higher the “priority” the higher the relative importance of the criteria. The most important criteria for each of the horizon analysts were coverage, quality, and efficiency, followed by accessibility and frequency.
Criteria Ranked According to AHP Priority
AHP, Analytic Hierarchy Process.
A cutoff score of ≤100 was selected. The thirty-five sources with their scores are set out in Table 4. Seven sources fell beneath the cutoff score of ≤100 and were recommended for removal from the NHSC horizon scanning program.
Total Scores for Each Scanning Source
DISCUSSION
We succeeded in developing evaluation criteria that were considered useful by the NHSC horizon analysts in assessing the value of current sources. As a result of our work, seven sources were removed from the NHSC horizon scanning program. From the NHSC perspective, the criteria could be categorized as primary (essential), for example, coverage, quality, and efficiency; and secondary (useful but less important), for example, accessibility, frequency of update, cost, memory, and contact details. Cost, for example, is not necessarily an individual concern as often a hosting institution (e.g., a university or health service) funds a source. Of course, for another HTA or EAA system with different funding arrangements or amounts, cost may well become a primary criterion.
The main limitation of this study was that the evaluation criteria were used by individual horizon analysts to assess their own assigned sources, perhaps leading to a subjective analysis of the source. Some high scores were unexpected. Two key medical journals, the Lancet and the Journal of the American Medical Association (JAMA), had scores of 210. Although there is no doubt that these sources provide high quality information, the high scores may be surprising, as they do not routinely yield a high number of “hits” for new and emerging health technologies. However, this study only took into account one horizon analyst's opinion of each source and when scoring, individuals may have made comparisons within their group of sources with no mechanisms for comparison of sources between horizon analysts. In addition, in terms of source “efficiency,” the time to find potentially relevant technologies and/or information will inevitably vary between individual horizon analysts and improve as experience is gained in scanning a specific source.
It will be necessary to further validate the criteria within the NHSC, by selecting a group of horizon analysts, assigning to them all a random selection of current sources for a set period of time (e.g., 4 to 6 months), and then asking them to assess their value. Such an exercise would require a substantial amount of individual time, duplicate scanning activity and may not take in account relative scanning experience within the team. For example, individuals may have more experience in scanning for drugs, than medical devices and diagnostics. So sources may need to be stratified, further increasing the complexity.
The study results could have been enhanced by the use of more qualitative methods such as interviewing experts and specialists in the field, perhaps using a Delphi questionnaire method, or by using artificial neural networking to recognize complex patterns in data.
There were similarities between our criteria judged as important, and the criteria identified in the four other studies (Reference Douw, Vondeling and Eskildsen2;Reference Perras and Topfer5;Reference Robert, Gabbay and Stevens7;Reference Thygesen, Vondeling and Douw10). The common criteria were accessibility (or ease of scanning), cost, efficiency, frequency, and quality (i.e. reliable, accurate, and objective information). We did not, however, measure the number of relevant “hits” for individual sources as Perras and Topfer did. Adding this could have increased the objectivity of our recommendations for removal. The evaluation criteria and scoring have the potential to be used as part of a more systematic and objective approach to the selection of identification sources, and its value needs to be established through future use. Since this study, the criteria have been tested on a new source. Formulary (3) was trialed for 3 months by a horizon analyst, who then used the criteria to assess the value of the source. The total score fell beneath the cutoff score, and it was rejected as a routine NHSC source.
It will be important that the evaluation criteria and scores are reviewed regularly to ensure that they are up-to-date and relevant to evolving NHSC requirements and a review of current sources could become a regular exercise.
CONCLUSIONS
The relevance of this work may be limited to comprehensive EAA systems that have a wide focus in terms of topic coverage. However, this study should serve as a starting point for discussion between all EAA systems (comprehensive or more focused) to develop, pilot, and validate common evaluation criteria to determine the value of sources. This could lead to the development of a common list of valuable sources, although the decision of which sources to scan will ultimately depend on the particular remit of the individual EAA system. It is recognized that different regions of the world will have access to different sources and that some sources will be more relevant to different regions, for example, the UK Gene Therapy Advisory Group and the USA Food and Drug Administration (FDA). Furthermore, the selection of sources to scan may also be cost dependent as some EAA systems may pay full subscription/registration costs, whereas others may be partially or fully funded through the institutions employing or hosting them. However, this type of assessment may be valuable as part of a process for efficiently assessing current sources and selecting new ones to scan.
CONTACT INFORMATION
Joanna Smith, BSc (Hons) (j.l.j.smith@bham.ac.uk), Senior Horizon Analyst, Alison Cook, BSc (Hons), PhD (a.m.cook@bham.ac.uk), Associate Director, Claire Packer, BM, BS, FFPH (c.packer@bham.ac.uk), Senior Clinical Lecturer in Public Health, Department of Public Health, Epidemiology and Biostatistics, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK
CONFLICT OF INTEREST
All authors report having no potential conflicts of interest.
