No CrossRef data available.
Article contents
Assessing the accuracy of forecasting: Applying standard diagnostic assessment tools to a health technology early warning system
Published online by Cambridge University Press: 01 August 2004
Abstract
Objectives: Early warning systems are an integral part of many health technology assessment programs. Despite this finding, to date, there have been no quantitative evaluations of the accuracy of predictions made by these systems. We report a study evaluating the accuracy of predictions made by the main United Kingdom early warning system.
Methods: As prediction of impact is analogous to diagnosis, a method normally applied to determine the accuracy of diagnostic tests was used. The sensitivity, specificity, and predictive values of the National Horizon Scanning Centre's prediction methods were estimated with reference to an (imperfect) gold standard, that is, expert opinion of impact 3 to 5 years after prediction.
Results: The sensitivity of predictions was 71 percent (95 percent confidence interval [CI], 0.36–0.92), and the specificity was 73 percent (95 percent CI, 0.64–0.8). The negative predictive value was 98 percent (95 percent CI, 0.92–0.99), and the positive predictive value was 14 percent (95 percent CI, 0.06–0.3).
Conclusions: Forecasting is difficult, but the results suggest that this early warning system's predictions have an acceptable level of accuracy. However, there are caveats. The first is that early warning systems may themselves reduce the impact of a technology, as helping to control adoption and diffusion is their main purpose. The second is that the use of an imperfect gold standard may bias the results. As early warning systems are viewed as an increasingly important component of health technology assessment and decision making, their outcomes must be evaluated. The method used here should be investigated further and the accuracy of other early warning systems explored.
Keywords
Information
- Type
- RESEARCH REPORTS
- Information
- International Journal of Technology Assessment in Health Care , Volume 20 , Issue 3 , August 2004 , pp. 381 - 384
- Copyright
- © 2004 Cambridge University Press
Early warning, horizon scanning or forecasting systems are the first stage of health technology assessment (HTA) (5). They facilitate the rational adoption of new technologies in health care by providing policy-makers with timely information on the potential consequences of the introduction of health technologies into the health-care system (2). Responses to early warning information include provision of HTA reports, modification of treatment protocols and improved financial planning. Many developed countries have early warning systems in place. EuroScan, the European Information Network on New and Changing Health Technologies (www.publichealth.bham.ac.uk/euroscan) has 12 member agencies, and there are many more organizations both established and developing with a remit covering early warning throughout the world. One of the earliest and most developed systems is that of the National Horizon Scanning Centre (NHSC) (www.publichealth.bham.ac.uk/horizon).
The NHSC predicts which new technologies are likely to have a significant impact on the National Health Service (NHS) in England and Wales. The method used is multifaceted, using several predetermined criteria (Box 1), expert opinion, industry information and corroboration with affiliated agencies. It aims to provide advance notice to the Department of Health for England and the National Assembly for Wales up to 3 years before launch of a technology.
Although forecasting systems are becoming more prevalent, there are differences in approaches to early warning between countries, suggesting a clear need for evaluation of methods (7). However, the validity of the results of any forecasting method is difficult to assess, as a control group is difficult to achieve (8). Although a retrospective qualitative analysis (6) has been conducted, there are generally few evaluations and no quantitative studies to date. This study reports the first quantitative assessment of accuracy of predictions by an early warning system.
The study investigates the accuracy of predictions made by the NHSC in the period 1997 to 1999 when 1,060 potentially interesting new technologies had been identified. Of these, 610 technologies were from six target clinical specialties: cardiology, diabetes, neurology, oncology, rheumatology, and obstetrics and gynecology.
METHODS
Predicting whether a technology will or will not have an impact on a health-care system is analogous to diagnosis. We used a modification of the Yerushalmy (9) approach to evaluate diagnostic tests, where the sensitivity and specificity of a test of interest are estimated with reference to an independently established standard diagnosis (4). The latter is ideally the true outcome or a gold standard test. We used an (imperfect) gold standard (3), which was a consensus of expert opinion 3 to 5 years after the event on whether a technology had made a significant impact on the NHS.
In 2002, we carried out stratified random sampling of the 610 technologies in the six target specialties. The stratification was by specialty and whether the technology had been predicted to have an impact by the NHSC. The resultant 117 technologies were equally divided between the specialties with the exception of rheumatology, for which there were 17 technologies. By design, 30 percent of the 117 technologies were predicted to have an impact on the NHS by the NHSC between 1997 and the end of 1999.
The list of technologies for each specialty group was sent to 25 consultants in that specialty (in England and Wales)—150 in total. The consultants were drawn from the West Midlands Regional Medical Advisory Committee and an Internet-based directory of UK specialists (http://www.specialistinfo.com). We asked the consultants to indicate those technologies that had made a significant impact according to the criteria listed in Box 1. We defined technologies as having had an impact if all respondents identified them (i.e., a 100 percent consensus). There were ten or more respondents for each of the specialties.
RESULTS
The prevalence of technologies having an impact according to consensus among responding experts was 6 percent (95 percent confidence interval [CI] 3–12 percent). Table 1 shows that there was agreement with experts for eighty-five technologies and disagreement for thirty-two technologies. The methods used by the NHSC to predict if a technology is likely to have a significant impact on the health service had a sensitivity of 71 percent (95 percent CI, 35–92 percent) and a specificity of 73 percent (95 percent CI, 64–80 percent). The negative predictive value (NPV) was 98 percent (95 percent CI, 92–99 percent) and the positive predictive value was 14 percent (95 percent CI, 6–30 percent).
DISCUSSION
The low prevalence of technologies having an impact suggests that the number of innovations having a major impact on health services is small. Concerning the accuracy of predictions made by an early warning system, the results suggest that the forecasting method used by the NHSC to select technologies that may have an impact on the health service is acceptable. The most important result is sensitivity, that is, the proportion of technologies truly having an impact on a health service that are correctly selected. Avoiding false negatives is essential to avoid evaluation by-pass where technologies enter into the health-care system without further assessment. In contrast, false positives (minimized by a high specificity) tend to overburden HTA and policy systems. Although the specificity of the NHSC's system appears adequate at 73 percent, the number of false positives may indicate an area of concern. The NPV was high at 97 percent. This finding could show that the NHSC's methods are good at filtering out those technologies that will not have a significant impact. The low positive predictive value indicates that the methods are more inclusive than exclusive.
In applying the diagnostic test evaluation technique to an early warning system, it is recognized that one of the difficulties in evaluating the accuracy of predictions is that there are several factors that interact to determine a technology's fate in the market place. One of the purposes of forecasting is to provide information that could be used to control adoption and diffusion. Therefore, in assessing the accuracy of the prediction, we need to consider that impact is not only determined by the characteristics of the technology but also by any controls that were implemented to influence adoption and diffusion. In this study, we gave experts a definition of significant impact (Box 1) and, thus, have framed the outcome and minimized evaluating the control component.
Another concern with the design of this study is reference test bias, inherent with the use of an “imperfect” gold standard. Ideally, the true outcome or a gold standard test is required to calculate accuracy. The true outcome in this case would involve prospectively monitoring a technology from the time of impact prediction using financial, health, and service data specific to the technology to determine whether it had made a significant impact on the health service. We believed that the opinion of practicing consultants after the event was a good proxy and had been used by others to elicit opinion on medical technologies (1;6). However, a particular problem with our imperfect standard is that the standard itself varies according to the percentage of responses required to determine a positive impact. We chose 100 percent as we considered that all practicing consultants in a particular specialty would be aware if a technology had truly made a significant impact in that specialty.
Clearly from our results there is opportunity for improving performance. However, forecasting is difficult, and the results reinforce that few technologies in development reach the market and even less have a significant impact on a health service. The NHSC predicted that 35 of the 117 technologies may have a significant impact but the experts believed that only 7 technologies had actually made an impact by 2002. There were thirty false positives. Although resource intensive, early warning systems are the first stage of HTA, so there are opportunities for technologies initially thought to have the potential to make an impact to be reconsidered as more data becomes available. The specificity of the method, therefore, is less important, particularly recognizing that false positives may include technologies that would have had inappropriate impact unless predicted. The NHSC are not only looking for technologies that will have a significant positive impact on the health service that could eventually be endorsed, but also for technologies that could have a significant negative impact.
CONCLUSIONS
We have used a method for assessing accuracy normally applied to diagnostic tests and have demonstrated it to have potential for evaluating the success of forecasting. The next step will be to use this method with others as a means of finessing particular approaches to early warning in HTA and comparing alternative methods, both of which will be prerequisites for considering efficiency.
POLICY IMPLICATIONS
Early warning systems are increasingly viewed as an integral component of HTA, and for this reason their outcomes must be evaluated. The method used here should be further investigated and the accuracy of other early warning systems explored. However, early warning systems are unlikely ever to be completely accurate, and this shortcoming too needs to be recognized. It is apparent that of the hundreds of new technologies developed and marketed each year, only a small percentage has a major impact on health services. It is important that the method of forecasting used identifies these most significant technologies.
The authors thank those consultants who participated in the study. Sue Simpson, Alison Cook, and Claire Packer are funded by the Department of Health; Andrew Stevens and Chris Hyde are funded by the National Health Service for England.
References
Fuchs V,
Sox H Jr.
2001
Physicians views of the relative importance of thirty medical innovations.
Health Aff.
20:
30–
42.Google Scholar
Hailey D,
Topfer L-A,
Wills F.
2001
Providing information on emerging health technologies to provincial decision makers: A pilot project.
Health Policy.
58:
15–
26.Google Scholar
Hui S,
Zhou X.
1998
Evaluation of diagnostic tests without gold standards.
Stat Methods Med Res.
7:
354–
370.Google Scholar
Knottnerus J,
Van Weel C,
Muris J.
2002
Evidence base of clinical diagnosis. Evaluation of diagnostic procedures.
BMJ.
324:
477–
480.Google Scholar
Milne R,
Clegg A,
Stevens A.
2003
HTA responses and the classic HTA report.
J Public Health Med.
25:
102–
106.Google Scholar
Robert G,
Stevens A,
Gabbay J.
1999
‘Early Warning systems’ for identifying new healthcare technologies.
Health Technol Assess.
3:
1–
108.Google Scholar
Simpson S,
Carlsson P,
Douw K, et al.
2002
A comparative analysis of early warning systems demonstrates differences in methods and structure: A survey of EuroScan member agencies.
In: The Challenge of Collaboration Proceedings of the 18th Annual Meeting of the International Society of Technology Assessment in Health Care,
Jena:
Urban & Fischer Verlag
189.
Spiby J.
1987
Future and emerging medical technology. A study to investigate advance in medical technology in the United Kingdom.
Submission to membership of the Faculty of Community Medicine.
London:
Faculty of Public Health Medicine, Royal College of Physicians
Yerushalmy J.
1947
Statistical problems in assessing methods of medical diagnosis with special reference to x-ray techniques.
Public Health Rep.
62:
1432–
1439.Google Scholar
Comparison of 100% Consensus of Expert Opinion on the Impact a Technology Has Had on the NHS in 2002 with the NHSC Prediction of Impact in 1997, 1998, or 1999
You have
Access
- 13
- Cited by