No CrossRef data available.
Article contents
The usefulness of the NHS Economic Evaluation Database to researchers undertaking technology assessment reviews
Published online by Cambridge University Press: 01 August 2004
Abstract
Objectives: Health-care technology reviews now increasingly include outcome costs as well as clinical effects. This study reports the findings and implications of a survey to explore the usefulness of the National Health Service Economic Evaluation Database (NHS EED) within this process.
Methods: Postal survey of lead authors of Technology Assessment Reviews (TARs) commissioned by the United Kingdom's National Institute for Clinical Excellence (NICE). The questionnaire investigated the usefulness of NHS EED in terms of (a) search strategy, (b) data extraction, (c) quality assessment, and (d) determining requirements for new modeling studies. Qualitative data were requested, including opinions regarding NHS EED.
Results: NHS EED was used in 90 percent of all identified reviews (n=46). The questionnaire response rate was 63 percent. The percentage of scores 3 or above (most useful), 2 or below (least useful), or N/A were, respectively, (a) search strategy=62 percent, 23 percent, 15 percent; (b) data extraction=23 percent, 27 percent, 50 percent; (c) quality assessment=38 percent, 19 percent, 42 percent; (d) determining requirements for new modeling studies=27 percent, 23 percent, 50 percent. The results were expanded further in the qualitative data from the respondents.
Conclusions: NHS EED is a useful tool for a variety of tasks in the NICE/TAR process but not, unsurprisingly, as a replacement for understanding primary studies. There is, however, a need to reduce the impact of time lags between the publication of economic evaluations and the appearance of abstracts relating to them on NHS EED. The results will inform future developments of the NHS EED database, which should increase its usefulness to researchers.
Information
- Type
- GENERAL ESSAYS
- Information
- International Journal of Technology Assessment in Health Care , Volume 20 , Issue 3 , August 2004 , pp. 249 - 257
- Copyright
- © 2004 Cambridge University Press
Increasing demands on health-care services and budgets from the well-established drivers of demographic change, new technologies and increased knowledge of health-care options among patients, mean that the outcomes considered in health-care technology assessments now increasingly include an appraisal of costs as well as clinical effects (1). In many countries, this fact has been fully recognized and subsequently integrated into the work of national regulatory bodies that approve the funding of health technologies and interventions and/or guidance on their use (5). These bodies include the National Institute for Clinical Excellence (NICE) in the United Kingdom (8), which followed the already well-established organizations in Australia (3) and Canada (2).
Health-care decision-makers and researchers are increasingly required to access, assess, and use economic and clinical evidence for interventions that are being considered by health-care providers. Recent studies in UK Health Authorities have shown that local decision-makers value sources such as the National Health Service Economic Evaluation Database (NHS EED), because it independently identifies, summarizes, and critiques evidence from economic evaluations (6;7;9). (Decision-makers also make use of other sources of assessed evidence, including the Cochrane Database of Systematic Reviews [CDSR; http://www.update-software.com/Cochrane/default.HTM], the Database of Abstracts of Reviews of Effects [DARE], and the Health Technology Assessment [HTA] database [both available at http://nhscrd.york.ac.uk].) Although this recent research reveals the relative merits of economic evidence and databases such as NHS EED for decision-makers, little work has been undertaken to date to assess the value of economic resources to researchers, although reviews of electronic databases have been undertaken recently (11;12). A particular group of researchers of interest to the UK context are those conducting systematic reviews of evidence for NICE health technology assessments (HTAs), known as Technology Assessment Reviews (TARs). Researchers involved in TARs typically adopt the following steps to locate, assess, and summarize economic evidence:
- Define the inclusion criteria for studies to be assessed in the TAR.
- Search for studies in a range of resources using a range of methods.
- Extract and record data from studies that meet the specified inclusion criteria.
- Assess the quality of included studies.
- Summarize the findings of all included studies.
- Determine the need for new modeling studies that synthesize the effectiveness and economic evidence derived from the review.
We hypothesized that NHS EED database might play a useful role in one or more of the stages of the TAR production process. We considered that, by searching a wide range of databases since the mid-1990s and filtering out only full economic evaluations, NHS EED could provide a useful option for researchers seeking the most relevant economic studies. In addition, we thought that researchers might use NHS EED as a form of independent validation of the researchers' own quality assessment of a published economic evaluation. Any wide divergence between the two assessments might be a cause for debate or reassessment. We also hypothesized that the structure of the NHS EED abstract, in effect, formed a checklist that TAR researchers might use to guide their own assessments. To discover whether our assumptions about the usefulness of NHS EED were valid, we surveyed the lead authors of TARs commissioned by NICE since 1999. Our methods and findings are reported below.
METHODS
The authors to be surveyed were identified from the NICE Web site (http://www.nice.org.uk/). The Web site lists completed technology appraisals where guidance has been issued. This source provides access to all relevant documents and related material, including guidance, TAR reports, patient information, and press releases. The TARs that are made available on the Web site are those originally submitted to NICE, most of which are subsequently published in the HTA monograph series. As not all submitted TARs had been published as HTAs at the time of the survey, it was decided that, to ensure consistency, only the original TARs would be used for the survey.
At the time of the study, there were forty-six available and complete TARs on the NICE Web site. The contact details of the lead authors (or teams) were obtained from the forty-six available TARs, and survey questionnaires were sent out in October 2002. A deadline for return of the questionnaires was set 1 month after this date. No incentives were used to increase the response rate, but reminders were issued to those lead authors that did not respond within the specified time.
The survey questionnaire consisted of four positive statements that required a response according to a scale from 1 (least useful) to 5 (most useful). An alternative option of “N/A” was available for situations in which respondents believed NHS EED was not applicable to a particular question. The statements were as follows:
- Search strategy: NHS EED was useful in locating or identifying economic studies that were included in the review.
- Data extraction: NHS EED abstracts were useful in extracting and summarizing data from the published economic studies.
- Quality assessment: NHS EED was useful in completing quality assessments (for example the Drummond 10-point or BMJ 35-point checklist) (4) for economic studies included in the review.
- Requirement for new modeling: NHS EED abstracts were useful in informing the decision to commission/not to commission a new modeling study.
To acquire both quantitative and qualitative data, each statement contained a second element, which invited open and explanatory comments about the statement. We also requested information about how NHS EED could be used, or improved, within future NICE TARs.
Qualitative data were analyzed by creating a table of similar responses under the general headings of “positive” and “negative” statements. Supporting verbatim comments and paraphrased responses were provided as examples and where they supported the qualitative statements presented in the table. Where a response or an explanation was feasible regarding each point, this statement is provided in the results section and is expanded upon in the discussion and conclusions.
RESULTS
Forty-six NICE TARs were identified and retrieved from the Web site, the details of which are shown in Table 1.
The search strategies reported in each TAR were then reviewed by CRD staff to establish whether NHS EED was reported to be part of the search strategy. A total of thirty-seven TARs explicitly stated that they searched NHS EED, a further three TARs searched the Cochrane Library, which includes NHS EED. Therefore, NHS EED was definitely used in 80 percent of TARs and possibly used in a further 6.5 percent. The questionnaire response rate was 63 percent. The results of the quantitative analysis are shown in Figure 1.
Results of the quantitative analysis of questions 1–4.
The most useful role played by NHS EED was found to be in terms of assisting with the search for economic evidence (62 percent of respondents gave a score of 3 or above). The next most useful role for NHS EED was that of literature quality assessment, with 38 percent of respondents giving NHS EED a score of 3 or above. However, a relatively high proportion of respondents (42 percent) did not find NHS EED relevant (N/A) to this element of the TAR process. Only 23 percent of respondents found NHS EED useful in data extraction. In terms of determining the requirements for new modeling studies, only 27 percent of the respondents scored NHS EED with 3 or more, and 50 percent of respondents believed that NHS EED was not applicable to this part of the process.
These quantitative results indicate that NHS EED was highly useful to researchers in searching for and locating relevant studies, and somewhat less useful in other aspects of its potential use within the HTA review process. However, the qualitative comments provide reasons that explain and expand upon the quantitative findings. These are summarized below according to questions 1–4.
1. Search strategy: NHS EED was useful in locating or identifying economic studies that were included in the review.
The majority of respondents reported a general lack of published economic evaluations within the area of interest of the TAR. This finding was not seen, however, as the fault of the NHS EED database but a reflection of the limited published research in the field being investigated. One respondent stated that the information obtained from NHS EED was more useful than the information obtained from a MEDLINE/EMBASE search in situations where several economic evaluations had been published in the area of interest.
Another respondent reported finding economic evaluations that were not contained on NHS EED. One of the respondents found what they classified as “poor quality” economic studies, which were published, but not found on NHS EED. Two other respondents highlighted the lack of company submissions on NHS EED. However, due to the nature of company submissions, it is not reasonable to expect them to be present on NHS EED. Some submissions will never get published and others are published at a later date.
2. Data extraction: NHS EED abstracts were useful in extracting and summarizing data from the published economic studies.
Where available, the abstracts were viewed by researchers as a useful reference point, summarizing the main points of the study and highlighting relevant details. The responses indicated that one of the purposes of NHS EED may be realized in that respondents reported that they often needed more detail than was contained within the NHS EED abstracts. NHS EED has been intended as a decision aid to help the researcher decide whether to obtain the full paper, rather than as a substitute for reading the full paper. The NHS EED abstract structure was described by one respondent as “very helpful for noneconomists, allowing them to gain some comprehension about the assessment of economic evaluations.” However, the majority of researchers believed that it was within the mandate of their project's protocol to extract the data directly from the primary studies and not be reliant on a third party to fulfill this task. Again, this is understandable as the database is not intended to be used as a substitute for reading the full economic evaluation.
3. Quality assessment: NHS EED was useful in completing quality assessments (for example the Drummond 10-point or BMJ 35-point checklist) for economic studies included in the review.
The abstracts were regarded as a useful reference point in completing checklists, but in many instances the primary study would be required to complete this task. The majority of respondents stated that they did complete checklists for the economic evaluations included in the TAR, but they did this with reference to the primary study and not just the abstract. As one respondent stated in this respect “Independent assessment of the nature and quality of economic evaluations would be seen as an important part of HTA, that is, could not rely on other assessments, although might well refer to them as a second stage procedure.” One respondent criticized the 10-point checklist as an instrument, claiming that it was “not sensitive enough to separate studies.”
4. Requirement for new modeling: NHS EED abstracts were useful in informing the decision to commission/not to commission a new modeling study.
One respondent stated that a “lack of the availability of abstracts highlighted the requirement for new modeling.” Another respondent believed strongly that a review of economic evaluations should be the first step, and if this highlighted a deficit in existing studies, then a new evaluation using modeling should be undertaken. It was not clear, however, if the respondent used NHS EED as a tool to aid the review process. Some of the respondents were involved in conducting TARs that did not include any modeling or were conducted before the more formal requirement for modeling to be included in TARs. Another stated that “modeling is an integral component of HTAs for NICE,” suggesting that, regardless of what evidence was available, a new model would be constructed. The respondents did not comment negatively on the usefulness of NHS EED in aiding the decision about commissioning new modeling studies.
5. Comment on how NHS EED could be used, or improved, within future HTA NICE reviews.
One of the respondents stated that there was a need to “make it more comprehensive, for example, put in industry submission models to NICE.” This change, however, is not feasible due to issues of confidentiality and/or timing and will be addressed further in the discussion. Another respondent suggested that there should be: “Expansion of the sections on relevance to the NHS. In particular, if a decision had to be made on the adoption of the technologies could it be made on the basis of that paper? If not, what were the principal limitations.”
The general consensus from responses was that, although it is difficult to quantify, NHS EED was generally found to be useful. One comment made by one respondent was with reference to the target audience of the database: “main requirement is for a comprehensive, well indexed, easily searchable database of HE evaluations, not abstracted info/quality assessment. The key issue is target audience. We are not decision-makers, but rather fellow researchers who probably have similar training and skills to your own (NHS EED) abstractors/reviewers. Concerning decision-makers, my own opinion is that the information is too complex to really encourage them to attempt to use the HE evaluations cited.”
Table 2 provides a summary of the main points emerging from the qualitative data analysis. The qualitative data showed that NHS EED was appreciated for its usefulness in identifying studies and could be more useful than just a search of MEDLINE/EMBASE. However, in some instances, reviewers stated that they had located economic studies that were not included on NHS EED. It was also valued as an independent source and reference point when economic evaluations were identified. However, researchers conducting TARs, as part of the NICE process, were required to review company submissions (which were usually confidential), and these submissions were understandably not available on NHS EED. In terms of data extraction, although NHS EED abstracts were seen as being a useful tool, the review process understandably requires researchers to retrieve and work from the full published papers. NHS EED was reported to be a useful source for nonhealth economists in that the structured abstracts helped them to understand the methodology and to interpret the results of economic evaluations. Finally, with regard to the completion of checklists, NHS EED was found to be useful, but, consistent with the comments on data extraction above, the NHS EED abstract was used in conjunction with the primary study as a complementary source of information regarding included economic evaluations.
DISCUSSION AND CONCLUSIONS
The results of the survey suggest that NHS EED is a useful aid to researchers undertaking TARs with some, perhaps anticipated, caveats. The limitations that have been highlighted are reasonable given the nature of the work being undertaken. The mandate of the researchers surveyed was to conduct an independent piece of research that will facilitate an evidence-based decision to formulate guidance. As such, it is reasonable to expect NHS EED abstracts to be used to supplement the original papers and to aid the search process, and the findings of this study support this view. However, there should not be an expectation that NHS EED can fill the gap regarding company submissions, as these often include recently available data and/or an original model. There may also be issues of confidentiality that mean evidence cannot be scrutinized in the public domain as in the case of published work.
Several of the respondents indicated that there were few or no economic evaluations in the area that they were reviewing. This may be explained by the finding that the majority of TARs are conducted to assess and review areas with new technologies. It is normal for the efficacy of a technology to be proven before its manufacture, but it is also the case that there is often a time lag between the effectiveness trial and the economic evaluation of any given technology (13). This finding can mean that, despite there being studies showing how effective a technology is, there may be to date no studies evaluating the new technology's cost-effectiveness.
Not all of the studies that were identified by review teams were found on NHS EED. There are three possible reasons for the absence of studies. First, there is a time lag between the publication and indexing of the studies on major databases and their abstracting for NHS EED. This lag means there may be papers being processed for NHS EED but not yet available as abstracts. Second, the study may not meet the NHS EED inclusion criteria for a full economic evaluation. Despite the database having quite liberal inclusion criteria (http://nhscrd.york.ac.uk/nfaq4.htm), not all economic studies meet the requirements and, as such, may not be abstracted. Third, company submissions are not available for abstract by NHS EED, because the work is often confidential. If, at some later date, the company submissions, or papers based on them, were to be published and meet the NHS EED inclusion criteria, an abstract would be produced.
NHS EED staff are addressing many of the suggestions raised by respondents. First, CRD continues to devote great effort to minimizing the time lag issue by recruiting abstractors and writing teams to produce abstracts. CRD welcomes inquiries from potential freelance abstractors with experience in health economics and writing teams based within organizations. The forthcoming installation of a new Web-based publishing system at CRD will also help in reducing time lags by streamlining the production process. It is hoped that this will all work to reduce the amount of time that a paper spends in the NHS EED system before being loaded onto the database as a completed abstract. Another recently introduced strategy to deal with time delays is fast tracking, used in conjunction with the identification of “provisional abstracts.” Within this process, a study is made visible to NHS EED users (provisional abstract) as soon as it has been classified as a full economic evaluation with a statement to the effect that an abstract will be produced in due course. However, users are able to request that the abstract be fast tracked, in which case the paper is abstracted ahead of others waiting in the system. Fast tracking is also applied to studies considered to be of increased value to the UK NHS, in particular those publications that fall under the umbrella of National Service Framework.
Additionally, to deal with the weakly reported studies in a way that is helpful to searchers, the database will now include bibliographic records for “partial economic evaluations.” The definition of a partial economic is a “study that meets the requirements of a full economic evaluation, namely that it reports costs and effects for an intervention and at least one comparator, but only reports costs or effects in a “global” manner.”
The response rate to the questionnaire (63 percent) may be a little lower that anticipated, and a higher return would clearly have helped to increase the validity of the findings. However, although reminders were used, it became evident from discussions with lead authors that several earlier NICE TARs did not benefit from the now fairly well-established methods of systematic review of economic studies applied to more recent TARs, and as such the economic elements were not as well focused as they now are. Additionally, the lack of economic evaluations in some NICE TARs may have influenced the motivation of researchers to respond, as NHS EED had not been relevant in these instances.
The survey results also indicate that the NHS EED team has work to do to promote the various ways in which it might meet the needs of different groups. The assumption by some of the respondents that the data in NHS EED should be enough to fulfill data extraction requirements indicates that the filtering and quality assessment roles of NHS EED need to be better communicated. The role of NHS EED as an independent source that offers critiques of economic evaluations, however, appears to be well accepted within the research community.
The survey has provided encouragement that NHS EED is largely on the right track in terms of its service to researchers preparing TARs, and has raised useful issues that the project team is now addressing. NHS EED is under constant development to improve its usability to both researchers and decision-makers. This finding is evidenced by the involvement of NHS EED as a principal player in the newly established European Network of Health Economic Evaluation Databases (EURONHEED) project (10), coordinated by the Collège des Économistes de la Santé in Paris. EURONHEED is bringing together health economists from a total of seventeen countries around Europe, and this process will clearly help in the advancement of reviewing and disseminating economic evidence to the benefit of researchers.
The NHS EED team can also use insights obtained form this survey to inform its closer working with TAR teams. NHS EED is seeking to work more collaboratively with TAR teams to maximize efforts being spent assessing economic evaluations. In this regard, two TAR teams have agreed to produce NHS EED abstracts for full economic evaluations that they are assessing as part of their TAR work. This relationship, as it develops, should be beneficial for both the database and the TAR process as the NHS EED template has been proposed and tentatively accepted as the standard data extraction form/critical summary for economic evaluations.
The findings of this survey of researchers, in conjunction with the developments outlined above, should assist the development of NHS EED in moving toward a more standardized and systematic approach to identifying, summarizing, and critiquing economic evaluations.
POLICY IMPLICATIONS
The implications of the present study center on the way in which structured abstracts contained on NHS EED should be used by reviewers undertaking TARs. NHS EED is primarily aimed at providing decision-makers in the UK NHS, and beyond, rapid access to the results of studies as well as a critique of the validity of the results, based on the methodology adopted by the authors. This aspect of the use of NHS EED has been investigated and disseminated through relevant publications, as outlined in the present paper. The results of this study show that NHS EED is used by the vast majority of lead authors making up the sample and is clearly valued by them, but its use in this context is complementary rather than providing a substitute for the review, which has to reach its own independent conclusions regarding the content and validity of included economic evaluations. The research findings, however, are highly valuable and will inform future developments of NHS EED, and related databases involved in the EURONHEED project, that may limit the disadvantages of NHS EED that were identified by those surveyed in this study. The question of including confidential reports in TARs that have not been scrutinized in the public domain is outside the remit of this study but is clearly an important issue that warrants further analysis and discussion.
References
Bentkover JD,
Corey R.
2002
Effective utilisation of pharmacoeconomics for decision makers.
Dis Manage Health Outcomes.
10:
75–
80.Google Scholar
CCOHTA.
1997
Guidelines for economic evaluation of pharmaceuticals: Canada.
2nd ed.
Ottawa:
Canadian Co-ordinating Office for Health Technology Assessment
Commonwealth of Australia.
1995
Guidelines for the pharmaceutical industry on preparation of submissions to the pharmaceutical benefits advisory committee: Including submissions involving economic analyses.
Canberra:
Australian Government Printing Office
Drummond M,
Jefferson T.
1996
Guidelines for authors and peer reviewers of economic submissions to the BMJ.
BMJ.
313:
275–
283.Google Scholar
Hjelmgren J,
Berggren F,
Andersson F.
2001
Health economic guidelines-similarities, differences and some implications.
Value Health.
4:
225–
250.Google Scholar
Hoffman C,
Matthias Graf von der Schulenburg J.
2000
The influence of economic evaluation studies on decision making: A European survey.
Health Policy.
52:
179–
192.Google Scholar
Hoffman C,
Nixon J,
Stoykova BA, et al.
2002
Do decision-makers find economic evaluations useful? results of recent focus group research in the UK.
Value Health.
5:
71–
78.Google Scholar
National Institute for Clinical Excellence.
2001
Technical guidance for manufacturers and sponsors on making a submission to a technology appraisal.
London:
National Institute for Clinical Excellence
Nixon J,
Phipps K,
Mugford M, et al.
2003
Using economic evidence to support decision-making: A case study of assertive community treatment within the UK National Service Framework for Mental Health.
Appl Health Econ Health Policy.
1:
179–
190.Google Scholar
Nixon J,
Ulmann P,
Glanville J, et al.
2004
The European Network of Health Economic Evaluation databases (EURONHEED) project.
Eur J Health Econom,
In press.
Royle P,
Waugh N.
2003
Literature searching for clinical and cost-effectiveness studies used in health technology assessment reports carried out for the National Institute for Excellence appraisal system.
Health Technol Assess.
7:
iii, ix-x,1-51.Google Scholar
Sassi F,
Archard L,
McDaid D.
2002
Searching literature databases for health care economic evaluations: How systematic can we afford to be?
Med Care.
40:
387–
394.Google Scholar
Stoykova BA,
Drummond MF,
Barbieri M,
Kleijnen J.
2003
The lag between effectiveness and cost-effectiveness evidence of new drugs: Implications for decision-making in health care.
Eur J Health Econom.
4:
313–
318.Google Scholar
Details of Identified NICE TARs Since 1999
Results of the quantitative analysis of questions 1–4.
Results of the Qualitative Analysis
You have
Access
- 6
- Cited by