With the current proliferation of healthcare technologies, it is a continued reality of the health technology assessment (HTA) environment that advice on health technologies will be required quickly by decision makers. This can conflict with the requirements of traditional HTAs, which may require substantial lengths of time to gather, analyze, interpret, review, and publish findings. This can create a tension between the demands of HTA and the imperatives of the policy-making process, where HTA input must be timely if it is to influence decisions (Reference Hailey, Corabian, Harstall and Schneider7).
Thus, the use of “rapid review” is increasing, driven primarily by this need to engage with policy makers, healthcare professionals, and consumers in a timely manner to provide evidence-based recommendations pertaining to healthcare activities and decisions. However, while this concept of rapid review has been prominent in the discourse surrounding HTA for some time, the HTA community is yet to reach a consensus regarding their validity and the most appropriate methodology to use in their preparation.
Within the context of a broader assessment of current methods and practice in HTA, the Australian Safety and Efficacy Register of New Interventional Procedures – Surgical (ASERNIP-S) has evaluated several aspects relating to the preparation of rapid reviews. Through the use of a survey tool and extensive Internet searching, current practice in the preparation of rapid reviews was assessed. This strategy was complemented by a systematic literature search designed to identify and examine the current evidence base pertaining to rapid review methodology. The assessment also incorporated a comparative element, examining the differences in essential conclusions between rapid and full reviews of the same topic; the results of this comparison are the focus of an independent article. The purpose of the review was to draw together data from multiple sources to inform debate, strengthen methodological development, and open international dialogue between agencies regarding this area of health technology assessment.
METHODS
Survey of HTA Organizations
Building upon an INAHTA survey undertaken in 2006, a new survey tool was developed and distributed to all International Network of Agencies for Health Technology Assessment (INAHTA) members (forty-six Agencies, see www.inahta.org/) in March 2007. Four other HTA agencies that are not members of INAHTA were also contacted due to their prominence in the field: National Institute for Clinical Excellence (NICE; United Kingdom); Malaysian HTA Program; Singapore Health Authority; and Agostino Gemelli University Teaching Hospital (Italy).
The survey was developed in consultation with an international panel of HTA experts and addressed a broad range of themes relating to the conduct of rapid reviews. Using both open-ended narrative and specific categorical questions, respondents were asked about the administration of the rapid review process, research strategies, composition of rapid review products, and the use of peer review and external experts in the process. Respondents were also asked to compare several features of their rapid review products with those of full reviews. For the purposes of this report, rapid review was defined as “any HTA report or systematic review that has taken between 1 and 6 months to produce which contains the elements of a comprehensive literature search.” The survey is available as an appendix to the full review written by ASERNIP-S on this topic in July 2007, from http://www.surgeons.org/Content/NavigationMenu/Research/ASERNI-PS/ASERNIPSPublications/Other_Reports1.htm.
The survey was distributed electronically by means of the INAHTA Secretariat, with reminder emails sent after 1 month. If a response was not received, further follow-up was undertaken.
Response data were collated by means of spreadsheet tabulation. The main outcomes were discussed narratively. Simple descriptive statistics were limited to the calculation of percentage values across the data set.
Identification of Literature on Rapid Review Methodology
A systematic search was conducted of MEDLINE (1950–March 2007), EMBASE (1980–March 2007), Australasian Medical Index (2004–March 2007), and the Cochrane Database of Methodology Reviews and Methodology Register (Issue 4, 2006). Search terms used in The Cochrane Library were: methodology AND (rapid OR accelerated OR quality assessment). More complex algorithms used in MEDLINE, EMBASE, and AMI are detailed in Table 1. The International Journal of Technology Assessment in Healthcare (1998–March 2007) was also hand-searched for germane articles, which were retrieved as appropriate.
Search Algorithms
*Subject heading.
KW, keyword.
Inclusion Criteria Articles were selected for inclusion if the abstract contained information on any one of several outcomes reported in either a qualitative or quantitative manner, including rapid review initiation and rationale, rapid review methodology, content/completion of the rapid review, and quality evaluation of the rapid review.
Comparative studies, guidelines, program evaluations, methods studies, commentaries, and surveys were all considered for inclusion. The bibliographies of all included studies were manually searched for relevant references that may have been missed in the database search (pearling). Searches were initially conducted without language restriction; however, foreign language articles were subsequently excluded on a case-by-case basis due to time and financial constraints.
RESULTS
Survey of HTA Organizations
Products, Rationale, and Commissioning Bodies. The twenty-three agencies who returned the survey collectively described thirty-six rapid review products. Of these, seventeen products took between 1 and 3 months to complete, sixteen products took between 3 and 6 months to complete, and three products did not fit into the designated categories and have been detailed separately. These categories are considered in more detail in the full report; the following results are generally composite. As respondents were able to tick more than one response, percentages do not consistently total 100 percent.
Collectively, the most commonly stated reason for conducting a rapid review was in response to political urgency and/or to support decisions (44 percent). Limited time and resources (33 percent) and to answer a question raised in a previous HTA and/or to answer a specific question (31 percent) were the next most frequent reasons given, followed by clinical urgency and uptake of technology (17 percent for both).
When examined collectively, many (69 percent) respondents indicated that macro-decision makers, such as governments and health ministries, commissioned the rapid reports. Organizations considered to be meso-decision makers, such as hospitals and community health agencies, were the commissioners in nearly half (47 percent) of the cases, and micro-decision makers (clinicians, patients, individual projects) in 39 percent.
Reflecting the commissioning bodies and frequent links between HTA agencies and governments, the macro-level decision makers were most often the intended audience for rapid reviews (72 percent). This finding was followed by micro-level audiences (58 percent) and meso-level audiences (31 percent).
Research Strategies. Approximately half of the respondents reported conducting a systematic search of many databases to identify evidence for rapid reviews, while the other half reported undertaking systematic searches of a restricted number of databases. Gray literature searching was reported by 42 percent of respondents, with hand-searching used by only 25 percent. Axiomatically, the time line of the review tended to indicate the level of literature searching undertaken: for example, in the 1- to 3-month category, a systematic search of many databases was reported by only 35 percent of responders, whereas in the 3- to 6-month category, 75 percent of the respondents reported a systematic search of many databases.
This finding contrasts with the research strategies used in the full systematic reviews. Thirteen agencies responded to the questions comparing rapid and full review products: all thirteen agencies reported that a systematic search of many databases was undertaken for the preparation of full systematic reviews, 69 percent conducted hand-searches and 77 percent examined gray literature.
The responses in relation to whether the rapid review products excluded specific types of studies revealed that no rapid review product excluded systematic reviews; 6 percent excluded randomized controlled trials (RCTs); 17 percent ex-cluded nonrandomized controlled trials; and 83 percent excluded case series. Thus, as the level of evidence of a study decreased, it was more likely to be excluded. Seven agencies indicated in the additional comments that the criteria for excluding studies were dependent on the availability of published data, and that lower levels of evidence would be used if there was nothing else available. Responses indicated that full reviews were less likely to exclude different study types: no full review excluded RCTs, only 8 percent excluded nonrandomized studies, and 31 percent excluded case series.
Composition and Peer Review. Clinical outcomes were included in nearly all of the rapid review products (94 percent). Economic factors were considered by 78 percent of the products, and 53 percent included social factors, such as ethics. Almost three quarters of the products included some form of quality assessment of the included evidence. When compared with rapid reviews, full systematic reviews were more likely to report clinical outcomes (100 percent) and examine economic and social factors (92 percent and 85 percent, respectively).
In total, more than half of the respondents (67 percent) reported the use of external experts during the preparation/evaluation of their rapid review products; of this group, 59 percent indicated that this lengthened the time the report took to complete. Comparison to full systematic reviews showed that reviews completed over a longer timeframe incorporated elements of peer review more commonly than rapid reviews.
Literature on Rapid Review Methodology
There was a paucity of information identified in the literature regarding specific methodologies undertaken in rapid reviews of health technologies. After evaluating 101 abstracts arising from the literature search, a total of twelve relevant studies were identified: one guideline abstract, three program evaluations, two comparative studies, two methods studies, three commentaries, and one survey.
Several authors acknowledged the complexities inherent in conducting rapid reviews, suggesting that it is difficult to accomplish the triad of responsiveness to short timeframes, scientific rigor, and transparency in a manner consistently acceptable to all stakeholders (Reference Eisenberg and Zarin5). However, despite these challenges, some HTA organizations reported producing a considerable number of rapid reviews. An overview of the Canadian HTA products produced between 1995 and 2001 indicated that 17 percent of the total HTA output were “short documents,” indicating the increasing importance placed upon responsiveness to short timeframes (Reference Lehoux, Tailliez, Denis and Hivon8). Furthermore, the increased opportunity to contextualize rapid reviews within specific healthcare settings is an acknowledged strength of this style of product (Reference Hailey, Corabian, Harstall and Schneider7;Reference McGregor and Brophy9).
Other authors urged caution in the production of rapid assessments, while acknowledging their potential influence on policy making. Rapid review was proposed as an important intermediary step in the assessment of emerging technologies, to be followed by a more comprehensive assessment (Reference Stevens, Milne and Burls12). Detailed HTAs, undertaken over months or even years, would remain essential to address certain questions (Reference Hailey6). A comparative study exploring the differences in the complexity and findings of rapid and full reviews evaluated the minimum information response required to accurately answer a specific policy question (Reference Scott and Harstall11; identified by means of personal communication). It was found that the extra clinical information included in a full assessment may be extremely valuable to clinicians, but is of less importance to policy or other nonclinical decision makers, highlighting the need to have a thorough understanding of the requirements of the intended audience before undertaking a rapid review.
Methodological differences that may distinguish a rapid review from a full HTA as discussed in published literature included the development of a limited research question and truncated literature searching. Aidelsburger et al. (Reference Aidelsburger, Felder and Wasem1) proposed that a rapid HTA should focus on the study question by exactly defining the technology, outcomes, and study population to be examined, thereby facilitating an accurate literature search and reducing the number of identified studies. Furthermore, in an examination of the use and impact of rapid HTAs, it was stated that the rapid reviews under consideration provided somewhat restricted advice, given that they were generally confined to addressing questions of efficacy or effectiveness (Reference Hailey, Corabian, Harstall and Schneider7). This contrasts with the broader research questions commonly addressed by full HTAs, where issues of efficacy/effectiveness are frequently considered in conjunction with those of safety, economic viability, legality, and ethics.
The potential impact that altering search methodology may have on the validity of the review findings was assessed by two groups. After examining the indexing characteristics of a variety of clinical trials, Egger et al. (Reference Egger, Juni, Bartlett, Holenstein and Sterne4) concluded that “systematic reviews that are based on a search of English language literature that is accessible in the major bibliographic databases will often produce results that are close to those obtained from reviews based on more comprehensive searches that are free from language restrictions.” The Cochrane Controlled Trials Register (CCTR) was found to be the single best source of randomized controlled trial references, with additional database searching retrieving only a small percentage of extra trials (Reference Royle and Milne10). Best et al. (Reference Best, Stevens and Colin-Jones2) also acknowledged that exhaustive data collection may have little effect on the final recommendation, although no empirical evidence was provided to support this claim.
Only one study could be identified that evaluated the validity and reliability of rapid reviews compared with more extensive follow-up reports (Reference Corabian and Harstall3). In five of six cases, the conclusions reached by the rapid review product (“Technotes”) was confirmed by later peer-reviewed reports. The only disagreement in conclusions was a single Technote concluding that an intervention was experimental, whereas a larger cost-effectiveness study indicated the intervention to be reported as safe and efficacious in the literature, although no reason was provided for this discrepancy.
DISCUSSION
Survey of HTA Organizations
The responses gathered from the survey indicated that there is currently no uniform description of what constitutes a rapid review product, with respondents reporting a variety of time lines, report components, search strategies, and evaluation methodologies among the range of rapid reviews produced. It is clear that there are difficulties in categorizing products as “rapid” or “full” simply based on the length of time taken to complete reviews.
The current survey demonstrated that there is no standardized methodology applied to the preparation of rapid reviews, although different trends were identified. Rapid review products were less likely to use systematic searches of many databases, conduct hand-searches, and use gray literature than full systematic reviews. These results are axiomatic and most likely stem from time constraints faced when preparing rapid reviews. As the duration of the review extended (i.e., from 1 to 3 months to 3 to 6 months), a trend for more rigorous search strategies emerged. The effect that truncated searching may have on the final conclusions of rapid reviews is still unclear, although it has been suggested that exhaustive data collections may have little effect on the final recommendation of a report (Reference Best, Stevens and Colin-Jones2).
Inclusion of different types of evidence in rapid reviews appeared to be done subjectively and reflected the availability of evidence more than the methodology of the review. Agencies indicated that higher level evidence is generally preferable but not always available. The idea that some rapid reviews may be based solely on existing systematic reviews, thus creating a rapid review of overviews rather than of primary evidence also emerged. This finding may indicate an important potential role for rapid reviews, allowing full review findings to be used in a local healthcare context by adding local perspective or national costing implications.
Rapid reviews were less likely to use external experts and peer review than full systematic reviews. This could result in rapid reviews (particularly those produced in 1 to 3 months), receiving little scrutiny from methodological and clinical experts. The impact of this on review quality is unknown and merits investigation, as high level decisions may be made based on these reviews.
Although the rapid reviews and full systematic reviews of all agencies included in this study have been described as two distinct groups, they are likely to constitute many discrete review products, encompassing differences in search strategies, composition, and included study types. The survey did not ask whether systematic reviews used quality assessment or critical appraisal. Although these may both be considered fundamental elements of a systematic review, confirmation of their existence would have strengthened the results. Full systematic reviews, like rapid reviews are not homogenous, which emphasizes the importance of detailing research methodologies for these reviews (as well as any other type of review). In the case of rapid reviews, this should be done to avoid making them appear more valid than they are. Failure to do this could result in full systematic reviews appearing redundant to those who do not understand the important differences between the two types of products. This may result in decision makers choosing to commission rapid reviews rather than full systematic reviews. This may not be appropriate in some instances, particularly for complex interventions that may need to be investigated in greater depth by a full systematic review.
The use of different types of reviews in determining and implementing policy decisions was only briefly touched on by the survey. Each of the review styles were commissioned by a variety of bodies, including private healthcare funds, hospital administrators, clinicians, and governments, although the survey did demonstrate that shorter rapid reviews were most likely to be written as a result of political urgency. Within the small data set considered, there was no clear correlation between the commissioning group and the use of the review findings in policy making. The impact of reviews on policy seemed to be more closely related to the intended audience or the commissioning body than the nature of the review, but given the relatively small sample size, definitive conclusions regarding the impact that rapid reviews have on policy making cannot be drawn.
Due to the limited survey group, the results of this analysis could not be grouped by country or type of healthcare system. Exploring whether there are any links between healthcare systems and the use of rapid reviews for policy making may provide interesting results and could contribute to revealing the international impact of these products.
Literature on Rapid Review Methodology
Evaluation of the available peer-reviewed literature confirmed the impression created by the survey results, namely that there is a lack of standardized, explicit methodology for the undertaking of rapid reviews. It may be that these products defy the description of prescriptive a priori methodology due to their flexible and adaptive nature, which allows them to be targeted to particular audiences within specific healthcare contexts. Hence, the development of explicit, standardized methodology regarding these products may be counterproductive, as it would limit one of their key strengths, namely their adaptiveness.
Proposed methods for conducting rapid reviews, such as limited searching and development of highly refined research questions are yet to be adequately validated in the literature. Clearly, refining the research question could mean that review findings are less able to be generalized to smaller patient populations or certain comparator interventions.
Truncated literature search strategies may often lead to uncertainty surrounding the conclusions of a review. Conversely, however, it has been suggested that extensive literature searching can actually introduce bias into an evidence base rather than limiting it, as trials that are harder to locate can be of lower quality (Reference Egger, Juni, Bartlett, Holenstein and Sterne4). It may be that resources would be more appropriately directed to thorough quality assessments of evidence available in major databases, rather than extensive searching.
Given the inherent variability of the size and composition of the evidence base between interventions, it may not be possible to validate methodological strategies for conducting rapid reviews and apply them to every subject. Rather, each topic must be evaluated by thorough scoping, and appropriate methodology defined. This may highlight a greater need for a panel of expert advisors to ensure that the nuances of each topic have been adequately considered.
POLICY IMPLICATIONS
This review has identified that the current rapid review products being produced by HTA agencies are not well-defined and are highly variable in their methodology. However, it is a reality of the HTA environment that there will continue to be pressure to produce reviews that are both timely and accurate to support the ever-increasing speed of the policy-making process in this area.
It is, therefore, recommended that, rather than developing a formalized methodology by which to conduct rapid reviews, which may be inappropriate and oversimplified, agencies should work to increase the transparency of the methods used for each review. It would, thus, be useful if HTA agencies could clearly identify their HTA products, with respect to the commissioning group and purpose of the review along with some general details outlining the methodologies used in their preparation. In addition, policy makers need to appreciate that certain parts of a comprehensive review (such as an independent and complete economic evaluation) may not realistically be completed in a rapid timeframe. Furthermore, methods for incorporating the advice of expert panels in a timely manner need to be developed to ensure that rapid reviews ask the correct questions and reach appropriate conclusions at both clinical and policy levels.
A rapid review should be written in answer to specific questions, rather than as a quick alternative to a comprehensive systematic review. In this manner, rapid reviews could be used to inform specific policy decisions in a timely manner without losing any of the important information that may be expected from a comprehensive review. It is perhaps the focus on appropriate use, along with suitable methodologies, of a rapid review that requires future consideration.
CONTACT INFORMATION
Amber Watt, BMedSc, GDPH (amber.watt@surgeons.org, Research Officer, Alun Cameron, BSc, PhD (alun.cameron@surgeons.org), Senior Research Manager, Lana Sturm, BSc (Hons) (lana01@internode.on.net), Research Officer, Timothy Lathlean, BBehSc, BSc (Hons) (tim.lathlean@surgeons.org), Research Officer, Wendy Babidge, BAppSc (Hons), PhD (wendy.babidge@surgeons.org), Director, Research, Audit & Academic Surgery Division, Australian Safety and Efficacy Register of New Interventional Procedures–Surgical (ASERNIP-S), Royal Australasian College of Surgeons, 38 Payneham Road, Stepney, South Australia, 5069
Stephen Blamey, MB, BS, FRACS, FACS (sblamey@netspace.net.au), Chair, Medical Services Advisory Committee, Department of Health & Ageing, GPO Box 9848, Canberra, ACT 2601, Australia
Karen Facey, BSc (Hons), PhD (k.facey@btinternet.com), University of Glasgow, University Avenue, Glasgow G12 8QQ, UK; Evidence-Based Health Policy Consultant, Woodlands Lodge, Buchanan Castle Estate, Drymen G63 0HX, UK
David Hailey, MSc, PhD (dhailey@ozemail.com), Senior Advisor, Institute of Health Economics, 1200 10405 Jasper Avenue, Edmonton AB T5J 3N4, Canada
Inger Norderhaug, PhD (inger.norderhaug@kunnskapssenteret.no), Head of Section, Norwegian Knowledge Centre for the Health Services, Box 7004, St Olavs Plass N-0130, Oslo, Norway
Guy Maddern, PhD, FRACS (asernips@surgeons.org), RP Jepson Professor of Surgery, Discipline of Surgery, University of Adelaide, North Terrace, Adelaide SA 5000 Australia; Scientific Director, Australian Safety and Efficacy Register of New Interventional Procedures–Surgical (ASERNIP-S), Royal Australasian College of Surgeons, P.O. Box 553, Stepney, SA 5069 Australia
