Rapid review is an evidence synthesis approach that can be conceptualized along a continuum in contrast to that of systematic review (Reference Ganann, Ciliska and Thomas1). While the concept of rapid evidence synthesis, or rapid review, is not novel, it remains a poorly understood and yet ill-defined set of disparate methodologies (Reference Gough, Thomas and Oliver2) supported by a paucity of published, available scientific literature (Reference Khangura, Konnyu, Cushman, Grimshaw and Moher3;Reference Harker and Kleijnen4). Despite this, rapid review approaches have been used to a great extent in the social science and policy arenas to support evidence-informed decision making (Reference Thigpen, Puddy, Singer and Hall5–Reference Perry7). Increasingly, rapid review is being used in the health sciences, and in particular, the field of health technology assessment (HTA), to support informed decisions in a timely manner (Reference Hailey, Corabian, Harstall and Schneider8).
HTA is a multidisciplinary field of policy analysis that has been established in response to the need for evidence to support healthcare decision making (9). HTA producers and decision makers generally operate in very different environments, though, and are faced with competing priorities that require HTA producers to remain agile in addressing the ever-changing needs of decision makers. As described in a 2008 report by Drummond et al. (Reference Drummond, Schwartz and Jonsson10) a recent evolution has been observed among HTA products from those which inform a range of decision makers, to those which inform a specific resource allocation decision. The report emphasizes the importance of ensuring that HTA products are linked to demonstrable and measurable actions, as such is their primary raison d'être.
One consistently identified barrier to implementing the evidence from HTA products is an incongruence between the time required to produce a full HTA that considers both clinical- and cost-effectiveness data (e.g., many, many months) and the time within which policy and other decision makers must render decisions (e.g., often weeks) (Reference Menon and Stafinski11).
Rapid review is an emerging approach within the field of HTA to address these challenges (Reference Hailey12). While rapid review approaches are being used by HTA producers across the world in a variety of ways (Reference Harker and Kleijnen4;13) an agreed-upon definition describing its constitution and methods is lacking (Reference Khangura, Konnyu, Cushman, Grimshaw and Moher3). This gap seems particularly salient when considering the potential implications that widespread use of a little-understood approach to evidence synthesis (in particular, one that generally makes methodological tradeoffs as compared to that of systematic reviews) might have for the production of HTA reports that subsequently inform health policies (Reference Gerhardus and Dintsios14) affecting millions of patients. And while HTA producers must prioritize their mandate to answer the time-sensitive needs of the health decision makers they serve, they must simultaneously ensure that the scientific imperative of methodological rigor is satisfied. To adequately address this inherent tension, the need for development of methodological standards that explicitly account for rapid review limitations in juxtaposition with systematic review, and delineate optimal approaches for managing these, has been identified (Reference Abrami, Borokhovski and Bernard15).
Canada's national HTA agency, the Canadian Agency for Drugs and Technologies in Health (CADTH) has progressively developed a rapid evidence synthesis approach, producing over 2,000 reports designed to meet the needs and timelines of health decision makers across Canada. Drawing from this applied experience, this study aims to contribute to growing efforts to clearly delineate rapid review methods used in HTA by describing those developed and implemented by CADTH. Ongoing efforts to promote transparency and clarity in this emerging area by explicating the methodological tradeoffs adopted by rapid review approaches will be critical toward assessing the validity and utility of rapid review products.
BACKGROUND
Funded by the Canadian federal, provincial, and territorial governments, CADTH is mandated to produce secondary evidence that supports informed decisions about the acquisition and use of health technologies for the nation's health decision makers and care providers. CADTH offers a variety of HTA products that are subject to continuous improvement measures; designed to ensure that scientific rigor and transparency are paramount, and ensure that the needs of Canadian decision makers are being met with credible evidence.
In 2002, as part of its ongoing needs assessment of Canadian health decision makers, and after several years of sustained and increasing demand for more rapid delivery of HTA reports, CADTH identified the need for a service dedicated to producing evidence-based rapid reviews. Launched in February 2005, the objective of CADTH's Rapid Response Service was to produce reports in response to urgent and/or time-sensitive inquiries regarding the evidence on existing healthcare technologies, such as drugs, devices, surgical and nonsurgical procedures, and diagnostic and screening tests.
CADTH'S RAPID REVIEW APPROACH
Rapid Response Reports
In an effort to meet the variable needs of the decision makers, CADTH offers a range of rapid review products, called Rapid Response reports, that span a continuum of time-to-delivery correspondent with the depth and breadth of scientific inquiry required (16). For example, while a Reference List is completed in approximately 1 week, a Systematic Review and Meta-Analysis (SR and MA) requires six months for completion (Table 1). While the production of all Rapid Response report types adhere to a common underlying process, the depth and breadth of steps involved in carrying out the review work and preparing the report are variable across report types (e.g., additional steps and time are required to complete those reports that demand greater scientific rigor) (Table 2).
CADTH's Rapid Response Products
*Adapted from the CADTH Web site.
Overview of CADTH'S Approach to Rapid Review
SR/MA, systematic review/meta-analysis.
Rapid Review Process and Methods
CADTH receives requests for rapid reviews from healthcare decision makers at the federal, provincial, regional, and local levels and refines these in consult with the requestor by clarifying purpose, (e.g., purchasing, policy or coverage decisions), scope (i.e., PICOS: population, intervention, comparator, outcomes, study design (Reference Wright, Brand, Dunn and Spindler17)), and the deadline for report completion. This refinement phase includes establishment of the most appropriate Rapid Response report type, development of the research question or questions which subsequently inform the literature search strategy, and identification of study selection criteria. Of note, SRs and MAs are supplemented by development of a formal research protocol (PRISMA-P) that is then published on the International Prospective Register of Systematic Reviews (PROSPERO) Web site (18;19).
Literature Search Strategy
An Information Specialist at CADTH works with the research team to develop a search strategy in accordance with information gathered during the refinement phase. To accommodate the abbreviated timeline, searches for rapid reviews are generally less extensive than those executed for full systematic reviews and HTAs. Except for SRs and MAs, fewer databases are searched, date and language limits are applied, fewer sites are searched for grey literature, and a peer-review of the search strategy may not be required. Default search limits for all CADTH Rapid Response reports include: publication dates limited to five years, language of publication limited to English-only, conference abstracts excluded, and clinical trial registries not searched. These limits may be modified on a case-by-case basis during topic refinement, if deemed necessary by the research team based on the scope and subject area of the report.
There are also differences in the extensiveness of searches between Rapid Response report types. For Reference Lists, Summaries of Abstracts and Summaries with Critical Appraisals, MEDLINE (by means of PubMed or OVID) is the primary bibliographic database searched. For SRs and MAs, Embase is searched routinely in addition to MEDLINE. Additional subject-specific databases, such as CINAHL or PsycINFO, may be added for any Rapid Response search on a case-by-case basis, depending on the report's topic. Automatic database search alerts are established for SRs and MAs only, to ensure the search remains current during the review process.
For Reference Lists, Summaries of Abstracts and Summaries with Critical Appraisals, grey literature is searched using an abridged version of CADTH's Grey Matters checklist (20). The abridged list contains forty-one key Web sites, including foremost Canadian and international HTA and guideline producers. Approximately half of these sites are compulsory and the other half are only searched for topics specific to pharmaceuticals, or if guidelines are required. For SRs and MAs, selected sections of the full Grey Matters checklist are searched based on the topic and the types of resources required, and a grey literature update is completed 2 weeks before the report is due.
Like full HTAs and systematic reviews, search strategies for SRs and MAs are always peer-reviewed by a second Information Specialist; however, other Rapid Response search strategies do not require peer review unless deemed necessary by the research team due to a subject matter's complexity.
Upon execution of the electronic search strategy for all Rapid Response reports, a reference management software program is used to compile the results and prepare the yield for study selection.
Study Selection
Study selection criteria are established during delineation of the PICOS as part of the topic refinement phase and are linked directly to the research question(s). The criteria are subsequently applied to the search yielded by one researcher for Reference Lists, Summaries of Abstracts, and Summaries with Critical Appraisal, while two researchers independently review and select articles for inclusion in SRs and MAs with discrepancies addressed by discussion and consensus.
Data Abstraction
Once again, the approach to data collection is dependent on the report type. While Reference Lists and Summary of Abstracts are not subject to formal data collection from full-text articles, information reported in the abstracts-only is summarized for a Summary of Abstracts. As data contained in the abstracts do not always accurately reflect the data in the full-text article, a statement to this effect is included in all Summary of Abstract reports. Data abstraction for both Summary of Abstracts and Summary with Critical Appraisal reports is conducted by one reviewer.
Data abstraction becomes more in-depth for Summaries with Critical Appraisal, as study characteristics, findings of interest to the requestor and authors’ conclusions are abstracted from the full-text articles and tabulated. Data are abstracted by the lead researcher assigned to the report to inform the summative, narrative synthesis.
SRs and MAs are guided by a structured data abstraction framework, developed a priori, informed by the research question(s), and subjected to duplicate data extraction with discrepancies between researchers resolved by discussion and consensus.
Evidence Synthesis
Once the included studies have been identified and their relevant data abstracted, the process of evidence synthesis follows. The depth of evidence synthesis will, again, depend on the level of report requested and the identified needs of the requestor; there is a balance between rigor and timeliness. Summaries with Critical Appraisal undergo a synthesis of the individual studies, while SRs and MAs may include a meta-analysis of quantitative data.
CADTH's narrative synthesis of the individual studies generally considers the primary study results, in particular the stated a priori outcomes of interest, and describes these in comparison with one another and in the context of the research question(s). This portion of the synthesis is both an art and a science as the researcher must consider the particular relevance of the data at hand to inform and support the purpose of the request, while making best efforts to limit bias (e.g., by working within an interdisciplinary research team and subjection of all Rapid Response reports to internal review) (Reference Tricco, Tetzlaff and Moher21).
In addition to a synthesis of the individual studies, SRs and MAs pool quantitative study findings, where appropriate. To determine if a meta-analysis is appropriate, clinical and methodological heterogeneity within the selected studies are assessed using standard methods (Reference Higgins and Green22). If the studies are deemed to be homogeneous, a meta-analysis is performed.
Critical Appraisal
With the exception of Reference Lists and Summaries of Abstracts, included studies in CADTH Rapid Response reports are critically appraised to systematically evaluate their strengths, limitations, reliability, validity and the relevance of their methods and findings. The research team selects a validated instrument that can be used appropriately with the studies included in the review (e.g., SIGN50 (23), Downs and Black (Reference Downs and Black24), QUADAS (Reference Whiting, Rutjes and Westwood25), AMSTAR (Reference Shea, Grimshaw and Wells26), and AGREE (Reference Brouwers, Kho and Browman27)). While studies in Reference Lists and Summaries of Abstracts are not critically appraised, they are arranged based on the hierarch of evidence (i.e., SRs and MAs, randomized controlled trials, etc.).
Report Production and Dissemination
Once the data have been assembled, the Rapid Response report is compiled in accordance with a standardized template and author guidelines to ensure consistency. All draft reports undergo internal review by either the clinical research manager or a CADTH researcher and a reference check by an Information Specialist. SR and MA draft reports undergo both an internal review and external peer-review process by content and/or methodological experts. External reviewers are identified according to their expertise specific to the research topic; industry feedback is also sought and considered for incorporation into the final report.
All reports contain a legal disclaimer that explicitly states the intent of the report, provides suggestion for considering other information sources and indicates that no guarantee of any effect is suggested by development and delivery of the report.
Finalized reports are submitted electronically to the original requestor and dissemination efforts that go beyond the original requestor include a monthly topic list of Rapid Response reports, in addition to copies of individual reports made publicly available on the CADTH Web site. This practice contributes to efforts that seek to mitigate duplicate work, by ensuring reports are accessible to other decision makers, both in Canada and internationally.
In addition, CADTH publishes Rapid Response reports with no studies eligible for inclusion, also known as “empty reviews.” Given the variation in how “empty reviews” are reported, Yaffe et al. recommend that guidelines be developed on the reporting of empty reviews (Reference Yaffe, Montgomery, Hopewell and Shepard28). The “empty reviews” by CADTH state that no relevant evidence that met the selection criteria was identified and are posted to the public Web site, to ensure transparency in reporting and avoiding publication bias.
RESULTS: OBSERVATIONS AND CASE STUDIES
CADTH's Rapid Response reports have received consistent pull from Canadian decision makers, with hundreds being requested each year. While a formal program evaluation of CADTH's Rapid Response Service remains ongoing and will be reported in detail elsewhere, Supplemental Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462313000664, offers a preliminary look at the number and type of reports produced annually between 2005 and 2013. Figure 1 illustrates the total number of Rapid Response reports completed and the number of reports specific to drug technologies during the same period. To briefly demonstrate pragmatic examples of the utility of CADTH's rapid review products, the case studies in Table 3 describe the development and implementation of evidence from two Rapid Response reports in real-world healthcare settings in Canada.
Two Case Studies that Illustrate the Impact of Rapid Response Reports
Number of Rapid Response Reports on Non-Drug Technologies and Total Reports between 2005 and 2013
DISCUSSION
Rapid review is emerging as an important approach to HTA that supports health decision makers and care providers contending with competing demands, arduous time constraints, and the need for relevant and rigorous evidence products. But the tension produced by these competing demands presents a unique challenge for HTA producers like CADTH and others (e.g., EUnetHTA's HTA Core Model for Rapid Assessment and the Cochrane Response Rapid Review). Despite the apparent relevance and utility of rapid review products, it cannot be ignored that they inherently harbor limitations (Reference Perry7) that demand careful consideration in assessing the nature and extent of that utility (e.g., restricted literature searches, narrow foci, lack of economic modeling and budget impact analysis, and limited generalizability).
Interest in these and other axiomatic differences between rapid reviews and systematic reviews is growing, yet challenges to performing comparative analyses begin with the lack of an agreed-upon definition for rapid review. But, to what extent can an approach like rapid review be either broadly or narrowly defined considering its range of approaches (Reference Abrami, Borokhovski and Bernard15;Reference Cameron, Watt, Lathlean and Sturm29)? CADTH's Rapid Response reports are demonstrative of this range, and while description of this kind of a typology and underlying methodologies is useful in considering extant approaches to rapid evidence synthesis, rapid review may better be conceptualized as a multiplicity of approaches that aspire to the scientific rigor of systematic review only insofar as their timelines will allow; making methodological tradeoffs as and where deemed appropriate.
Extending from this, then, questions arise concerning which tradeoffs in rapid review are appropriate; at what times, and; in which contexts? And perhaps most importantly: what are the downstream effects of these tradeoffs, especially with regard to decision making and development of health policy? And how do these effects compare with those that have limited the value of full HTAs and systematic reviews to health decision makers who require evidence quickly to inform urgent decisions (Reference Menon and Stafinski11)? These and other questions around the implications of truncating systematic review's methodology remain largely unanswered.
Nonetheless, efforts have been made to begin addressing these questions by drawing comparisons between rapid review and systematic review: an Australian Safety and Efficacy Register of New Interventional Procedures – Surgical (ASERNIP-S) current practice report on the use of rapid review in HTA (Reference Cameron, Watt, Lathlean and Sturm29), and a study by Watt et al. (Reference Watt, Cameron and Sturm30) both attempted to compare and contrast whether the essential conclusions of rapid and systematic reviews differed to any significant degree. Remarkably, both reports concluded that, while full systematic reviews contain a greater degree of depth and detail, the overall conclusions of those rapid reviews and systematic reviews included in their studies were not significantly different. The authors of both reports note that, while the exhaustive data included in full HTAs, systematic reviews and meta analyses may be of critical importance to supporting micro-level clinical decisions, these may be of lesser importance in certain macro-level policy decisions; suggesting that directionality of an effect rather than precise point estimates of that effect may be sufficient, at least some of the time. This suggestion emphasizes the importance of having a clear understanding of the audience, policy context and intended use of the HTA product in advance of carrying out the work needed to develop and answer the question (Reference Cameron, Watt, Lathlean and Sturm29).
In accordance with this kind of contextualized approach, Thomas et al. also recognize the challenges that timelines present to completion of a systematic review that must meet social policy-making deadlines (Reference Thomas, Newman and Oliver31). The authors propose a range of approaches to reviewing the literature according to timelines, resources and requirements (e.g., a limited research team could conduct a standard systematic review if resources are limited, time is not an issue). The research team, therefore, would have the opportunity to carefully consider the policy context and issues and interpret the study findings (Reference Thomas, Newman and Oliver31).
CONCLUSIONS
The current presents describes one approach to rapid review within an HTA context. While there is no consensus yet on the definition or methodology for rapid review, ultimately, HTA producers and other evidence synthesis experts must continue to contend with how best to provide health systems, their multiple, various stakeholders and needs, with the evidence required to optimize patient health outcomes. Rapid review is one approach that is emerging to bring evidence to the fore of healthcare decision making in a way that is often considered to be timely and relevant, but it comes at a price of methodological trade-offs as compared to systematic reviews. As such, the development, evaluation, refinement, and transparent reporting of rapid review methods is imperative; vigilance will be necessary on the part of evidence synthesis methods experts, and methodological transparency on the part of HTA producers will be of paramount importance.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462313000664
CONTACT INFORMATION
Sara Khangura, BA, MPH(c), Research Manager, University of Ottawa, Ottawa, Canada
Julie Polisena, MSc (juliep@cadth.ca), Department of Epidemiology & Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada; Scientific Advisor, Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada
Tammy J. Clifford, PhD, Adjunct Professor, Department of Epidemiology & Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada; Vice President, Strategic Initiatives and Chief Scientist, Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada
Kelly Farrah, MLIS, Information Specialist, Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada
Chris Kamel, BSc, MSc, Clinical Research Manager, Canadian Agency for Drugs and Technologies in Health, Ottawa, Canada
CONFLICTS OF INTEREST
All authors are employed at CADTH; otherwise they report not having potential conflicts of interest.
