Highlights
What is known?
-
• The methods, or approaches, used to design and run searches have not changed substantially over the last 25+ years.
-
• Search experts have provided a lot of searching guidance through methodological handbooks and online web guides.
-
• Alternate approaches have been proposed that may improve search conduct through efficiency gains.
What is new?
-
• The long-established practice of searching is still a suitable approach; however, searching could be made more efficient by integrating some new approaches into current practice.
-
• Search experts now know which approaches they use or recommend that are backed by evidence and those which are lacking in evidence.
-
• Four approaches were identified that may make search conduct more efficient; however, additional robust studies to verify their benefit are needed.
Potential impact for RSM readers
-
• This review assesses and quantifies the impact different search approaches have on the recall (relevant studies found), the precision (number of irrelevant studies found), and the time required to conduct a search.
-
• This will enable systematic reviewers, or search designers, to identify potentially more efficient approaches when conducting their searches.
-
• This can lead to a productivity saving as a search that still finds all relevant studies, but fewer irrelevant studies, means a reduction in the amount of screening required.
1 Background
Searches to find relevant studies for inclusion in systematic and scoping reviews have been conducted using a standard search approach (SSA) for many years.Reference Higgins, Thomas, Chandler, Cumpston, Li and Page1 This approach aims to find all the relevant studies addressing a specific question with a search of multiple bibliographic databases (e.g., MEDLINE and Embase) with or without a check of reference lists of included studies. The databases are searched using a combination of words and subject terms, known as a search string.Reference Clark, Sar and Williams2 This search string is designed by the searcher by identifying the review concepts to be searched along with the relevant text words and controlled vocabulary for those concepts (often in consultation with a subject expert). The searcher combines the terms and concepts using Boolean operators. The search string is then translated to run in other databases.Reference Lefebvre, Manheimer, Glanville, Higgins and Green3– Reference MacFarlane, Russell-Rose and Shokraneh7
The SSA can be time-consuming, and quality is reliant upon the skill of the search specialist. For an experienced search specialist, the average time to conduct a search using the SSA is 13 hours, but it can take 50 hours or even longer.Reference Bramer, Rethlefsen, Mast and Kleijnen8 Further, the SSA can identify large numbers of irrelevant studies, which creates a large screening burden. The median precision of searches in reviews using the SSA is 0.93%.Reference Borah, Brown, Capers and Kaiser9 A precision of 0.93% equates to a number needed to read (NNR) of 108, meaning 108 records need to be screened to identify one relevant study.
Alternate approaches to searching for studies for inclusion in a review have been developed, with the aim of improving the efficiency of database search string formulation and minimising screening burden. Some of these approaches are modifications of the standard approach, some add components to it, and some aim to replace the SSA altogether. The comparative performance of these alternate search approaches (ASAs) and the SSA has not been systematically evaluated. To inform current searching practice, we conducted a systematic review of studies comparing ASAs to the SSA used to identify studies for inclusion in systematic reviews.
2 Methods
2.1 Overview
We conducted a systematic review of studies comparing ASAs to the SSA for identifying studies for inclusion in a healthcare-related systematic or scoping reviews. The review is reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines.Reference Page, McKenzie and Bossuyt10 The PRISMA checklist is available in Supplement 1 of the Supplementary Material.
The protocol for this review was registered on 13 October 2020 and is available via the Open Science Framework (OSF): https://osf.io/dct9m/.
2.2 Eligibility criteria
We included studies:
-
• Comparing ASAs and SSAs in five or more healthcare-related review searches.
-
• Comparing the ASA to the SSA where the SSA search results were obtained from a search of two or more bibliographic databases, +/− a check of the reference list. Studies including an SSA that performed searches in two or more databases but analysed the results from only one of the database searches (e.g., PubMed search only) were also included.
-
• Evaluating an ASA accessible to and usable by most searchers, e.g., a study evaluating an ASA that uses simple, freely available software was considered eligible, while a study evaluating complex computer code not available or understandable by most searchers was excluded.
-
• Evaluating ASAs that are not variations of the SSA, e.g., citation searching is an ASA and would be included, while varying the combinations of bibliographic databases is a variation of the SSA and was excluded.
-
• Comparing the results of a search conducted using the ASA to the results of a search conducted using the SSA. Studies comparing the results of a search conducted using the ASA to the studies included in a review without ensuring the studies included in the review were found by the SSA were excluded.
-
• Reporting at least one of the outcomes of interest: (1) recall (number of relevant studies found); (2) precision (number of irrelevant studies found); or (3) the time required to design and/or run a search.
-
• Written in any language, peer reviewed, and published in an academic journal.
2.3 Search strategy
To identify relevant studies, we searched PubMed (via the National Library of Medicine) and the Library, Information Science and Technology Abstracts (LISTA) (via EBSCO) databases on 20 September 2023, and the search was then updated on 22 January 2025. This was supplemented by browsing the studies in Summarized Research in Information Retrieval for HTA (SuRe Info) website11 on 19 March 2019, conducting a single-iteration backwards and forwards citation search conducted on 13 October 2023 (using the included studies from the 20 September 2023 search), and screening the reference lists of other reviews (conducted on 3 March 2020). To develop the search strings for our database searches, we obtained seed studies by screening for eligibility of the studies listed in the SuRe Info website.11 We also obtained seed studies by screening the reference lists of known relevant reviewsReference Cooper, Booth, Britten and Garside12, Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre13 and the searching chapter of the Cochrane Handbook.Reference Higgins, Thomas, Chandler, Cumpston, Li and Page1 An experienced information specialist (JC) used these seed studies to design a PubMed search string, which was translated to be used in the LISTA database using the Polyglot Search Translator,Reference Clark, Sanders and Carter14 although the PubMed MeSH terms were manually converted to the appropriate LISTA subject terms. A backwards and forwards citation search of the included studies was conducted following the TARCiS guidance statementReference Hirt, Nordhausen, Fuerst, Ewald and Appenzeller-Herzog15 using the SpiderCite tool from the Systematic Review Accelerator (SRA).Reference Clark, Glasziou, Del Mar, Bannach-Brown, Stehlik and Scott16 The full search strings for each database are in Supplement 2 of the Supplementary Material.
2.4 Study selection
The search results were deduplicated using the ‘focused’ algorithm, combined with a manual review, in the Deduplicator, which is available via the SRA.Reference Forbes, Greenwood, Carter and Clark17 The deduplicated results were then screened using the Screenatron tool in the SRA.Reference Clark, Glasziou, Del Mar, Bannach-Brown, Stehlik and Scott16 This was done independently by the first author (J.C.) paired with the other authors (C.F., L.-F.K., and S.S.) against the eligibility criteria, initially via title and abstract, then by full text with discrepancies resolved through discussion.
2.5 Data extraction
A data extraction form was designed, piloted on five studies, and revised. We extracted details of the study design, the ASA evaluated, the SSA, and the recall, precision, and time required to conduct the search by each approach. Data were extracted independently by the first author (J.C.) paired with the other authors (E.B., C.F., L.-F.K., and S.S.), with discrepancies resolved through discussion.
2.6 Risk-of-bias (RoB) assessment
We assessed the risk of bias (RoB) for each included study using a modified version of the Quality Assessment of Diagnostic Accuracy Studies tool version 2 (QUADAS-2).Reference Whiting, Rutjes and Westwood18 QUADAS-2 is a tool for assessing bias and applicability concerns in studies of diagnostic accuracy. This tool was selected based on the similarities between the studies evaluating diagnostic tests and studies evaluating search performance. Major modifications were made to the wording of the domains to suit this review. Comparisons between the original QUADAS-2 and the modified version, and the guidance for using our modified QUADAS-2 are available in Supplement 3 of the Supplementary Material.
The modified QUADAS-2 tool assesses RoB arising from:
-
• Domain 1A: Search strategy selection, e.g., were the search strategies used in the evaluations randomly selected?
-
• Domain 2A: ASA conduct, e.g., was the ASA designed by searchers without knowledge of the results of the SSA?
-
• Domain 3A: SSA conduct, e.g., did the study investigators compare the ASA to the results of the complete SSA, e.g., were the results of searching all databases comprising the SSA used in the evaluation, or did they use the search results of only searching one of the databases?
-
• Domain 4: Differences between the ASA and SSA, e.g., were the skill and experience levels of the searchers conducting the ASA and the SSA similar?
The modified QUADAS-2 tool assesses applicability and usability across the included studies, with the domains covering the following:
-
• Domain 1B: Applicability, e.g., can the study findings be applied to all review types?
-
• Domain 2B: ASA usability, e.g., can the ASA be easily used by review teams?
-
• Domain 3B: SSA usability, e.g., can the SSA be easily used by review teams?
Each domain was rated as low, high, or unclear. RoB assessment was conducted independently by the first author (J.C.) paired with the other authors (C.F., L.-F.K., and S.S.) against the eligibility criteria with discrepancies resolved through discussion. RoB in one included study authored by members of the review team was assessed by an external reviewer (i.e., a person not involved with this review or an author on the study), and a reviewer who was not an author.
2.7 Synthesising the data
Characteristics of the included studies were tabulated, with the ordering of presentation based on where in the search process each ASA is used, following a hierarchical framework of search approaches.Reference Clark, Beller, Glasziou and Sanders19
We planned to meta-analyse data from studies comparing the SSA with similar ASAs, however, due to large variations in ASAs, meta-analysis was not considered appropriate. Instead, study findings were synthesised without meta-analysis in text and tables.
We extracted and calculated the following outcome measures:
Recall: a measure of whether all the available evidence has been found.
• Recall = relevant reports found by the search/total number of relevant reports (reports included in the review)
Recall is expressed as a percentage, e.g., if a search strategy finds 8 relevant reports, from a pool of 10 relevant reports its recall is 80%.
Mean difference (%) in recall was calculated as: ASA recall−SSA recall. Because a higher recall is better, a positive mean difference in recall indicates better performance of the ASA.
NNR: a measure of the precision of the search or the number of irrelevant records found by the search.
-
• Number needed to read (NNR) = total records found/found relevant reports
NNR is expressed as a number, e.g., if there a search finds 1,000 records and in the search results there are 10 relevant reports, the NNR is 100. This means that 100 records need to be read to find a single relevant study.
Mean difference in the NNR was calculated as: ASA NNR−SSA NNR. Because a lower NNR is better, a positive mean difference in NNR indicates better performance of the SSA.
Time: a measure of the time taken to design and/or run a search expressed in minutes. We reported the time taken to design and to run the searches in combination, and separately when available.
Mean difference in time was calculated as: ASA time−SSA time. Because a lower time is better, a positive mean difference in time indicates better performance of the SSA.
If an ASA improved on the SSA across all outcomes, then it would have a higher recall, a lower NNR, and a lower time.
We present study findings according to whether an ASA was: (1) judged to potentially improve search conduct; (2) judged to potentially make search conduct worse; or (3) the impact on search conduct was too difficult to judge.
These judgements were based on our consideration of the relative importance and impact on search conduct of the outcomes reported by the study or studies evaluating that ASA.Reference Boluyt, Tjosvold, Lefebvre, Klassen and Offringa20 When determining whether an ASA improved or worsened search results, we attributed the most importance to recall, with the NNR and time being secondary considerations, e.g., (1) if an ASA improved recall and made a minor impact on the NNR or time, we determined it improved search conduct; (2) If the recall for the ASA and SSA was similar, we considered changes to the NNR and time to make a determination; (3) if an ASA improved recall but made the NNR or time significantly worse, we determined the ASA resulted in worse search conduct. These judgements regarding whether a search is considered ‘similar’, or ‘significantly worse’ are based on the searching knowledge and experience of the search specialist authors on the review (JC, EB, and SS).
3 Results
3.1 Results of the search
We included 14 studies after screening 3,520 unique records retrieved from a searching research website, checking reference lists of known reviews and a book chapter, as well as bibliographic database and citation searches (Figure 1).Reference Moher, Liberati, Tetzlaff and Altman21 A full list of studies excluded at the full-text screening stage is available in Supplement 4 of the Supplementary Material.
PRISMA flowchart of included studies.
Figure 1. Long description
The flowchart is organized into three vertical phases: Identification, Screening, and Included.
Phase 1: Identification
* Left Track (Databases and Registers): 2173 records identified (PubMed n = 1831, L I S T A n = 342). 93 duplicate records were removed before screening.
* Right Track (Other Methods): 1635 records identified (Websites n = 165, Citation searching n = 1470). 195 duplicate records were removed before screening.
Phase 2: Screening
* Left Track: 2080 records screened, leading to 2022 records excluded. 58 reports were sought for retrieval, and all 58 were retrieved. 58 reports were assessed for eligibility, resulting in 47 exclusions. Reasons for exclusion include: Inadequate data (n = 2), Approach unusable (n = 7), Wrong comparator (n = 10), Wrong intervention (n = 7), Wrong outcomes (n = 8), Wrong population (n = 3), Wrong pub. type (n = 1), and Wrong study type (n = 9).
* Right Track: 12 reports were sought for retrieval, and all 12 were retrieved. 12 reports were assessed for eligibility, resulting in 9 exclusions. Reasons for exclusion include: Approach unusable (n = 1), Wrong intervention (n = 3), Wrong outcomes (n = 2), Wrong sample size (n = 1), and Wrong study type (n = 2).
Phase 3: Included
* Both tracks converge at the bottom. A total of 14 studies were included in the review, corresponding to 14 reports of included studies.
3.2 Characteristics of included studies
Of the 14 included studies, one was a randomised controlled trial,Reference Clark, Sanders and Carter14 and the remaining 13 were non-randomised comparative studies.Reference Bramer, Giustini and Kramer22– Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 The single trial randomised searchers to translate database search strings either manually or using an automatic translation tool. In the non-randomised studies, investigators either reran the SSA or used the search results from SSAs reported in published reviews. The SSA was either rerun in full (e.g., all the database searches were rerun) or in part (e.g., only the PubMed search was rerun). Either the reported results or the results of the rerun SSA were compared with the search results of the ASA, which was performed at a similar time, or with the search adjusted to account for the time difference. The included studies compared search approaches used to identify studies for a range of review types: eight studies evaluated search approaches for any type of review,Reference Clark, Sanders and Carter14, Reference Bramer, Giustini and Kramer22, Reference Bramer, Giustini, Kramer and Anderson23, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29, Reference Wang, Scells, Clark, Koopman and Zuccon32, Reference Wang, Scells, Koopman and Zuccon33 three for diagnostic test accuracy reviews,Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 two for intervention reviews,Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31 and one for a drug intervention review.Reference McDonagh, Thakurta and Peterson28 Thirteen of 14 (93%) studies reported recall,Reference Bramer, Giustini and Kramer22– Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 10 (71%) studies reported precision,Reference Bramer, Giustini and Kramer22, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24– Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31– Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 and three (21%) reported on the time required to conduct the SSA and ASAReference Clark, Sanders and Carter14, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 (Table 1).
The included studies evaluated a range of ASAs. Three studies evaluated ‘using seed studies to design a search’.Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 ‘Searching Google Scholar’Reference Bramer, Giustini and Kramer22, Reference Bramer, Giustini, Kramer and Anderson23 and ‘using diagnostic study search filters’Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 were evaluated by two studies each. Single studies evaluated ‘simple searches combined with the PubMed similar articles’Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31; ‘using search words only, no subject terms, in a search’Reference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26; ‘using subject terms only, no search words, in a search’Reference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26; ‘automation of search string translation’Reference Clark, Sanders and Carter14; ‘contacting manufacturers’Reference McDonagh, Thakurta and Peterson28; ‘adding backwards citation searching’Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30; ‘adding backwards and forwards citation searching’Reference Wang, Scells, Clark, Koopman and Zuccon32; ‘using backwards and forwards citation searching only’Reference Wang, Scells, Clark, Koopman and Zuccon32; ‘generative AI–derived searches’Reference Wang, Scells, Koopman and Zuccon33 (Table 1).
3.3 Risk of bias
All 13 (100%) of the non-randomised studies were at high or unclear RoB in at least one domain. The single randomised trial was judged to be at low RoB (as many of the review team were authors on this trial, an external reviewer conducted the assessment with a member of the review team who was not an author on this trial). There were applicability concerns for sevenReference Bramer, Giustini and Kramer22, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference McDonagh, Thakurta and Peterson28, Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 of the 14 included studies (50%), and there was concern about usability of the ASA approaches evaluated in two studies.Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25 From the 14 included studies, sevenReference Bramer, Giustini and Kramer22– Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 had high or unclear RoB in Domain 1A: search strategy selection, mostly due to the studies using a convenience rather than random sample. FourReference Bramer, Giustini and Kramer22, Reference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26, Reference McDonagh, Thakurta and Peterson28, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31 had high or unclear RoB in Domain 2A: conduct of the ASA, generally caused by searchers knowing the results of the SSA before running the ASA. Ten studiesReference Bramer, Giustini, Kramer and Anderson23– Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29, Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30, Reference Wang, Scells, Clark, Koopman and Zuccon32– Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 had high RoB in Domain 3A: conduct of the SSA, due to use of the results obtained by only rerunning a single database search, rather than the results obtained by rerunning searches of all the databases comprising the SSA. Three studiesReference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Wang, Scells, Clark, Koopman and Zuccon32, Reference Wang, Scells, Koopman and Zuccon33 were at high RoB in Domain 4: differences between the ASA and SSA, due to significant skill differences between the searchers conducting the ASA and SSA searches. SevenReference Bramer, Giustini and Kramer22, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference McDonagh, Thakurta and Peterson28, Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30, Reference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 had high or unclear concern in Domain 1B: applicability, due to small numbers of searches used to compare the ASA and SSA. TwoReference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25 had concerns around Domain 2B: ASA usability, due to needing highly skilled search experts and all studies had low concern in Domain 3B: SSA usability (Figures 2 and 3).
Characteristics of included studies by alternate search approach evaluated
Table 1. Long description
The table consists of six columns: Study I D, Study design, Alternate search approach (A S A) description, Standard search approach (S S A) description, Review type evaluated, and Reported outcomes. The data is grouped by the following thematic categories:
* Using seed studies to design a search: Includes Hausner 24, Hausner 25, and Paynter 29. These studies compare Ovid M E D L I N E or PubMed searches designed with pre-identified articles against Cochrane or published review searches. Outcomes include recall, precision, and time.
* Using diagnostic study search filters: Includes Leeflang 27 and Whiting 34. Both use PubMed or Ovid M E D L I N E searches with diagnostic filters compared to standard published review searches for diagnostic test accuracy reviews.
* Adding backwards citation searching: Preston 30 compares database searches plus reference list checking against standard H T A review searches.
* Automation of search string translation: Clark 14 evaluates a randomized controlled trial using an automation tool for translation versus manual translation, measuring time.
* Searching Google Scholar: Bramer 23 and Bramer 22 compare Google Scholar results (limited to 1,000 or unlimited) against PubMed or Embase plus M E D L I N E.
* Using simple searches plus similar articles: Rice 31 compares minimal synonym searches combined with PubMed similar articles against multi-database searches.
* Using search words only: Leblanc 26 compares title/abstract word searches against MeSH terms plus search words.
* Using subject terms only: Leblanc 26 compares MeSH-only searches against standard PubMed searches.
* Using backwards and forwards citation searching only: Wang 32 compares the SpiderCite automation tool against searches by information specialists.
* Generative A I-derived searches: Wang 33 compares ChatGPT zero-shot prompts against standard database searches.
* Adding backwards and forwards citation searching: A second entry for Wang 32 compares specialist searches plus SpiderCite against specialist searches alone.
* Contacting manufacturers: McDonagh 28 evaluates adding pharmaceutical manufacturer contact to standard grey literature and database searches for drug intervention reviews.
Risk of bias and applicability per study. ASA, alternate search approach; SSA, standard search approach.
Figure 2. Long description
The table is divided into two main sections: Risk of Bias (four domains) and Applicability Concerns (three domains). Ratings are indicated by colored circles: red with an X for High, green with a plus for Low, and yellow with a minus for Unclear.
Column Headers:
* Risk of Bias: D 1 A: Search selection, D 2 A: A S A conduct, D 3 A: S S A conduct, D 4: A S A / S S A differences.
* Applicability Concerns: D 1 B: Applicability, D 2 B: A S A usability, D 3 B: S S A usability.
Data Rows by Category:
* Using seed studies to design a search:
- Hausner 2015: High, Low, High, Low | High, High, Low.
- Hausner 2016: High, Low, High, High | Low, High, Low.
- Paynter 2021: Low, Low, High, Low | Low, Low, Low.
* Using diagnostic study search filters:
- Leeflang 2006: Low, Low, High, Low | High, Low, Low.
- Whiting 2011: Unclear, Low, High, Low | High, Low, Low.
* Adding backwards citation searching:
- Preston 2015: High, Low, High, Low | High, Low, Low.
* Automation of search string translation:
- Clark 2020: Low, Low, Low, Low | Low, Low, Low.
* Searching Google Scholar:
- Bramer 2013: Unclear, Low, High, Low | Low, Low, Low.
- Bramer 2016: High, High, Low, Low | High, Low, Low.
* Using simple searches plus similar articles:
- Rice 2017: High, High, Low, Low | High, Low, Low.
* Using search words only, no subject terms, in a search:
- Leblanc 2024: Low, High, High, Low | Low, Low, Low.
* Using subject terms only, no search words, in a search:
- Leblanc 2024: Low, High, High, Low | Low, Low, Low.
* Using backwards and forwards citation searching only:
- Wang 2022: Low, Low, High, High | Low, Low, Low.
* Generative A I derived searches:
- Wang 2023: Low, Low, High, High | Low, Low, Low.
* Adding backwards and forwards citation searching:
- Wang 2022: Low, Low, High, High | Low, Low, Low.
* Contacting manufacturers:
- McDonagh 2018: Low, Unclear, Low, Low | High, Low, Low.
Risk of bias and applicability summary. ASA, alternate search approach; SSA, standard search approach.
Figure 3. Long description
A horizontal stacked bar chart with an X-axis ranging from 0 percent to 100 percent. A legend at the bottom identifies green as Low, yellow as Unclear, and red as High risk or concern.
From top to bottom, the domains are:
* D 1 A Search selection: approximately 55 percent Low, 15 percent Unclear, and 30 percent High.
* D 2 A A S A conduct: approximately 70 percent Low, 5 percent Unclear, and 25 percent High.
* D 3 A S S A conduct: 25 percent Low and 75 percent High.
* D 4 A S A or S S A differences: 75 percent Low and 25 percent High.
* D 1 B Applicability: approximately 58 percent Low and 42 percent High.
* D 2 B A S A usability: approximately 88 percent Low and 12 percent High.
* D 3 B S S A usability: 100 percent Low.
3.4 Alternate search approaches judged to potentially improve search conduct
Four ASAs were judged to potentially improve search conduct and could be considered for use by people designing searches: (1) using seed studies to design a search, (2) using a diagnostic search filter, (3) adding backwards searching and (4) automating search string translation.
The first of these ASAs, using seed studies to design a database search, was evaluated in three studies.Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 Seed studies are a set of pre-identified studies that meet or are close to meeting review eligibility criteria and are used to identify and test search terms that may be used in bibliographic database searches. Recall was improved with the ASA in two studies and was slightly worse than the SSA in another, and the NNR was similar (Table 2). Two studiesReference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 found that using seed studies saved time (Table 2). TwoReference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25 of the four studies were considered to be at high RoB due to the method used to select the searches used in the evaluation and threeReference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt24, Reference Hausner, Guddat, Hermanns, Lampert and Waffenschmidt25, Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 out of four were at high RoB arising from the conduct of the SSA.
The second ASA, using diagnostic search filters, was evaluated in two studies.Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 These studies found that using the best-performing filters resulted in similar recall (one study reported 0%, the other reported −4% mean difference), but improved NNR compared to the SSA (5 and 20 more articles needed to be read with the SSA) (Table 2). Both studies were judged to be at high RoB due to conduct of the SSA, and there were concerns regarding the applicability of the ASA to all types of reviews.
The third ASA involved adding backwards citation searching of the included studies to the results of the bibliographic database search. Evaluated in one study,Reference Preston, Carroll, Gardois, Paisley and Kaltenthaler30 the ASA showed improved recall compared to the SSA (MD +8%), but the study did not report the impact on NNR, so it is unclear what impact this ASA has on review workload (Table 2). This study was considered to be at high RoB due to selection of the searches, conduct of the SSA, and there was high concern regarding the applicability of the ASA to all review types.
The fourth ASA, the automatic translation of search strings, was evaluated in one study, a randomised trial. This study did not report on recall or NNR but found that using the automatic translation tool saved time while conducting searches (Table 2). Fourteen minutes were saved when search string translation was automated. This study was judged to be at low RoB on all domains.Reference Clark, Sanders and Carter14
3.5 Alternate search approaches judged to potentially make search conduct worse
Six ASAs were judged to potentially make search conduct worse. The first, evaluated in two studies,Reference Bramer, Giustini and Kramer22, Reference Bramer, Giustini, Kramer and Anderson23 involved searching Google Scholar instead of searching bibliographic databases. Recall with the ASA was similar or worse than the SSA, and the one study reporting the NNR found that searching Google Scholar alone was significantly worse than the SSA. In this study, the mean difference in NNR was +3,292, meaning 3,292 extra records need to be screened to identify a single relevant record (Table 2). Both studies were at high or unclear RoB arising from the method used to select the searches used in the evaluation. The other five ASAs that were judged to potentially make search conduct worse were using simple searches plus similar articlesReference Rice, Ali, Fitzpatrick-Lewis, Kenny, Raina and Sherifali31 as a replacement to bibliographic database searching; using search words only, no subject terms, in a searchReference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26; using subject terms only, no search words, in a search; using backwards and forwards citation searching onlyReference Wang, Scells, Clark, Koopman and Zuccon32; and using generative AI–derived searches instead of human derived searches.Reference Wang, Scells, Koopman and Zuccon33 The recall of all these ASAs was less than the recall of the SSA, sometimes substantially so, meaning, the ASA missed too many relevant studies to be usable (Table 2). Of the four studies evaluating these five ASAs, threeReference Leblanc, Hamroun, Bentegeac, Le Guellec, Lenain and Chazard26, Reference Leeflang, Scholten, Rutjes, Reitsma and Bossuyt27, Reference Whiting, Westwood, Beynon, Burke, Sterne and Glanville34 had high RoB arising from the conduct of the SSA.
Results of studies reporting average recall, NNR, and time by type of alternate search approach
Table 2. Long description
The table is divided into three main categories based on the impact of A S A.
1. A S A s that probably improve search conduct:
- Using seed studies: Hausner 24 and 25 show A S A recall of 96 percent and 97 percent respectively, outperforming S S A. Hausner 25 also reduced search time by 60 minutes.
- Paynter 29: S S A recall was higher at 92 percent versus 85 percent for A S A, but A S A saved 420 minutes.
- Diagnostic filters: Leeflang 27 and Whiting 34 show A S A s with lower N N R than S S A s.
- Citation searching: Preston 30 shows A S A recall of 93 percent versus S S A 85 percent.
- Automation: Clark 14 shows A S A saved 14 minutes in search string translation.
2. A S A s that probably make search conduct worse:
- Google Scholar: Bramer 22 shows that screening all results (G S all) leads to a massive N N R of 3,333 compared to S S A's 42.
- Simple searches: Rice 31 shows S S A had 100 percent recall while A S A had 83 percent.
- Subject terms only: Leblanc 26 shows a significant drop in recall to 36 percent for A S A compared to 78 percent for S S A.
- Generative A I: Wang 33 shows a very low A S A recall of 9 percent compared to S S A 78 percent.
3. A S A s impact difficult to judge:
- Wang 32: Adding citation searching increased recall to 93 percent but also increased N N R to 200.
- McDonagh 28: Contacting manufacturers provided a 7 percent mean difference in recall.
ASA, alternate search approach; SSA, standard search approach; Recall, the number of relevant studies found out of the total pool of relevant studies that could be found. Higher recall is better and a positive mean difference in recall means the ASA is better than the SSA; NNR, the number of articles that need to be read to identify a single relevant article; MD, mean difference. Lower NNR is better and a negative mean difference in NNR means the ASA is better than the SSA; Mean differences presented in bold indicates the ASA performed better than the SSA on that outcome; GS 1000, screening the first 1,000 search results returned by Google Scholar; GS all, screening all the search results returned by Google Scholar.
3.6 Alternate search approaches with difficult to judge search impact
There were two ASAs for which the data were incomplete or inconclusive with regard to the impact the ASA has on search performance. The first of these, evaluated in one study, involved contacting manufacturers.Reference McDonagh, Thakurta and Peterson28 This study investigated contacting pharmaceutical manufacturers and requesting additional studies to be added to the results of bibliographic database searching. This approach was found to improve recall (MD +7%) (Table 2), but the study did not report the NNR or the additional time required to contact manufacturers. This study was judged at unclear RoB due to the conduct of the ASA. There was also concern over the applicability of the ASA to all review types. The second ASA was adding backwards and forwards citation searching.Reference Wang, Scells, Clark, Koopman and Zuccon32 This study found improved recall (MD +7%) but worse NNR (MD +150 meaning 150 additional records had to be screened to identify one relevant record) (Table 2). This study had high RoB arising from how the SSA was conducted and differences between the conduct of the ASA and SSA. The difficulties in interpreting the results of these two studies are covered in more detail in the discussion.
4 Discussion
We included 14 studies comparing 12 ASAs to the SSA. Eight of the 12 ASAs evaluated were assessed in single studies only. Only one study evaluated an ASA using a randomised trial design. The other 13 studies were non-randomised comparative studies and were all judged to be at high RoB in at least one domain. Most of the studies evaluated the ASAs impact on recall and/or precision, with only three reporting on the time required to conduct the ASA. Four ASAs were judged to potentially perform better than the SSA, six were judged to potentially perform worse, and for two ASAs, it was not possible to determine their impact.
There are other summaries and reviews of alternative existing search approaches, but none are systematic reviews. A notable summary of research on searching is the SuRe Info website, which provides summaries of research but does not assess the quality of the research.11 One existing reviewReference Cooper, Booth, Varley-Campbell, Britten and Garside35 investigated search guidance in nine documents and found similarity in what constitutes an SSA. It also looked at search design research, but most of the evidence was opinion based and therefore not eligible for inclusion in our review. A different reviewReference Cooper, Booth, Britten and Garside12 on searching for qualitative studies found that the approaches currently used are only supported by case studies and poor empirical studies. They did not conduct quality assessment due to heterogeneity between the studies and the lack of a common appraisal instrument. They suggest contacting manufacturers for additional studies, handsearching specific journals, searching trial registers, and web searching, but none of the studies they used to make these recommendations met the eligibility criteria of our systematic review. Two relevant scoping reviews were also done as part of a PhD thesis.Reference Mathisen36 The first review in the thesis looked at the selection of search terms and development of simpler search strategies, and most (11/13) studies they found were not eligible for our review. Their review suggests that there are benefits from automatic search term selection and simple searches with small numbers of search terms. The second review in the thesis was about automated retrieval methods. They found 13 articles relevant to the topic: all 13 were ineligible for our review. They covered text mining, machine learning, and ranking algorithms and concluded that automated search approaches can’t replace the conventional Boolean search approach and that skilled human search specialists are still required; this aligns with our findings.
Our review has strengths and limitations. The primary strength is that we only included comparative studies that compared the results of the ASA and the SSA for at least one of three outcomes: recall, precision, or time. We identified many studies testing an ASA against the studies included in the original review, not the results of a search. These studies are likely to give a false measure of how effective the SSA is because it was assumed the SSA was perfect, with 100% recall. Another strength is that we conducted quality assessment of the included studies, something that is currently lacking in search evaluation. A limitation of the review is we could not obtain the precision around the estimates of effect from the data of many of our included studies, and so, meta-analysis of studies evaluating the same or similar ASAs was not possible. A further weakness is the quality of the studies included. Many of the included studies only compared the ASA to the PubMed search from the original review instead of all the databases and used a small number of searches in the evaluation. Finally, one of the ASAs evaluated in this review was designed and evaluated by many of the authors of this review. To minimise potential bias, the process of extraction and synthesis was prespecified and standardised, while the RoB assessment was conducted by a reviewer who was not an author on the original study and an external reviewer who was also not an author on the original study.
Our review findings could impact the design and conduct of future searching research. We searched multiple sources to identify research on best approaches to searching. We retrieved and manually assessed over 3,500 records, of which only a small number, 14, met our inclusion criteria. This suggests a current lack of robust evaluations of search approaches. This may be due to unavailability of funding for such research, as we could only identify a single study that appears to have been directly funded.Reference Paynter, Featherstone, Stoeger, Fiordalisi, Voisin and Adam29 Considering the importance of evidence identification in all research, not just evidence synthesis, this significant funding gap is of concern, and we suggest funding bodies look at the benefit that improved searches could have on research productivity. We also suggest designers of searching studies review the RoB form we used in this review. Then, when designing their studies, they attempt to ensure their methodology would lead to a low RoB in as many domains as possible.
The findings of our review can guide the future conduct of searching. Four ASAs showed results that were potentially beneficial: (1) using seed studies; (2) using study design filters; (3) adding backwards citation searching to bibliographic database searching; and (4) automation of search string translation. People designing searches, who currently do not use these ASAs, could examine whether they are suitable to be incorporated into their searching workflow. Six ASAs were judged to make searches worse and negatively impact on search performance. People designing searches, who are thinking of whether they should use these ASAs, should be able to now easily decide they are not worth adding into their searching workflows. Two ASAs looked promising, but on closer inspection they are probably not helpful in most cases. The first was contacting manufacturers for additional studies. Although the study found an improvement in recall, the group doing the contacting was the Drug Effectiveness Review Project (DERP), which is a collaboration of state Medicaid agencies. It is doubtful that drug manufacturers would respond at the same level to review teams, so it is unlikely review teams would see results as good as those obtained by DERP. The second ASA was adding backwards and forwards citation searching. Although the recall was improved, it was a similar improvement to adding backwards citation searching only. Also, the NNR was significantly worse and increased screening workload by a factor of four. Therefore, although this ASA looks beneficial, only doing backwards citation searching gets similar results with a lower NNR.
The findings of our review may be used to guide search design and inform the design of search evaluation studies. It has identified potentially promising ASAs, which provide a focus for further research in this area, as these are the ASAs that should be evaluated further in the future. The findings of this systematic review can help those conducting searches make evidence-informed decisions about the approach that best suits their needs.
5 Conclusion
We found that efficiency gains may be obtained from (1) using seed studies to design searches, (2) adding study design filters, (3) adding backwards citation searching to the results of bibliographic searching, and (4) using automation tools for translating searches, although additional and stronger evidence is needed before definitive recommendations could be made.
Acknowledgements
We would like to thank Dr. Natalia Krzyzaniak for conducting the risk-of-bias assessment on one of our included studies.Reference Clark, Sanders and Carter14 This enabled a study that most of the review team were authors on to be assessed by two reviewers who were not authors of that study.
Author contributions
Conceptualisation: J.C., E.B., S.S.; Data curation: J.C.; Formal analysis: J.C., E.B., S.S.; Investigation: J.C., E.B., C.F., L.F.-K., S.S.; Methodology: J.C., E.B., C.F., L.F.-K., S.S.; Project administration: J.C.; Supervision: E.B., S.S.; Validation: J.C., E.B., C.F., L.F.-K., S.S.; Visualisation: J.C.; Writing—original draft: J.C., E.B., S.S.; Writing—review and editing: J.C., E.B., C.F., L.F.-K., S.S.
Competing interest statement
J.C., E.B., C.F., and S.S. developed the search translation tool, the Polyglot Search Translator. The evaluation of the tool is one of the studies included in this article, which has three of the authors from this systematic review (J.C., E.B., and S.S.). The remaining author declares no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.
Data availability statement
The authors confirm that the data supporting the findings of this study are available within the article and its Supplementary Material.
Funding statement
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Supplementary material
To view supplementary material for this article, please visit http://doi.org/10.1017/rsm.2026.10112.
Open access
