What is already known?

• • Systematic reviews (SRs) are time-consuming and resource-intensive, with manual title and abstract screening acting as the primary bottleneck.

• • Existing automation tools and large language models (LLMs) show promise but historically struggle to achieve the near-perfect recall required for safe evidence synthesis.

What is new?

• • MechaScreener is a zero-shot, LLM-based screening tool that utilises a deterministic 1–5 ordinal relevance scoring mechanism based on Population, Intervention, Comparator, Outcome, Type of study (PICOT) criteria.

• • Across 15 SR datasets, MechaScreener achieved perfect recall (100%). In the 10 evaluation reviews (pooled record-level recall 95% CI: 0.98–1.00), it reduced the manual screening workload by an average of 61% (macro-average specificity 0.61, cluster bootstrap 95% CI: 0.46–0.75).

Potential impact for RSM readers

• • Researchers can deploy this tool as a highly reliable ‘first-pass’ filter to safely eliminate more than half of irrelevant literature without missing critical studies, allowing resources to be reallocated to full-text review and data extraction.

2.1. System architecture

MechaScreener development started in 2024 using a Node.js/TypeScript backend environment with a Vue 3 frontend. The system processes reference citations in batches, extracting the title and abstract and pairing them with a user-provided screening criteria string; formatted as Population, Intervention, Comparator, Outcome, and Type of study (PICOT) criteriaReference Speckman and Friedly ¹⁰ or general inclusion/exclusion rules.

The user-provided criteria are passed to the LLM verbatim within a fixed prompt template. MechaScreener performs no natural language processing, language standardisation, or rewriting of the user’s input; the only transformation is insertion of the criteria text and each reference’s title and abstract into the prompt template described below. Consequently, the clarity and completeness of the user-supplied criteria directly affect performance (see Section 4.2).

At the time of this evaluation, the underlying LLM was Qwen3-235B-A22B-Instruct-2507 (Alibaba’s Qwen series), accessed through the OpenRouter API. Because MechaScreener is model-agnostic, the underlying model can be substituted as newer models are released; the results reported here are specific to this model version. All queries used a temperature of 0 and a top_k of 1 with max_tokens set to 5, and the assistant response was prefilled with the token ‘Score:’ to constrain the output to a single ordinal value.

2.2. Prompting and scoring

The MechaScreener system utilises a structured prompt consisting of:

1. 1. System prompt: Defined instructions for the LLM to act as a screening assistant.

2. 2. Reference data: Title and abstract of the article.

3. 3. Criteria: The eligibility criteria for the specific review.

The verbatim system prompt and the full prompt template used in this study are reproduced in the Supplementary Material.

In the context of LLMs, temperature is a parameter that controls the randomness or creativity of the model’s output distribution, while top_k restricts the model to sample only from the k most probable next tokens. To ensure that the screening process is completely deterministic and highly reproducible for research purposes, we utilised a model configuration with a temperature of 0 and top_k of 1, effectively forcing the model to always select the single most likely response.

The model is instructed to provide a score from 1 (low relevance) to 5 (high relevance).

We emphasise that this 1–5 value is an ordinal relevance score rather than a calibrated probability: it has not been validated to correspond to a true numerical probability of inclusion, and it is used only to rank references and to set an inclusion threshold. The observed distribution of scores across each evaluation library is reported in the Supplementary Material.

2.3. Experimental design

The evaluation of MechaScreener was conducted in two phases:

• • Development dataset: Five control reference libraries were used to calibrate the prompt and scoring threshold aimed at maximising recall over specificity.

• • Evaluation dataset: Ten additional reference libraries were used for independent validation of the tool’s performance across different medical and social science domains (e.g., COVID-19, diabetes, and bone health). We included five SRs focused on randomised controlled trials (RCTs) and five SRs focused on non-randomised controlled trial (non-RCT) studies. All 15 reference libraries are available online in our GitHub repository.

The eligibility criteria prompts provided to the LLM were developed by three individuals with varying levels of SR experience (a novice reviewer, a PhD student, and an experienced reviewer). Each library was assigned to a single individual, who authored one eligibility-criteria prompt for that library; there was no process of merging or compiling multiple prompts per library. The single prompt used for each of the 15 libraries is therefore reproduced verbatim in the Supplementary Material. The exact eligibility criteria provided to the LLM for each library are detailed in the Supplementary Material. This file can also be used as a prompt guide for using MechaScreener. The 10 evaluation studies were sourced from previous Cochrane reviews. As title/abstract screening results are not available, we used the final full-text inclusion decisions to determine what the relevant studies were.

To compile the set of reviews for the evaluation, we reran searches from 10 randomly selected Cochrane reviews. The sampling frame was created by searching PubMed for Cochrane reviews using the following query:

 ((Interventions[Title/Abstract] OR Intervention[Title/Abstract])
 AND (randomized[Title/Abstract] OR randomised[Title/Abstract])
 AND "controlled trials"[Title/Abstract]
 AND "Cochrane Database Syst Rev"[Journal])

From this result set, 10 reviews were selected by generating a random number (using the Google random number generator) between one and the total number of search results found. For example, if 1,000 results were found, the random number was between 1 and 1,000. The search result that corresponded to this number was checked to see if it met our inclusion criteria of 1) having searched three or more databases; 2) reporting search strategies for all databases; 3) having an identifiable list of included studies; 4) having a minimum of 6 to a maximum of 50 included studies; and 5) for five of the reviews, including RCTs, and for the other five, including study types other than RCTs. Any references missing title or abstract data were excluded from the evaluation.

2.4. Metrics

The performance of MechaScreener is evaluated primarily using two metrics: recall (sensitivity) and specificity (selectivity).

Recall (or sensitivity) measures the proportion of actual relevant studies that were correctly identified by the tool. In the context of SRs, maximising recall is paramount to ensure that no critical literature is missed. It is defined as

(1) $$ \begin{align} \text{Recall} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}. \end{align} $$

Specificity (or selectivity) measures the proportion of irrelevant studies that were correctly excluded. A higher specificity indicates a greater reduction in the manual screening workload, as fewer irrelevant articles are passed to the human reviewers. It is defined as

(2) $$ \begin{align} \text{Specificity} = \frac{\text{True Negatives (TN)}}{\text{True Negatives (TN)} + \text{False Positives (FP)}}. \end{align} $$

Wilson Score Confidence Interval (95% CI) is used to calculate the confidence bounds for recall and specificity. Standard Wald confidence intervals fail mathematically when proportions reach 0% or 100% (as seen with perfect recall). The Wilson Score Interval is robust for these extreme values.Reference Wilson ¹¹ It is calculated as

(3) $$ \begin{align} \text{CI} = \frac{\hat{p} + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \end{align} $$

where $\hat {p}$ is the observed proportion, n is the total number of observations, and z is the z-score (1.96 for a 95% confidence level).

Pooling and estimands. We summarise performance across the evaluation libraries in two ways. The macro-average is the unweighted mean of the per-review proportions, treating each SR as the unit of analysis; this is the estimand reported as the ‘mean’ recall and specificity in the abstract and Section 3. The micro-average (pooled) statistic instead sums the true positives, false positives, true negatives, and false negatives across all reviews and computes a single proportion, treating each reference as the unit of analysis. Because records are clustered within reviews (and each review has its own criteria prompt), a micro-average Wilson interval treats the roughly 58,000 records as independent and therefore understates uncertainty at the review level. To account for this clustering, we additionally report a cluster bootstrap 95% CI for the macro-average, resampling whole reviews (with replacement, 10,000 iterations) so the interval reflects between-review heterogeneity. For recall, we further distinguish the record-level estimand (the proportion of relevant records identified) from the review-level estimand (the probability of zero missed inclusions in a new review), reporting the number of reviews in which recall was 1.00 together with a Wilson interval on that proportion.

2.5. Development results

Five control libraries were used for development to maximise recall (proportion of references correctly included) and specificity (proportion of references correctly excluded). These libraries were compiled based on previous reviews conducted at the Institute for Evidence-Based Healthcare (IEBH). The developmental results for these libraries are presented in Table 1.

Table 1.

Results across five development libraries

Table 1 Long description

The table consists of four columns and five data rows.

Column headers are: Library name, No. of Refs, Include studies Correctly included, and Exclude studies Correctly excluded.

Row 1: Antibiotic Prescribing and Telehealth. 934 references. 13/13 (100%) correctly included. 612/921 (66%) correctly excluded.

Row 2: Long Covid. 1,216 references. 16/16 (100%) correctly included. 552/1,200 (46%) correctly excluded.

Row 3: Natural History Primary Care. 5,913 references. 11/11 (100%) correctly included. 3,169/5,902 (54%) correctly excluded.

Row 4: Non Drug Interventions. 5,342 references. 35/35 (100%) correctly included. 3,972/5,307 (75%) correctly excluded.

Row 5: Salt Substitution. 889 references. 23/23 (100%) correctly included. 733/866 (85%) correctly excluded.

To maximise the recall score, the threshold was optimised to the highest possible score that still maintained a perfect recall of 1.0. An LLM was used to rank each article on a scale from 1 to 5. Empirically, setting the inclusion threshold in the development set at a score of two or higher and excluding only the articles that scored a one, achieved the target 1.00 recall while preserving a reasonable degree of specificity. This prioritised finding all relevant evidence so that the researcher can focus on making fine-grained decisions about what to exclude.

2.6. Evaluation dataset

The evaluation set for the performance of MechaScreener used 10 reference libraries that included a diverse range of topics shown below (five studies that were RCTs and five studies that were non-RCTs; Table 2) .

Table 2.

Overview of references and missing abstracts in evaluation libraries

Table 2 Long description

The table consists of three columns: Library name, Total references, and Missing abstracts. It is divided into two horizontal sections.

Section 1: R C T studies

* Balneotherapy for Chronic Venous: 338 total references, 11 missing abstracts.

* Calcium Vitamin D for Bones: 5,756 total references, 255 missing abstracts.

* Methylxanthine for Apnea: 2,781 total references, 334 missing abstracts.

* Phosphodiesterase 5 Inhibitors: 2,672 total references, 338 missing abstracts.

* Topical and Oral Steroids for OME: 4,292 total references, 583 missing abstracts.

Section 2: Non-R C T studies

* C O V I D-19 Measures in Schools: 2,162 total references, 21 missing abstracts.

* Diabetes as a T B Risk Factor: 5,177 total references, 165 missing abstracts.

* Falls Interventions: 18,098 total references, 1,095 missing abstracts.

* Topical Fluoride causing Dental Fluorosis: 11,829 total references, 2,527 missing abstracts.

* Sanitation for Prevention Diarrhoea: 10,770 total references, 297 missing abstracts.

4.1. Performance variance and specificity

While the recall remained stable at 100% across both the development and evaluation datasets, we observed notable variance in specificity depending on the SR topic and study design. This variance demonstrates that the LLM’s capacity to confidently discard irrelevant literature is highly dependent on the precision of the research question and the standardisation of the field’s terminology.

For instance, highly specific pharmacological interventions such as Phosphodiesterase 5 Inhibitors Reference Maltez, Maxwell and Rirash ²⁰ achieved a specificity of 0.91, effectively eliminating over 90% of irrelevant studies. Clinical trials in these domains typically feature highly structured abstracts and unambiguous PICOT elements, allowing the LLM to easily detect off-target papers.

Conversely, broader or more complex topics, particularly those involving non-RCTs like COVID-19 Measures in Schools Reference Littlecott, Krishnaratne and Burns ²² or Diabetes as a TB Risk Factor,Reference Franco, Bongaerts and Metzendorf ²³ exhibited lower specificities (0.21 and 0.39, respectively). These public health and observational fields often involve diverse study designs, varied outcome measures, and less structured reporting. When abstract information is vague, lacks explicit PICOT details, or touches on related thematic concepts without matching precise criteria, MechaScreener conservatively assigns a higher relevance score. This built-in conservatism prevents the accidental exclusion of relevant studies, correctly prioritising a perfect recall over maximal workload reduction when faced with ambiguity.

4.2. Limitations

Despite the robust performance, several limitations must be acknowledged. First, as noted in the methodology, the evaluation dataset’s ground truth for ‘relevant’ studies was based on final full-text inclusion decisions rather than initial human title and abstract screening decisions. Because abstracts often lack complete methodological details, an automated title and abstract screener may correctly flag a study as potentially relevant based on the abstract alone, only for it to be counted as a false positive because the study was ultimately excluded during the human full-text review phase. Consequently, the reported specificity may artificially underestimate the tool’s true specificity at the abstract screening level.

Second, the current methodology relies heavily on the availability of study abstracts. As highlighted in Table 2, thousands of references across the tested libraries (e.g., over 2,500 in the Topical Fluoride causing Dental Fluorosis library) lacked abstract data. Studies without abstracts are inherently difficult to screen automatically and often default to human review, reducing the overall workload savings.

Third, MechaScreener does not replace human screening. It functions strictly as a ‘first-pass’ filter, meaning that any results the tool flags as potentially relevant still need to be manually screened by a human reviewer to confirm eligibility. Furthermore, MechaScreener is designed explicitly as a title and abstract screening assistant and does not perform full-text screening, meaning human reviewers must also assess the remaining literature in full detail. Finally, while we utilised a deterministic approach (i.e., temperature = 0) to ensure reproducibility, the system’s performance remains fundamentally tied to the quality, clarity, and comprehensiveness of the user-provided eligibility criteria. Poorly defined criteria may lead to lower specificity as the model struggles to differentiate borderline studies.

Because published SRs do not report the full list of records excluded at title and abstract screening, we could not reuse the original reviews’ screening sets. Instead, we re-ran each review’s reported search strategy to reconstruct a comparable reference library. The resulting counts therefore differ from the numbers reported in the original reviews, sometimes substantially, because of differences in the databases searched, the search date, and the deduplication method applied; for example, the number of references we obtained for a given library will not necessarily match the post-deduplication title/abstract count reported in the source review. Our recall and specificity estimates should be interpreted relative to the reconstructed libraries rather than the original reviews’ screening logs. We did not retain the individual PubMed query, search date, and total-results count for each reconstructed search; however, the full reconstructed reference libraries (titles, abstracts, and ground-truth labels) are archived in our GitHub repository so that the exact inputs to MechaScreener can be inspected and reused.

A further limitation is the possibility of training-data contamination. The evaluation reviews were published in 2023–2024, and the underlying LLM (Qwen3-235B-A22B-Instruct-2507) was released in 2025. Although its knowledge cutoff is not publicly documented, its training corpus plausibly extends into this period, so the model may have encountered the included primary studies, and potentially the source SRs themselves, during pre-training. If so, some of the tool’s ability to identify relevant records could reflect memorisation rather than zero-shot generalisation. We were unable to quantify this effect, and it should be assessed prospectively on reviews published after the model’s training cutoff.

Additionally, our reconstructed searches returned some records published after the source review was conducted. These records could not have been screened by the original review authors, yet they were screened by MechaScreener and counted according to the same ground-truth rule (treated as ‘exclude’, and hence as true negatives when the tool scored them below the threshold). Across the evaluation set, these post-publication records accounted for approximately 4.6% of all true negatives and 3.2% of all false positives. Because they contribute to both the numerator and denominator of specificity, their net effect on the reported specificity is small; and to the extent that they are enriched for off-topic records, their inclusion is at least as likely to depress specificity (by adding false positives) as to inflate it. We therefore judge the impact on our conclusions to be minimal.

4.3. Future directions

Future research should explore the integration of MechaScreener directly into standard SR workflows and reference management platforms. Evaluating its performance across a broader range of foundational LLMs, as well as refining the prompt architecture to better handle title-only references, are key next steps. Furthermore, expanding the system’s capabilities to assist with automated full-text screening and data extraction could further streamline the evidence synthesis pipeline. Overall, these findings solidify the role of LLMs as powerful, dependable augmentations to traditional SR methodologies.