Highlights
What is already known?
-
• Systematic literature reviews (SLRs) are essential to research but remain slow and costly. As publication volume accelerates, researchers face a key bottleneck in having to screen thousands of abstracts.
-
• Large language models (LLMs) can automate title/abstract screening to lighten manual workloads while maintaining accuracy. However, human-in-the-loop screening remains standard because studies consistently report strong recall with variable precision.
-
• Prompt engineering matters: although studies have identified common prompting strategies, no factorial, cross-model evaluation of prompt $\times $
model
$\times $
criteria interaction has yet been conducted.
What is new?
-
• To quantify model–prompt interaction effects in automated citation screening, we conduct a structured cross-LLM comparison of prompting strategies.
-
• Using realistic screening tasks, we map each leading model’s strengths, weaknesses, and consistency/inconsistency.
-
• We conduct a cost-performance analysis that motivates a staged, cost-aware workflow suitable for at-scale screening.
Potential impact for RSM readers
-
• We identify model–prompt configurations that (1) achieve balanced performance at target recall with (2) manageable false-positive rates.
-
• We offer concrete budgeting advice to make at-scale screening more feasible, even for those with constrained resources.
-
• We establish transparent, reproducible evaluation practices, and reusable components that support transfer across domains and more robust SLR automation.
1 Introduction
Systematic literature reviews (SLRs) are a foundational method used in evidence-based research to provide transparent and reproducible syntheses of existing knowledge. The SLR process typically unfolds in five stages: (1) searching for relevant studies, (2) screening titles and abstracts to assess eligibility, (3) retrieving information from full texts, (4) synthesizing findings, and (5) writing results in a structured manner.Reference Han, Susnjak and Mathrani 1 Among these, the screening stage is especially demanding, as researchers must evaluate many publications precisely and consistently. Indeed, the rapid growth of scientific output has made manual screening increasingly difficult, which has in turn limited SLRs’ scalability and timeliness.Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou 2 – Reference Marshall and Wallace 4 We recognise that “systematic review” and “evidence synthesis” are more common terms in some review-methodology communities. However, we retain SLR in this study because (1) our classification framework is organised around formally defined SLR stages (searching, screening, retrieval, synthesis, and writing) and (2) “SLR automation” is widely used in the computing and information-systems literature from which the dataset was constructed.Reference Han, Susnjak and Mathrani 1
Advances in artificial intelligence, particularly large language models (LLMs), present new opportunities to improve the screening stage of the SLR process.Reference Li, Sun and Tan 5 LLMs excel at natural language understanding, reasoning, and classification, which are capabilities that align with screening demands.Reference Issaiy, Ghanaati and Kolahi 6 Early studies have already demonstrated LLMs’ potential to automate abstract screening by reducing manual workloads while maintaining accuracy.Reference Nori, King, McKinney, Carignan and Horvitz 7 , Reference Huang, Altosaar and Ranganath 8 By leveraging LLMs, researchers can likely increase screening efficiency while also scaling their evidence syntheses more effectively.Reference Sanghera, Thirunavukarasu and El Khoury 9
To best harness LLMs for SLR tasks, prompt engineering (deliberately designing inputs to steer model behavior) is critical. Prompting strategies range from simple formats (zero-shot and few-shot) to more advanced methods (chain-of-thought [CoT] and self-reflection); they aim to improve reasoning and calibration by eliciting intermediate steps or self-evaluation.Reference Wei, Wang and Schuurmans 10 , Reference Madaan, Tandon and Gupta 11 Although researchers have explored LLMs across several SLR tasks, recent work has focused most on automation of title/abstract screening.Reference Li, Sun and Tan 5 , Reference Issaiy, Ghanaati and Kolahi 6 , Reference Syriani, David and Kumar 12 , Reference Cao, Sang and Arora 13 Within automation of title/abstract screening, one key gap remains: a factorial, cross-model evaluation that jointly varies prompt type, LLM family/version, and eligibility-criteria formulation to quantify interaction effects on accuracy, precision–recall balance, and screening efficiency in SLR automation.
As such, this study comparatively evaluates prompting strategies for literature screening across multiple LLMs. We assess zero-shot, few-shot, CoT, and self-reflection prompting with six representative LLMs available at the time of evaluation: GPT-4o, 14 GPT-4o-Mini, 15 DeepSeek-Chat-V3, 16 Llama-4-Maverick, 17 Gemini-2.5-Flash, 18 and Claude-3.5-Haiku. 19 By systematically analyzing model–prompt interactions, we identify performance trade-offs, optimal configurations, and practical strategies to integrate LLMs into SLR workflows. This study makes four key contributions to the automation of literature screening:
-
• Systematic evaluation of prompting strategies: Comprehensively assessing how different prompting approaches influence LLM performance in automated screening.
-
• Benchmarking of representative LLMs available at the time of evaluation: Providing comparative evidence of leading models’ strengths, weaknesses, and consistencies/inconsistencies in classifying systematic review tasks.
-
• Identification of effective model–prompt configurations: Offering practical guidance for LLM deployment in screening workflows by pinpointing model–prompt combinations that yield robust, balanced performance.
-
• Advancing methodological understanding through evidence: Providing evidence-based insight into multiple dimensions of literature screening automation by examining interactions among prompting strategies, LLMs, and classification tasks.
2 Literature review
2.1 Automating literature screening
SLRs are widely used to synthesise research evidence; however, they remain very resource-intensive, particularly at the title-and-abstract screening stage. In screening literature, researchers assess up to thousands of retrieved citations against predefined inclusion and exclusion criteria. Multiple reviewers often must double-screen to ensure rigor, thus expending substantial time and effort before analysis can even commence.Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou 2 , Reference Borah, Brown, Capers and Kaiser 3 Consequently, academics rely increasingly on automation to reduce workload and enhance methodological rigor, transparency, and reproducibility.
Early work used supervised machine learning (ML) methods (support vector machines, naïve Bayes, and logistic regression) to classify abstracts.Reference Wallace, Trikalinos, Lau, Brodley and Schmid 20 , Reference Thomas, McNaught and Ananiadou 21 Relying on bag-of-words or TF-IDF features, these models required large labeled datasets and struggled with semantic refinement as well as cross-domain generalization. Later, semi-supervised and active learning approaches reduced labeling demands by prioritizing informative records.Reference Cohen, Hersh, Peterson and Yen 22 , Reference Miwa, Thomas, O’Mara-Eves and Ananiadou 23 Active learning is now central to screening, with tools like ASReview reducing workload up to 70% while maintaining recall.Reference Van De Schoot, De Bruin and Schram 24 Advances include (1) stopping heuristics (like the SAFE procedure) that balance efficiency with comprehensivenessReference Boetje and van de Schoot 25 and (2) ensemble methods that improve stability with classifier combinations.Reference Marshall, Noel-Storr, Kuiper, Thomas and Wallace 26 In parallel, platforms like Rayyan, EPPI-Reviewer, SWIFT-Active Screener, and Research Screener embed ML-driven prioritization to focus reviewer effort.Reference Stansfield, Stokes and Thomas 27 , Reference Chai, Lines, Gucciardi and Ng 28 Benchmarking efforts (like the CLEF eHealth 2019 Technology-Assisted Review task) also establish frameworks for assessing generalizability.Reference Kanoulas, Li, Azzopardi and Spijker 29 More recently, deep learning and transformer architectures have improved semantic modeling, though reproducibility and cross-domain transfer remain challenging.Reference Bannach-Brown, Hair, Bahor, Soliman, MacLeod and Liao 30
LLM application constitutes the most significant recent shift in research. Exploratory studies using GPT-3.5, GPT-4, and related models show strong recall but variable precision, making such models appropriate triage aids that still cannot replace dual human screening.Reference Dennstädt, Zink, Putora, Hastings and Cihoric 31 , Reference Oami, Okada and Nakada 32 Hybrid strategies that combine LLMs with active learning or re-ranking approaches yield efficiency gains while preserving recall safeguards.Reference Landschaft, Antweiler and Mackay 33 , Reference Matsui, Utsumi, Aoki, Maruki, Takeshima and Takaesu 34 These findings point toward human-in-the-loop workflows, where AI supports prioritization and filtering while humans oversee final inclusion decisions.
2.2 Language models used in systematic literature reviews
Transformer-based architecture (most notably BERT) has reshaped natural language processing by enabling contextualized word representations and deeper semantic modeling.Reference Devlin, Chang, Lee and Toutanova 35 Domain-specific variants like BioBERTReference Lee, Yoon and Kim 36 and SciBERTReference Beltagy, Lo and Cohan 37 extend these gains to biomedical and scientific corpora to improve screening and classification.Reference Du, Soysal and Wang 38 , Reference Ambalavanan and Devarakonda 39 This foundation facilitates SLR automation by showcasing transformers’ capacity to parse complex abstracts and study reports. LLMs (like GPT-3, GPT-4, Claude, Gemini, and LLaMA) have since introduced broad generalizations with minimal fine-tuning, which allow zero-shot and few-shot applications where labeled data are scarce.Reference Brown, Mann and Ryder 40 – Reference Touvron, Martin and Stone 44 Early evaluation reports their capacity to conduct screening, retrieval, and synthesis tasks framed as classification or question answering.Reference Nori, King, McKinney, Carignan and Horvitz 7 , Reference Khraisha, Put, Kappenberg, Warraitch and Hadfield 45 , Reference Sujau, Wada, Vallee, Hillis and Sušnjak 46 Prospective studies also report strong sensitivity but variable precision, again indicating that LLMs work well as prioritization aids but cannot replace dual human reviewers.Reference Dennstädt, Zink, Putora, Hastings and Cihoric 31 , Reference Oami, Okada and Nakada 32 Furthermore, hybrid designs embedding LLMs into active learning or re-ranking pipelines improve efficiency while safeguarding recall.Reference Landschaft, Antweiler and Mackay 33 , Reference Matsui, Utsumi, Aoki, Maruki, Takeshima and Takaesu 34
LLMs also contribute to broader SLR automation (for instance, information extraction, synthesis, and drafting).Reference Guo, Gupta, Deng, Park, Paget and Naugler 47 – Reference Susnjak, Hwang, Reyes, Barczak, Mcintosh and Ranathunga 49 Retrieval-augmented generation (RAG) enhances domain specificity and reduces hallucination, thereby promoting reproducibility.Reference Han, Susnjak and Mathrani 1 , Reference Ali, Mohtasim, Mosharrof and Krishna 50 However, performance depends on prompt design, model selection, and choice of evaluation metric; thus, prompt engineering and human oversight remain critical. In sum, while LLMs mark a major advancement in SLR automation, accuracy, transparency, and rigor still necessitate human-in-the-loop oversight. Full automation will demand further development, systematic testing, and validation. For now, LLMs serve best as intelligent assistants that accelerate screening and synthesis; indeed, they complement but cannot replace human judgment.
2.3 Prompt engineering for LLMs and its applications in SLR automation
Prompt engineering involves deliberately designing inputs to elicit accurate and contextually appropriate outputs; across tasks, it remains key to enhanced LLM performance.Reference Reynolds and McDonell 51 Once considered an ad-hoc practice, prompt engineering has become a structured field with theoretical foundations, design principles, and empirical validation.Reference White, Fu and Hays 52 Task framing, input format, and instruction specificity can significantly change LLM performance, so SLR automation relies heavily on prompt engineering. Basic strategies include (1) zero-shot prompting, where models complete tasks without examples, and (2) few-shot prompting, where limited exemplars guide output.Reference Brown, Mann and Ryder 40 These methods showcase LLMs’ generalization capabilities, so they remain widely used in classification, summarization, and question answering. More advanced techniques extend model reasoning: (1) CoT prompting elicits a step-by-step explanation that improves performance on an arithmetic, common-sense, or logical task,Reference Wei, Wang and Schuurmans 10 , Reference Kojima, Gu, Reid, Matsuo and Iwasawa 53 while (2) self-reflection prompting enables iterative critique and revision that makes complex problem solving more robust.Reference Madaan, Tandon and Gupta 11 , Reference Shinn, Cassano, Gopinath, Narasimhan and Yao 54 Such approaches move LLMs beyond surface-level pattern recognition toward structured reasoning, which is essential for evidence synthesis.
Although early research largely addressed domains like mathematics, common-sense reasoning, and programming,Reference Kojima, Gu, Reid, Matsuo and Iwasawa 53 , Reference Zhou, Schärli and Hou 55 recent studies have applied prompt engineering to SLR workflows, particularly in the screening stage. These studies show that three levers shape abstract LLM screening: (1) prompting strategy, (2) model selection, and (3) expression of eligibility criteria. Early single-model work with GPT-3.5 used PICOS-anchored, zero-shot prompts before applying a relevance threshold to protect recall.Reference Issaiy, Ghanaati and Kolahi 6 In software-engineering corpora, zero-shot prompts that clearly list exclusion criteria outperform few-shot variants and exhibit improved specificity.Reference Syriani, David and Kumar 12 Broader evaluations across multiple models converge on structured templates (either (a) criterion-by-criterion Boolean checks in a rigid output format or (b) framework-style reasoning) with deterministic decoding.Reference Li, Sun and Tan 5 , Reference Cao, Sang and Arora 13 Large multi-model and ensemble studies further demonstrate that small changes to bias wording in otherwise identical prompts shift the recall–precision balance (each model has a different optimal bias). Ensembling can secure very high sensitivity with workable precision.Reference Sanghera, Thirunavukarasu and El Khoury 9 Live deployments report high recall using exclusion-first, PICO-grounded Boolean prompts with tightly controlled generation settings.Reference Homiar, Thomas and Ostinelli 56 A cross-domain assessment also highlights that the wording and granularity of inclusion and exclusion criteria can change results as much as prompt style or model upgrades.Reference Delgado-Chaves, Jennings and Atalaia 57 Moreover, systems with added retrieval components illustrate how topic and corpus characteristics interact with prompt structure and model selection while also motivating recall-protective decision rules.Reference Chen, Liu and Mo 58
2.4 Research questions
Although LLMs have been explored in the context of screening automation, few studies systematically compare prompting strategies across models and screening criteria. Consequently, an important question remains: how do different model–prompt combinations influence performance, as assessed under standard evaluation metrics? Importantly, practical deployment requires accuracy but remains subject to operational constraints: screening typically spans tens of thousands of abstracts, token-expanding prompts materially increase spend and latency, and model choice can vary costs by several orders of magnitude. We address these problems by systematically assessing several prompting strategies and LLMs to provide empirical evidence and actionable guidance. Therefore, we ask the following questions:
RQ1: How do prompting strategies and LLM choice individually and jointly affect screening performance across standard metrics (accuracy, precision, recall, and F1) and screening criteria?
RQ2: Given the scale and budget sensitivity of real-world screening, what are different model–prompt combinations’ cost-performance implications for large-scale use, and which configurations deliver the best balance between accuracy and operational efficiency?
3 Methodology
We examine how different LLMs perform with different prompting strategies in literature-screening automation. We structure our research (Figure 1) around four main components: (1) dataset construction, (2) prompting strategy design, (3) model selection, and (4) evaluation metrics. The following sections describe each component in detail.
Workflow of research methodology.
Figure 1 Long description
The flowchart shows four sequential components connected by arrows. Component 1, Dataset Construction: papers retrieved from six databases, duplicates removed, 1,376 studies manually screened and annotated for LLM use. Component 2, Prompting Strategy Design: five prompt types designed (zero-shot, few-shot, CoT, CoT-few-shot, self-reflection). Component 3, Model Selection: six LLMs evaluated (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick). Component 4, Evaluation Metrics: accuracy, precision, recall, and F1 measured at two classification levels.
3.1 Dataset construction
To enable a comprehensive evaluation, we curated a dataset of academic publications on SLR automation. We retrieved papers published between 2014 and 2024 from major databases (Scopus, ScienceDirect, ACM Digital Library, IEEE Xplore, Web of Science, and Semantic Scholar), using APIs where available. Search queries targeted terms such as “literature review automation” and “systematic review automation.” Our initial search returned over 5,000 records. After removing duplicates and excluding non-research items, we were left with 1,376 unique studies for manual screening.
We screened titles and abstracts for eligibility; studies were eligible if they addressed automation of any SLR stage (searching, screening, retrieval, synthesis, and report writing) (Figure 2). We then annotated studies for LLM use. As shown in Figure 3, screening involved two levels of classification: (1) overall relevance to SLR automation (yes/no) and (2) task-specific annotation (searching, screening, retrieval, synthesis, and report writing) and detection of LLM use (yes/no). We formalized all labeled records into a structured JSON dataset that established a ground truth. To ensure reliable annotation, two researchers with LLM and systematic-review experience independently labeled all records. Annotation followed a written coding guide that defined Level 1 relevance to SLR automation and Level 2 task-stage labels. At Level 1, records were labelled relevant if they addressed automation or computational support for at least one systematic review stage. At Level 2, relevant records were labelled for searching, screening, retrieval, synthesis, writing, and LLM use; these labels were not mutually exclusive. Disagreements were reviewed against the coding guide and resolved through consensus discussion. Before consensus adjudication, the two annotators achieved 91% raw agreement and Cohen’s kappa of 0.809 for Level 1 relevance classification; disagreements occurred in 118 records and mainly involved abstracts with broad or ambiguous descriptions of review-support technologies. Common ambiguous cases included distinguishing ordinary systematic reviews from studies of SLR automation, separating searching from screening, and distinguishing general AI/NLP use from explicit LLM use.
Stages of SLR process.
Figure 2 Long description
The diagram illustrates the SLR workflow as five sequential stages in a linear progression. Stage 1: Searching. Stage 2: Screening. Stage 3: Retrieval. Stage 4: Synthesis. Stage 5: Report Writing. Each stage represents a step in conducting a systematic literature review and forms the basis for the task classification categories used in this study.
Screening process flowchart.
Figure 3 Long description
The flowchart shows two classification levels. Level 1, Relevance Classification: each abstract is classified as relevant or not relevant to SLR automation. Level 2, Task-Stage Classification: relevant abstracts are assigned to one or more of six task categories: Searching, Screening, Retrieval, Synthesis, Writing, and Using LLMs. Studies can belong to multiple categories. The process produces a multi-label classification dataset of 491 relevant studies from 1,376 screened.
As shown in Figure 4, of the 1,376 screened studies, we identified 491 (35.7%) as relevant. Screening automation accounted for the largest share (260, 53%), followed by retrieval (189, 38%), searching (147, 30%), synthesis (109, 22%), and finally writing (34, 7%). These categories were not mutually exclusive, as many studies addressed multiple SLR stages. The distribution above reflects task complexity: labor-intensive processes (like screening) received greater attention, while conceptually demanding tasks (like synthesis and writing) received less attention. Notably, 127 studies (26%) explicitly examined LLM applications; this is a substantial proportion, given LLMs’ recent emergence.
Distribution of studies relevant to SLR automation tasks.
Figure 4 Long description
The figure is a vertical bar chart showing the distribution of papers across systematic literature review automation categories. The horizontal axis lists seven categories, and the vertical axis shows the number of papers, ranging from 0 to 500. Exact counts are displayed above each bar.The first bar, shown in green, represents all papers identified as relevant to SLR automation and has the highest count, at 491. The remaining purple bars show the number of papers associated with specific automation tasks. Screening automation is the largest task category, with 260 papers, followed by retrieval automation with 189 and searching automation with 147. Application of LLMs includes 127 papers, while synthesis automation includes 109. Writing automation is the smallest category, with 34 papers.The bars generally decrease in height from left to right, indicating progressively fewer papers across the listed task categories. Because papers could be assigned to more than one automation task, the task-specific counts are not mutually exclusive and should not be summed to obtain the overall total.
This corpus should be interpreted as a specialised screening dataset focused on SLR automation. Because the search strategy and eligibility criteria targeted automation of SLR stages, relevant abstracts may contain explicit lexical cues such as “systematic review,” “automation,” “screening,” or “LLM.” This may make the relevance-classification task less ambiguous than screening tasks in domains where eligibility criteria are more nuanced and lexical separation between relevant and irrelevant records is weaker.
3.2 Prompting strategy design
To systematically assess input design’s impact on LLM performance, we evaluated five prompting strategies that represent the primary paradigms used in both practice and research:
-
• Zero-shot prompting: Provides the model only task instructions, without examples. Prompts for Level 1 relevance classification (Appendix B.1) and Level 2 task classification (Appendix B.3) include inclusion/exclusion criteria and contextual information on SLR stage. This baseline (1) tests the model’s capacity to generalize from task descriptions alone and (2) reflects the most scalable deployment scenario, as real-world pipelines often begin with minimal customization.
-
• Few-shot prompting: Supplements task instructions with a small number of annotated examples. This examines whether domain-specific exemplars enhance model calibration, thus simulating cases where limited labeled data are available.
-
• Chain-of-thought prompting: Instructs the model to generate step-by-step reasoning prior to classification. CoT prompts for Level 1 (Appendix B.2) and Level 2 classifications (Appendix B.4) encourage logical consistency, transparency, and better handling of ambiguous cases, which are qualities essential for literature screening.
-
• CoT few-shot prompting: Combines exemplar-based guidance with explicit reasoning, providing worked examples of both process and output. As the most structured strategy, COT few-shot aims to reduce error propagation while balancing precision and recall in complex tasks.
-
• Self-reflection prompting: The model produces an initial response, critiques it, and revises it as needed. By appending a reflection check to a CoT prompt (Appendix B.5), self-reflection promotes calibration and error correction in an attempt to minimize false positives and negatives. Though less established, it reflects an emerging direction in meta-cognitive prompting.
We paired all prompts with a system message (Appendix B.6) instructing the model to assume the role of an expert in ML and SLR automation. This contextual grounding reduced ambiguity, strengthened task alignment, and improved classification consistency. Collectively, such strategies ranged from minimal intervention (zero-shot) to highly structured reasoning and self-correction (CoT few-shot and self-reflection). This breadth enabled us to evaluate (1) overall accuracy and (2) prompting paradigms’ effect on the recall–precision trade-off central to SLR automation.
3.3 Large language models evaluated
We evaluated six representative LLMs available at the time of evaluation drawn from major providers, including OpenAI, Anthropic, Google DeepMind, DeepSeek, and Meta. Table 1 summarizes the evaluated models, their providers, their intended use contexts, and their key strengths. Our selection included flagship systems (optimized for reasoning and comprehension) as well as lightweight variants (designed for cost efficiency and scalability). This diversity helped us assess (1) overall accuracy and (2) performance–efficiency trade-offs crucial to large-scale screening pipelines. All six models demonstrated strong capabilities in classification, reasoning, and retrieval-style tasks, making them well-suited for systematic review automation. They also boasted complementary strengths (e.g., advanced reasoning, efficiency, and open-source adaptability), thereby facilitating a representative and balanced cross-architectural evaluation of prompting strategies’ influence over performance. To ensure reproducibility and consistency, we accessed all models via API endpoints during August and September 2025, with requests routed through the OpenRouter multi-provider gateway. We used fixed generation settings where supported by the provider to reduce output variability, and no model was fine-tuned for this study. Outputs were requested in structured JSON format and checked for parseability before metric calculation.
Overview of evaluated LLMs for literature screening automation
Table 1 Long description
The table has five columns: Model, Provider, Scale/Variant, Intended Use, and Key Strengths. GPT-4o (OpenAI, Flagship): advanced reasoning and comprehension; strong performance on reasoning-intensive tasks. GPT-4o-mini (OpenAI, Lightweight): cost-efficient, scalable screening; reduced computational cost. DeepSeek-Chat-V3 (DeepSeek, mid-scale efficiency-focused): general-purpose with efficiency emphasis; strong cost-performance balance. Gemini-2.5-Flash (Google, Flagship): high-performance research; competitive recall in complex tasks. Claude-3.5-Haiku (Anthropic, Lightweight): fast, cost-efficient text classification; strong reasoning relative to size. Llama-4-Maverick (Meta, open-source): reproducible academic research; adaptable and transparent.
3.4 Evaluation metrics
We evaluated model predictions against our expert-annotated dataset using four widely adopted classification metrics:
-
• Recall: The proportion of true positives (TPs) among all actual positives (model sensitivity).
-
• Precision: The proportion of TPs among all positive predictions.
-
• Accuracy: The overall proportion of correctly classified records.
-
• F1-Score: The harmonic mean of precision and recall, balancing both dimensions.
For each LLM-prompting strategy combination, these computed metrics allowed cross-model and cross-prompt comparisons. Overall, our methodology provides a rigorous and reproducible basis for assessing interactions between prompting strategy and LLM architecture in literature-screening automation. Although we report recall, precision, accuracy, and F1 across all analyses, their interpretation depends on the classification level. For Level 1 relevance classification, recall/sensitivity is the primary metric because false negatives (FNs) may lead to missed relevant studies. Precision is interpreted mainly as a workload-related measure, while F1 and accuracy are treated as secondary descriptive summaries. For Level 2 task-stage classification, the objective is different: records already identified as relevant are assigned to one or more SLR task categories. In this multi-label categorisation setting, F1 and accuracy are given greater interpretive weight because they summarise task-label correctness and balanced classification performance across categories. Recall and precision are still reported to show whether models tend to miss task-relevant studies or over-assign task labels.
Finally, each model–prompt configuration was evaluated as an independent static classifier applied to the full dataset. We did not simulate an operational screening workflow involving active learning, ranking, prioritisation, stopping criteria, or escalation of borderline records. Therefore, the staged, cost-aware workflow implications should be interpreted as directions for future workflow design rather than as workflow-level validation in the present study.
4 Results
We organize our results by the two classification levels used to construct our ground truth dataset (Figure 3). The first part reports on Level 1 classification, which determines a study’s relevance to SLR automation. The second part covers Level 2 classification, which (1) identifies specific SLR stages addressed (searching, screening, retrieval, synthesis, and writing) and (2) detects LLM use. Each part includes sections on overall performance, comparisons across prompting strategies and models, and analyses of their interactions. Across both levels, we evaluate six LLMs (GPT-4o, GPT-4o-Mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under four prompting strategies (zero-shot, few-shot, CoT, and CoT-few-shot). We include self-reflection prompting only for Level 1 classification, given its weaker performance in preliminary testing. We conduct our evaluation using four standard metrics (recall, precision, accuracy, and F1), and we supplement it with descriptive and statistical sub-analyses to capture differential effects of model choice and prompting strategy.
4.1 Classification of relevance to SLR automation
The following evaluations granularly assess how prompt design and model choice jointly shape automated identification of relevant studies (Figure 3, Level 1 Classification). Table A1 (in Appendix A) reports performance metrics for each prompt–model combination, while Figure 5 summarizes these results, enabling systematic comparisons across prompt types and models in terms of recall and precision. Appendix B presents the corresponding prompt templates.
Recall-precision performance across models by prompt type.
Figure 5 Long description
The scatter plot displays recall on the y-axis (approximately 0.85 to 1.00) and precision on the y-axis (approximately 0.75 to 0.96), with each point representing one model-prompt combination. Zero-shot and CoT points cluster in the high-recall, lower-precision region. CoT-few-shot points shift toward higher precision. GPT-4o and DeepSeek-Chat-V3 points are concentrated in the upper-right quadrant, indicating the strongest combined recall-precision performance. Self-reflection points fall at lower precision values across most models.
4.1.1 Effects of prompting strategies on relevance classification
Prompting strategies substantially shape LLMs’ predictive behavior in SLR automation. To disentangle prompt design’s effects from model-specific characteristics, we first analyze prompt types’ performance variations within each LLM before conducting a macro-averaged evaluation across all models. This two-stage approach allows us to assess (1) model-level sensitivity to prompt choice and (2) general trends in prompting effectiveness across the entire LLM ensemble.
Classification performance variation of prompt types by LLMs: We used Friedman’s nonparametric repeated-measures test as an exploratory omnibus analysis to examine whether prompt type was associated with differences in record-level classification correctness within each LLM. The unit of observation was the individual article, and the repeated conditions were the five prompt types applied to the same records. TPs, true negatives (TNs), false positives (FPs), and FNs were used to describe classification outcomes, but were not treated as ordered categories. The Friedman results are therefore used to indicate possible prompt sensitivity, while interpretation is based mainly on recall, precision, F1, and confidence intervals (CIs) in Table A1. The exploratory Friedman results suggested prompt-related variation for Claude-3.5-Haiku (
$\chi ^2(4)=134.6,\ \mathrm {p}<1.0\times 10^{-25}$
), GPT-4o-Mini (
$\chi ^2(4)=40.7,\ \mathrm {p}<1.0\times 10^{-7}$
), Gemini-2.5-Flash (
$\chi ^2(4)=38.3,\ \mathrm {p}<1.0\times 10^{-7}$
), and Llama-4-Maverick (
$\chi ^2(4)=27.4,\ \mathrm {p}<1.0\times 10^{-5}$
). By contrast, prompt-type effects were weaker for GPT-4o (
$\chi ^2$
(4) = 8.2, p = 0.085) and negligible for DeepSeek-Chat-V3 (
$\chi ^2$
(4) = 1.2, p = 0.88). Table A1 shows that prompt type mainly changed the balance between recall and precision. Zero-shot and CoT prompting generally gave higher recall and are therefore more relevant to first-pass screening, where FNs are the main concern. CoT-few-shot often improved precision and F1, but in some cases at the expense of recall. It is therefore better viewed as a more selective option for later triage or workload reduction. Self-reflection showed weaker performance, mainly because it increased FPs without a clear recall benefit.
Macro-averaged evaluation metrics of prompt types across models: To isolate prompt design’s effects from model-specific variation, we examined macro-averaged performance across all models (Table 2). Prompting strategies (types) exert a substantial influence on classification outcomes.
Macro-averaged evaluation metrics for each prompt type (across all models)
Table 2 Long description
The table has five columns: Prompt Type, Recall, Precision, Accuracy, and F1. Zero-shot: Recall 0.971, Precision 0.834, Accuracy 0.917, F1 0.895. CoT: Recall 0.964, Precision 0.855, Accuracy 0.927, F1 0.905. Few-shot: Recall 0.954, Precision 0.875, Accuracy 0.934, F1 0.912. Self-reflection: Recall 0.946, Precision 0.753, Accuracy 0.862, F1 0.835. CoT-few-shot: Recall 0.930, Precision 0.901, Accuracy 0.937, F1 0.913. Zero-shot achieves the highest recall; CoT-few-shot the highest precision and accuracy.
Because screening is recall-oriented, recall is treated as the primary metric; precision, F1, and accuracy are used to describe selectivity, workload, and classification balance. Zero-shot prompting achieved the highest observed recall (0.971), followed by CoT (0.964) and few-shot prompting (0.954). These results suggest that zero-shot and CoT prompting are more suitable for first-pass screening, where the aim is to minimise missed relevant studies. This higher sensitivity came with reduced precision, especially for zero-shot prompting (0.834), which would pass more irrelevant records to later review stages. CoT-few-shot achieved the highest observed precision (0.901) and F1 (0.913), with few-shot prompting very close in F1 (0.912). This small F1 difference should not be over-emphasised. More importantly, CoT-few-shot had lower recall (0.930) than zero-shot and CoT. If used as a sole exclusion mechanism, this reduction in recall could be problematic in a systematic review workflow. Self-reflection had the lowest precision (0.753) and F1 (0.835), without a meaningful recall advantage over zero-shot or CoT. The results therefore do not support a single “best” prompt. Instead, zero-shot and CoT are more appropriate when recall is the main priority, whereas CoT-few-shot may be useful when reducing FPs and reviewer workload is more important.
4.1.2 Effects of model choice on relevance classification
While prompt design can indeed drive predictive outcomes, choice of LLM can also introduce systematic variation. To disentangle the effects of model architecture and training from those of prompting strategies, we evaluate each LLM’s performance under identical prompt conditions. This section analyzes model-level variation for each respective prompt type before aggregating results to examine macro-level patterns across all prompting strategies.
Classification performance variation of LLMs by prompt types: We also used Friedman’s nonparametric repeated-measures test to explore whether model choice was associated with differences in record-level correctness under each prompt type. The unit of observation was again the individual article, and the repeated conditions were the six LLMs applied to the same records under a given prompt. TP, TN, FP, and FN were used descriptively rather than as ordered categories. The test results indicate possible model-related variation, while Table A1 provides the main basis for interpreting recall, precision, F1, and uncertainty. The exploratory Friedman results suggested cross-model variation under self-reflection (
$\chi ^2(5)=786.8,\ \mathrm {p}=8.1\times 10^{-168}$
), CoT (
$\chi ^2(5)=177.7,\ \mathrm {p}=1.7\times 10^{-36}$
), and zero-shot prompting (
$\chi ^2(5)=120.3,\ \mathrm {p}=2.8\times 10^{-24}$
). Few-shot prompting showed more moderate variation (
$\chi ^2(5)=24.1,\ \mathrm {p}=2.0\times 10^{-4}$
), while CoT-few-shot showed only marginal variation (
$\chi ^2(5)=11.5,\ \mathrm {p}=0.042$
). These results indicate that model choice mattered more under some prompts than others, but statistical significance alone does not establish practical superiority. Table A1 shows that Gemini-2.5-Flash and Llama-4-Maverick often produced more inclusive predictions, with high recall but lower precision. GPT-4o and DeepSeek-Chat-V3 were more balanced, with stronger precision and F1 but sometimes lower recall than the most inclusive models. Thus, models with higher recall may be preferable for first-pass screening, while more balanced models may be more useful for triage or workload reduction.
Macro-averaged evaluation metrics of LLMs across prompt types: To isolate choice-of-model effects from prompt-specific variations, we examined macro-averaged performance across all prompt types. In doing so, we took a broader look at the way model selection influences classification outcome when averaged across prompting strategies. Indeed, our analysis showcased (1) consistently strong performers and (2) models whose performances depended heavily on prompts.
Recall is treated as the main screening metric, while precision, F1, and accuracy describe workload and classification balance. As shown in Table 3, Gemini-2.5 achieved the highest observed macro-average recall (0.984), followed by Llama-4-Maverick (0.980) and Claude-3.5-Haiku (0.967). These models therefore produced more inclusive screening decisions, which may be useful in first-pass screening. Their high recall was accompanied by lower precision, particularly for Llama-4-Maverick (0.795), Claude-3.5-Haiku (0.803), and Gemini-2.5 (0.818), indicating greater downstream review burden. GPT-4o and DeepSeek-Chat-V3 showed a more balanced pattern. GPT-4o had the highest observed macro-F1 (0.918), followed closely by DeepSeek-Chat-V3 (0.914), and both had the highest precision values. However, the F1 difference between them is small and should not be treated as evidence that one model is clearly better. Their recall values (0.928 and 0.925) were also lower than those of the most inclusive models. These results suggest that model choice should be tied to the screening objective. Gemini-2.5, Llama-4-Maverick, and Claude-3.5-Haiku may be more useful when sensitivity is prioritised, whereas GPT-4o and DeepSeek-Chat-V3 may be preferable where a stronger balance between recall and precision is needed.
Macro-Averaged Evaluation Metrics For Each Model (Across All Prompt Types)
Table 3 Long description
The table has five columns: Model Type, Recall, Precision, Accuracy, and F1. Gemini-2.5: Recall 0.984, Precision 0.818, Accuracy 0.915, F1 0.893. Llama-4-Maverick: Recall 0.980, Precision 0.795, Accuracy 0.899, F1 0.876. Claude-3.5-Haiku: Recall 0.967, Precision 0.803, Accuracy 0.901, F1 0.877. GPT-4o-Mini: Recall 0.934, Precision 0.831, Accuracy 0.898, F1 0.874. GPT-4o: Recall 0.928, Precision 0.910, Accuracy 0.941, F1 0.918. DeepSeek-Chat-V3: Recall 0.925, Precision 0.904, Accuracy 0.938, F1 0.914.
4.1.3 Interactions between prompting and models
Table A2 and Figure 6 show that prompt–model combinations differed in how they balanced recall and precision. Because screening prioritises minimising FNs, recall is the main criterion for interpreting these combinations; precision and F1 indicate selectivity and workload.
Evaluation metrics of observed best-performing prompt types by model (relevance classification).
Figure 6 Long description
The figure is a heatmap that displays four metrics for each LLM under its observed best prompt. GPT-4o (Zero-shot): Accuracy 0.952, Precision 0.921, Recall 0.947, F1 0.934. DeepSeek-Chat-V3 (CoT): Accuracy 0.943, Precision 0.901, Recall 0.945, F1 0.923. Llama-4-Maverick (CoT-few-shot): Accuracy 0.939, Precision 0.879, Recall 0.961, F1 0.918. Gemini-2.5 (Few-shot): Accuracy 0.935, Precision 0.860, Recall 0.980, F1 0.916. GPT-4o-Mini (Few-shot): Accuracy 0.936, Precision 0.880, Recall 0.951, F1 0.914. Claude-3.5-Haiku (CoT-few-shot): Accuracy 0.927, Precision 0.859, Recall 0.953, F1 0.903.
Several configurations produced recall-oriented results. Gemini-2.5 with few-shot prompting achieved recall of 0.980, although with some precision trade-off (0.860). Llama-4-Maverick with CoT-few-shot also achieved high recall (0.961), with precision of 0.879. DeepSeek-Chat-V3 with CoT prompting provided a more balanced result, with recall of 0.945 and precision of 0.901. GPT-4o with zero-shot prompting achieved the highest observed accuracy (0.952), precision (0.921), and F1 (0.934). This is a strong result, but it should not be treated as universally preferable. In screening, a configuration with slightly lower F1 may still be more appropriate if it provides higher recall. Similarly, Claude-3.5-Haiku’s stronger result under CoT-few-shot reflects a model-specific pattern rather than a general rule. The main implication is that prompt–model combinations should be selected according to the intended screening role. Some configurations favour recall for first-pass screening; others favour precision and may be more useful for triage or workload reduction. Small differences in F1 or accuracy should be interpreted cautiously, particularly where CIs overlap.
4.1.4 Confidence intervals of evaluation metrics
As shown in Table A1, we report 95% CIs for all metrics. We estimated Accuracy, Precision, and Recall CIs with Wilson score intervals, and we obtained F1 CIs with bootstrap resampling at the abstract level (5,000 replicates) to account for the joint variability of Precision and Recall. A large denominator made accuracy intervals narrow, whereas Precision and Recall spanned wider ranges, reflecting class imbalance. F1 intervals were typically broader than Accuracy intervals but provided realistic measures of uncertainty. These results show that small differences, particularly those around one to two percentage points, often fall within overlapping CIs. Such differences should be treated as descriptive unless they imply a meaningful change in recall, FN risk, workload, or cost.
4.2 Classification of SLR stage(s) and LLM use
The following evaluations assess how variations in prompt design and model selection influence performance across six classification tasks (Figure 3): five SLR automation stages (searching, screening, retrieval, synthesis, and writing) and detection of LLM use. We report results as (1) task-specific model and prompt sensitivities and (2) macro-averages across conditions (where appropriate). Given our dataset composition (491 relevant articles with notable task-specific imbalances, that is, 260 articles about screening and only 34 about writing), results interpretation must account for task prevalence.
4.2.1 Effects of prompting methods on tasks classification outcomes
Prompt design and model choice jointly shape performance across six Level-2 tasks (searching, screening, retrieval, synthesis, writing, and detecting LLM use). To disentangle these effects, we first assess task-specific prompt sensitivity within each model using Friedman’s nonparametric test. In doing so, we identify points across tasks where prompt design significantly alters performance. We then report prompt types’ macro-averaged performance metrics across models and tasks, highlighting systematic precision–recall trade-offs for each prompting strategy.
Classification performance of prompt types by models across tasks: For each Level-2 task and within each LLM, we used Friedman’s nonparametric repeated-measures test to explore whether prompt type was associated with differences in task-label correctness. The unit of observation was the article-task decision, and the repeated conditions were the four prompt types applied to the same records. TP, TN, FP, and FN were used descriptively rather than as ordered categories. Because Level-2 labels were multi-label and imbalanced, the results are interpreted with reference to task-specific recall, precision, F1, CIs, and task prevalence. As summarized in Table 4, prompt sensitivity varied across tasks and models. Claude-3.5-Haiku and GPT-4o-Mini showed stronger responsiveness to prompt design across several tasks, whereas Gemini-2.5-Flash and GPT-4o were generally less prompt-sensitive. DeepSeek-Chat-V3 and Llama-4-Maverick showed more task-dependent variation. These results should not be interpreted as showing that one prompt type is generally superior across Level-2 tasks. Instead, prompt choice changed the recall–precision balance within particular task labels as shown in Table A6. Higher-recall prompts may be preferable where missing task-relevant studies is most problematic, while higher-precision prompts may be more useful for reducing FPs in later categorisation or triage. This distinction is especially important for low-prevalence labels such as writing, where small numerical changes may reflect only a few records.
Friedman statistics (
$\chi ^2$
with df = 3) and p-values across tasks and LLMs
Table 4 Long description
The table has columns for six LLMs (Claude-3.5-Haiku, DeepSeek-Chat-V3, Gemini-2.5-Flash, GPT-4o-Mini, GPT-4o, Llama-4-Maverick), each with chi-squared and p-value sub-columns (df=3, four prompt types). Rows cover six tasks: Retrieval, Screening, Searching, Synthesis, Writing, and Using LLMs. Bold values indicate p<0.05. Claude-3.5-Haiku and GPT-4o-Mini show significant prompt-related variation across most tasks. DeepSeek-Chat-V3 and Gemini-2.5-Flash show fewer significant effects. GPT-4o shows significance only for Searching. Full data is available in the table.
Note: Friedman tests compare performance across four prompt types within each LLM for a given task; df
$=3$
. Bold indicates
$p<0.05$
; highlighted
$\chi ^2$
matches those significant p-values.
Macro-averaged evaluation metrics of prompt types across models and tasks: According to our macro-averaged evaluation across all models and tasks (Table 5 and Figure 7), prompting strategies systematically shape the recall–precision trade-off. Accuracy remained consistent across methods (0.872–0.877), indicating stable baseline reliability, but precision, recall, and F1 diverged. Zero-shot prompting achieved the highest recall (0.838) and would prove particularly valuable in early-stage screening, where under-inclusion (of relevant literature) would cost more than over-inclusion. However, zero-shot’s low precision (0.713) increased FPs and would burden downstream reviewers. In contrast, CoT-few-shot prompting yielded the highest precision (0.774) and a competitive F1 (0.752) by balancing exemplar guidance and reasoning. This gain in precision reduced spurious classifications, though at the cost of moderately lower recall (0.736). Few-shot prompting delivered mid-range performance (F1 = 0.754), especially benefiting models that align well with contextual exemplars. CoT prompting raised recall (0.812) relative to few-shot prompting but sacrificed precision (0.718); it tended toward over-inclusion when no exemplars grounded its reasoning. In short, F1 scores were tightly clustered (0.752–0.766), indicating no uniformly dominant strategy.
Precision–recall trade-off for prompt strategies.
Figure 7 Long description
The scatter plot shows recall on the x-axis and precision on the y-axis for four prompt types, averaged across all models and tasks. Zero-shot is positioned at the highest recall (0.838) with the lowest precision (0.713). CoT-few-shot occupies the highest precision position (0.774) with the lowest recall (0.736). CoT and Few-shot fall at intermediate positions. The plot illustrates a clear negative trade-off between recall and precision as prompting shifts from zero-shot toward structured approaches.
The plot in Figure 7 reveals a clear precision–recall trade-off across prompt strategies: recall rises toward zero-shot, precision peaks with CoT-few-shot, and few-shot and CoT occupy the middle ground. The dashed line highlights a consistent downward trend from higher-precision/lower-recall (CoT-few-shot) to lower-precision/higher-recall (zero-shot): incremental precision losses typically accompany gains in recall. Practically, strategy should vary with stage: use zero-shot for early, high-recall passes; transition to few-shot or CoT to balance workload and errors; and deploy CoT-few-shot when precision is paramount. This progression aligns model behavior with screening objectives, reducing overall workload while controlling risk of error.
4.2.2 Effect of LLM choice on tasks classification outcomes
Beyond assessing individual models’ prompt sensitivities, each LLM’s performance need also be evaluated under the same prompting conditions. As such, we apply Friedman’s nonparametric test to measure cross-model variation for each task, with datasets organized by prompt type. A macro-level LLM evaluation complements our analysis by revealing (1) precision–recall trade-offs and (2) shifts in overall model effectiveness when averaged across prompt types and tasks.
Classification performance of LLMs by prompt types across tasks: To examine model-related variation in Level-2 task classification, we used Friedman’s nonparametric repeated-measures test for each task and prompt type. The unit of observation was the article-task decision, and the repeated conditions were the six LLMs applied to the same records. TP, TN, FP, and FN were used to describe outcomes, not as ordered categories. Because Level-2 labels were imbalanced and some categories had few positive cases, the results were interpreted alongside task-specific recall, precision, F1, CIs, and task prevalence. As shown in Table 6, the exploratory Friedman results suggested that model-related variation differed across task-prompt combinations. This indicates that LLMs behaved differently across searching, screening, retrieval, synthesis, writing, and LLM-use detection. However, statistical significance alone should not be read as practical superiority, particularly when differences are small or when the task label is rare. The task-specific performance results in Table A6 provide the main basis for interpretation. Models with higher recall may be more appropriate for sensitivity-focused identification of task-relevant studies, whereas models with higher precision or F1 may be more useful for later workload reduction or categorisation. For rare labels such as writing, results should be treated cautiously because small changes in counts can have a visible effect on the metrics.
Macro-averaged evaluation metrics for each prompt (across models and tasks)
Table 5 Long description
The table has five columns: Prompt Type, Accuracy, Precision, Recall, and F1. CoT: Accuracy 0.872, Precision 0.718, Recall 0.812, F1 0.758. CoT-few-shot: Accuracy 0.877, Precision 0.774, Recall 0.736, F1 0.752. Few-shot: Accuracy 0.877, Precision 0.741, Recall 0.775, F1 0.754. Zero-shot: Accuracy 0.876, Precision 0.713, Recall 0.838, F1 0.766. Accuracy is similar across all prompts (0.872 to 0.877). Zero-shot achieves the highest recall; CoT-few-shot the highest precision. Self-reflection was excluded from task classification analysis.
Friedman statistics (
$\chi ^2$
with df = 5) and p-values across tasks and prompt types
Table 6 Long description
The table has columns for four prompt types (CoT, CoT-few-shot, Few-shot, Zero-shot), each with chi-squared and p-value sub-columns (df=5, six models). Rows cover six tasks. Bold values indicate p<0.05. CoT and CoT-few-shot show significant model-related variation across most tasks. Few-shot shows fewer significant differences, particularly not for Retrieval (p=0.871) or Searching (p=0.169). Zero-shot shows significance primarily for Screening and Using LLMs. Using LLMs shows significance across all four prompt types. Full data is available in the table.
Note: Bold indicates
$p<0.05$
; highlighted
$\chi ^2$
corresponds to significant p-values.
Macro performance and precision–recall trade-offs by model: Model selection significantly influences classification outcomes, particularly by shaping the precision–recall trade-off (Table 7 and Figures 8a and 8b). GPT-4o had the highest observed overall performance (accuracy = 0.886, F1 = 0.778), combining the highest precision (0.757) with strong recall (0.808). Its compact variant, GPT-4o-mini, achieved comparable balance (F1 = 0.772). Gemini-2.5-Flash (F1 = 0.763) and DeepSeek-Chat-v3 (F1 = 0.759) constituted a second performance tier. Gemini emphasized recall (0.807) at the cost of lower precision (0.729), while DeepSeek provided more balance (precision = 0.755, recall = 0.771). Llama-4-Maverick (F1 = 0.740) and Claude-3.5-Haiku (F1 = 0.732) consistently underperformed across prompt types, with weaker precision and greater instability. In all models, accuracy and recall exceeded precision, which suggests a systematic bias toward over-inclusion (i.e., more reliably capturing relevant items than excluding FPs). Overall, OpenAI models (GPT-4o and GPT-4o-mini) stay most stable across tasks, while Gemini and DeepSeek remain competitive but more prompt-sensitive. Claude and Llama are recall-oriented but often sacrifice precision.
Macro-averaged evaluation metrics for each model (across prompts and tasks)
Table 7 Long description
The table has five columns: Model, Accuracy, Precision, Recall, and F1. GPT-4o: Accuracy 0.886, Precision 0.757, Recall 0.808, F1 0.778. GPT-4o-mini: Accuracy 0.880, Precision 0.746, Recall 0.816, F1 0.772. Gemini-2.5-Flash: Accuracy 0.879, Precision 0.729, Recall 0.807, F1 0.763. DeepSeek-Chat-v3: Accuracy 0.878, Precision 0.755, Recall 0.771, F1 0.759. Llama-4-Maverick: Accuracy 0.869, Precision 0.725, Recall 0.766, F1 0.740. Claude-3.5-Haiku: Accuracy 0.860, Precision 0.708, Recall 0.773, F1 0.732.
The precision–recall trade-off chart (Figure 8b) reinforces these patterns. GPT-4o and GPT-4o-mini sustain a strong equilibrium between inclusivity and selectivity. Gemini achieves comparable recall but lower precision, reflecting over-inclusivity, while DeepSeek maintains a mid-range balance. Llama and Claude demonstrate the weakest trade-off, with both precision and recall falling at the lower end. Taken together, no model dominates across dimensions, but GPT-4o and GPT-4o-mini consistently provide the most reliable equilibrium.
F1 and precision–recall trade-off across models.
F1 by model.
Precision–recall by model.
Figure 8 Long description
The figure contains two side-by-side panels comparing six large language models using F1 score, precision, and recall. Panel (a) is a horizontal bar chart titled “F1 by model.” Models are ordered from highest to lowest F1 score. GPT-4o has the highest F1 score at 0.778, followed by GPT-4o-Mini at 0.772, Gemini-2.5-Flash at 0.763, DeepSeek-Chat-V3 at 0.759, Llama-4-Maverick at 0.740, and Claude-3.5-Haiku at 0.732. The horizontal axis ranges from 0.70 to 0.80.Panel (b), titled “Precision–recall by model,” is a scatter plot with precision on the horizontal axis, ranging from 0.70 to 0.78, and recall on the vertical axis, ranging from 0.76 to 0.83. Each model is represented by a labelled cross. GPT-4o-Mini has the highest recall, approximately 0.816, with precision around 0.746. GPT-4o has recall around 0.808 and the highest precision, approximately 0.757. Gemini-2.5-Flash has recall around 0.807 and precision around 0.729. DeepSeek-Chat-V3 has precision around 0.755 and recall around 0.771. Claude-3.5-Haiku has the lowest precision, about 0.708, with recall around 0.773. Llama-4-Maverick has precision around 0.725 and the lowest recall, approximately 0.766.
4.2.3 Effect of classification tasks
Performance varied markedly across the six classification tasks, reflecting both dataset balance and task complexity (Table 8).
Macro-averaged evaluation metrics for each task (across prompts and models)
Table 8 Long description
The table has five columns: Task, Accuracy, Precision, Recall, and F1. Retrieval: Accuracy 0.851, Precision 0.792, Recall 0.831, F1 0.810. Screening: Accuracy 0.861, Precision 0.874, Recall 0.864, F1 0.868. Searching: Accuracy 0.875, Precision 0.745, Recall 0.797, F1 0.766. Synthesis: Accuracy 0.837, Precision 0.631, Recall 0.662, F1 0.642. Writing: Accuracy 0.953, Precision 0.640, Recall 0.797, F1 0.702. Use of LLMs: Accuracy 0.960, Precision 0.908, Recall 0.946, F1 0.925. LLM-use detection and Screening show the highest F1; Synthesis the lowest.
High-performing tasks: Among traditional SLR stages, screening produced the strongest results (F1 = 0.868), consistent with its prevalence in our dataset (260 examples); this provided stable training signals. Retrieval also performed well (F1 = 0.810), balancing precision and recall effectively.
Moderate performance: Searching produced less reliable results (F1 = 0.766), despite a reasonable sample size (147). This is likely because variable semantic descriptions (of “searching”) hindered generalization.
Low-performing tasks: Synthesis (F1 = 0.642) and writing (F1 = 0.702) yielded the weakest results. Both were underrepresented in our dataset (109 and 34 examples, respectively) and demanded higher-order reasoning (summarization, integration, and generative framing) that was less lexically anchored. Writing also derived unusually high accuracy (0.953) but very low precision (0.640); this suggests systematic over-classification of irrelevant literature.
LLM use as a distinct dimension: LLM-use detection achieved the highest overall performance (F1 = 0.925; accuracy = 0.960). This reflects (1) the ubiquity of such research (127 examples) and (2) the salience of lexical cuing (e.g., “ChatGPT,” “large language model,” and “transformer”) to reduce ambiguity. Precision (0.908) and recall (0.946) were jointly high: models seldom missed or over-predicted LLM-related literature.
Generally, task performance follows a gradient from concrete and lexically anchored (LLM use, screening, and retrieval) to abstract and conceptually demanding (synthesis and writing). LLM-use detection showed the strongest observed classification profile in this corpus, whereas synthesis and writing remain constrained by data scarcity and semantic abstraction.
4.2.4 Interactions among prompts, models, and classification tasks
While macro-averages isolate the respective effects of prompting strategy and model choice, their interaction shapes overall performance. Each model responds differently to a given prompting method, and each task responds differently to a given prompt-LLM pairing. This section examines (1) prompt performance across models, (2) prompt performance across tasks, and (3) model performance across tasks. Unsurprisingly, we find that the optimal configuration varies with context.
Model-specific prompting strategies: Table A3 (in Appendix A) and Figure 9 report the observed best-performing prompting method for each LLM. They reveal distinct model-specific preferences; no universally optimal strategy exists. With CoT-few-shot prompting, GPT-4o achieved the highest overall accuracy (0.907); thus, structured reasoning and exemplar calibration seem to pair well for large-scale models. By contrast, GPT-4o-mini performed best with CoT prompting and scored the highest F1 among compact models (0.830). As such, smaller architectures may benefit disproportionately from explicit reasoning instructions (to offset limited representational capacity). Claude-3.5-Haiku, DeepSeek-Chat-v3, and Gemini-2.5-Flash performed optimally with zero-shot prompting, reaching competitive accuracies (0.876–0.901) and consistently high recall (0.850–0.857). These models appear to generalize effectively without additional scaffolding. However, their lower precision (0.713–0.774) indicates a higher FP risk (thereby limiting their precision-sensitive applications). Llama-4-Maverick aligned best with CoT-few-shot (accuracy = 0.887; F1 = 0.774), where reasoning and exemplar guidance came together to counteract Llama’s recall–precision imbalance.
Prompt performance for each LLM (tasks classifications).
Figure 9 Long description
The figure is a heatmap showing the observed best-performing model–prompt combinations for six LLMs in task classification. Rows list each model with its prompt strategy, and columns show Accuracy, Precision, Recall, and F1. The colour scale ranges from 0.70 to 0.90, with darker purple indicating higher scores. GPT-4o / CoT-few-shot has the highest Accuracy and Precision, while GPT-4o-mini / CoT has the highest Recall and F1. Zero-shot is shown as the best-performing prompt for Claude-3.5-Haiku, DeepSeek-Chat-v3, and Gemini-2.5-Flash.
In sum, our findings suggest a bifurcation: reasoning-oriented prompts (CoT and CoT-few-shot) enhance performance in GPT models and Llama-4, while zero-shot remains most effective for Claude, DeepSeek, and Gemini. Importantly, for prompting to prove effective, model architecture and the prompt’s cognitive scaffolding must align; such harmony matters more than prompt complexity.
Classification-task-specific prompting strategies: As shown in Table A4 (Appendix A) and Figure 10, the observed optimal prompting strategy varies by task. For LLM-use detection, CoT prompting achieved near-perfect results (accuracy = 0.969; F1 = 0.941); accordingly, structured reasoning matters very much to this task. In retrieval, zero-shot prompting performed best, yielding strong recall (0.861) and competitive accuracy (0.865); however, its moderate precision (0.803) reflects inclusivity at the expense of selectivity. Screening benefited most from few-shot prompting, which balanced high precision (0.888) with strong recall (0.865): annotated exemplars minimized FPs and negatives alike. Similarly, few-shot optimized searching, but precision remained lower (0.740): broad retrieval patterns improved recall (0.816) while reducing specificity. Overall, synthesis remains the weakest task. Zero-shot prompting yielded low precision (0.612) and F1 (0.662) because abstract integration requires structured reasoning or richer contextual input. Writing, by contrast, performed best under CoT-few-shot conditions, achieving high accuracy (0.961) but only moderate precision (0.730); these metrics reflect frequent over-inclusion despite reliable detection.
Prompt performance for each task.
Figure 10 Long description
The figure is a heatmap showing the best-performing prompt strategy for each SLR task. Rows show task–prompt combinations, and columns show Accuracy, Precision, Recall, and F1. The colour scale ranges from 0.60 to 0.95, with darker purple indicating higher scores. Use of LLMs with CoT performs best overall, with the highest F1 score of 0.941. Screening with Few-shot and Retrieval with Zero-shot also perform well. Synthesis and Writing have lower F1 scores, showing that prompt performance varies across tasks.
Generally, reasoning-oriented prompts (CoT and CoT-few-shot) excel in conceptually demanding tasks (LLM detection and writing classification). Screening and searching classification benefit most from example-based prompts (few-shot) because such tasks require balanced trade-offs. Zero-shot prompting works well for retrieval but remains inadequate for synthesis classification because it requires deeper contextual reasoning.
Classification-task-specific LLMs: As seen in Table A5 (Appendix A) and Figure 11, the observed best-performing LLM systematically varies by SLR stage/LLM-use detection. GPT-4o achieved the highest performance in LLM-use detection (accuracy = 0.978; F1 = 0.957) because it could capture explicit lexical markers of model usage. For retrieval classification, DeepSeek-Chat-v3 provided the best balance (precision = 0.847; recall = 0.847; F1 = 0.847); it was broadly inclusive but did not excessively count FPs. Gemini-2.5-Flash (F1 = 0.905) best supported screening classification; it combined high precision (0.895) with strong recall (0.915), benefiting borderline inclusion decisions. For searching classification, GPT-4o-mini delivered the strongest results (accuracy = 0.923; recall = 0.929), though its lower precision (0.803) suggests a bias toward over-retrieval. GPT-4o-mini also performed best in writing classification (accuracy = 0.974; recall = 0.971) but was only moderately precise (0.733) and tended toward over-inclusivity. Synthesis classification was weakest because it involved semantically complex content; GPT-4o led this task (F1 = 0.698), driven by recall (0.807) but limited by low precision (0.615).
LLM performance for each task.
Figure 11 Long description
The heatmap shows Accuracy, Precision, Recall, and F1 for the best-performing LLM in each SLR task category. For LLM-use detection, GPT-4o scores highest (F1 0.957). For retrieval, DeepSeek-Chat-v3 leads (F1 0.847). For screening, Gemini-2.5-Flash performs best (F1 0.905). For search, GPT-4o-mini leads (F1 0.861). GPT-4o performs best for synthesis (F1 0.698), while GPT-4o-mini leads writing (F1 0.835). The figure shows task-specific model strengths rather than one LLM dominating all tasks.
In aggregate, clear patterns emerge: GPT-4o and GPT-4o-mini dominate tasks anchored by explicit lexical cues (LLM use, searching, and writing), while Gemini-2.5-Flash and DeepSeek-Chat-v3 excel in tasks requiring interpretive judgment (screening and retrieval). Across models, synthesis classification remains a persistent limitation: to improve precision in abstract-reasoning tasks, more advanced approaches are required.
4.2.5 Confidence intervals for task data
As shown in Table A6 (Appendix A), accuracy CIs were uniformly narrow, reflecting large denominators and metric stability. Precision and recall produced wider intervals, especially in tasks with fewer positives (e.g., synthesis and writing) because class imbalance magnifies variability. F1 CIs, derived via bootstrap resampling, were broader than accuracy CIs but accurately reflected uncertainty. In brief, stable tasks (like screening) tend to yield tighter intervals, while conceptually complex and data-sparse tasks tend to yield broader intervals; thus, we should cautiously interpret small performance differences.
5 Discussion
In this section, we (1) synthesize our main findings, (2) outline cost implications, (3) acknowledge key limitations, and (4) suggest avenues for future research. We use accuracy as a measure of overall correctness, and we use F1 to take a balanced look at precision and recall. In doing so, we clarify trade-offs inherent in different prompting strategies and model choices.
5.1 Classification of relevance to SLR automation
To answer RQ1, we now discuss how prompt design and model choice individually and jointly affect screening performance. To begin, by evaluating prompting strategies and model-specific effects in relevance classification, we yield several practical insights.
First, zero-shot and CoT prompting are most appropriate for recall-critical first-pass relevance screening because they reduce the risk of missing relevant studies. CoT-few-shot remains valuable, but mainly as a more selective strategy when the aim is to reduce FPs and reviewer workload.
Second, CoT-few-shot achieved stronger precision and F1, but it should not be treated as the default sole exclusion strategy for Level 1 screening. A layered approach remains practical: use a high-recall prompt, such as zero-shot or CoT, for broad inclusion, followed by CoT-few-shot or few-shot for refinement, triage, or workload reduction.
Third, self-reflection prompting underperforms by over-including FPs without commensurate recall gains. Self-reflection also exhibits high cross-model variance, which suggests strong prompt–model interaction. Mechanistically, reflection encourages justificatory reasoning weakly anchored to explicit decision rules; without exemplar grounding, a model can rationalize initial errors without correcting them. By contrast, CoT-few-shot stabilizes decisions by pairing stepwise reasoning with concrete examples, thus yielding a better precision–recall balance.
Finally, our results highlight the importance of model–prompt alignment. Some configurations are more suitable for high-recall first-pass screening, while others are better suited to precision-oriented refinement. Thus, prompt design should be tailored to model strengths and to the intended screening role, rather than relying on a universally superior configuration.
5.2 Classification tasks corresponding to SLR stages and LLM use
We now continue to answer RQ1 by moving from relevance classification to task classification. Across task dimensions, prompting strategies systematically influence the recall–precision balance: zero-shot maximizes inclusivity, while CoT-few-shot achieves the most reliable equilibrium. GPT-4o and GPT-4o-mini perform most consistently across tasks, while Gemini and DeepSeek remain competitive but more prompt-sensitive. Claude and Llama emphasize recall but require structured prompting to attain adequate precision (Figure 12, left panel).
Comparative performance of observed best prompts and LLMs across tasks.
Figure 12 Long description
The figure contains three bar-chart panels reporting F1 scores. The left panel shows the best prompt method for each LLM, with GPT-4o-mini using CoT highest at 0.830. The middle panel shows the best prompt method for each task: CoT for LLM-use, Zero-shot for retrieval and synthesis, Few-shot for screening and search, and CoT-few-shot for writing. The right panel shows the best LLM for each task: GPT-4o for LLM-use and synthesis, DeepSeek-Chat-v3 for retrieval, Gemini-2.5-Flash for screening, and GPT-4o-mini for search and writing. Overall, the figure shows that the strongest configuration varies by task and model.
Performance varies considerably by task. Screening and retrieval classification yield the highest accuracy, reflecting clearer operational definitions and stronger dataset representation. By contrast, conceptual complexity and limited data constrain synthesis and writing classification; therefore, such tasks remain most challenging. LLM-use detection outperforms all other categories, achieving very high precision and recall. We may attribute this success to distinctive lexical markers (e.g., “ChatGPT” and “large language models”) (Figure 12, mid and right panels). Therefore, no universally optimal configuration exists: outcomes depend on interactions among task characteristics, model behavior, and prompt design. A layered strategy (using zero-shot to maximize recall before using CoT-few-shot to maximize precision) emerges as a pragmatic semi-automation approach.
In sum, LLMs already provide reliable support for screening classification, retrieval classification, and LLM-use detection. However, more abstract tasks (synthesis classification and writing classification) require additional methodological innovation and larger datasets before they can reach comparable automation levels.
5.3 Cost analysis
To answer RQ2, we begin by evaluating different model–prompt combinations’ cost-performance implications. In Table 9, we report costs of relevance classification per 1,000 abstracts across model–prompt pairings, alongside corresponding F1 scores. Cost estimates were calculated from the recorded input and output token counts for each model–prompt configuration, using the API pricing available at the time of experimentation. Input tokens included the system message, task instructions, prompt template, examples where applicable, and title–abstract text, while output tokens included the model-generated response and final classification output. We find that across model–prompt pairings, absolute-dollar cost differences remain substantial. In particular, GPT-4o-Mini incurred very low costs across all prompt types (holding costs equal, GPT-4o-Mini’s F1 was highest for 4/5 prompts). Llama-4-Maverick only led cost savings when paired with CoT-few-shot (+0.008), relative to larger models (e.g., GPT-4o and Claude-3.5-Haiku). Structured prompts (CoT and CoT-few-shot) delivered acceptable F1 on GPT-4o-Mini, despite increased token usage. Accordingly, prompts can be prototyped by pairing GPT-4o-Mini with CoT/CoT-few-shot (and, where useful, brief self-reflection) to inexpensively explore instruction variants (before using a higher-capacity model to conduct targeted validation). Furthermore, to take a first pass over a large volume of abstracts, low-cost models (e.g., GPT-4o-Mini and DeepSeek-Chat-V3) can be paired with structured prompts (CoT and CoT-few-shot) to favorably balance cost and performance; they should only escalate borderline and contentious cases for adjudication by a stronger model (e.g., GPT-4o). This staged strategy preserves most attainable accuracy and lowers absolute expenditure.
Cost of relevance classification per 1,000 abstracts (tokens are totals; cost in USD)
Table 9 Long description
The table lists prompt types as rows and reports input tokens, output tokens, and then cost (USD) and F1 for each of six LLMs. Token counts range from 419,060 input tokens (Zero-shot) to 874,060 (CoT-few-shot). GPT-4o-Mini is consistently the lowest-cost model (Zero-shot: $0.09; CoT-few-shot: $0.18). Llama-4-Maverick matches GPT-4o-Mini in cost. DeepSeek-Chat-V3 is also low-cost (Zero-shot: $0.13). GPT-4o is the most expensive (Zero-shot: $1.43; CoT-few-shot: $3.04) and achieves the highest F1 under Zero-shot (0.934). Self-reflection is generally costly with lower F1. Full data is in the table.
5.4 Limitations and future directions
Domain scope is the primary limitation of this study. We curated the dataset around SLR automation, so it does not fully represent the range of screening tasks typically encountered in systematic reviews. In many screening tasks, eligibility decisions depend on nuanced domain-specific criteria, such as population, intervention, comparator, outcome, study design, setting, or context, and relevant and irrelevant abstracts may use similar terminology. By contrast, abstracts in our corpus may contain more salient topic-specific lexical markers, such as “systematic review,” “screening,” “automation,” “machine learning,” or “LLM.” These cues may make the classification task easier than screening in domains with weaker lexical separation and more ambiguous eligibility criteria. Consequently, the absolute performance estimates reported here may overestimate performance in broader systematic review screening contexts. Our findings should therefore be interpreted primarily as evidence of prompt–model interaction effects within corpora similar to the SLR-automation corpus examined in this study, rather than as general estimates of LLM screening performance across all evidence-synthesis domains.
We evaluated manually refined prompting strategies but not systematic iterative methods, meta-prompting methods,Reference Shinn, Cassano, Gopinath, Narasimhan and Yao 54 , Reference Zhang, Yuan and Yao 59 or declarative prompt optimizationReference Susnjak 60 (which might enable more effective, efficient prompt refinement). We did not test RAG, although RAG may improve precision at a fixed, high recall and yield more stable, transferable predictions across heterogeneous abstracts by grounding decisions in task-relevant passages. In addition, class balance and prevalence varied across Level-2 tasks (e.g., we encountered relatively few “writing” and “synthesis” papers); this may have increased uncertainty and depressed F1 in abstraction-heavy tasks. Also, despite consensus labeling, manual annotation may have introduced residual subjectivity, especially in under-represented categories (like synthesis and writing). Finally, our evaluation emphasizes quantitative metrics without directly assessing qualitative dimensions (e.g., interpretability and trustworthiness).
A further limitation is the rapid pace of LLM development. The models evaluated here were contemporary and widely used at the time of experimentation, but they should not be treated as a fixed or permanent set of “state-of-the-art” systems. Newer model versions may differ in performance, cost, context length, and API behaviour. Therefore, the absolute performance and cost estimates reported in this study may become dated as model ecosystems evolve. Because the evaluated LLMs are closed or externally maintained systems, we cannot fully determine whether any records or closely related abstracts appeared in their pretraining data. Although no fine-tuning was performed and the evaluation used an independently annotated dataset, potential pretraining exposure cannot be ruled out.
Future work should extend this study in several directions. First, to address the domain-scope limitation, the same model–prompt configurations should be validated on standardised and cross-domain screening benchmarks beyond SLR automation, including datasets with subtler lexical signals, more ambiguous eligibility criteria, and domain-specific screening tasks such as PICO-guided clinical reviews or evidence syntheses in social policy, education, or ecology contexts. Second, future studies should examine whether RAG, prompt optimisation, and more adaptive prompting strategies can improve recall while controlling FPs and cost. Third, these approaches should be evaluated under realistic human-in-the-loop screening conditions, including prioritisation, reviewer feedback, stopping rules, FN monitoring, workload reduction, and cost tracking. Such work would help determine whether the prompt–model interaction patterns observed in this specialised SLR-automation corpus transfer to broader evidence-synthesis screening settings while supporting the development of more practical, reliable, and cost-aware LLM-assisted review workflows. Finally, the same evaluation should be replicated with newer models and updated API versions to determine whether the observed prompt–model interaction patterns remain stable.
5.5 Practical recommendations
Based on our task-specific evaluation of prompting strategy and LLM performance, several practical recommendations emerge for LLM integration into SLR workflows:
-
• Account for task complexity: Supply detailed inclusion/exclusion criteria and contextual framing to scaffold reasoning when targets are less lexically anchored.
-
• Use high-recall prompts for first-pass screening: For Level 1 relevance screening, use zero-shot or CoT prompting when the priority is to minimise missed relevant studies.
-
• Use CoT-few-shot for precision-oriented refinement: After a high-recall first pass, use CoT-few-shot or few-shot prompting to reduce FPs and reviewer workload.
-
• Align prompts with model characteristics: GPT-4o and DeepSeek remain robust across prompting strategies, while Gemini and Llama benefit most from precision-enhancing prompts (e.g., CoT-few-shot).
-
• Prefer reasoning-based prompts for abstract tasks: Use CoT or CoT-few-shot for classifications that demand deeper contextual reasoning.
-
• Cost-aware prompt development: Prototype on a small calibration set using a low-cost model (e.g., GPT-4o-mini), applying CoT or CoT-few-shot to tune instructions. Validate on a higher-capacity model (e.g., GPT-4o) only if doing so would deliver a measurable performance gain for your intended use.
-
• First-pass at scale: When confronted with a large volume of abstracts, combine a low-cost model (e.g., GPT-4o-mini or DeepSeek-Chat-V3 where available) with structured prompts (CoT/CoT-few-shot) to achieve a favorable cost-performance balance. Escalate only borderline/contentious cases to a stronger model.
Generally, clearly defined instructions and contextual cues lead to effective prompting, while LLM-specific and task-specific characteristics guide strategy selection.
6 Conclusion
According to our cross-model evaluation, prompting strategy, model choice, and task type jointly determine screening effectiveness. With regard to prompting strategy, zero-shot and CoT are most suitable for high-sensitivity first-pass relevance screening, while CoT-few-shot offers stronger precision and F1 for later triage or workload reduction; self-reflection performs worst because it exhibits high model-specific variance and over-inclusivity. With regard to model choice, GPT-4o and DeepSeek deliver strong all-round results, while GPT-4o-Mini performs competitively at a markedly lower cost. Our relevance-classification cost analysis confirms substantial absolute differences among model–prompt pairings. Thus, it suggests that future workflows could incorporate staged, cost-aware procedures, using lower-cost, high-recall configurations for initial screening and reserving higher-cost or more precision-oriented configurations for later triage or uncertain records. These conclusions apply primarily to SLR-automation corpora with salient lexical cues; they may or may not transport well to PICO-anchored/less explicit domains. Future work should include cross-domain external validation and evaluation of retrieval-augmented and prompt-optimization methods under human-in-the-loop conditions with prospective workload and cost tracking.
Author contributions
Conceptualization: B.H. and T.S.; Data curation: B.H.; Formal analysis: B.H.; Funding acquisition: A.M. and T.S.; Investigation: B.H.; Methodology: B.H.; Project administration: B.H.; Resources: B.H.; Software: B.H.; Supervision: A.M. and T.S.; Validation: B.H.; Visualization: B.H.; Writing—original draft: B.H.; Writing—review and editing: B.H., A.M., V.U.W., and T.S. All authors approved the final submitted draft.
Competing interests
The authors declare that no competing interests exist.
Data availability statement
Where redistribution of bibliographic records is restricted by database licensing, derived metadata, labels, prompts, and code are available from the corresponding author upon reasonable request to support reproducibility.
Funding statement
This research was partially supported by the Massey University College of Sciences REaDI funds 2025.
Ethical standards
The research meets all ethical guidelines, including adherence to the legal requirements of the study country. This manuscript used OpenAI’s GPT-4 to refine sentence structure. All intellectual property in this document belongs solely to the authors. GPT-4 did not contribute to the original ideas, insights, or intellectual content of the work.
A Additional data tables and their brief analysis
Evaluation metrics for prompt–model combinations in relevance classification
Table A1 Long description
The table reports performance metrics for all combinations of six LLMs and five prompt types (30 rows), with 95% confidence intervals for each metric. Notable results: GPT-4o with Zero-shot achieves the highest F1 (0.934, CI [0.918, 0.949]); Llama-4-Maverick with Zero-shot achieves the highest recall (0.998, CI [0.994, 1.000]); GPT-4o with COT-few-shot achieves the highest precision (0.952, CI [0.932, 0.971]); GPT-4o with Zero-shot achieves the highest accuracy (0.952, CI [0.940, 0.963]); Self-reflection generally yields lower F1. CoT-few-shot shifts results toward higher precision at some cost to recall across most models.
Note: All metrics are reported with 95% confidence intervals in brackets. Boldface highlights the best-performing point estimate within each prompt type and metric (CIs not bolded). PT = Prompt type (CoT, CoT-few-shot, Few-shot, Self-reflection, and Zero-shot).
A.1 Evaluation metrics for prompt–model combinations in relevance classification
Table A1 reports relevance-classification performance (accuracy, precision, recall, and F1 with 95% CIs) for six LLMs across five prompting strategies (CoT, CoT-few-shot, Few-shot, Self-reflection, and Zero-shot). Patterns indicate strong model–prompt interactions: GPT-4o attains the highest F1 under Zero-shot (0.934) and Few-shot (0.927), while DeepSeek leads under CoT-few-shot (0.921). CoT-few-shot generally shifts the operating point toward higher precision at some cost to recall (e.g., GPT-4o Precision 0.952 with Recall 0.886), whereas Zero-shot and CoT favor very high recall (often
$\geq $
0.98 for Gemini and Llama) with correspondingly lower precision and middling F1. Self-reflection underperforms relative to simpler prompts (notably for GPT-4o-Mini and Llama), suggesting limited benefit for this task. In short, if minimizing FPs is paramount, CoT-few-shot is preferable; if maximizing inclusivity (recall) is the goal, Zero-shot/CoT are more suitable, with GPT-4o consistently the most robust.
A.2 Observed best-performing prompt types by model for relevance classification
Table A2 shows evaluation metrics of observed best-performing prompt types by model for relevance classification. GPT-4o attains the top result under Zero-shot (Acc 0.952, Prec 0.921, and F1 0.934), showing strong out-of-the-box calibration. Peaks vary by model, DeepSeek-Chat-V3 with CoT (F1 0.923), Llama-4-Maverick with CoT-few-shot (F1 0.918), indicating prompt–model interactions. Gemini-2.5’s best Few-shot setting prioritizes recall (0.980) over precision (0.860), whereas GPT-4o’s Zero-shot is more precision-balanced. In brief, best-of-prompt F1s cluster tightly (0.903–0.934), with a modest but consistent frontier-model edge.
Observed best-performing prompt types by model (relevance classification)
Table A2 Long description
The table has five columns: Model, Prompt Type, Accuracy, Precision, Recall, and F1. GPT-4o: Zero-shot (Accuracy 0.952, Precision 0.921, Recall 0.947, F1 0.934). DeepSeek-Chat-V3: CoT (Accuracy 0.943, Precision 0.901, Recall 0.945, F1 0.923). Llama-4-Maverick: CoT-few-shot (Accuracy 0.939, Precision 0.879, Recall 0.961, F1 0.918). Gemini-2.5: Few-shot (Accuracy 0.935, Precision 0.860, Recall 0.980, F1 0.916). GPT-4o-Mini: Few-shot (Accuracy 0.936, Precision 0.880, Recall 0.951, F1 0.914). Claude-3.5-Haiku: CoT-few-shot (Accuracy 0.927, Precision 0.859, Recall 0.953, F1 0.903).
A.3 Observed best-performing prompt method for each LLM of tasks classifications
Table A3 lists, for each LLM, the prompt method that yielded its best task-classification performance and the associated metrics. Zero-shot is optimal for Claude-3.5-Haiku, DeepSeek-Chat-v3, and Gemini-2.5-Flash, reflecting strong transfer with minimal prompt design but a recall-leaning profile (e.g., Claude Recall 0.853 vs. Precision 0.713). Structured prompting benefits others: Llama-4-Maverick and GPT-4o peak under CoT-few-shot, with GPT-4o achieving the highest Accuracy (0.907) and Precision (0.834), while GPT-4o-mini attains the highest Recall (0.877) and F1 (0.830) under plain CoT. The pattern suggests that frontier or architecture-sensitive models exploit reasoning structure for better calibration, whereas several models still reach their best with simple zero-shot prompting.
Observed best-performing prompt method for each LLM (tasks classifications)
Table A3 Long description
The table has five columns: Model, Best Prompt, Accuracy, Precision, Recall, and F1. Claude-3.5-Haiku: Zero-shot (Accuracy 0.876, Precision 0.713, Recall 0.853, F1 0.775). DeepSeek-Chat-v3: Zero-shot (Accuracy 0.901, Precision 0.774, Recall 0.850, F1 0.806). Gemini-2.5-Flash: Zero-shot (Accuracy 0.895, Precision 0.753, Recall 0.857, F1 0.799). Llama-4-Maverick: CoT-few-shot (Accuracy 0.887, Precision 0.783, Recall 0.771, F1 0.774). GPT-4o: CoT-few-shot (Accuracy 0.907, Precision 0.834, Recall 0.789, F1 0.810). GPT-4o-mini: CoT (Accuracy 0.902, Precision 0.795, Recall 0.877, F1 0.830).
A.4 Observed best-performing prompt method for each task of automation tasks classifications
Table A4 identifies, for each task, the prompt that yields the best performance. CoT dominates LLM use with the top scores across all metrics (Accuracy 0.969, Precision 0.929, Recall 0.954, and F1 0.941), indicating prominent clues. Screening and Searching favor Few-shot, with Screening achieving the strongest F1 (0.876; Precision 0.888, Recall 0.865), suggesting exemplars calibrate decision boundaries effectively. Retrieval and Synthesis peak under Zero-shot (F1 0.830 and 0.662, respectively), implying broad generalization without examples can help, though Synthesis remains the hardest (low Precision 0.612). Writing prefers CoT-few-shot with very high Accuracy (0.961) but modest F1 (0.715), consistent with class imbalance or threshold sensitivity in generative judgments.
Observed best-performing prompt method for each task
Table A4 Long description
The table has five columns: Task, Best Prompt, Accuracy, Precision, Recall, and F1. LLM use: CoT (Accuracy 0.969, Precision 0.929, Recall 0.954, F1 0.941). Retrieval: Zero-shot (Accuracy 0.865, Precision 0.803, Recall 0.861, F1 0.830). Screening: Few-shot (Accuracy 0.870, Precision 0.888, Recall 0.865, F1 0.876). Searching: Few-shot (Accuracy 0.877, Precision 0.740, Recall 0.816, F1 0.774). Synthesis: Zero-shot (Accuracy 0.836, Precision 0.612, Recall 0.726, F1 0.662). Writing: CoT-few-shot (Accuracy 0.961, Precision 0.730, Recall 0.709, F1 0.715).
A.5 Observed best-performing LLM for each task of automation tasks classifications
Table A5 names the top LLM for each task and its scores. Performance leadership is task-specific: GPT-4o leads LLM use (Accuracy 0.978; Precision 0.953; F1 0.957) and Synthesis (F1 0.698), though the latter’s low Precision (0.615) underscores task difficulty. Retrieval is best with DeepSeek-Chat-v3 (balanced Precision/Recall 0.847), while Screening favors Gemini-2.5-Flash (F1 0.905 from high Recall 0.915 with strong Precision 0.895). GPT-4o-mini excels in Searching and Writing, driven by very high Recall (0.929 and 0.971) but more modest Precision (0.803 and 0.733), making it well-suited for inclusive first-pass triage at the cost of more FPs. On balance, the winners differ by task, reflecting clear precision–recall trade-offs rather than a single universally best model.
Observed best-performing LLM for each task
Table A5 Long description
The table has six columns: Task, Best LLM, Accuracy, Precision, Recall, and F1. LLM use: GPT-4o (Accuracy 0.978, Precision 0.953, Recall 0.960, F1 0.957). Retrieval: DeepSeek-Chat-v3 (Accuracy 0.882, Precision 0.847, Recall 0.847, F1 0.847). Screening: Gemini-2.5-Flash (Accuracy 0.899, Precision 0.895, Recall 0.915, F1 0.905). Searching: GPT-4o-mini (Accuracy 0.923, Precision 0.803, Recall 0.929, F1 0.861). Synthesis: GPT-4o (Accuracy 0.846, Precision 0.615, Recall 0.807, F1 0.698). Writing: GPT-4o-mini (Accuracy 0.974, Precision 0.733, Recall 0.971, F1 0.835).
Evaluation metrics across tasks, prompt types, and models
Table A6 Long description
The table reports four metrics with 95% CIs for all combinations of six tasks, four prompt types, and six LLMs. Easier tasks (LLM-use, Screening) yield uniformly high scores; Synthesis remains the most challenging with lower precision and F1 across all configurations. Zero-shot and CoT maximise recall; CoT-few-shot shifts toward higher precision at some cost to recall. GPT-4o generally achieves the strongest macro F1; DeepSeek-Chat-V3 is close behind; Gemini-2.5-Flash often posts the highest recall. Full data with confidence intervals for all combinations is in Appendix A.
Note: All metrics are reported with 95% confidence intervals in brackets. Boldface highlights the best-performing point estimate within each prompt type (under each task) and metric (CIs not bolded). The first two columns (TK, PT) are displayed vertically to save width. TK = Classification tasks. PT = Prompt type.
A.6 Evaluation metrics across tasks, prompt types, and models
Table A6 summarizes Accuracy, Precision, Recall, and F1 (with 95% CIs) across all tasks, prompt types, and models. Patterns indicate strong prompt-task interactions: Zero-shot and CoT tend to maximize recall (useful for inclusive first-pass screening), while CoT-few-shot shifts toward higher precision at some cost to recall. Easier tasks (LLM use, Screening) show uniformly high scores, whereas Synthesis remains challenging with lower precision and F1. Among models, GPT-4o generally attains the strongest macro F1 with DeepSeek close behind; Gemini often posts the highest recall, and smaller/open models trail mainly on precision. In short, there is no single best prompt—choose recall-leaning prompts for sensitivity and CoT-few-shot when FPs are costlier.
B Prompt templates
B.1 Zero-shot prompt template for SLR relevance classification
You are analyzing academic articles to determine whether they are relevant to the automation of the systematic literature review (SLR) process.
For each article, you are provided with the title and abstract. Based on this information, answer the following question:
Does the article discuss, study, or present methods related to the automation of systematic literature reviews or their component tasks?
Definition of the SLR process:
The systematic literature review (SLR) process includes the following five tasks:
- Searching—retrieving relevant articles from databases or repositories.
- Screening—selecting or filtering articles based on inclusion/exclusion criteria.
- Retrieval—extracting information or data from article content.
- Synthesis—summarizing, integrating, or organizing findings from selected studies.
- Writing—generating narrative or structured text for the review.
Automation methods may include algorithms, rule-based systems, traditional machine learning, or large language models (LLMs), such as GPT, BERT, etc.
Inclusion criteria:
Classify the article as **relevant** if **any** of the following apply:
1. The article discusses or investigates the automation of systematic literature reviews or literature reviews, including conceptual discussions, tool development, system design, or frameworks, even if specific SLR tasks are not mentioned.
2. The article discusses, studies, or applies automation methods to perform **any** of the five defined SLR tasks—even if the article’s primary purpose is not described as automating literature reviews—as long as the tasks are performed on academic or scientific literature.
Exclusion criteria:
The article is **not relevant** if:
- It is a systematic review or literature review about automation applied in other domains (e.g., robotics, healthcare, and smart factories), but does not address automation of literature reviews themselves.
Response format:
For each article, respond with: ‘‘Yes’’ if it meets the inclusion criteria, ‘‘No’’ if it matches the exclusion criteria or if there is insufficient information.
Now, analyze the following list of articles (each includes a title and abstract):
{title_abstract_pairs}
Return your analysis in JSON format, structured exactly like the following:
[
{
"Title": "<article title>",
"Relevant to SLR automation": "Yes" | "No"
}
]
Please strictly follow the JSON format, do not output any text other than the JSON objects. Make sure that double quotes inside value strings to be escaped as required by JSON.
B.2 CoT prompt template for SLR relevance classification
You are analyzing academic articles to determine whether they are relevant to the automation of the systematic literature review (SLR) process.
Each article includes a title and abstract. Follow the reasoning steps below and then answer the final question.
Step-by-step reasoning:
Step 1—Identify automation of literature review:
Is the article discussing, studying, or presenting methods related to automating literature reviews or systematic literature reviews?
The article is relevant **as long as it discusses any aspect of automating literature reviews**—including tool development, system architecture, conceptual frameworks, workflows, or general discussion—even if it does not specify which SLR task is automated.
Step 2—Check for automation of SLR tasks:
Does the article describe or apply automation techniques to perform **any** of the following five tasks, specifically when applied to academic or scientific literature?
- Searching—retrieving relevant articles from databases or repositories.
- Screening—selecting or filtering articles based on inclusion/exclusion criteria.
- Retrieval—extracting information or data from article content.
- Synthesis—summarizing, integrating, or organizing findings from selected studies.
- Writing—generating narrative or structured text for the review.
The article is relevant even if it does not mention literature reviews explicitly, as long as it automates one or more of these tasks on academic or scientific publications.
Step 3—Apply exclusion rules:
The article is not relevant if:
- It is a literature review or systematic review of automation in a domain unrelated to literature reviews (e.g., automation in medicine, robotics, or manufacturing).
- It uses automation for tasks unrelated to literature review and does not apply methods to academic or scientific literature.
Step 4—Conclusion*:
Based on the steps above, decide whether the article is relevant to automation of literature reviews or automates any part of the SLR process.
Final question:
**Is the article relevant to the automation of systematic literature review (SLR) or any of its defined tasks?** Respond with: ‘‘Yes’’ if it meets the inclusion criteria, ‘‘No’’ if it matches the exclusion criteria, or if there is insufficient information.
Response format:
Now, analyze the following list of articles. Each article includes a title and abstract:
{tittle and abstract pairs }
For each article, provide your reasoning and final decision in the following JSON format:
[
{{
"Title": "<article title>",
"Reasoning": "<brief explanation of your reasoning steps>",
"Relevant to SLR automation": "Yes" | "No"
}}
]
Please strictly follow the JSON format, do not output any text other than the JSON objects. Make sure that double quotes inside value strings to be escaped as required by JSON.
B.3 Zero-shot prompt template for SLR tasks classification
You are analyzing academic articles that have already been identified as relevant to the automation of systematic literature reviews (SLRs).
Each article includes a title and abstract. Your task is to determine:
1. Which SLR tasks are discussed or automated in the article? Select all applicable tasks from the list below:
- Searching—fetching or retrieving candidate articles from databases, repositories, or search engines** (e.g., PubMed and Scopus).
- Screening—selecting or filtering articles from the search results, based on inclusion/exclusion criteria, relevance, or study characteristics.
- Retrieval—extracting specific information or structured data from article content.
- Synthesis—summarizing, integrating, or organizing findings across multiple studies.
- Writing—generating narrative or structured text for the review.
2. Does the article mention the use of large language models (LLMs)?
Identify whether tools such as GPT, BERT, T5, or other LLMs are mentioned in the context of automating any of the above tasks.
Response format:
Respond clearly and concisely using the format below.
Now, analyze the following list of articles (each includes a title and abstract):
{title and abstract pairs}
Return your analysis in JSON format, structured exactly like the following:
[
{{
"Title": "<article title>",
"Searching automation": "Yes" | "No",
"Screening automation": "Yes" | "No",
"Retrieval automation": "Yes" | "No",
"Synthesis automation": "Yes" | "No",
"Writing automation": "Yes" | "No",
"Using LLMs": "Yes" | "No"
}}
]
Please strictly follow the JSON format, do not output any text other than the JSON objects. Make sure that double quotes inside value strings to be escaped as required by JSON.
B.4 CoT prompt template for SLR tasks classification
You are analyzing academic articles that have already been identified as relevant to the automation of systematic literature reviews (SLRs).
Each article includes a title and abstract. Your task is to determine:
1. Which SLR tasks are discussed or automated in the article?
Select from the following tasks (choose all that apply):
- Searching—fetching or retrieving candidate articles from databases, repositories, or search engines** (e.g., PubMed and Scopus).
- Screening—selecting or filtering articles from the search results, based on inclusion/exclusion criteria, relevance, or study characteristics.
- Retrieval—extracting specific information or structured data from article content.
- Synthesis—summarizing, integrating, or organizing findings across multiple studies.
- Writing—generating narrative or structured text for the review.
2. Does the article mention the use of large language models (LLMs)?
Determine whether the article discusses LLMs, such as GPT, BERT, T5, or similar, in the context of automating any part of the SLR process.
Reasoning instructions:
For each article, follow this reasoning process:
- First, briefly summarize what the article is about based on the title and abstract.
- Then, assess whether any of the five SLR tasks are described as being automated or supported.
- For each task, explain why you think it is (or is not) discussed.
- Finally, decide whether the article mentions the use of LLMs, citing keywords or model names if applicable.
Response format:
Then provide your structured output in the required JSON format.
Now, analyze the following list of articles. Each article includes a title and abstract:
{title and abstract pairs}
For each article, provide your reasoning and final decision in the following JSON format:
[
{{
"Title": "<article title>",
"Searching automation": "Yes" | "No",
"Screening automation": "Yes" | "No",
"Retrieval automation": "Yes" | "No",
"Synthesis automation": "Yes" | "No",
"Writing automation": "Yes" | "No",
"Using LLMs": "Yes" | "No"
}}
]
Please strictly follow the JSON format, do not output any text other than the JSON objects. Make sure that double quotes inside value strings to be escaped as required by JSON.
B.5 Self-reflection check in self-reflection prompt template for SLR relevance classification
Self-reflection check:
Before finalizing your decision, reflect on your reasoning:
- Did you mistakenly classify the article based only on keywords like “automation” or “systematic review” without considering the actual context?
- Did you overlook whether the automation applies to academic or scientific literature?
- Did you misclassify a literature review that focuses on automation in another domain (e.g., robotics and healthcare) as relevant, even though it is not about automating reviews themselves?
- Did you apply the inclusion and exclusion rules consistently?
If you find any mistakes or inconsistencies, revise your answer.
B.6 System message
You are an expert in machine learning, with specialized knowledge in large language models (LLMs) and the automation of systematic literature reviews (SLRs). You understand the structure of the SLR process and are able to identify whether and how automation techniques—particularly those involving LLMs—are applied to specific review tasks, such as searching, screening, information extraction, synthesis, and writing.
Open access
