Highlights
What is already known?
-
• Automation-assisted screening is routinely evaluated using metrics such as recall, precision, and workload reduction, and results are increasingly summarized across studies to support cumulative methodological conclusions.
What is new?
-
• Screening metrics estimate workflow-induced targets. Nominally similar metrics may therefore estimate different quantities across studies. Cross-study comparison is warranted only when targets align across representation, inference, governance, evaluation, and the reference-decision set.
Potential impact for RSM readers
-
• The framework helps readers distinguish variation in method performance from variation in what is being estimated, and specifies when existing or prospective evaluations can support synthesis.
1 Introduction
Systematic reviews and meta-analyses are central to cumulative science, but screening large candidate sets has become increasingly resource-intensive, particularly as growth in the published literature has outpaced feasible manual processing.Reference Borah, Brown, Capers and Kaiser1, Reference Bannach-Brown, Przybyła, Thomas, Rice, Ananiadou and Liao2 Automation is therefore being adopted as infrastructure for research synthesis, including retrieval, screening, and data extraction.Reference van de Schoot, de Bruin and Schram3 The rationale is primarily practical: automation is used to preserve feasibility and timeliness as evidence bases expand.Reference Clark, Barton and Albarqouni4
As automation has become more common, evaluation has become the central methodological problem. Studies assess automated retrieval, screening, and extraction using measures such as recall, precision, and workload reduction, often framed in terms of substituting for or augmenting human effort.Reference Wallace, Trikalinos, Lau, Brodley and Schmid5– Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7 This work spans diverse review topics, data sources, and workflow configurations, creating an expectation that evaluation evidence should support cumulative learning about the reliability and limits of automation in research synthesis.Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference O’Connor, Tsafnat and Gilbert8 In practice, comparison or aggregation of evaluation results across studies remains challenging, even when evaluations address similar tasks and report similar metrics.Reference Olorisade, Brereton and Andras9 Recent systematic mappings of this literature document the scale of the variation: across dozens of evaluations, methods, datasets, reference-standard constructions, and reported metrics differ enough to impede cumulative comparison.Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10– Reference Blaizot, Veettil and Saidoung12 Methodological evaluations describe the same problem in different terms, characterizing the field’s evaluation methods as non-standardized in ways that prevent valid comparison across studies.Reference Spiero, Leeuwenberg, Moons, Hooft and Damen13 The problem is not a lack of evaluations, but the definition and interpretation of their results across heterogeneous workflows.
1.1 A motivating example
Consider two hypothetical evaluations of automation-assisted screening. Both report a recall of 0.95, and each is internally valid on its own terms. Study A evaluates a fully automated classifier applied exhaustively to a candidate set of 10,000 records. An independent reference set, constructed by expert adjudicators who reviewed every record without knowledge of the classifier’s predictions, identifies 500 as relevant. The classifier correctly identifies 475 of these, giving recall = 475/500 = 0.95. Study B evaluates a ranking algorithm with early stopping, applied to a candidate set of comparable size. Records are ranked by estimated relevance. A reviewer screens them in ranked order and stops when a statistical criterion estimates that 95% of relevant records have been identified. The 1,800 screened records contain 95 relevant items, and all 95 have been found. The remaining 8,200 records receive no reference decision. Recall is reported as 95/100, where 100 is the stopping procedure’s estimated total number of relevant records.
The two values are both labeled “recall = 0.95,” but they are not estimates of the same quantity. Study A’s recall is the proportion of independently adjudicated relevant records that the classifier correctly identified, defined over the full candidate set of 10,000 records. Study B’s reported recall is a stopping-conditioned estimate. The 95 relevant records are observed within the 1,800 screened records, while the denominator of 100 is an estimated total number of relevant records under the stopping procedure. The observed reference-decision set is therefore the screened subset, but the reported target domain is a latent or model-based domain extending beyond that subset. Performance among the 8,200 unscreened records is not directly observed. It enters the reported value only through the assumptions of the stopping procedure. The denominators differ. The adjudication procedures differ. The sets over which the metric is defined differ. Treating the two estimates as equivalent would confuse a shared metric label with a shared target quantity. This article argues that such treatment reflects how evaluation is organized rather than failures of individual studies, and that cumulative inference about automation-assisted screening requires clarifying what is being estimated in each case.
The example is stylized, but the distinction is general: reported screening metrics are conditional on the workflow that produces them, and workflow variation can change the quantity being estimated.
2 Problem statement
Cross-study synthesis requires reported screening results to estimate comparable quantities. In applied screening research, that presupposition is often unmet. Two studies can report the same metric, with the same numerical value, and have those numbers refer to different underlying quantities.Reference Bannach-Brown, Przybyła, Thomas, Rice, Ananiadou and Liao2, Reference Wallace, Trikalinos, Lau, Brodley and Schmid5, Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10 Syriani and colleagues, for example, report per-corpus metrics across five software-engineering systematic reviews and find that the same classifier produces different performance profiles on different corpora, with corpus-level features such as the conflict ratio between human reviewers predicting tool behavior.Reference Syriani, David and Kumar14 The ambiguity is not statistical heterogeneity around a common target. It is a difference in what is being estimated.
A metric is a rule for computing a value from observed data. Recall, precision, specificity, and workload reduction are metrics in this sense. An evaluation target is the underlying quantity the metric is intended to estimate. The reported result is the study-specific realization of the metric. The terms “evaluation target” and “estimand” are used interchangeably in this article, by analogy to the ICH E9(R1) sense in which the estimand is the quantity targeted by estimation, defined by the operational specifications of its constituent components.15 In inherited or operational workflows, the target is not prospectively specified but retrospectively induced by workflow design. This mirrors the contrast in treatment-effect studies such as clinical trials, where the estimand is specified prospectively by design before data are observed, rather than induced retrospectively as here. “Evaluation target” is used where this induced character is the emphasis. In a fully adjudicated fixed corpus, a reported metric is a descriptive functional of that corpus, becoming an estimate when interpreted as targeting performance beyond that finite reference set. Both senses are covered.
Each metric has operational components. For recall, the denominator is the sum of true positives and false negatives. That denominator is only well-defined once the evaluation has operationalized what counts as a record, what counts as relevant, which records are candidates for adjudication, and under what authority reference decisions are generated.Reference Marshall and Wallace6, Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference Olorisade, Brereton and Andras9 Each of these operationalizations is shaped by four dimensions of choice (representation, inference, governance, and evaluation), together with the reference-decision set the configuration produces or specifies.Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference Callaghan and Müller-Hansen16, Reference Wallace, Noel-Storr, Marshall, Cohen, Smalheiser and Thomas17 When studies operationalize choices across these elements differently, the constituent components of the metric count different things.Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference Olorisade, Brereton and Andras9, Reference Callaghan and Müller-Hansen16 The recall statistic, computed identically in both studies, then refers to different underlying quantities. The same holds for precision, specificity, and workload-reduction metrics, which depend on analogous operationalizations across the same elements.
Two analytically distinct failure modes arise when evaluation targets go unspecified. The first concerns the construction of the reference standard. Reference decisions may be generated by a process whose properties are not independent of the method being evaluated.Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10, Reference Howard, Phillips and Miller18 When the same reviewers who produce the reference labels are also the reviewers the method is designed to assist, apparent accuracy conflates agreement with the reviewers and agreement with an external criterion of relevance. This failure mode arises primarily in governance and evaluation. The second failure mode concerns observability. Even when reference decisions are independently constructed, the adjudicated set may itself be shaped by the workflow.Reference Callaghan and Müller-Hansen16, Reference Howard, Phillips and Miller18 Routing, prioritization, or stopping decisions made by the method under evaluation, or by the workflow in which it is embedded, can determine which records reach adjudication at all. Records never surfaced for adjudication are never counted as missed. The evaluation target is defined over the induced reference-decision set rather than over the full candidate set. This failure mode arises primarily in inference and is reflected in the reference-decision set. The two modes often co-occur, especially when evaluations inherit labels from operational review workflows.Reference Bannach-Brown, Przybyła, Thomas, Rice, Ananiadou and Liao2, Reference Howard, Phillips and Miller18
Evaluation targets are induced by configurations across the framework, with particular attention here to evaluations that inherit labels from operational review workflows. The conditions under which such evaluations can be compared with each other, and under which their results can cumulate, are specified as alignment across the four dimensions and the reference-decision set. The framework is developed for the Identification and Screening stages of the PRISMA 2020 flow diagram,Reference Page, McKenzie and Bossuyt19 where selection under partial information is central.Reference Marshall and Wallace6, Reference Kelly and Sugimoto20, Reference Tsafnat, Dunn, Glasziou and Coiera21 Within the Screening stage, the framework treats both screening decisions: records screened (typically titles and abstracts) and reports assessed for eligibility (typically full text). The same logic extends in principle to data extraction and downstream stages, where selection under partial information also occurs. Those extensions are noted where relevant but are not developed here.
3 Decomposing screening evaluation targets
To compare screening evaluations, one must specify the workflow and evaluation choices that define the evaluation target. Four dimensions of choice recur: representation, inference, governance, and evaluation. Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference O’Connor, Tsafnat and Gilbert8, Reference Tsafnat, Dunn, Glasziou and Coiera21 Each is shaped by procedural choices made within the evaluation and by structural facts the evaluation inherits, including corpus coverage, indexing depth, and available reviewer expertise. Representation concerns what units are screened and what information is visible at the point of decision. Inference covers how decisions are generated, including prioritization, thresholding, and stopping. Governance addresses who holds decision authority and how escalation, verification, and disagreement resolution operate. Evaluation covers what is being estimated and how the reference is used: which metric is computed, which comparator the method is judged against, and whether reference labels are treated as fixed or revisable. The configurations across these four dimensions produce or specify a reference-decision set, the set of records for which reference decisions are available, over which reported metrics are computed. The framework is represented schematically in Figure 1.
The four workflow dimensions and the reference-decision set they produce or specify together determine the evaluation target.
Figure 1. Long description
The flowchart is organized into three vertical levels.
At the top level, four rectangular boxes represent workflow dimensions, arranged from left to right.
1. Representation: Information available at the point of decision.
2. Inference: Decision rule, prioritization, stopping.
3. Governance: Decision authority and adjudication.
4. Evaluation: Metric, comparator, reference status.
Arrows from each of these four boxes point downward and converge on the middle level.
The middle level contains a single dashed-line rectangle labeled Reference-decision set, with the subtext: Records receiving reference decisions.
A single downward arrow connects this middle box to the bottom level. This arrow is labeled: metric computed over this set.
The bottom level consists of a solid gray rectangle labeled Evaluation target, with the subtext: estimand.
The framework is a minimal working decomposition: each dimension is required to specify the target, none is reducible to another, and common evaluation-design features can be located within one or more of them. They need not be treated as fully independent. Corpus construction and database coverage are operational activities that shape both representation and the reference-decision set. Prevalence structure is a property of the reference-decision set. Reviewer expertise is a property of governance. Choice of stopping rule is an operational activity within inference. Choice of comparator and gold-standard treatment are activities within evaluation. Other decompositions are possible, but any that omits one of the four dimensions risks conflating evaluations whose targets differ on it.
The framework is orthogonal to review phase. A phase such as title–abstract screening involves representation, inference, governance, and evaluation choices simultaneously, all of which contribute to the reference-decision set produced or specified at that phase. Conversely, a dimension such as governance operates across phases. Table 1 presents this orthogonality as a grid. Rows are the four dimensions and the reference-decision set. Columns are PRISMA 2020 stages in which screening-related choices arise.Reference Page, McKenzie and Bossuyt19 These are Identification, where database searches and pre-screening reduction happen, and two stages within Screening: the first screening decision (records screened), and the second screening decision (reports assessed for eligibility). Identification is included because candidate-pool construction defines the retrieval universe over which downstream screening misses can later be observed and counted. Cells contain the operational activities that express the row at the column’s phase.
Operational activities within each element of the framework across the Identification and Screening stages of PRISMA 2020
Table 1. Long description
The table is organized into four columns. The first column lists the Element, while the next three columns fall under the PRISMA 2020 stage header: Identification, Screening: records screened, and Screening: reports assessed for eligibility.
* Representation: In Identification, it involves database indexing fields, controlled vocabulary versus free text, and document metadata format. In Screening (records), it involves title only, title plus abstract, enriched metadata, or full text. In Screening (reports), it involves full text of retrieved reports, supplementary materials, and author information.
* Inference: In Identification, it involves query construction, refinement, and retrieval rules. In Screening (records), it involves exhaustive screening, ranking with thresholds or stopping rules, and active-learning loops. In Screening (reports), it involves full-text inclusion/exclusion rules and linkage to first-pass decisions.
* Governance: In Identification, it involves search standards, peer review, and documentation. In Screening (records), it involves reviewer independence, blinding, dual-review, and automation roles. In Screening (reports), it involves adjudication of disagreements and final-inclusion authority.
* Evaluation: In Identification, it involves choice of comparator and scope of inference. In Screening (records), it involves metric selection like recall, precision, or W S S. In Screening (reports), it involves interpretation of method-reference disagreement.
* Reference-decision set: In Identification, it involves databases searched, filters, and deduplication. In Screening (records), it defines which records advance to full-text retrieval. In Screening (reports), it defines which records receive final reference decisions.
a Full text at this stage is not conventional in human-conducted reviews but appears in emerging automation-assisted designs.
Operational choices within these cells determine what the components of reported metrics count. The opening vignette contrasted two hypothetical evaluations that both reported recall of 0.95. In Study A, the reference-decision-set cell at Identification establishes a candidate pool through independent database searches and documented deduplication. The inference cell at Screening: records screened uses exhaustive screening with no stopping rule. The governance cell at Screening: reports assessed for eligibility resolves disagreements through dual independent review with arbitration. Study A’s denominator for computing recall, the sum of true positives and false negatives, is counted over the full documented candidate pool. In Study B, the candidate pool is identical in principle. The inference cell at Screening: records screened uses a ranking algorithm with an early-stopping rule triggered at an estimated 95% recall target. The reference-decision-set cell at the same stage therefore restricts adjudication to the records screened before stopping. Study B’s observed numerator is counted within the screened subset, while its denominator is estimated under the stopping procedure. The reported metric therefore combines observed adjudication with an inferential claim about unscreened records.
The two recall statistics are computed by the same formula but refer to different underlying quantities, because the operational choices in the cells that fix the denominator have differed between the two studies. The numerator for precision depends on operationalizations within governance (what is adjudicated as a correct positive) and within the reference-decision set (the pool over which positives are assessed). The true-negative basis for specificity depends on operationalizations within the reference-decision set at Identification (which records enter the candidate pool) and within inference at Screening (which records are judged negative and by what procedure). In each case, the same formula applied to differently operationalized cells produces values that refer to different underlying quantities. The metric label and formula conceal this difference.
Established frameworks for evaluating diagnostic test accuracy and prediction models, such as QUADAS-2,Reference Whiting, Rutjes and Westwood22 assess bias and applicability conditional on a specified estimand. The framework here is analytically prior to such assessment. QUADAS-2’s domains (patient selection, index test, reference standard, and flow and timing) overlap with several elements of the present framework, with index-test concerns involving both representation and inference, reference-standard concerns involving governance and evaluation, and flow-and-timing concerns involving observability and reference-decision-set construction. The mapping is partial rather than one-to-one. The framework here addresses a prior and cross-study question: whether reported metrics are targeting the same quantity in the first place. Risk-of-bias assessment remains necessary, but it does not by itself establish target alignment for cumulative synthesis. A QUADAS-2 assessment applied to an inherited-adjudication evaluation can be internally consistent and still mischaracterize what the evaluation estimates, because the framework operates on the workflow-induced estimand without diagnosing how that induction occurred.
3.1 Representation and its consequences for evaluation
Representation specifies the information available at the screening decision and therefore the errors that can be made. Evaluations based on titles differ from those based on abstracts, enriched metadata, or full text, even when they address the same review question.Reference Marshall and Wallace6, Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10 Ambiguity that could be resolved with more information remains unresolvable when only sparse information is available, and relevance judgments made under sparse representations do not generalize to richer ones.Reference Marshall and Wallace6, Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10 Inherited workflows often combine multiple representational levels across stages, such that different records are judged on the basis of different information.Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10, Reference Wallace, Noel-Storr, Marshall, Cohen, Smalheiser and Thomas17 Without representational specification, reported performance conflates method behavior with information availability.
3.2 Inference, prioritization, and stopping
Inference strategies determine how available information is used to make screening decisions and how far screening proceeds. Automated screening systems often rank records by estimated relevance, then use thresholds or stopping rules to decide which items are reviewed manually. These decisions shape what remains unobserved, and therefore what falls outside the reference-decision set.Reference Callaghan and Müller-Hansen16, Reference Yu, Agarwal, Johnston and Cohen23– Reference Boetje and van de Schoot25 An evaluation conducted under exhaustive screening estimates a different quantity from one conducted under early stopping, even when both report recall or workload reduction. Some stopping rules make no provision for the unscreened records. These include heuristic rules, such as consecutive-irrelevant thresholds or minimum screening proportions, and operational truncation, where screening ends because of labor or cost limits and the evaluated set reflects those constraints rather than the screening method. Under either, directly observed performance is limited to records screened before stopping, and unscreened records cannot contribute observed false negatives. Under statistical stopping designs, unscreened records may instead enter the reported quantity through model-based assumptions or probabilistic guarantees.Reference Callaghan and Müller-Hansen16, Reference Boetje and van de Schoot25 Two recent works illustrate the conditioning explicitly. Hou and TiptonReference Hou and Tipton26 propose a resampling-based stopping procedure whose evaluation target is specified a priori as a probabilistic recall guarantee, P(Recall ≥ c) ≥ R, rather than inherited from a completed workflow, making the estimated quantity invariant to the choice of underlying prioritization algorithm. At the metric level, the Work Saved over Sampling measure introduced by Cohen et al.Reference Cohen, Hersh, Peterson and Yen27 is examined by Kusa et al.,Reference Kusa, Lipani, Knoth and Hanbury28 who show that metric-level choices have a parallel effect. The maximum value of Work Saved over Sampling depends on the class balance of the dataset, so reported WSS scores describe different quantities across datasets, and normalization (equivalent to the true negative rate) restores a measure whose definition does not vary across them. Both show that technical design choices can define the estimated quantity.
3.3 Governance and decision authority
Governance arrangements specify who makes screening decisions, how disagreements are resolved, and the role automation plays relative to human judgment. Governance combines procedural choices about who adjudicates with structural facts about who is available to adjudicate. In some workflows, automated systems substitute for one human screener. In others, they function as triage tools that route records to experts for adjudication. These configurations define different targets because they specify different human or workflow functions for automation to replace.Reference O’Connor, Tsafnat, Thomas, Glasziou, Gilbert and Hutton7, Reference Perlman-Arrow, Loo, Bobrovitz, Yan and Arora29, Reference Thomas, Noel-Storr and Marshall30 An evaluation that measures agreement with final inclusion decisions estimates a different quantity from one that measures agreement with routing or escalation decisions, even when both operate on the same records.Reference Wallace, Noel-Storr, Marshall, Cohen, Smalheiser and Thomas17, Reference Thomas, Noel-Storr and Marshall30 In inherited workflows, reference labels embed the verification and override structure of the original process.Reference Bannach-Brown, Przybyła, Thomas, Rice, Ananiadou and Liao2, Reference Wallace, Trikalinos, Lau, Brodley and Schmid5, Reference Howard, Phillips and Miller18
3.4 Evaluation choices and the estimand
Evaluation covers what the evaluation purports to estimate and how the reference is used. The dimension includes the choice of metric (recall, precision, work saved over sampling, sensitivity at fixed thresholds, others), the status assigned to reference labels (fixed gold standard, revisable estimate, or one input among several), the comparator against which the method is judged (single-reviewer screening, dual-independent screening, an established benchmark, an idealized exhaustive review, or no explicit comparator), and the inferential scope of the reported metric (performance over the available reference-decision set, the retrieval universe, a target population of similar reviews, or deployment conditions). Evaluation choices operate on the reference-decision set rather than constructing it, but they determine what the reported metric is interpreted as estimating. Two studies that share representation, inference, governance, and reference-decision set can still produce metrics that estimate different quantities if their evaluation choices differ. Recent work treats reference labels as revisable rather than fixed, allowing the evaluation to estimate a posterior over the underlying relevance structure rather than agreement with a single gold standard.Reference Chernikova, Stadler, Melev and Fischer31, Reference Courvoisier, Buitrago-Garcia, Buclin, Bürgisser, Iudici and Mongin32
3.5 Reference-decision set and observability
The reference-decision set is the set of records for which reference decisions are available. It can be produced by the configured workflow, as when adjudication is restricted to records that advance through screening, or it can be specified independently of the workflow, as when an external relevance set is constructed for evaluation. Its boundaries determine which errors can be observed. When only records advanced from earlier stages receive full adjudication, exclusions remain unverified. The reference-decision set in such cases is a proper subset of the candidate set,Reference Wallace, Trikalinos, Lau, Brodley and Schmid5, Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10, Reference Howard, Phillips and Miller18 and false negatives that fall outside the adjudicated subset are unobservable. Inherited workflows amplify this effect, as reference labels reflect prior routing, stopping, and adjudication decisions rather than an independently constructed reference-decision set.Reference Callaghan and Müller-Hansen16, Reference Howard, Phillips and Miller18 Under these conditions, directly observed components of metrics such as recall are defined relative to the observed subset of records unless the evaluation explicitly specifies a model-based or probabilistic target domain.
Three sets warrant explicit distinction. The candidate set comprises all records entering the screening workflow. The reference-decision set comprises the subset with observed reference decisions. The target domain is the set over which the reported metric is interpreted as estimating performance. Under exhaustive independent adjudication, the three sets coincide. Under inherited or stopped workflows, the reference-decision set may be a workflow-induced subset of the candidate set, while the target domain may be the reference-decision set, the candidate set, or a model-based latent domain depending on the evaluation design. Stopping procedures with statistical guarantees, for example, may target a probabilistic recall claim over the candidate set rather than a deterministic claim over the reference-decision set. Misalignment between the reference-decision set and the target domain is a frequent source of confusion in reported screening metrics.
Table 2 lists key questions for each dimension and for the reference-decision set. The questions are not exhaustive but identify choices that most reliably distinguish evaluation targets across studies. Together they define the minimal specification required to identify what quantity a reported screening evaluation result estimates.
Minimal interpretive specification of screening evaluation targets (estimands)
Table 2. Long description
The table consists of two columns: Element and Key questions.
* Representation: Includes five questions regarding the unit screened (title, abstract, or full text), metadata completeness, pre-processing transformations, derived features like similarity scores, and consistency of representation across stages.
* Inference: Includes five questions about decision rules (filters, classifiers, or language models), whether rules are fixed or learned, termination criteria (stopping rules or budgets), targets for stopping (recall or workload reduction), and the handling of uncertain predictions.
* Governance: Includes four questions concerning authority distribution between humans and automation, resolution of disagreements, override protocols, and reviewer expertise or training.
* Evaluation: Includes six questions about metrics (recall, precision, or work saved), the status of reference labels as gold standards, comparators (single or dual screening), interpretation of disagreements, the scope of the reported metric, and the unit of analysis.
* Reference-decision set: Includes five questions about which records entered the candidate set, how reference decisions were generated (independently or inherited), and how excluded or unscreened records were verified relative to the target domain.
4 Illustrative configurations
In practice, these elements recur in patterned combinations. These combinations define evaluation configurations. Two of these patterns recur at the study level, where they shape how individual evaluations specify reference decisions and observability. The third recurs at the synthesis level, where the consequences of unaligned study-level configurations surface as pooling problems. The configurations were developed through purposive reading of the screening-evaluation literature rather than a formal scoping or systematic review. Reading drew from key methodological venues, including Systematic Reviews, BMC Medical Research Methodology, Research Synthesis Methods, and the Journal of Clinical Epidemiology, with attention to publications from 2006 onward, beginning with Cohen et al.’s introduction of automated classification for workload reduction in screening.Reference Cohen, Hersh, Peterson and Yen27 Reference lists were examined and citations chained forward and backward to identify additional studies illustrating recognizable evaluation configurations. The directly relevant reading covered automation-assisted screening evaluations, reviews, and methodological commentaries, supported by adjacent-field reading in information retrieval evaluation, diagnostic test accuracy, and estimand specification. The aim was conceptual clarification of recurring target-defining patterns, not exhaustive scoping or prevalence estimation.
Configurations were noted as candidates when the same pattern of representation, inference, governance, evaluation, and the reference-decision set recurred across studies with otherwise different methods and topics. The three configurations presented below are those that recurred most visibly and that illustrate analytically distinct kinds of evaluation-target induction. They are conceptual ideal types and may overlap in practice. A real evaluation may exhibit features of more than one. The illustrative sources cited under each configuration include both primary evaluations used as unit-level exemplars and reviews or meta-analyses used as field-level documentation. Other configurations were identified during reading and are not developed here, including configurations in which the reference-decision set is shared across studies but governance differs because of variation in adjudication authority, and configurations in which prospective and retrospective evaluations on the same corpus estimate different quantities. The sections that follow examine the three illustrated configurations across recognizable evaluation designs.
5 Illustrative configuration 1: endogenous adjudication
Endogenous adjudication arises when the evaluation reference labels are produced by the same operational workflow being evaluated, rather than by an independently constructed reference set.Reference Wallace, Trikalinos, Lau, Brodley and Schmid5, Reference Cohen, Hersh, Peterson and Yen27, Reference Yao, Low and Sivajohanathan33 The evaluative object is the local workflow. Records that advance receive adjudication, while exclusions typically remain unverified, rendering the reference-decision set conditional on the routing behavior of the screening process itself. Reported performance therefore estimates agreement with workflow-produced labels, not accuracy against an independent reference standard (see Figure 2).
Endogenous adjudication generates the reference-decision set from the workflow being evaluated, so reported metrics estimate agreement with workflow-produced labels.
5.1 Screening-dependent adjudication in semi-automated citation screening
Wallace et al. studied a semi-automated citation screening workflow designed to reduce reviewer workload by embedding supervised classification within an active learning pipeline.Reference Wallace, Trikalinos, Lau, Brodley and Schmid5 The evaluation was conducted over three previously completed reviews, with screening decisions at the title–abstract stage determining which records advanced to full-text review. Citations retrieved in full text were treated as relevant for evaluation purposes, while citations excluded at screening were not independently verified. This restricted adjudication to the subset induced by screening decisions. The authors are clear that their evaluation target is not predictive accuracy on an independent hold-out dataset. Rather, they evaluate whether the screening workflow succeeds in identifying all citations included in the completed reviews, operationalizing this objective through two bespoke metrics, yield and burden, that prioritize sensitivity within the induced adjudication domain.
5.2 Endogenous adjudication through training-set–dependent evaluation in preclinical screening
Bannach-Brown et al.Reference Bannach-Brown, Przybyła, Thomas, Rice, Ananiadou and Liao2 evaluated machine-learning–assisted citation screening in a large preclinical systematic review of animal studies, with the objective of reducing screening workload while maintaining high sensitivity for inclusion. Screening was conducted on title–abstract representations using dual independent human reviewers, with disagreements resolved through reconciliation. Relevance was defined operationally by inclusion decisions generated within this workflow, and records excluded at screening were not independently verified against an external reference standard. Machine learning models were trained to approximate these reconciled human decisions and were evaluated relative to them, with performance targets set to achieve at least 95% sensitivity. Evaluation metrics, including workload reduction and screening error rates, were therefore defined with respect to adjudicated subsets induced by the construction of training and validation sets within the review workflow.
5.3 Endogenous adjudication at scale in the screening evaluation literature
O’Mara-Eves et al. synthesize 44 evaluations of text-mining–assisted screening and document a rapidly expanding but structurally heterogeneous evidence base.Reference O’Mara-Eves, Thomas, McNaught, Miwa and Ananiadou10 Across the reviewed studies, reference decisions are inherited from completed review workflows, most commonly through retrospective simulation against human include/exclude decisions generated during screening, with no independent verification of records excluded at earlier stages. Evaluation designs, metrics, and thresholds vary widely, and replication is rare. As the authors conclude, the resulting diversity of datasets, designs, and outcome measures precludes reliable comparison of performance across studies.
6 Illustrative configuration 2: inherited adjudication with stopping-conditioned observability
This configuration arises when inherited reference decisions are combined with prioritization or stopping. Under this configuration, the estimand is fixed by adjudication decisions embedded in the workflow, which determine which records ever receive verification. Subsequent stopping or prioritization strategies further restrict observability by truncating manual review. Reported metrics are therefore defined over domains that are both inherited and truncated. Figure 3 diagrams this configuration.
Inherited adjudication with stopping-conditioned observability: the reported metric is conditioned on inherited labels and on which records remain observable after stopping.
Figure 3. Long description
The flowchart consists of several interconnected boxes and a central large container.
* On the far left, a dashed box labeled Prior workflow (reference decisions) points an arrow labeled inherited labels to the right into the central container.
* The central container is a large rounded rectangle labeled Candidate set at the top. It is divided horizontally by a dashed line labeled stopping boundary.
* The top half of the Candidate set is shaded gray and labeled Screened subset, with the subtext inherited reference decisions observed.
* The bottom half of the Candidate set is white and labeled Not screened, with the subtext false negatives unobserved unless modeled or sampled.
* To the right of the Candidate set, a box labeled Method / ranking rule points an arrow to the left, meeting the stopping boundary. This arrow is labeled prioritizes and stops.
* At the bottom of the entire diagram is a wide gray box labeled Target: reported performance conditioned on inherited labels and stopping.
Aggregated heterogeneous targets: pooling across studies whose reference-decision sets differ produces a summary that does not estimate a single shared target.
6.1 Inherited adjudication with stopping-conditioned observability in statistical stopping designs
Callaghan and Müller-HansenReference Callaghan and Müller-Hansen16 evaluate statistical stopping criteria for machine-learning–assisted screening using datasets drawn from previously completed systematic reviews, in which inclusion and exclusion labels are known a priori and treated as fixed for simulation. Their contribution is framed around the problem of deciding when screening may be stopped while still achieving a user-specified recall target under conditions where recall is unobservable until screening is complete. They formulate stopping as a statistical hypothesis test, using a hypergeometric model to estimate recall over the unseen portion of the corpus. Reliance on labels inherited from completed reviews permits retrospective evaluation in simulation but entails irreversibly truncated observability in deployment. Under this design, performance estimates characterize trade-offs between efficiency and confidence within a historically fixed reference-decision set rather than screening performance over a fully adjudicated retrieval universe.
6.2 Heuristic stopping under inherited adjudication in active learning-based screening
Boetje and van de Schoot propose the SAFE procedure, a conservative set of stopping heuristics for active learning–based screening designed to guide reviewers’ decisions about when to end screening under uncertainty.Reference Boetje and van de Schoot25 The procedure assumes a standard workflow in which relevance judgments are generated by human screeners and inherited as screening progresses, without construction of an external reference standard. SAFE combines several heuristic criteria to reduce the risk of premature stopping, including minimum screening proportions, estimates of total relevant records, consecutive-irrelevant thresholds, and key-paper validation. The authors emphasize that all thresholds are context-dependent and illustrative, and that stopping decisions necessarily balance efficiency against the risk of missing relevant records. Observability is conditioned by both inherited adjudication and stopping heuristics, and performance is characterized in terms of practical decision governance rather than identification of a cumulative screening performance estimand. SAFE’s conservatism reflects that recall is not identifiable at stopping, with additional workload protecting against unobserved misses.
6.3 Tool-integrated screening with sensitivity-based stopping rules
Yao et al.Reference Yao, Low and Sivajohanathan33 prospectively evaluated Covidence’s AI functions during literature updates for two oncology clinical practice guidelines, reporting workload savings for Stage I title–abstract screening at 95% and 100% sensitivity and assessing the consequences of missed included references. The evaluation corpus was defined by an ongoing guideline update workflow, with reference decisions generated through routine title–abstract screening and downstream full-text adjudication rather than reconstructed for evaluation. Full-text adjudication was therefore available only for records that advanced past Stage I screening, while records excluded at screening were not independently verified. Performance was evaluated under sensitivity-based stopping thresholds using metrics including workload savings, time savings, and work saved over sampling, all defined relative to this inherited reference-decision set. At 95% sensitivity, workload was halved while one included study per guideline was missed. Enforcing 100% sensitivity markedly reduced efficiency gains. Reported performance characterizes screening behavior within a truncated reference-decision set conditioned by both inherited adjudication and stopping rules.
7 Illustrative configuration 3: aggregated heterogeneous targets
The third pattern is synthesis-level rather than study-level. It arises when results are pooled across reviews or datasets whose workflow-induced targets differ. The pooled summary then aggregates distinct screening contexts treated as jointly informative. The result is an average over heterogeneous targets, not an estimate of a common target. This pattern is presented alongside the two study-level configurations because it shows what follows when configuration variation is not addressed at synthesis time (see Figure 4).
7.1 Cross-dataset aggregation in practitioner-oriented screening evaluation
Marshall and WallaceReference Marshall and Wallace34 evaluate machine-learning classifiers for identifying randomized controlled trials using training and validation data drawn from distinct screening workflows, including the Cochrane Crowd and Clinical Hedges datasets. These datasets embody different adjudication pipelines and inclusion criteria, and reference labels are inherited from historically constructed workflows rather than specified for evaluation. Performance is summarized using metrics such as area under the receiver operating characteristic curve, sensitivity, specificity, and number needed to screen, and results are presented across multiple model architectures and use cases. The authors acknowledge dataset dependence and prevalence effects, but performance estimates are treated as jointly informative for guiding practice across contexts.
7.2 Large-scale descriptive aggregation across heterogeneous screening evaluations
Van Dinter et al. synthesize 41 primary studies on the automation of systematic literature reviews published between 2006 and 2020, spanning multiple domains, review objectives, and stages of the review process.Reference van Dinter, Tekinerdogan and Catal11 Across the reviewed studies, datasets and reference decisions are inherited from completed review workflows and differ widely in application domain, screening step, database coverage, feature representations, and performance objectives. The authors document substantial variation and limited replication, but do not attempt quantitative synthesis, instead treating the evidence base as structurally heterogeneous.
7.3 Pooled evaluation across reviews and domains in automated screening
Feng et al. conducted a systematic review and diagnostic accuracy meta-analysis of artificial-intelligence–assisted literature screening, synthesizing results from 17 primary evaluations drawn from distinct medical systematic reviews.Reference Feng, Liang and Zhang35 Reference decisions were defined by human screening outcomes generated within individual review workflows rather than by a shared adjudication protocol, and datasets differed in topic area, corpus size, inclusion prevalence, algorithm class, and database coverage. Despite this heterogeneity, recall, specificity, and precision were pooled using a bivariate diagnostic meta-analysis model, implicitly treating these distinct screening contexts as instances of a common evaluative object. Although the authors adopt a cautious analytic posture and acknowledge heterogeneity, the pooled estimates necessarily characterize average performance across heterogeneous reference-decision sets rather than performance with respect to a single, well-defined estimand.
8 Limits and conditions of synthesis
8.1 Non-aligned targets
The configurations show why nominally identical metrics need not estimate the same quantity. Apparent agreement across studies may arise from metrics defined on different reference-decision sets, while apparent disagreement may reflect differences in observability, adjudication, or stopping rather than differences in screening effectiveness.Reference Olorisade, Brereton and Andras9, Reference Callaghan and Müller-Hansen16, Reference Howard, Phillips and Miller18 When targets diverge across the framework, aggregation cannot distinguish method performance from target variation. Internal validity within studies does not resolve target nonalignment across studies.
8.2 Aligned targets
Workflow-induced targets do not preclude comparison. Many workflow differences do not change the estimated quantity. The claim is narrower: comparison fails when workflow differences change any of the representation, inference, governance, and evaluation dimensions, or the reference-decision set they produce. When workflow differences leave the four dimensions and the reference-decision set congruent across studies, comparison is warranted on the quantity the shared configuration defines.
The alignment conditions map onto the framework directly. Representational alignment requires that the information available at the point of screening decision be comparable across studies, so that performance under sparse representations is not being compared with performance under richer ones. Inferential alignment requires that screening either be exhaustive in both studies or be truncated by the same stopping criterion applied under comparable conditions, so that recall denominators refer to the same set. Governance alignment requires that reference decisions be generated by comparable adjudication structures, so that apparent disagreement with the index method is not conflated with apparent disagreement with routing or escalation choices. Evaluation alignment requires that the metric, the comparator, the treatment of reference labels, the unit of analysis, and the intended scope of inference be comparable across studies. A shared metric label then reflects the same quantity rather than concealing differences in what the reported value estimates. Reference-decision-set alignment requires that the rule by which records receive reference decisions, and the relationship between the candidate set, observed reference-decision set, and target domain, be comparable across studies or transformable under explicit assumptions. When all five conditions hold, a reported recall in one study and a reported recall in another estimate a common underlying quantity, and standard comparison is appropriate. When any condition fails, the quantities diverge, and the framework in the preceding sections identifies which element is responsible and what features a study would need to report for conditional or restricted comparison to be possible.
Configuration alignment is sufficient for target alignment but not necessary. Different configurations may target the same quantity in several ways. Configurations that differ in form but produce identical reference-decision sets target the same observable quantity. Configurations whose differences operate on a dimension that does not affect the measured quantity are functionally equivalent for that metric. Statistical inference from a sampled or stopped configuration may target the same estimand as direct observation in an exhaustive configuration, under that procedure’s model assumptions. Work such as Hou and Tipton,Reference Hou and Tipton26 which specifies the estimated quantity a priori as a probabilistic recall guarantee, illustrates this case. Retrospective designs may be deliberately constructed to match the configuration of a prospective comparator, producing a common target by design. Different joint distributions of relevance and classification may yield the same metric value, although this is rare in practice. The diagnostic question is therefore whether configuration differences are target-preserving, not whether they exist, and cannot be answered without examining the configuration.
This has implications for standard heterogeneity statistics in meta-analyses of screening evaluations. Statistics such as I 2 and τ2 quantify variation among estimates assumed to target a common, possibly random-effects distributed, quantity. When the estimated quantities differ, observed heterogeneity conflates performance variation with target variation. A large I 2 in a meta-analysis of screening performance may reflect genuine variability in method effectiveness, systematic differences in what the evaluations are estimating, or both, and the statistic itself does not distinguish these. The framework diagnoses target variation before standard heterogeneity statistics are interpreted. Alignment conditions can be assessed at the level of study design before pooling, and residual heterogeneity among estimates judged to target a common quantity can then be summarized in the usual way. At the metric level, Kusa et al.Reference Kusa, Lipani, Knoth and Hanbury28 show an instance of this logic. Work Saved over Sampling has a maximum value that depends on the class balance of the dataset, so pooled WSS scores average performance over a quantity that itself varies across datasets. Their normalization (equivalent to the true negative rate) restores a metric whose definition does not vary, and cross-dataset aggregation becomes interpretable.
9 Methodological implications for screening evaluation
In treatment-effect synthesis, specification of an estimand is recognized as a prerequisite for meaningful aggregation and interpretation.15, Reference Hernán and Robins36 Workflow-induced evaluation targets play an analogous role in screening evaluation. When evaluation targets are not aligned across the framework, aggregate summaries may be informative about reported results but cannot support cumulative inference.
When labels are inherited from completed workflows, the workflow defines the evaluation target through its routing rules, adjudication asymmetries, representational regimes, and termination conditions. Making these features explicit allows assessment of when results are conditionally comparable, when cautious contrast is warranted, and when aggregation is unwarranted.Reference Olorisade, Brereton and Andras9 Such characterization provides a principled basis for interpreting machine-assisted screening results across studies and specifying the limits of cumulative inference.
10 Practical guidance for designing and interpreting evaluations
Recent studies respond to comparability problems in different ways. Some studies report performance separately across evaluation datasets or screening contexts and treat the variability as a finding.Reference Spiero, Leeuwenberg, Moons, Hooft and Damen13, Reference Syriani, David and Kumar14 Others alter reference-decision-set features experimentally to produce tighter comparisons. König and colleagues, for example, vary the prevalence of relevant abstracts by random subsampling and produce 84,000 artificial collections from 21 published meta-analyses.Reference König, Zitzmann and Hecht37 The manipulation improves algorithm comparability but creates domains that do not correspond to actual review workflows. The contrast between the two approaches reflects an unresolved tension between preserving the reference-decision set induced by the workflow and altering that set to enable controlled comparison. The framework identifies the workflow features at stake.
Several recent designs illustrate the framework’s diagnostic value. Gates et al. pool evaluations across governance structures, combining dual-reviewer systematic reviews with single-reviewer rapid reviews.Reference Gates, Gates, Sebastianski, Guitard, Elliott and Hartling38 Hamel et al. separate title–abstract recall from final-include recall, distinguishing workflow decisions that are otherwise conflated.Reference Hamel, Kelly, Thavorn, Rice, Wells and Hutton39 The recent literature also contains designs at the boundary of the configurations as currently described. Two studies in particular indicate that the assumption of fixed reference decisions may not hold universally. One quantifies human reviewer accuracy at 100% recall and 15% precision and uses the quantification as a calibration benchmark rather than treating reference decisions as ground truth.Reference Chernikova, Stadler, Melev and Fischer31 Another revises the human gold standard on the basis of LLM disagreement and reports the original gold standard’s accuracy at 96.3% for classification and 98.3% for extraction against the revised reference.Reference Courvoisier, Buitrago-Garcia, Buclin, Bürgisser, Iudici and Mongin32 These designs remain rare, but they extend the governance domain to revisable rather than fixed reference decisions.
A separate set of studies introduces targeted design modifications that make specific configuration features explicit. Explicit prospective evaluations of endogenous adjudication remain uncommon, although retrospective reliance on workflow-produced labels is widespread, and recent studies extend the design space in useful directions.Reference Valizadeh, Moassefi and Nakhostin-Ansari40, Reference Holland, Oakes, Sarna, Chai, Ng and Moore41 One example runs two independent screening pipelines on the same corpus and reconciles discrepancies through full-text review, producing reference decisions that are not anchored to either pipeline.Reference Holland, Oakes, Sarna, Chai, Ng and Moore41 Several inherited-adjudication studies introduce techniques aligned with specific features of that configuration. Trad and colleagues deliberately exclude the final-included articles from the training set used to calibrate a commercial tool, addressing a contamination concern that other inherited-adjudication evaluations leave unexamined.Reference Trad, Yammine and Charafeddine42 Fuller-Tyszkiewicz and colleagues sample from the auto-rejected pool and re-screen those records to provide a prospective check on the most consequential failure mode.Reference Fuller-Tyszkiewicz, Jones and Vasa43 Another recent study matches reference decisions to the title–abstract eligibility level rather than to final-include status, aligning the reference decision with the decision the tool is being asked to make.Reference Dennstädt, Zink, Putora, Hastings and Cihoric44 These techniques operate at different points in the screening lifecycle and respond to different features of the configurations. None eliminates the reliance on inherited or endogenous reference decisions, but each makes specific governance or evaluation-domain features explicit that simpler designs leave unobserved.
Two practical questions follow: when is conditional synthesis possible, and how should prospective evaluations specify targets?
Three options are available for conditional synthesis, differing in the strength of inference they support. Stratification by configuration is appropriate when a sufficient number of studies share a configuration to support pooled estimates within strata. This allows conditional statements of the form “across evaluations in configuration X, the pooled metric is Y,” while preserving the distinction between configurations whose targets differ. Sensitivity analysis is appropriate when the primary concern is the effect of a single misalignment source, typically stopping-rule truncation. Modeling the distribution of unobserved records under plausible stopping behavior can bound the effect of truncation on apparent recall, and can indicate whether reported differences between studies are robust to this source. Descriptive pooling is appropriate when heterogeneity is substantial and neither stratification nor sensitivity analysis recovers a credible common target. In this case, results may be summarized for reference without any claim that they estimate a shared quantity, provided the absence of that claim is made explicit in the presentation. Metric-level normalization is a fourth option where it is available. Kusa et al.Reference Kusa, Lipani, Knoth and Hanbury28 demonstrate this for Work Saved over Sampling, showing that the metric’s dependence on class balance can be removed by normalization, which restores a quantity whose definition does not vary across datasets and whose aggregation is therefore interpretable.
Prospective evaluations should specify the evaluation target before reporting metrics, parallel to the estimand specification now required in treatment-effect studies.15 Specification should identify, at minimum, the representational regime at the point of decision, the inferential termination rule, the governance of reference decisions, and the set of records over which those decisions are defined. An evaluation whose target is specified in these terms can be compared with other evaluations that specify the same target, and the conditions for comparison can be checked rather than inferred. The work of Hou and TiptonReference Hou and Tipton26 illustrates what this can look like at the level of algorithm design. Their resampling procedure provides a probabilistic recall guarantee of the form P(Recall ≥ c) ≥ R, where (c, R) are stated as parameters of the procedure rather than properties of a completed dataset. Two studies using the procedure with the same (c, R) parameters estimate the same quantity and can be compared directly, even when their underlying prioritization algorithms differ. Studies that adopt target-specified procedures of this kind provide naturally alignable evaluation targets and are particularly well suited to cumulative synthesis.
A compact procedure for applying the framework to a set of candidate studies follows from the preceding discussion. For each study under consideration:
-
1. Identify the representational regime at the screening decision point. Record the information available at each screening pass. Title and abstract, enriched metadata, and full text are different regimes.
-
2. Identify the inferential termination rule. Record whether screening was exhaustive, thresholded, or stopped, and specify any stopping criterion.
-
3. Identify the governance of reference decisions. Record who generated reference labels, whether adjudication was independent of the method, and how disagreements were resolved.
-
4. Identify the evaluation choices. Record the metric reported, the comparator the method is judged against, the status assigned to reference labels (fixed or revisable), and what the metric is taken to estimate.
-
5. Identify the reference-decision set. Record which records received reference decisions and whether that set reflects workflow operation or evaluation design.
Then, across studies:
-
6. Compare the five properties identified in steps 1–5. Studies that align across the framework can be treated as sharing an evaluation target. Studies that diverge on one or more elements are estimating different quantities, and the points of divergence indicate the source.
-
7. Choose an aggregation strategy consistent with the pattern of alignment. Stratify when multiple studies share a configuration. Use sensitivity analysis when a single misalignment source dominates, typically stopping-rule truncation. Apply metric-level normalization where a normalized form is available, as with Work Saved over Sampling.Reference Kusa, Lipani, Knoth and Hanbury28 Use descriptive pooling when no other option recovers a credible common target, provided the absence of a shared-target claim is made explicit.
11 Implications for future research
The framework also defines a meta-research agenda. Structured reviews that characterize evaluation configurations rather than screening performance could delineate the domain spaces over which existing results are interpretable and potentially comparable.Reference O’Connor, Tsafnat and Gilbert8, Reference Olorisade, Brereton and Andras9 This approach would support conditional synthesis and disciplined comparison without introducing new benchmarks or standards. Emerging evaluations that use language-model-generated summaries of full-text articles as the representation for classificationReference Song, Huang and Thapa45 alter choices within one element of the framework without displacing the others. The representation at the decision point is no longer inferable from the workflow stage, and explicit documentation of what was used would support cross-study comparison.
A complementary direction concerns what screening evaluation should estimate. Kusa and colleaguesReference Kusa, Zuccon, Knoth and Hanbury46 argue that traditional recall and precision treat all relevant records as equivalently important, when in fact the influence of an included record on the downstream synthesis varies with the weight it carries in the eventual meta-analysis. Their outcome-based evaluation framework reweights retrieval performance by the effect of each record on review conclusions, producing a measure that more closely tracks what screening is ultimately for. The two lines of work are compatible: identifying a workflow’s induced evaluation target and specifying what that target should be given the eventual use of the review are separable questions.
Recent work on systematic review workflow automation reflects growing recognition of these limitations. Wittenborg et al.,Reference Wittenborg, Karras and Auer47 for example, propose a requirements-driven framework that specifies workflow stages, tasks, and tool roles across the review lifecycle and evaluates tools relative to these requirements rather than aggregate performance metrics. Meaningful synthesis of screening evaluation evidence therefore depends on characterizing the evaluation targets those workflows induce, not on the assumption that shared metric labels imply shared quantities.
12 Limitations
12.1 Several limitations apply
The framework is conceptual and does not estimate the prevalence of configurations in the literature. The illustrated configurations are not exhaustive, and other workflow patterns will require additional analysis. Alignment judgments may require interpretation when reporting is incomplete, since the framework identifies the choices that matter but cannot supply missing operational details. The framework diagnoses target nonalignment but does not by itself adjudicate which target is substantively correct, a question that depends on the use to which the screening results are put. The development is focused on the Identification and Screening stages of PRISMA 2020, with the data extraction and downstream synthesis stages noted only briefly. Statistical adjustment, sensitivity analysis, or principled restriction of inference may sometimes recover interpretable cross-study comparisons even when raw alignment fails, but only under additional assumptions that the framework does not specify.
13 Conclusion
Cross-study cumulation depends on whether reported screening results estimate comparable quantities. Metric labels alone do not establish comparability. The framework here makes the operational specifications behind each metric explicit and provides a basis for assessing alignment across studies. Where alignment holds, comparison and synthesis are appropriate. Where it fails, the points of misalignment indicate what must be reported, normalized, or restricted before aggregation. The framework is not anti-aggregation but a condition-setting framework for justified aggregation. Characterizing workflow-induced targets is a prerequisite for cumulative inference about automation-assisted screening.
Author contributions
Conceptualization: J.A.N.; Data curation: J.A.N.; Formal analysis: J.A.N.; Writing—original draft: J.A.N.; Writing—review and editing: J.A.N.
Competing interest statement
The author declares that no competing interests exist.
Data availability statement
No quantitative primary-study data were extracted or statistically analyzed. Published studies were used as conceptual exemplars, and no study-level dataset was generated.
Funding statement
The author declares that no specific funding has been received for this article.
Use of artificial intelligence (AI) tools
Artificial intelligence tools were used during manuscript preparation. During initial preparation in 2025, GPT-5.1 Pro (OpenAI) was used for clerical checks, citation–reference consistency verification, reference renumbering, and copy editing. During the 2026 revision, Claude Opus 4.7 (Anthropic) was used for response auditing, phrasing support, and structural review. Both are commercially available tools accessible through their respective providers. The author reviewed, revised, and approved all manuscript content and is responsible for its accuracy, originality, and substance. No AI tool was used to generate or analyze data, develop the framework, conduct literature searches, or assess the substance of cited works.
Open access
