Knowledge synthesis involves bringing together findings from individual research studies to answer a specific question or explore a particular topic, often at a global level. People with lived or living experience of health conditions or of accessing healthcare services (PWLE), such as patients and their families or friends, can provide valuable perspectives throughout this process. PWLE involvement can help shape the design and conduct of a knowledge synthesis to better reflect real-world concerns through a process called patient engagement. Patient engagement is the meaningful and active involvement of PWLE throughout the research process. While previous research shows that PWLE can contribute meaningfully when appropriate supports are in place, there is limited practical guidance on how to enact patient engagement in knowledge syntheses. This tutorial paper aims to guide researchers and PWLE who would like to engage in knowledge syntheses together. Building on existing research, our team (comprised of PWLE, researchers, and an academic librarian) shares lessons we have learned from working together throughout the stages of a knowledge synthesis. We present considerations for successful patient engagement when research teams are: (1) planning to engage, (2) recruiting PWLE, (3) establishing roles and rapport, (4) supporting capacity development, (5) conducting the knowledge synthesis, and (6) mobilizing findings. We also identify 12 common barriers to patient engagement and offer strategies to address them, alongside recommendations for integrating PWLE across every stage of the knowledge synthesis process.

Sharing, harmonizing, and analyzing participant-level data is of central importance in the rapid research response to emerging pathogens. Individual participant data meta-analyses (IPD-MAs), which synthesize participant-level data from related primary studies, have several advantages over pooling study-level effect estimates in a traditional meta-analysis. IPD-MAs enable researchers to more effectively separate spurious heterogeneity related to differences in measurement from clinically relevant heterogeneity from differences in underlying risk or distribution of factors that modify disease progression. This tutorial describes the steps needed to conduct an IPD-MA of an emerging pathogen and how IPD-MAs of emerging pathogens differ from those of well-studied exposures and outcomes. We discuss key statistical issues, including participant- and study-level missingness and complex measurement error, and present recommendations. We review how IPD-MAs conducted during the COVID-19 response addressed these statistical challenges when harmonizing and analyzing participant-level data related to an emerging pathogen. The guidance presented here is based on lessons learned in our conduct of IPD-MAs in the research response to emerging pathogens, including Zika virus and COVID-19.

Intraclass correlation coefficient (ICC) estimates are necessary for several statistical techniques. Researchers need accurate ICC estimates when conducting prospective power analyses for clustered data scenarios. In addition, meta-analysts require reasonable ICC values when adjusting effect size estimates to account for clustered primary study data or to correct for psychometric artifacts when using the ICC as a reliability measure. The validity of these analyses hinges on the accuracy of the ICC estimate. Beyond these secondary analyses, ICC estimates have been used as the focal outcome of meta-analysis itself to obtain a pooled measure of agreement, reliability, or the influence of a cluster’s effect. This study evaluates how well meta-analytically pooled ICC estimates recover the population ICC parameter value when using different ICC variance formulas as the inverse variance weights used in the pooling. We found that the variance formula that uses a normalizing transformation performs best across most conditions.

The I-squared index was proposed in 2002 as a measure to help understand the consistency of study results in a meta-analysis. It was developed to overcome some of the limitations of existing measures, principally the chi-squared test for heterogeneity and the between-study variance as estimated in a random-effects meta-analysis. I-squared measures approximately the proportion of total variability in results that is due to true heterogeneity rather than random error; it is also conveniently interpreted as a measure of inconsistency in the results of the studies. The index has become extremely widely used, although it is often misinterpreted as an absolute measure of the amount of heterogeneity, which it is not. Here, we discuss the I-squared index and the different ways it can be defined, computed, and interpreted. We introduce a new interpretation of I-squared as a weighted sum of squares, which we propose may be helpful when setting up simulation studies. We discuss some of the extensions and repurposes that have been proposed for I-squared and offer some recommendations on the appropriate use of the index in practice.

Identifying studies for inclusion in realist syntheses using title and abstract screening is challenging given the need to unearth explanatory insights to build context–mechanism–outcome configurations. Such insights may only be uncovered through full-text paper reading. To address this issue, a novel approach for identifying studies has been developed called Reverse Chronology Quota Record Screening (RCQRS). Results of database searches are screened in reverse chronology, and in stages, to fill quotas matching the requirements of the review. RCQRS may be beneficial in any of the following circumstances: (a) the timeframe to complete the synthesis is short; (b) the scope of inquiry is not fully defined, (c) the availability of suitable literature is beyond the screening capacity of the reviewers; or (d) the availability of literature is sparse and reviewers seek to extrapolate insights from other areas. In contrast to RCQRS, exhaustive title and abstract screening (i.e., screening entire database results) may hamper study selection due to being overwhelming in volume and time-intensive, resulting in a causally thin cohort of papers for analysis. RCQRS used in stages, and in conjunction with other search strategies (e.g., hand searching, backward citation tracking, and expert solicitation) can support creative, robust analytical insights with causally rich extrapolations. Using the Horizon-EU funded SERENITY study on deprescribing in palliative care as a case example, the benefits and limitations of RCQRS are explored. Finally, a checklist template is offered for teams who wish to reflect on, and transparently report, the use of RCQRS in their realist synthesis.

It is difficult to understand the safety profile of drugs based on a single clinical trial since clinical trials are often designed to prove efficacies, and sample size is not powered for safety assessment. Thus, meta-analysis would be a valuable tool to infer the safety profiles utilizing multiple studies. Individual clinical trials usually report the incidence proportions of adverse events (AEs) observed in the study. The follow-up duration may be study-specific, and furthermore different between the treatment groups within a single study. It often occurs in oncology clinical trials and if this is the case, it is hard to interpret the aggregated relative risk of AEs and compare the risk of AEs between the treatment groups with the standard meta-analysis techniques. The progression-free survival or the overall survival is often used as the primary endpoint in oncology clinical trials and the Kaplan–Meier estimates of the survival functions for the primary endpoint are often demonstrated graphically, which give us information of the follow-up duration of the AEs. We propose novel meta-analysis methods for AEs that address differences in follow-up durations by efficiently utilizing the Kaplan–Meier estimates of the primary endpoint. We adapt our approach using both simulated data and real data from a meta-analysis of bevacizumab. Simulation studies demonstrate that the proposed methods perform well when follow-up time differs between trials and groups.

Exponential growth in scientific literature has heightened the demand for efficient evidence-based synthesis, driving the rise of the field of automated meta-analysis (AMA) powered by natural language processing and machine learning. This PRISMA systematic review introduces a structured framework for assessing the current state of AMA, based on screening 13,216 papers (2006–2024) and analyzing 61 studies across diverse domains. Findings reveal a predominant focus on automating data processing (52.5%), such as extraction and statistical modeling, while only 16.4% address advanced synthesis stages. Just one study (approximately 2%) explored preliminary full-process automation, highlighting a critical gap that limits AMA’s capacity for comprehensive synthesis. Despite recent breakthroughs in large language models and advanced AI, their integration into statistical modeling and higher-order synthesis, such as heterogeneity assessment and bias evaluation, remains underdeveloped. This has constrained AMA’s potential for fully autonomous meta-analysis (MA). From our dataset spanning medical (67.2%) and non-medical (32.8%) applications, we found that AMA has exhibited distinct implementation patterns and varying degrees of effectiveness in actually improving efficiency, scalability, and reproducibility. While automation has enhanced specific meta-analytic tasks, achieving seamless, end-to-end automation remains an open challenge. As AI systems advance in reasoning and contextual understanding, addressing these gaps is now imperative. Future efforts must focus on bridging automation across all MA stages, refining interpretability, and ensuring methodological robustness to fully realize AMA’s potential for scalable, domain-agnostic synthesis.

Effective public health decision-making relies on rigorous evidence synthesis and transparent processes to facilitate its use. However, existing methods guidance has primarily been developed within clinical medicine and may not sufficiently address the complexities of public health, such as population-level considerations, multiple evidence streams, and time-sensitive decision-making. This work contributes to the European Centre for Disease Prevention and Control initiative on methods guidance development for evidence synthesis and evidence-based public health advice by systematically identifying and mapping guidance from health and health-related disciplines.

Structured searches were conducted across multiple scientific databases and websites of key institutions, followed by screening and data coding. Of the 17,386 records identified, 247 documents were classified as ‘guidance products’ providing a set of principles or recommendations on the overall process of developing evidence synthesis and evidence-based advice. While many were classified as ‘generic’ in scope, a majority originated from clinical medicine and focused on systematic reviews of intervention effects. Only 41 documents explicitly addressed public health. Key gaps included approaches for rapid evidence synthesis and decision-making and methods for synthesising evidence from laboratory research, disease burden, and prevalence studies.

The findings highlight a need for methodological development that aligns with the realities of public health practice, particularly in emergency contexts. This review provides a key repository for methodologists, researchers, and decision-makers in public health, as well as clinical medicine and health care in Europe and worldwide, supporting the evolution of more inclusive and adaptable approaches to public health evidence synthesis and decision-making.

Automating data extraction from full-text randomized controlled trials for meta-analysis remains a significant challenge. This study evaluates the practical performance of three large language models (LLMs) (Gemini-2.0-flash, Grok-3, and GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customized prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customized prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.

Flexible meta-regression is an important facet of biomedical research. These methods employ basis expansions to model the relationship between exposure and outcome. Examples include splines, fractional polynomial, and local regression models. The extent to which flexible meta-regression has been appropriately employed and reported in the past has not been studied. This study aims to provide an overview of flexible meta-regression for researchers as well as summarize the practical use and reporting of these methods in the epidemiological literature. A systematic literature search of EMBASE, MEDLINE, and the Cochrane Library identified systematic reviews published between 2011 and 2021 which employed flexible meta-regression methods. Data on study characteristics were extracted and validated by two independent researchers. Given the lack of guidance on reporting standards for flexible meta-regression, we proposed a novel 9-item checklist. This was used to assess reporting quality in included studies. A total of $N=346$ eligible studies were identified and $N=86$ were randomly sampled for full data extraction and analysis. The number of reviews using flexible regression methods increased over time. Spline models were the most frequently employed class of methods. There were no apparent trends in the use of the three methods over country of affiliation, number of authors, or clinical context. In over two thirds of reviews sampled, methods were found to have not been reported consistently. Further work is needed to improve the reporting of these methods to ensure research transparency and reproducibility.

Critical interpretive synthesis was introduced in 2006 to address various shortcomings of systematic reviews such as their limitations in synthesizing heterogeneous data, integrating diverse study types, and generating theoretical insights. This review sought to outline the methodological process of conducting critical interpretive syntheses by identifying the methods currently in use, mapping the processes that have been used to date, and highlighting directions for further research. To achieve this, a scoping review of critical interpretive syntheses published between 2006 and 2023 was conducted. Initial searches identified 1628 publications and after removal of duplicates and exclusions, 212 reviews were included in the study. Most reviews focused on health-related subjects. Authors chose to utilize the method due to its iterative, inductive, and recursive nature. Both question-based and topic-based reviews were conducted. Literature searches relied on electronic databases and reference chaining. Mapping to the original six-phase model showed most variability in use of sampling and quality assessment phases, which were each done in 50.7% of reviews. Data extraction utilized a data extraction table. Synthesis involved constant comparison, critique, and consolidation of themes into constructs, and a synthesizing argument. Refining critical interpretive synthesis methodology and its best practices are important for optimizing the utility and impact and ensuring findings are relevant and actionable for informing policy, practice, and future research.

The ROBINS-I tool is a commonly used tool to assess risk of bias in non-randomised studies of interventions (NRSI) included in systematic reviews. The reporting of ROBINS-I results is important for decision-makers using systematic reviews to understand the weaknesses of the evidence. In particular, systematic review authors should apply the tool according to the guidance provided. This study aims to describe how ROBINS-I guidance is currently applied by review authors. In January 2023, we undertook a citation search and screened titles and abstracts of records published in the previous 6 months. We included systematic reviews of non-randomised studies of intervention where ROBINS-I had been used for risk-of-bias assessment. Based on 10 criteria, we summarised the diverse ways in which reviews deviated from or reported the use of ROBINS-I. In total, 492 reviews met our inclusion criteria. Only one review met all the expectations of the ROBINS-I guidance. A small proportion of reviews deviated from the seven standard domains (3%), judgements (13%), or in other ways (1%). Of the 476 (97%) reviews that reported some ROBINS-I results, only 57 (12%) reviews reported ROBINS-I results at the outcome level compared with 203 reviews that reported ROBINS-I results at the study level alone. Most systematic reviews of NRSIs do not fully apply the ROBINS-I guidance. This raises concerns around the validity of the ROBINS-I results reported and the use of the evidence from these reviews in decision-making.

With the rapid growth of scholarly literature, efficient artificial intelligence (AI)–aided abstract screening tools are becoming increasingly important. This study evaluated 10 different machine learning (ML) algorithms used in AI-aided screening tools for ordering abstracts according to their estimated relevance. We focused on assessing their performance in terms of the number of abstracts required to screen to achieve a sufficient detection rate of relevant articles. Our evaluation included articles screened with diverse inclusion and exclusion criteria. Crucially, we examined how characteristics of the screening data—such as the proportion of relevant articles, the overall frequency of abstracts, and the amount of training data—impacted algorithm effectiveness. Our findings provide valuable insights for researchers across disciplines, highlighting key factors to consider when selecting an ML algorithm and determining a stopping point for AI-aided screening. Specifically, we observed that the algorithm combining the logistic regression (LR) classifier with the sentence-bidirectional encoder representations from transformers (SBERT) feature extractor outperformed other algorithms, demonstrating both the highest efficiency and the lowest variability in performance. Nonetheless, the algorithm’s performance varied across experimental conditions. Building on these findings, we discuss the results and provide practical recommendations to assist users in the AI-aided screening process.

A meta-analysis of diagnostic test accuracy (DTA) studies typically synthesizes study-specific test sensitivity ($Se$) and specificity ($Sp$) to quantify the accuracy of an index test of interest. The bivariate linear mixed effects model with logit transformation of $Se$ and $Sp$ (BLMM-Logit) is commonly used to make statistical inferences, but may lead to misleading results due to the need for Haldane–Anscombe correction and an approximate estimation of variance within the study. Alternative models based on the arcsine square root and Freeman–Tukey double arcsine transformation have been proposed to address these issues; however, they still rely on approximate variance estimation, which is suitable only for large sample sizes. The bivariate generalized linear mixed effects model (BGLMM) is another option, but it faces convergence issues with small meta-analyses or sparse primary studies. To address these limitations, we proposed an exact within-study variance calculation method that does not require Haldane–Anscombe correction and is applicable regardless of the transformation used or the number of studies and participants. We evaluated this method against existing approaches using real-life and simulated DTA meta-analyses. The methods were comparable for large meta-analyses. However, BLMM-Logit demonstrated substantial negative bias in estimating variances between studies and consistently underestimated summary $Se$ and $Sp$ in all simulation scenarios. In contrast, the proposed exact methods (Exact-Logit, Exact-ASR, and Exact-FTDA) and BGLMM had minimal bias and better performance metrics, particularly for meta-analyses with sparse primary studies. Thus, the proposed exact methods should be preferred for DTA meta-analyses with small or sparse studies.

Scoping reviews are increasingly used for evidence synthesis, yet a comprehensive overview of benchmarks for their conduct is lacking. This study aimed to extract and analyze key metrics of preregistered scoping reviews to aid researchers in planning, resource allocation, methodological rigor, and innovation. We examined 2,038 scoping review protocol registrations on the Open Science Framework between April and August 2024, identifying 891 corresponding publications. Extracted variables included review type (e.g., rapid scoping review), information about the number of studies in the screening process, methodological practices, and funding. Using Stata, we calculated process metrics such as review duration, yield rate, and team composition. Qualitative analysis was applied to further metrics, including aims and eligibility criteria frameworks. Among the 891 publications, 91.47% were published as scoping reviews. On average, reviews involved 6.11 authors and took 80.41 weeks from registration to publication, with rapid scoping reviews completed in 55.63 weeks. Most publications (80.13%) originated from high-income countries, where funding was more common. An average of 5.19 databases were searched, identifying a mean of 6,920.66 potential inclusions, with 92.64 studies ultimately included (yield rate: 5.68%). Endnote was the most frequently employed screening tool. Language restrictions were reported in 37.60% of cases, and 75.42% did not conduct or report critical appraisal. This study provides valuable insights into common practices within scoping review processes, offering researchers a foundation to allocate resources more efficiently and enhance research quality through methodological benchmarks. It underscores the need for greater transparency and improved methodological rigor.

Network meta-analysis is the established method to pool evidence from multiple clinical trials and make direct and indirect comparisons between different treatments. To ensure its validity, one of the major assumptions requiring examination is that the different sources of information are consistent, which is to say that the direct and indirect effect estimates agree. There are at least three different aspects to consider: (1) the original effect sizes of the direct and indirect treatment effects and their relative contribution to the total evidence; (2) the difference between them and its associated uncertainty/significance; and (3) the type of difference between them, that is, whether the direct and indirect estimates agree that a treatment is beneficial or harmful. Current visualization approaches typically use forest plots or heat maps, but these are limited as at least one of the above aspects is usually absent. Furthermore, as the number of treatments in the network increases, these visualizations can become difficult to understand. We present a visualization that combines the three aspects without being too difficult to interpret, outline the mathematical background and provide the code to produce it in R.

Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.

Dissemination bias can occur when qualitative research is published selectively, potentially reducing the confidence in qualitative evidence. This retrospective cohort study aims to quantify the extent of non-dissemination of qualitative health research by following 1,123 conference abstracts. The proportion of non-dissemination, the time to publication, as well as associations between author or study characteristics and full publication were examined. For 22.8% of these studies, no full publication could be identified within at least 6 and up to 8 years after their presentation. For those that were published, median time to publication was 11 months (95% CI 10 to 12). Studies from authors affiliated with institutions in Australia were more likely to be published than those from North America (OR 4.47; 95% CI 1.58 to 18.74). Oral presentations were more likely to be published than poster presentations (OR 3.40; 95% CI 1.57 to 8.20). Studies that used two qualitative data collection methods were more likely to be published than studies that used one qualitative method only (OR 1.53; 95% CI 1.01 to 2.38). Conference abstracts that reported no funding were less likely to be published than those which reported funding (OR 0.71; 95% CI 0.51 to 0.99). Publicly funded research was more likely to be published than privately funded research (OR 2.24; 95% CI 1.16 to 4.28). Given the considerable proportion of unpublished health-related qualitative studies, there is a reason to believe that dissemination bias may impact negatively on qualitative evidence synthesis. This can, in turn, impair decision-making that uses qualitative evidence.

This study investigated information retrieval of preprint records in the context of evidence synthesis work and compared 12 sources used to discover preprints. Identification of grey literature is often required or recommended in evidence synthesis guidance, and preprints are categorized as grey literature. The purpose of this work is to inform how and where to search for preprints to maximize coverage (through exploration of preprint server across aggregators and databases) while balancing search efficiency. Authors selected aggregators and databases hosting two or more preprint servers and then tested search functionality and extracted characteristics and features. Authors analyzed and compared the selected sources, tabulated the number of essential features, and created comparison tables reflecting database and aggregator features. The study protocol was registered in Open Science Framework registries. Preprint aggregators and databases differ in their content coverage, and their ability to design a comprehensive and reproducible search strategy. Limitations such as character or word limits for queries, limited advanced search operators, and missing export functionality affect the usability of aggregators for evidence synthesis searches. Ongoing updates to search interfaces and functionality and differing approaches to versioning make it challenging to study discovery of preprints across sources. The recommendations and scenarios in this article will assist searchers engaged in evidence synthesis to make informed decisions about where to search for preprints.