Meta-analysis is often viewed as a gold standard for synthesizing available evidence, but it faces challenges as evidence becomes available gradually and studies may be disseminated selectively and at different times. When publication timing is systematically related to study results, meta-analytic conclusions at any given time can be biased; for example, studies with statistically significant or larger effects may be published earlier than those with null or smaller effects. This phenomenon, commonly referred to as time-lag bias, is a key reason why early meta-analytic conclusions may later weaken, reverse, or become more heterogeneous as additional evidence accrues. To address the limitations of existing definitions and detection approaches, which are largely tied to monotone, effect-size-driven dissemination, we broaden the concept of time-lag bias beyond the traditional yet simple notion that “positive” studies are published earlier. Specifically, we view time-lag bias more generally as a manifestation of time-dependent evidence availability, whereby the timing of study dissemination can distort meta-analytic conclusions at different stages of evidence accrual. We illustrate this broader perspective through several real-world case studies, including classical time-lag bias, inverse time-lag bias, non-monotone temporal patterns, and anomalous evidence localized within specific time windows. These examples show that time-lag bias can arise through multiple dissemination mechanisms and may exaggerate, attenuate, or even reverse pooled estimates over time. They highlight the importance of explicitly accounting for the temporal dimension of evidence accumulation and motivate the routine use of diagnostic and sensitivity strategies to assess robustness to selective dissemination over time.

Meta-analyses in social science fields face multiple methodological challenges arising from how primary research studies are designed and reported. One challenge is that many primary studies report multiple relevant effect size estimates, leading to a data structure with dependent observations. Another is selective reporting bias, which arises when the availability of study findings is influenced by the statistical significance of results. Although many selective reporting diagnostics and bias-correction methods have been proposed, few are suitable for meta-analyses involving dependent effect sizes. Among available methods, step-function selection models are conceptually appealing and have shown promise in previous simulations, but tend to understate uncertainty in parameter estimates if dependent effects are ignored. We study methods for estimating step-function models from data involving dependent effect sizes, focusing specifically on estimating parameters of the marginal distribution of effect sizes and accounting for dependence using cluster-robust variance estimation or bootstrap resampling. We describe two estimation strategies, demonstrate them by re-analyzing data from a previous synthesis on ego depletion effects, and evaluate their performance through an extensive simulation study under both univariate and multivariate selection processes. Simulation findings indicate that selection models provide low-bias estimates of average effect size and clustered bootstrap confidence intervals provide acceptable coverage levels for meta-analyses with at least 30 studies. Although unadjusted estimates are more efficient when selective reporting is absent or minimal, the proposed methods provide a robust and useful approach for evaluating the potential biases arising when selective reporting is present in meta-analyses with dependent effects.

Dissemination bias can occur when qualitative research is published selectively, potentially reducing the confidence in qualitative evidence. This retrospective cohort study aims to quantify the extent of non-dissemination of qualitative health research by following 1,123 conference abstracts. The proportion of non-dissemination, the time to publication, as well as associations between author or study characteristics and full publication were examined. For 22.8% of these studies, no full publication could be identified within at least 6 and up to 8 years after their presentation. For those that were published, median time to publication was 11 months (95% CI 10 to 12). Studies from authors affiliated with institutions in Australia were more likely to be published than those from North America (OR 4.47; 95% CI 1.58 to 18.74). Oral presentations were more likely to be published than poster presentations (OR 3.40; 95% CI 1.57 to 8.20). Studies that used two qualitative data collection methods were more likely to be published than studies that used one qualitative method only (OR 1.53; 95% CI 1.01 to 2.38). Conference abstracts that reported no funding were less likely to be published than those which reported funding (OR 0.71; 95% CI 0.51 to 0.99). Publicly funded research was more likely to be published than privately funded research (OR 2.24; 95% CI 1.16 to 4.28). Given the considerable proportion of unpublished health-related qualitative studies, there is a reason to believe that dissemination bias may impact negatively on qualitative evidence synthesis. This can, in turn, impair decision-making that uses qualitative evidence.

Do academic publication standards reflect or determine research results? The article proposes minimal criteria for distinguishing useful ‘unpublishable’ results from low-quality research, and argues that the virtues of negative results have been overlooked. We consider the fate these results have suffered thus far, review arguments for and against their publication and introduce a new initiative – a journal to disseminate negative results and advance debate on their recognition and use.

Qualitative research addresses important healthcare questions, including patients’ experiences with interventions. Qualitative evidence syntheses combine findings from individual studies and are increasingly used to inform health guidelines. However, dissemination bias—selective non-dissemination of studies or findings—may distort the body of evidence. This study examined reasons for the non-dissemination of qualitative studies. We identified conference abstracts reporting qualitative, health-related studies. We invited authors to answer a survey containing quantitative and qualitative questions. We performed descriptive analyses on the quantitative data and inductive thematic analysis on the qualitative data. Most of the 142 respondents were female, established researchers. About a third reported that their study had not been published in full after their conference presentation. The main reasons were time constraints, career changes, and a lack of interest. Few indicated non-publication due to the nature of the study findings. Decisions not to publish were largely made by author teams. Half of the 72% who published their study reported that all findings were included in the publication. This study highlights researchers’ reasons for non-dissemination of qualitative research. One-third of studies presented as conference abstracts remained unpublished, but non-dissemination was rarely linked to the study findings. Further research is needed to understand the systematic non-dissemination of qualitative studies.

Until a few years ago, moderate alcohol consumption was thought to have (mild) beneficial effects on health. However, some recent studies have suggested that “there is no safe level” of alcohol intake. Consequently, public health institutions have responded by advising against any level of alcohol use and suggesting governments a number of policies to reduce overall alcohol consumption. Nonetheless, medical studies suffer from a variety of intrinsic limitations that could undermine the reliability of their findings, especially when focusing on low-intake levels. On the one hand, we show that the literature on alcohol consumption may suffer from publication bias; such a problem is known to be present in the scientific literature in general. On the other hand, we discuss other potential sources of bias, which are inevitable due to the infeasibility of randomized controlled trials. We assess a sample of articles for the presence of omitted variable bias, miscalculation of alcohol intake, use of linear in place of non-linear models, lack of validation of Mendelian randomization assumptions, and other possible weaknesses. We conclude that the claim that “there is no safe level” of alcohol intake is not sufficiently supported based on our current scientific knowledge.

A key parameter estimated by lab and field experiments in economics is the individual discount rate—and the results vary widely. We examine the extent to which this variance can be attributed to observable differences in methods, subject pools, and potential publication bias. To address the model uncertainty inherent to such an exercise we employ Bayesian and frequentist model averaging. We obtain evidence consistent with publication bias against unintuitive results. The corrected mean annual discount rate is 0.33. Our findings also suggest that discount rates are independent across domains: people tend to be less patient when health is at stake compared to money. Negative framing is associated with more patience. Finally, the results of lab and field experiments differ systematically, and it also matters whether the experiment relies on students or uses broader samples of the population.

A growing body of research has found that talking to young children is positively associated with language outcomes. However, there is tremendous heterogeneity in the design of these studies, which could potentially affect the strength and reliability of this association. The present meta-analysis, comprising 4760 participants across 71 studies, goes beyond prior research by including: 1) more recent studies, 2) non-English-speaking populations, 3) more fine-grained categorization of measures of input, 4) additional moderators, and 5) a multilevel model design allowing us to consider multiple effect sizes per study. We find a moderate association between input and outcomes (R2=0.04-0.07) across four input measures, with some evidence of publication bias. We find no differences in effect size across any of the input measures. Child age and study duration moderated some effects of input. Our findings suggest that language input-outcome associations remain robust but modest across a multitude of contexts and measures.

When the process of publication favors studies with small p-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailed p-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.

To evaluate how study characteristics and methodological aspects compare based on presence or absence of industry funding, Hughes et al. conducted a systematic survey of randomized controlled trials (RCTs) published in three major medical journals. The authors found industry-funded RCTs were more likely to be blinded, post results on a clinical trials registration database (ClinicalTrials.gov), and accrue high citation counts.1 Conversely, industry-funded trials had smaller sample sizes and more frequently used placebo as the comparator, used a surrogate as their primary outcome, and had positive results.

This paper examines the replicability of studies in design research triggered by the replication crisis in psychology. It highlights the importance of replicating studies to ensure the robustness of research results and examines whether the description in a publication is sufficient to replicate. Therefore, the publication of a reference study was analysed and a replication study was conducted. The design of the replication study appears similar to the reference study, but the results differ. Possible reasons for the differences and implications for replication studies are discussed.

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships, but the identifying assumptions required by an IV design are demanding, and assessing their validity remains challenging. In this paper, we replicate 67 articles published in three top political science journals from 2010 to 2022 and identify several concerning patterns. First, researchers often overestimate the strength of their instruments due to non-i.i.d. error structures such as clustering. Second, IV estimates are often highly uncertain, and the commonly used t-test for two-stage-least-squares (2SLS) estimates frequently underestimate the uncertainties. Third, in most replicated studies, 2SLS estimates are significantly larger in magnitude than ordinary-least-squares estimates, and their absolute ratio is inversely related to the strength of the instrument in observational studies—a pattern not observed in experimental ones—suggesting potential violations of unconfoundedness or the exclusion restriction in the former. We provide a checklist and software to help researchers avoid these pitfalls and improve their practice.

It is a long known problem that the preferential publication of statistically significant results (publication bias) may lead to incorrect estimates of the true effects being investigated. Even though other research areas (e.g., medicine, biology) are aware of the problem, and have identified strong publication biases, researchers in judgment and decision making (JDM) largely ignore it. We reanalyzed two current meta-analyses in this area. Both showed evidence of publication biases that may have led to a substantial overestimation of the true effects they investigated. A review of additional JDM meta-analyses shows that most meta-analyses conducted no or insufficient analyses of publication bias. However, given our results and the rareness of non-significant effects in the literature, we suspect that biases occur quite often. These findings suggest that (a) conclusions based on meta-analyses without reported tests of publication bias should be interpreted with caution and (b) publication policies and standard research practices should be revised to overcome the problem.

Are difficult decisions best made after a momentary diversion of thought? Previous research addressing this important question has yielded dozens of experiments in which participants were asked to choose the best of several options (e.g., cars or apartments) either after conscious deliberation, or after a momentary diversion of thought induced by an unrelated task. The results of these studies were mixed. Some found that participants who had first performed the unrelated task were more likely to choose the best option, whereas others found no evidence for this so-called unconscious thought advantage (UTA). The current study examined two accounts of this inconsistency in previous findings. According to the reliability account, the UTA does not exist and previous reports of this effect concern nothing but spurious effects obtained with an unreliable paradigm. In contrast, the moderator account proposes that the UTA is a real effect that occurs only when certain conditions are met in the choice task. To test these accounts, we conducted a meta-analysis and a large-scale replication study (N = 399) that met the conditions deemed optimal for replicating the UTA. Consistent with the reliability account, the large-scale replication study yielded no evidence for the UTA, and the meta-analysis showed that previous reports of the UTA were confined to underpowered studies that used relatively small sample sizes. Furthermore, the results of the large-scale study also dispelled the recent suggestion that the UTA might be gender-specific. Accordingly, we conclude that there exists no reliable support for the claim that a momentary diversion of thought leads to better decision making than a period of deliberation.

A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short “null results reports” online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.