The target article noted that we (Francis & Thunell, Reference Francis and Thunell2022) previously reported that many multi-study publications on object-based attention seem “too good to be true” in the sense that the success rate of the experimental outcomes (e.g. finding a significant result) dramatically exceeded the estimated power of those experiments, based on the reported effects and sample sizes. Possible explanations for this inconsistency include that failed studies were not reported (publication bias) and that studies were being run incorrectly (questionable research practices), which are both causes of psychology’s replication crisis over the past decade. Whatever the reason, the published studies likely exaggerate the true effect.

We further noted that none of the analyzed studies on object-based attention reported a power analysis; not even those performing a direct replication. Here, we perform a power analysis based on the large-sample study reported by Francis and Thunell (Reference Francis and Thunell2022) where the object-based attention effect (the difference between response times for a same object condition and a different object condition) amounts to a statistically significant effect of 14 ms (first row of Table 1). If this finding is representative of the population object-based attention effect, a sample size of n = 222 participants would be needed to give 80% power for a dependent t-test (Francis, Reference Francis2024a). The mean sample size in the studies analyzed by Francis and Thunell (Reference Francis and Thunell2022) is 17 (last column of Table 1), which is thus nowhere near large enough to reliably show this effect. To make things worse, the studies did not aim to merely show the basic object-based effect, but rather modulations thereof, which entail lower power still. As an example, suppose some special population group has a reduced object-based attention effect that is only half as large (7 ms) as that of controls. If we compare this reduced effect to the standard 14 ms effect using an independent samples t-test, 1756 participants per group are required to achieve 80% power (Table 1; some scientists advocate for higher power, so we include calculations also for 90% power).

Table 1.

Power analyses for five attention paradigms. The independent test compares the basic effect in a control group against an experimental group that has a basic effect with half the magnitude of the control group. Commonly used sample sizes in experimental studies in the meta-analysis articles (far right column) are much too small to reliably show effects of interest

We conducted a similar analysis for four other common experimental paradigms in attention (Spatial Cueing, the Stroop Effect, the Simon Effect, and the Emotional Stroop Effect). We performed power analyses based on data from online versions of these paradigms where thousands of students participated (Francis & Neath, Reference Francis and Neath2015; Francis, Reference Francis2024b). Comparing the estimated sample sizes needed to achieve reasonable power (Table 1; column 5) with the sample sizes used for experiments in recent meta-analyses in the respective fields, it is clear that, also in these fields, sample sizes are often too small. The paradigm that fares best is the Stroop Effect: According to our power analysis, samples of n = 9 are sufficient to show the basic effect with 80% power, and a recent meta-analysis of the Stroop Effect (Šömen, et al., Reference Šömen, Peskar, Wollesen, Gramann and Marusic2023) indicates that samples of size n = 50 are common. However, modern studies typically do not measure just the basic Stroop Effect but instead look for modulations of the Stroop Effect, which generally reduces power: A study investigating, for example, half a difference in the effect across independent samples needs 52 participants per sample to have 80% power (Table 1).

For the three other considered paradigms, the studies included in meta-analyses (Cespón, Hommel, Korsch, & Galashan, Reference Cespón, Hommel, Korsch and Galashan2020; Cisler, et al., Reference Cisler, Wolitzky-Taylor, Adams, Babson, Badour and Willems2011; Landry & Parker, Reference Landry and Parker2013) generally used samples much too small to consistently show even the basic effects (Table 1, last column). Strikingly, scientists commonly run experiments on the Emotional Stroop Effect with sample sizes similar to those used for the Stroop Effect, even though the magnitude of the Emotional Stroop Effect (4 ms) is estimated to be only about 2% of the classic Stroop Effect (185 ms). These small sample sizes should not be taken as evidence that the effects are not real, but it seems unlikely that the reported modulations have been accurately measured. We suspect that many of the conclusions derived from these studies are chasing noise, and will not replicate in new studies.

Thus, while we agree with the target article that there is a crisis in visual attention, we are not convinced that the field fits the first of Kuhn’s (Reference Kuhn1962) signs of a crisis: that there be significant anomalies that must be explained. Rather, if the visual attention paradigms discussed here (Table 1) are representative, the observed “anomalies” might simply be due to noise. Indeed, the apparent consistency of some effects across multiple studies highlights a different kind of crisis: Scientists likely do not publish their (many) failed experimental results, which means that the published studies likely misrepresent the true effects. It is impossible to build good theories (based on summary statistic encoding or otherwise) until there is good experimental data with which those theories can be calibrated and compared.

Fortunately, it is relatively easy to address these problems by running informed power analyses that consider the effect sizes and the planned comparisons. Our large-sample data sets (Table 1) are good starting points to identify appropriate sample sizes for these paradigms; but, we suspect that scientists could do better with lab-based experiments, more trials per participant, and optimization of stimuli that lead to bigger effects; these methodological improvements may allow for smaller sample sizes than what is listed in Table 1. On the other hand, it is common for investigations to base their conclusions on a complex set of hypothesis test outcomes, and ensuring a high probability that all relevant outcomes are statistically significant requires larger samples. For paradigms where a single large dataset is not already available, scientists can use published effect sizes to perform power analyses. Given our concerns that the literature is biased with overestimated effects, this initial power analysis might indicate sample sizes that are still too small to give adequate power. Our hope is that new unbiased experiments will better estimate reality and allow for future experiments to identify appropriate sample sizes and thereby contribute to a better understanding of visual attention.