Problems with reproducibility in psychology

Lack of reproducibility might seem like a non-issue; after all, it may seem like a guarantee that running the same analysis on the same data would give the same result. However, there is increasing evidence that reproducibility of published results in psychology is relatively low.

Checking reproducibility of reported results in psychology is greatly impeded by a common failure to share data (Vanpaemel et al. Reference Vanpaemel, Vermorgen, Deriemaecker and Storms2015; Wicherts et al. Reference Wicherts, Borsboom, Kats and Molenaar2006). Even when data are available, they are often of poor quality or not usable (Kidwell et al. Reference Kidwell, Lazarevic, Baranski, Hardwicke, Piechowski, Falkenberg, Kennett, Slowik, Sonnleitner, Hess-Holden, Errington, Fiedler and Nosek2016). Yet some issues with reproducibility can be assessed by scrutinizing papers. Studies repeatedly showed that roughly half of all published psychology articles contains at least one inconsistently reported statistical result, wherein the reported p value does not match the degrees of freedom and test statistic; in roughly one in eight results this may have affected the statistical conclusion (e.g., Bakker & Wicherts, Reference Bakker and Wicherts2011; Nuijten et al. Reference Nuijten, Hartgerink, Van Assen, Epskamp and Wicherts2016; Veldkamp et al. Reference Veldkamp, Nuijten, Dominguez-Alvarez, van Assen and Wicherts2014; Wicherts et al. Reference Wicherts, Bakker and Molenaar2011). Furthermore, there is evidence that roughly half of psychology articles are inconsistent with the given sample size and number of items (Brown & Heathers Reference Brown and Heathers2017), coefficients in mediation models often do not add up (Petrocelli et al. Reference Petrocelli, Clarkson, Whitmire and Moon2012), and in 41% of psychology articles reported degrees of freedom do not match the sample size description (Bakker & Wicherts Reference Bakker and Wicherts2014).

Problems that can be detected without having the raw data, are arguably just the tip of the iceberg of reproducibility issues. Studies that intended to reanalyze data from published studies also often ran into problems (e.g., Ebrahim et al. Reference Ebrahim, Sohani, Montoya, Agarwal, Thorlund, Mills and Ioannidis2014; Ioannidis et al. Reference Ioannidis, Allison, Ball, Coulibaly, Cui, Culhane, Falchi, Furlanello, Game, Jurman, Mangion, Mehta, Nitzburg, Page, Petretto and van Noort2009). Beside the poor availability of raw data, papers usually do not contain details about the exact analytical strategy. Researchers often seem to make analytical choices that are driven by the need to obtain a significant result (Agnoli et al. Reference Agnoli, Wicherts, Veldkamp, Albiero and Cubelli2017; John et al. Reference John, Loewenstein and Prelec2012). These choices can be seemingly arbitrary (e.g., choice of control variables or rules for outlier removal; see also Bakker et al. [Reference Bakker, van Dijk and Wicherts2012] and Simmons et al. [Reference Simmons, Nelson and Simonsohn2011]), which makes it hard to retrace the original analytical steps to verify the result.

Suggested solution

Performing a replication study in a novel sample to establish the reliability of a certain result is time consuming and expensive. It is essential that we avoid wasting resources on trying to replicate a finding that may not even be reproducible from the original data. Therefore, we argue that it should be standard practice to verify the original results before any direct replication is conducted.

A first step in verifying original results can be to check whether the results reported in a paper are internally consistent. Some initial screenings can be done quickly with automated tools such as “statcheck” (Epskamp & Nuijten Reference Epskamp and Nuijten2016; http://statcheck.io), “p-checker” (Schönbrodt Reference Schönbrodt2018), and granularity-related inconsistency of means (“GRIM” [Brown & Heathers Reference Brown and Heathers2017]). Especially if such preliminary checks already flag several potential problems, it is crucial that data and analysis scripts are made available for more detailed reanalysis. One could even argue that if data are not shared in such cases, the article should be retracted.

If a result can successfully be reproduced with the original data and analyses, it is interesting to investigate its sensitivity to alternative analytical choices. One way to do so is to run a so-called multiverse analysis (Steegen et al. 2016), in which different analytical choices are compared to test the robustness of the result. When a multiverse analysis shows that the study result is present in only a limited set of reasonable scenarios, you may not want to invest additional resources in replicating such a study. Note that a multiverse analysis still does not require any new data, and is therefore a relatively cost-effective way to investigate reliability.

Reanalysis of existing data is a crucial tool in investigating reliability of psychological results, so it should become standard practice to share raw data and analysis scripts. Journal policies can be successful in promoting this (Giofrè et al. Reference Giofrè, Cumming, Fresc, Boedker and Tressoldi2017; Kidwell et al Reference Kidwell, Lazarevic, Baranski, Hardwicke, Piechowski, Falkenberg, Kennett, Slowik, Sonnleitner, Hess-Holden, Errington, Fiedler and Nosek2016; Nuijten et al. Reference Nuijten, Borghuis, Veldkamp, Dominguez-Alvarez, Van Assen and Wicherts2017), so we hope that more journals will start requiring raw data and scripts.

In our proposal, the assessment of replicability is a multistep approach that first assesses whether the original reported results are internally consistent, then sets out to verify the original results through independent reanalysis of the data using the original analytical strategy, followed by a sensitivity analysis that checks whether the original result is robust to alternative choices in the analysis, and only then involves the collection of new data.