Open science

Open science is a collaborative network to future-proof our field by promoting transparency, replicability, and accessibility. Open science has many benefits, such as reducing or resolving questionable research practices, promoting collaboration and data sharing, increasing the accessibility of nonpublished research, and extending the public’s trust in science. Some actions to promote open science include preregistration of quantitative and qualitative studies (which can reduce questionable research practices such as p hacking), sharing code and data (which can increase research replicability; Hagger, Reference Hagger2021), and contributing to online archives for nonpublished studies (Aguinis et al., Reference Aguinis, Banks, Rogelberg and Cascio2020). Additionally, open science has the potential to enable us to verify the exact procedures and justification for the analysis while also increasing the reliability of results.

Our field has wrestled with the file drawer problem (i.e., the threat of bias within empirical literature due to “null effect” studies not being disseminated; Card, Reference Card and Salkind2010); however, using open science practices allows nonpublished or “null-effect” studies to be uploaded and shared online, which promotes the publication of the findings regardless of the statistical effect (Allen & Mehler, Reference Allen and Mehler2019). Furthermore, open science allows for increased accessibility to other datasets on platforms such as Open Science Framework (OSF; osf.io). Requiring graduate students to preregister studies and share research plans, data, and code via the OSF as part of coursework helps build a culture of open science and promotes multidisciplinary research by facilitating collaboration with other disciplines and researchers. Graduate students can benefit from learning the values of open science while also gaining practical experience by practicing in courses. Ultimately, open science practices make our work more credible, reliable, and collaborative. We envision the future of I-O research to be a more accessible and collaborative process while increasing the trust in the field as a science. The first step is to provide graduate students with tangible opportunities to engage with open science practices.

Open-source software

Despite the acknowledgment that R is a highly capable, flexible, and free software that could become the “default” statistical program for I-O researchers, many undergraduate programs, graduate programs, and research labs still use paid-license alternatives like SPSS, SAS, Stata, MPlus, and so on as their go-to software for training and use. Those options certainly have unique strengths and weaknesses for specific projects or processes, so exposure to multiple statistical programs should be encouraged. However, I-O psychology departments can work to future-proof statistical software by implementing more comprehensive coursework and training in open-source software like R, Jamovi, JASP, and/or Python. R is a particularly appealing software due to its variety of shared libraries and searchable code. Instruction in these languages as distinct topics, rather than as a “means to an end” (i.e., to complete research projects or course assignments within I-O), should shift students’ views on coding and data analysis more broadly too (i.e., show its applicability to jobs, research, and hobbies outside of I-O).

Improved graduate training in open-source software future-proofs the field in multiple regards. First, greater fluency in coding provides the next generation of researchers with the skill set to adapt more quickly to constant changes in technology. Second, collectively training the next generation of researchers in the same software can make collaboration and scrutinization of research more efficient. Finally, the resulting shift in the field to free (or low-cost) software options like R, Jamovi, or JASP could make methodological processes more accessible to a broader population in the future. In other words, a more diverse group of academics, practitioners, students, and members of the public can learn and implement the field’s newest methodological techniques regardless of their financial resources or institutional affiliation.

Bayesian statistics

As pointed out by Hyland, overreliance on null hypothesis significance testing (NHST) may be limiting our ability to theorize and examine research questions. As mentioned above, graduate statistics rely heavily upon the frequentist paradigm of statistical thinking and NHST. However, the Bayesian statistical paradigm offers several benefits that have yet to be fully appreciated in I-O psychology such as providing direct support for any model, incorporating prior empirical evidence, and representing uncertainty with probability distributions. More recently, statistical software, such as JASP (JASP Team, 2021) and R, has greatly improved the accessibility of Bayesian analysis. Along with the technological gaps, most I-O psychologists are entrenched in frequentist philosophical thinking and are unable to provide Bayesian courses to students.

Bayesian methods provide additional benefits that can improve graduate statistical reasoning and analysis. First, Bayesian methods view probability as a degree of belief and represent uncertainty in probabilistic distributions that are often more thoughtful representations of events. This contrasts with the dichotomous, all-or-nothing thinking of p values used in NHST where we are limited to “rejecting the null” or “failing to reject the null hypothesis” (Hoekstra et al., Reference Hoekstra, Finch, Kiers and Johnson2006; Rosnow & Rosenthal, Reference Rosnow and Rosenthal1989). Second, a Bayesian analysis can support the finding of no difference and even compare multiple alternative models through Bayes factors that tell us how much better one model is at explaining the observed data relative to another model. One way to facilitate a transition to Bayesian thinking might be to teach null hypothesis Bayesian testing to graduate students in addition to or as an alternative to NHST so that students can quantify results and support null findings (e.g., Dienes, Reference Dienes2014). Third, a Bayesian analysis allows for the inclusion of known information to inform the data analysis (e.g., data from a meta-analysis or possible parameter values). Not only is this beneficial for explicitly modeling assumptions and prior beliefs, but it also is beneficial for teaching graduate students how to handle smaller samples. Oftentimes datasets with small samples are not used due to an increase in Type II error (i.e., failing to reject the null when it is false), but in the Bayesian framework, we can supplement small samples with carefully selected prior distributions (e.g., van de Schoot et al., Reference van de Schoot, Broere, Perryck, Zondervan-Zwijnenburg and van Loey2015).

There is a growing consensus among various disciplines that we need to train graduate students to be successful in their statistical reasoning. Taking a multidisciplinary approach by providing Bayesian statistics courses would future-proof statistical inference by equipping graduate students with another statistical tool that has many potential advantages for I-O psychology.