One of the essential elements of a rigorous and robust science is that it has the ability to self-correct (Harms, Credé, & DeSimone, Reference Harms, Credé and DeSimone2018). The focal article by Köhler et al. (Reference Köhler, Gonzales-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) has a great deal to say about how to improve the review process, but ultimately we feel that their suggestions fall short. Peer review may be the “cornerstone” of the profession, but in its current form, that cornerstone is cracked. In the words of Drummond Rennie, editor of the Journal of the American Medical Association, “If peer review was a drug it would never be allowed onto the market” (Smith, Reference Wright2010). Köhler et al. acknowledge many of the failings of the current review process, that it can be biased, unreliable, overly negative and critical, unnecessarily time-consuming, and blocks ingenuity, and that it gives editors and reviewers too much power to shape messages to their fit their own agendas (see also Hunter, Reference Hunter2012). However, the thrust of their article is try to offer patches to a broken model. They are not the first to do so, nor are they likely to be the last. The problems with traditional peer review (TPR) seem intractable. For this reason, we suggest that a critical paradigm shift is needed within industrial and organizational (I-O) psychology to bring reviewing into the 21st century.
Twenty-five years ago, with the rise of the world wide web, Earl Hunt made a radical suggestion that the nature of publishing would and should change to embrace technology, reduce journal clutter, and direct attention toward important articles rather than a limited set of high-profile journals (Hunt, Reference Hunt1995). Hunt argued that articles could be uploaded to a central repository, which would be searchable based on key words and then ranked based on readership. “Be read or be gone!” would replace “publish or perish” as a maxim for research publications. To an extent, Hunt’s vision has been fulfilled by Google Scholar, with the exception that the repositories are still mostly being screened by journal publishers engaged utilizing TPR and often limiting access to paid subscribers. The system is still broken.
But we live in an age that Hunt could only imagine, and it is time to update his vision to match what we know is possible based on our experiences with crowdsourcing, consumer websites, and advances in the peer-review process that have been pioneered in other fields. For that reason, we suggest that Hunt’s model could be updated utilizing best practices in post-publication peer review (PPPR). PPPR already exists within I-O in a limited form, in that articles from our journals can be reviewed, questioned, and discussed on forums such as PubPeer. However, reviews on such forums are determined by the interests of particular readers, tend to focus on errors and miscalculations, and are not readily linked to the publications themselves.
Our proposal would echo Hunt’s vision of a central repository, but one that accepted that it would simply not be feasible to allow everyone to upload whatever they wished onto the website. Instead, published articles would be identified by members of an editorial board, triggered by high citation rates, or self-nominated for review by the authors themselves. Articles would then be tagged with key words and rated on dimensions such as methodological rigor, contribution, practical value, and so on. Automated statistical checks of the accuracy of reported results could supplement the human evaluations (see Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016). Reviews could be added and discussed openly with authors, with the option of making corrections and clarifications if necessary.
The technology to do so already exists and has been demonstrated to work. For instance, in the of field biology, the Faculty of 1000 (www.facultyof1000.com) provides a widely used forum for top researchers to engage in post-publication peer review and to help select, sort, and evaluate the overwhelming numbers of articles being generated that field (Wets, Weedon, & Velterop, Reference Wets, Weedon and Velterop2003). Combined with the resources available on an already-well-used platform such as the Open Science Framework (www.osf.io), it is possible to have data, syntax, the article itself, and the reviews all in the same place. In other words, it is open science in its most complete and robust form (Teixeira da Silva & Dobranszki, Reference Teixeira da Silva and Dobranszki2015).
The potential benefits of embracing PPPR are many (see Teixeira da Silva, Reference Teixeira da Silva2015). It helps to place the focus on individual articles and their contributions rather than on a limited number of prestigious journals. This also serves as an effective way of ensuring that members of that community are able to track major innovations in their field. Plus, studies have demonstrated that PPPR is better at identifying valuable contributions than TPR (Herron, Reference Herron2012). PPPR ratings could be used to inform hiring, tenure, and promotion decisions as well as evaluations by grant agencies. Further, it could provide a path for practitioners who rarely publish themselves to get more involved in the research evaluation processes and allow them to indicate what issues are of most importance in the working world (Paterson & Harms, Reference Paterson and Harms2019; Teixeira da Silva & Dobranszki, Reference Teixeira da Silva and Dobranszki2015). PPPR ratings could also be useful for garnering media attention and allowing the field to demonstrate its value and commitment to robust research outside the immediate academic discipline. Finally, PPPR serves as a second, sober look at published articles and can be used to identify and correct mistakes when they are made (Teixeira da Silva, Al-Khatib, & Dobranszki, Reference Teixeira da Silva, Al-Khatib and Dobranszki2017). This is particularly important for those of us in the organizational sciences. Recent assessments have found that the base rates of misreported results in our top journals are substantially higher than other disciplines of psychology (Nuijten et al., Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016; see also Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017; Credé & Harms, Reference Credé and Harms2019). And even when reported accurately, many previously reported results may no longer be supported when utilizing more advanced statistical techniques (Credé & Harms, Reference Credé and Harms2015). Because the applied nature of our work means that such mistakes are more likely to impact the lives of others, we need to maintain vigilance. PPPR can serve this function.
In terms of creating incentives for participation in the PPPR process, we expect that a number of individuals would participate out of a sheer love for the discipline they work in, but extrinsic rewards could be used as well. Reviewers could be rewarded with points or badges for service, and quality of reviews could be ensured by having the reviews themselves rated by other readers. These ratings themselves could be used to inform hiring, promotion, and tenure decisions. To this end, we would expect PPPR to attract high quality, high effort reviews (Teixeira da Silva et al., Reference Teixeira da Silva, Al-Khatib and Dobranszki2017). It has also been suggested that institutions and departments could encourage participation by taking journal club discussions that are already being done and then posting the comments on a collective institutional account. This could potentially serve as valuable information about the quality and culture of graduate programs for potential applicants.
We acknowledge that a switch to a PPPR-based review culture would be difficult, but the change does need to be immediate. For now, TPR serves a useful purpose and might still have a place in the publication process in the future. But we believe it is clear that PPPR can serve as a useful supplement to TPR (Ware, Reference Ware2008). We also anticipate (and have seen) strong pushback from journal editors who wish to exercise control over the academic discourse (e.g., Fiske, Reference Fiske2016; Wright, Reference Wright2016). This is to be expected. Change is difficult, and criticisms leveled against systems you create or control might be perceived as threatening. Some will also argue that open peer reviews will unnecessarily expose junior scholars to potential retaliation (Bastion, Reference Bastion2014). Although this is possible, other disciplines have implemented PPPR without such consequences. Rather, open discussion has spurred greater transparency and civility. As we have seen with the open science movement, radical change is sometimes needed to fix what is broken and to create a more robust and rigorous science. We believe PPPR is a necessary step in that direction.
You have
Access
One of the essential elements of a rigorous and robust science is that it has the ability to self-correct (Harms, Credé, & DeSimone, Reference Harms, Credé and DeSimone2018). The focal article by Köhler et al. (Reference Köhler, Gonzales-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) has a great deal to say about how to improve the review process, but ultimately we feel that their suggestions fall short. Peer review may be the “cornerstone” of the profession, but in its current form, that cornerstone is cracked. In the words of Drummond Rennie, editor of the Journal of the American Medical Association, “If peer review was a drug it would never be allowed onto the market” (Smith, Reference Wright2010). Köhler et al. acknowledge many of the failings of the current review process, that it can be biased, unreliable, overly negative and critical, unnecessarily time-consuming, and blocks ingenuity, and that it gives editors and reviewers too much power to shape messages to their fit their own agendas (see also Hunter, Reference Hunter2012). However, the thrust of their article is try to offer patches to a broken model. They are not the first to do so, nor are they likely to be the last. The problems with traditional peer review (TPR) seem intractable. For this reason, we suggest that a critical paradigm shift is needed within industrial and organizational (I-O) psychology to bring reviewing into the 21st century.
Twenty-five years ago, with the rise of the world wide web, Earl Hunt made a radical suggestion that the nature of publishing would and should change to embrace technology, reduce journal clutter, and direct attention toward important articles rather than a limited set of high-profile journals (Hunt, Reference Hunt1995). Hunt argued that articles could be uploaded to a central repository, which would be searchable based on key words and then ranked based on readership. “Be read or be gone!” would replace “publish or perish” as a maxim for research publications. To an extent, Hunt’s vision has been fulfilled by Google Scholar, with the exception that the repositories are still mostly being screened by journal publishers engaged utilizing TPR and often limiting access to paid subscribers. The system is still broken.
But we live in an age that Hunt could only imagine, and it is time to update his vision to match what we know is possible based on our experiences with crowdsourcing, consumer websites, and advances in the peer-review process that have been pioneered in other fields. For that reason, we suggest that Hunt’s model could be updated utilizing best practices in post-publication peer review (PPPR). PPPR already exists within I-O in a limited form, in that articles from our journals can be reviewed, questioned, and discussed on forums such as PubPeer. However, reviews on such forums are determined by the interests of particular readers, tend to focus on errors and miscalculations, and are not readily linked to the publications themselves.
Our proposal would echo Hunt’s vision of a central repository, but one that accepted that it would simply not be feasible to allow everyone to upload whatever they wished onto the website. Instead, published articles would be identified by members of an editorial board, triggered by high citation rates, or self-nominated for review by the authors themselves. Articles would then be tagged with key words and rated on dimensions such as methodological rigor, contribution, practical value, and so on. Automated statistical checks of the accuracy of reported results could supplement the human evaluations (see Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016). Reviews could be added and discussed openly with authors, with the option of making corrections and clarifications if necessary.
The technology to do so already exists and has been demonstrated to work. For instance, in the of field biology, the Faculty of 1000 (www.facultyof1000.com) provides a widely used forum for top researchers to engage in post-publication peer review and to help select, sort, and evaluate the overwhelming numbers of articles being generated that field (Wets, Weedon, & Velterop, Reference Wets, Weedon and Velterop2003). Combined with the resources available on an already-well-used platform such as the Open Science Framework (www.osf.io), it is possible to have data, syntax, the article itself, and the reviews all in the same place. In other words, it is open science in its most complete and robust form (Teixeira da Silva & Dobranszki, Reference Teixeira da Silva and Dobranszki2015).
The potential benefits of embracing PPPR are many (see Teixeira da Silva, Reference Teixeira da Silva2015). It helps to place the focus on individual articles and their contributions rather than on a limited number of prestigious journals. This also serves as an effective way of ensuring that members of that community are able to track major innovations in their field. Plus, studies have demonstrated that PPPR is better at identifying valuable contributions than TPR (Herron, Reference Herron2012). PPPR ratings could be used to inform hiring, tenure, and promotion decisions as well as evaluations by grant agencies. Further, it could provide a path for practitioners who rarely publish themselves to get more involved in the research evaluation processes and allow them to indicate what issues are of most importance in the working world (Paterson & Harms, Reference Paterson and Harms2019; Teixeira da Silva & Dobranszki, Reference Teixeira da Silva and Dobranszki2015). PPPR ratings could also be useful for garnering media attention and allowing the field to demonstrate its value and commitment to robust research outside the immediate academic discipline. Finally, PPPR serves as a second, sober look at published articles and can be used to identify and correct mistakes when they are made (Teixeira da Silva, Al-Khatib, & Dobranszki, Reference Teixeira da Silva, Al-Khatib and Dobranszki2017). This is particularly important for those of us in the organizational sciences. Recent assessments have found that the base rates of misreported results in our top journals are substantially higher than other disciplines of psychology (Nuijten et al., Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016; see also Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017; Credé & Harms, Reference Credé and Harms2019). And even when reported accurately, many previously reported results may no longer be supported when utilizing more advanced statistical techniques (Credé & Harms, Reference Credé and Harms2015). Because the applied nature of our work means that such mistakes are more likely to impact the lives of others, we need to maintain vigilance. PPPR can serve this function.
In terms of creating incentives for participation in the PPPR process, we expect that a number of individuals would participate out of a sheer love for the discipline they work in, but extrinsic rewards could be used as well. Reviewers could be rewarded with points or badges for service, and quality of reviews could be ensured by having the reviews themselves rated by other readers. These ratings themselves could be used to inform hiring, promotion, and tenure decisions. To this end, we would expect PPPR to attract high quality, high effort reviews (Teixeira da Silva et al., Reference Teixeira da Silva, Al-Khatib and Dobranszki2017). It has also been suggested that institutions and departments could encourage participation by taking journal club discussions that are already being done and then posting the comments on a collective institutional account. This could potentially serve as valuable information about the quality and culture of graduate programs for potential applicants.
We acknowledge that a switch to a PPPR-based review culture would be difficult, but the change does need to be immediate. For now, TPR serves a useful purpose and might still have a place in the publication process in the future. But we believe it is clear that PPPR can serve as a useful supplement to TPR (Ware, Reference Ware2008). We also anticipate (and have seen) strong pushback from journal editors who wish to exercise control over the academic discourse (e.g., Fiske, Reference Fiske2016; Wright, Reference Wright2016). This is to be expected. Change is difficult, and criticisms leveled against systems you create or control might be perceived as threatening. Some will also argue that open peer reviews will unnecessarily expose junior scholars to potential retaliation (Bastion, Reference Bastion2014). Although this is possible, other disciplines have implemented PPPR without such consequences. Rather, open discussion has spurred greater transparency and civility. As we have seen with the open science movement, radical change is sometimes needed to fix what is broken and to create a more robust and rigorous science. We believe PPPR is a necessary step in that direction.