One of the key characteristics of a rigorous and robust science is that it has the ability to self-correct when mistakes are made. The focal article by Grand et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) has plenty to say about what faithful actors can do to ensure a more robust industrial and organizational (I-O) psychology literature, but we were surprised at how little attention the issue of correcting the research record was given. To be clear, the target article mentions the issue of research misconduct and even cites an article from Leadership Quarterly (Atwater, Mumford, Schriesheim, & Yammarino, Reference Atwater, Mumford, Schriesheim and Yammarino2014) that deals specifically with the issue, but it does not include guidance on this issue for editors or publishers. This is not surprising. The issue of misconduct is one that produces a great deal of discomfort in academic circles (Wager, Reference Wager2015), and when it is discussed, it is usually only in the vaguest terms (e.g., Banks et al., Reference Banks, O'Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016). Moreover, as Grand and colleagues point out, it could be argued that the “gathering storm” of questionable research that has enveloped other fields such as social psychology seems distant. Perhaps to reassure us, the authors point to several articles showing that results of our field seem to be replicable, robust, and relevant. But are they?
There are many reasons to believe that research in I-O psychology is not as robust as it seems. A recent comprehensive review of the correspondence of effects found between laboratory and field studies (Vanhove & Harms, Reference Vanhove and Harms2015) reported a much lower estimate of relationship (r = .61) than the earlier estimate suggested by Mitchell (Reference Mitchell2012) and reported in the focal article. A review of gross errors and inconsistencies in reporting p-values found that not only did Journal of Applied Psychology have the highest base rates of problematic results, but that the frequency of these problems was approximately twice that of Journal of Personality and Social Psychology (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016). Further, a recent review of structural equation model (SEM) testing in the organizational sciences found that a substantial number of studies drew incorrect or unsupported conclusions from their data, and some even reported mathematically impossible findings (Credé & Harms, Reference Credé and Harms2015). Other reviews have found discrepancies in reported degrees of freedom in 38% of studies in Academy of Management Journal and Journal of Applied Psychology (Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017). Our own review of other journals such as Journal of Management, Journal of Organizational Behavior, and Personnel Psychology found even higher base rates (56.6%) of missing degrees of freedom in SEM models, and a substantial number (36%) reported root mean square error of approximation (RMSEA) values that were incompatible with the reported chi-square, degrees of freedom, and sample size. Rather than patting ourselves on the back for a job well done, we should be panicking.
The number of errors and discrepancies in the published record suggests that the peer-review process is failing our scientific discipline. This is, of course, not surprising. Editors and reviewers are overworked, reviewers are not compensated for the accuracy of their work, and both editors and reviewers are often being asked to review work characterized by a statistical complexity that is difficult to evaluate for all but the most quantitatively inclined among us. Similarly, authors may feel compelled to perform statistical analyses that they are not well equipped to perform in order to impress reviewers and readers with their level of sophistication. It is therefore not surprising that so many errors slip through the peer review process.
In light of the failure of the status quo review process, it becomes critical that the collective expertise of the I-O psychology and management community be harnessed to ensure the integrity of our scientific output and general discipline. This can be partially achieved by asking authors to publish all research materials and data alongside the article when possible. When this would be difficult (e.g., due to the use of proprietary data, in cases of ongoing data collection or analysis), authors should be willing to make data available to other researchers for the purposes of replication and/or quality checking. In doing so, we recommend that authors conform to the standards for data sharing outlined in section 1.08 of the American Psychological Association (APA) manual (APA, 2010), which includes provisions for ensuring that authors and requestors agree on the acceptable uses for the shared data.
It is similarly important for journals or publishers to allow, encourage, and facilitate a robust system for online postpublication review, whereby errors in analyses or interpretation that are detected by readers of a journal can be corrected or at least disseminated to other readers of the journal. Some may fear that such a system would result in a hostile research environment in which researchers snipe at each other via postpublication review. We do not share this concern because it should be relatively easy to exert some editorial discretion over the postpublication review process to ensure a basic level of civility. Journals could also opt to make the names of authors of postpublication review comments public so that a greater incentive for the accuracy of such review comments is assured. By linking postpublication review comments directly to published articles (rather than publishing them as stand-alone articles), it would also be possible to provide some credit to the authors of postpublication reviews without negatively affecting the impact factor of journals.
Some editors have begun expressing some concerns about the integrity of our field and how the failure to correct errors may undermine the public's faith in our work (e.g., Harley, Faems, & Corbett, Reference Harley, Faems and Corbett2014). Others remain reticent on the issue of corrective actions and even somewhat dismissive as to whether a problem exists. For example, in an interview with Retractionwatch.com, one prominent editor of a management journal seemed to suggest that the reputation of scholars should be prioritized over that of the scientific record because errors in the organizational research literature have less severe consequences than those in medical research. We feel that this sends a potentially dangerous signal to both new and established researchers regarding the attitudes of the field's gatekeepers toward research misconduct and the value of our research in general.
I-O psychologists and those in the related organizational behavior (OB)/human resources (HR) fields often ponder why practitioners do not read our research or implement our findings (e.g., Rynes, Colbert, & Brown, Reference Rynes, Colbert and Brown2002). But should they? The evidence seems to suggest that a substantial amount of our research contains errors and that the editorial and review process that is intended to ensure research quality often turns a blind eye to this issue. We fear that unless we take corrective actions, the “gathering storm” threatens to sweep us all away as practitioners increasingly come to see our work as “fake news” and stop listening to us entirely.
Fortunately, the publication process contains mechanisms by which researchers can acknowledge and correct mistakes. Unfortunately, we suspect that publication of corrigenda and retractions is viewed as a career death knell by many researchers, selection committees, and promotion and tenure committees. Correcting research mistakes need not be as scary or threatening as we make it out to be. Although it is certainly undesirable for a researcher to retract a large proportion of his or her CV, isolated corrections of honest mistakes should be celebrated as signs of a healthy, transparent, and robust field. Accordingly, it is important that we ensure that the punishment fits the crime. Self-correction of a minor reporting mistake should not carry the same stigma as article retraction due to questionable research practices such as data fabrication. There is even evidence that demonstrating a willingness to self-correct and even self-retract when one makes a mistake actually enhances the reputation of the researcher in question (Lu, Jin, Uzzi, & Jones, Reference Lu, Jin, Uzzi and Jones2013). As the focal article noted, a robust science needs to have a higher tolerance for honest mistakes. It is difficult for science to self-correct when egos and reputations are considered more important than the quality and trustworthiness of research. But a more robust I-O psychology will require us to put such issues aside. Changing perceptions about the undesirability of corrigenda and retractions that result from misinterpretations of data will take time but can be facilitated by explicit actions taken by gatekeepers and norm setters. For example, journal editors could craft policies and editorial statements that encourage authors to correct errors when they are identified, whereas professional associations and conferences could organize panels in which editors and researchers affected by corrections and retractions discuss the process and the (hopefully minor) impact on their careers and research programs.
One of the key characteristics of a rigorous and robust science is that it has the ability to self-correct when mistakes are made. The focal article by Grand et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) has plenty to say about what faithful actors can do to ensure a more robust industrial and organizational (I-O) psychology literature, but we were surprised at how little attention the issue of correcting the research record was given. To be clear, the target article mentions the issue of research misconduct and even cites an article from Leadership Quarterly (Atwater, Mumford, Schriesheim, & Yammarino, Reference Atwater, Mumford, Schriesheim and Yammarino2014) that deals specifically with the issue, but it does not include guidance on this issue for editors or publishers. This is not surprising. The issue of misconduct is one that produces a great deal of discomfort in academic circles (Wager, Reference Wager2015), and when it is discussed, it is usually only in the vaguest terms (e.g., Banks et al., Reference Banks, O'Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016). Moreover, as Grand and colleagues point out, it could be argued that the “gathering storm” of questionable research that has enveloped other fields such as social psychology seems distant. Perhaps to reassure us, the authors point to several articles showing that results of our field seem to be replicable, robust, and relevant. But are they?
There are many reasons to believe that research in I-O psychology is not as robust as it seems. A recent comprehensive review of the correspondence of effects found between laboratory and field studies (Vanhove & Harms, Reference Vanhove and Harms2015) reported a much lower estimate of relationship (r = .61) than the earlier estimate suggested by Mitchell (Reference Mitchell2012) and reported in the focal article. A review of gross errors and inconsistencies in reporting p-values found that not only did Journal of Applied Psychology have the highest base rates of problematic results, but that the frequency of these problems was approximately twice that of Journal of Personality and Social Psychology (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, Reference Nuijten, Hartgerink, van Assen, Epskamp and Wicherts2016). Further, a recent review of structural equation model (SEM) testing in the organizational sciences found that a substantial number of studies drew incorrect or unsupported conclusions from their data, and some even reported mathematically impossible findings (Credé & Harms, Reference Credé and Harms2015). Other reviews have found discrepancies in reported degrees of freedom in 38% of studies in Academy of Management Journal and Journal of Applied Psychology (Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017). Our own review of other journals such as Journal of Management, Journal of Organizational Behavior, and Personnel Psychology found even higher base rates (56.6%) of missing degrees of freedom in SEM models, and a substantial number (36%) reported root mean square error of approximation (RMSEA) values that were incompatible with the reported chi-square, degrees of freedom, and sample size. Rather than patting ourselves on the back for a job well done, we should be panicking.
The number of errors and discrepancies in the published record suggests that the peer-review process is failing our scientific discipline. This is, of course, not surprising. Editors and reviewers are overworked, reviewers are not compensated for the accuracy of their work, and both editors and reviewers are often being asked to review work characterized by a statistical complexity that is difficult to evaluate for all but the most quantitatively inclined among us. Similarly, authors may feel compelled to perform statistical analyses that they are not well equipped to perform in order to impress reviewers and readers with their level of sophistication. It is therefore not surprising that so many errors slip through the peer review process.
In light of the failure of the status quo review process, it becomes critical that the collective expertise of the I-O psychology and management community be harnessed to ensure the integrity of our scientific output and general discipline. This can be partially achieved by asking authors to publish all research materials and data alongside the article when possible. When this would be difficult (e.g., due to the use of proprietary data, in cases of ongoing data collection or analysis), authors should be willing to make data available to other researchers for the purposes of replication and/or quality checking. In doing so, we recommend that authors conform to the standards for data sharing outlined in section 1.08 of the American Psychological Association (APA) manual (APA, 2010), which includes provisions for ensuring that authors and requestors agree on the acceptable uses for the shared data.
It is similarly important for journals or publishers to allow, encourage, and facilitate a robust system for online postpublication review, whereby errors in analyses or interpretation that are detected by readers of a journal can be corrected or at least disseminated to other readers of the journal. Some may fear that such a system would result in a hostile research environment in which researchers snipe at each other via postpublication review. We do not share this concern because it should be relatively easy to exert some editorial discretion over the postpublication review process to ensure a basic level of civility. Journals could also opt to make the names of authors of postpublication review comments public so that a greater incentive for the accuracy of such review comments is assured. By linking postpublication review comments directly to published articles (rather than publishing them as stand-alone articles), it would also be possible to provide some credit to the authors of postpublication reviews without negatively affecting the impact factor of journals.
Some editors have begun expressing some concerns about the integrity of our field and how the failure to correct errors may undermine the public's faith in our work (e.g., Harley, Faems, & Corbett, Reference Harley, Faems and Corbett2014). Others remain reticent on the issue of corrective actions and even somewhat dismissive as to whether a problem exists. For example, in an interview with Retractionwatch.com, one prominent editor of a management journal seemed to suggest that the reputation of scholars should be prioritized over that of the scientific record because errors in the organizational research literature have less severe consequences than those in medical research. We feel that this sends a potentially dangerous signal to both new and established researchers regarding the attitudes of the field's gatekeepers toward research misconduct and the value of our research in general.
I-O psychologists and those in the related organizational behavior (OB)/human resources (HR) fields often ponder why practitioners do not read our research or implement our findings (e.g., Rynes, Colbert, & Brown, Reference Rynes, Colbert and Brown2002). But should they? The evidence seems to suggest that a substantial amount of our research contains errors and that the editorial and review process that is intended to ensure research quality often turns a blind eye to this issue. We fear that unless we take corrective actions, the “gathering storm” threatens to sweep us all away as practitioners increasingly come to see our work as “fake news” and stop listening to us entirely.
Fortunately, the publication process contains mechanisms by which researchers can acknowledge and correct mistakes. Unfortunately, we suspect that publication of corrigenda and retractions is viewed as a career death knell by many researchers, selection committees, and promotion and tenure committees. Correcting research mistakes need not be as scary or threatening as we make it out to be. Although it is certainly undesirable for a researcher to retract a large proportion of his or her CV, isolated corrections of honest mistakes should be celebrated as signs of a healthy, transparent, and robust field. Accordingly, it is important that we ensure that the punishment fits the crime. Self-correction of a minor reporting mistake should not carry the same stigma as article retraction due to questionable research practices such as data fabrication. There is even evidence that demonstrating a willingness to self-correct and even self-retract when one makes a mistake actually enhances the reputation of the researcher in question (Lu, Jin, Uzzi, & Jones, Reference Lu, Jin, Uzzi and Jones2013). As the focal article noted, a robust science needs to have a higher tolerance for honest mistakes. It is difficult for science to self-correct when egos and reputations are considered more important than the quality and trustworthiness of research. But a more robust I-O psychology will require us to put such issues aside. Changing perceptions about the undesirability of corrigenda and retractions that result from misinterpretations of data will take time but can be facilitated by explicit actions taken by gatekeepers and norm setters. For example, journal editors could craft policies and editorial statements that encourage authors to correct errors when they are identified, whereas professional associations and conferences could organize panels in which editors and researchers affected by corrections and retractions discuss the process and the (hopefully minor) impact on their careers and research programs.