The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) develops a highly compelling competency framework for excellent reviewing. It is certainly a key piece of a robust and open science ecosystem (Banks et al., Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019). The framework can and should be used as a guide for novice and seasoned reviewers “in defining what they should pay attention to, upon what their review should touch, how they should word their review, what kind of advice they need to provide, and how they should overall go about peer reviewing” (Köhler et al., Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020, p. 17). Creating specific reviewer training around the competencies takes this framework to the next level by actively teaching reviewers how to implement the proposed guidelines. Developing these skills is a critical component for bettering the publication process by enabling reviewers to provide feedback to colleagues that is more organized, reliable, and helpful.
The focal article, by design, has an individual focus. Thus, this commentary will serve as an extension to the focal article, pushing the call for better reviewing further by making the case for results-blind reviews (RBR) as an environmental mechanism that better supports the framework and training of competencies discussed in the focal article.
A conducive environment
Literature on transfer-of-training explores under what conditions knowledge and skills learned in training are applied in action (Baldwin & Ford, Reference Baldwin and Ford1988). This area of research emphasizes the importance of situational cues and environmental factors, noting that these largely determine whether or not learned competencies are actually applied in the workplace (Blume, Ford, Baldwin, & Huang, Reference Blume, Ford, Baldwin and Huang2010; Colquitt, Lepine, & Noe, Reference Colquitt, Lepine and Noe2000). It is our contention that the training of competencies alone may not be enough for successful transfer of reviewing skills as situation/context matters to what behaviors are exhibited by individuals (Reference Dalal, Alaybek, Sheng, Holland and TomassettiDalal, Alaybek, Sheng, Holland, & Tomassetti, in press). Furthermore, the training focuses on explicit behaviors of reviewers, but does not target the implicit or unconscious biases that may exist in the system itself.
Bias in the dominant peer-review system
The current dominant peer review system in the fields of industrial and organizational psychology (I-O), organizational behavior (OB), and human resources (HR) requires authors to submit full manuscripts, including results, to a journal. There is good reason to believe that manuscript results can bias reviewers’ judgments toward the rest of the article. This, in turn, can undermine the efficacy of the focal article’s framework.
Research suggests that favoritism toward significant and novel findings in the organizational sciences is widespread (Antonakis, Reference Antonakis2017; Murphy & Aguinis, Reference Murphy and Aguinis2019; O’Boyle, Banks, Carter, Walter, & Yuan, Reference O’Boyle, Banks, Carter, Walter and Yuan2019; O’Boyle, Banks, & Gonzalez-Mulé, Reference O’Boyle, Banks and Gonzalez-Mulé2017), and null findings are typically undervalued (Coursol & Wagner, Reference Coursol and Wagner1986; Emerson, Warme, Wolf, Heckman, Brand, & Leopold, Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010). Emerson et al. (Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010) illustrated this bias, having conducted a randomized experiment where two versions of a fabricated manuscript were submitted to peer review at two different journals. The manuscripts were identical, with the only difference being that one version reported positive, statistically significant results while the other version reported no significant differences (aka, the null-outcome version). The authors also purposely placed five identical errors throughout the two versions. Reviewers at both journals favored the positive-outcome manuscript version, more frequently recommending it for publication than the null-outcome version (97% vs. 80%). Moreover, reviewers detected more of the errors in the null-outcome version than the positive-outcome manuscript. Furthermore, the null-outcome version received lower scores for methodological validity than the positive-outcome manuscript. These results suggest that reviewers often interpret statistically significant findings as a consequence of methodological rigor and are more likely to recommend positive-outcome manuscripts for publication (Emerson et al., Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010). This ultimately contributes to a publication bias in the literature, where knowledge is not entirely representative of research.
As the focal article suggests, training reviewers to look past the significance of findings and focus solely on the rigor of the research design could potentially reduce this bias. However, is a cautionary warning truly enough to eliminate implicit bias? It is our contention that blinding the research results of a manuscript under review will invariably reduce the bias caused by significant, positive, and novel results—as the results are not shown prior to the decision. This is akin, for example, to what was found in the sex-based hiring study conducted by Goldin and Rouse (Reference Goldin and Rouse2000). The researchers investigated systematic bias in the selection of symphony orchestras in the United States by implementing blind auditions, concealing candidates’ identities from the orchestra’s selection committee. The authors demonstrated that the blind audition reduced sex-based implicit biases in hiring decisions, increasing the probability of a women being selected to the orchestra. Blind auditioning promoted impartiality in the committees’ hiring decisions, increasing the proportion of women in the symphony orchestra by roughly 50%. Other examples of studies like this exist across the selection, performance appraisal, and decision-making literature that strengthen the case for changing the system to yield better and more accurate outcomes.
Results blind reviews
There are two stages in RBR. In Stage 1 of RBR, researchers send in a manuscript including an introduction, theoretical/conceptual rationale, relevant literature review, hypotheses, detailed methodology, and plan for analyses (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018). The actual results do not matter since compelling questions answered in appropriate ways is what should drive what is published, not the actual findings. On the basis of these criteria, reviewers may reject, invite for further revision, or recommend an in-principle acceptance to the researchers. An in-principle acceptance is a formal agreement stating that the research will be published as long as the procedures are followed and the proposed/agreed upon analyses are conducted as described. Overall, RBR is intended to alter the lens through which researchers and reviewers approach and evaluate research (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018). Assuming the researchers fulfilled the promises indicated in their manuscript at Stage 1, their article would be accepted—regardless of supported hypotheses or significant findings.
In Stage 2, only upon receiving an in-principle acceptance, the researchers submit the full manuscript, including the results and discussion. The manuscript is assigned to the same reviewers from Stage 1. Assuming there are no unjustifiable deviations from the original proposal and all analyses are handled competently, the manuscript is accepted and published. Reviewers must focus on the quality and rationale of the study, evaluating on the basis of the appropriateness of research questions and hypotheses, the practical or theoretical importance, and the rigor of the methodology without knowledge of (or being biased by) results. It is important to note that RBR processes are highly akin to how we train our graduate students. Namely, students create thesis and dissertation proposals. The theoretical rational, research questions/hypotheses, and proposed methodology are highly scrutinized. But, once accepted by the committee (similar to an in-principle acceptance), the students’ passing of these milestones is not actually affected by what is found in the research.
There is another potential benefit of the RBR process. It could lead to authors being more transparent and honest in their reporting of results. The preference for significant results may incentivize researchers to engage in questionable research practices (QRPs), such as presenting post hoc findings as a priori, HARKing, selectively reporting fit indices, and strategically deleting outliers for the purpose of reaching statistical significance (Banks, Rogelberg, Woznyj, Landis, & Rupp, Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016). The QRPs mentioned above lead to a publication bias (i.e., file drawer problem) in academic journals, potentially hindering the advancement of knowledge by only displaying supportive findings or meaningful outcomes. The RBR process generally eliminates implicit or explicit incentives for researchers to engage in QRPs, instead encouraging scholars to submit manuscripts regardless of whether they found significant results (Button et al., Reference Button, Bal, Clark and Shipley2016).
Woznyj, Grenier, Ross, Banks, and Rogelberg (Reference Woznyj, Grenier, Ross, Banks and Rogelberg2018) explored the field’s reactions (including practitioners) to the RBR initiative and found that a majority of stakeholders in the organizational sciences (roughly 75%) believes the field would benefit from blind reviewing. Seventy-two percent of stakeholders believe the initiative would help advance cumulative knowledge, while 65% believe RBR would combat against questionable research practices. In regard to RBR benefiting the peer-review process, about one-fourth (22%) of stakeholders indicated blind results would shift how reviewers evaluate research. One respondent stated that the initiative would “focus reviewer attention on theory and the testing of theory rather than on statistical support,” while another respondent noted “reviewers and editors would probably focus a lot more on the quality of the arguments and hypothesis development than they currently do” (Woznyj et al., Reference Woznyj, Grenier, Ross, Banks and Rogelberg2018, p. 567).
With this all said, it is important to acknowledge some potential unknowns with the RBR process that warrant further examination. First, will reviewers, both academics and practitioners, like to review articles in this fashion? It is our contention that they will for a couple of reasons: (1) The Stage 1 manuscript is actually shorter than a full manuscript, thus it should take less time to read and review; and (2) RBR processes are basically identical to what is done for a thesis and dissertation process in graduate school. For example, if students effectively carry out their proposal (akin to a Stage 1 manuscript), they are given an in-principal acceptance—they just need to deliver on what was approved during the initial proposal meeting. Thus, the RBR process in many regards should feel comfortable and familiar. Second, does the RBR process require a longer reviewer commitment? This is certainly worthy of examination. We do not believe it will. Namely, the RBR process just changes the rhythms of reviewing. If a Stage 1 manuscript is given an in-principal acceptance, reviewer time with the Stage 2 manuscript should be absolutely minimal, as reviewers are just confirming the article was carried out as described in the Stage 1 manuscript and that the discussion section is appropriate and well-done. Typically, the Stage 2 review processes should be completed in one or, at most, two rounds.
Conclusion
The focal article develops a much-needed competency framework for training reviewers, offering concrete advice for how we can improve our reviewing skills. Training reviewers via the framework is a great step toward solving some of the issues with the peer-review process in the organizational sciences. This commentary extends the focal argument by suggesting a complementary solution, results-blind review, that addresses the scientific ecosystem at large. Using RBR, reviewers can more easily and successfully enact the competencies proposed in the focal article, ultimately advancing our science and improving the published research that represents our field.
You have
Access
The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) develops a highly compelling competency framework for excellent reviewing. It is certainly a key piece of a robust and open science ecosystem (Banks et al., Reference Banks, Field, Oswald, O’Boyle, Landis, Rupp and Rogelberg2019). The framework can and should be used as a guide for novice and seasoned reviewers “in defining what they should pay attention to, upon what their review should touch, how they should word their review, what kind of advice they need to provide, and how they should overall go about peer reviewing” (Köhler et al., Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020, p. 17). Creating specific reviewer training around the competencies takes this framework to the next level by actively teaching reviewers how to implement the proposed guidelines. Developing these skills is a critical component for bettering the publication process by enabling reviewers to provide feedback to colleagues that is more organized, reliable, and helpful.
The focal article, by design, has an individual focus. Thus, this commentary will serve as an extension to the focal article, pushing the call for better reviewing further by making the case for results-blind reviews (RBR) as an environmental mechanism that better supports the framework and training of competencies discussed in the focal article.
A conducive environment
Literature on transfer-of-training explores under what conditions knowledge and skills learned in training are applied in action (Baldwin & Ford, Reference Baldwin and Ford1988). This area of research emphasizes the importance of situational cues and environmental factors, noting that these largely determine whether or not learned competencies are actually applied in the workplace (Blume, Ford, Baldwin, & Huang, Reference Blume, Ford, Baldwin and Huang2010; Colquitt, Lepine, & Noe, Reference Colquitt, Lepine and Noe2000). It is our contention that the training of competencies alone may not be enough for successful transfer of reviewing skills as situation/context matters to what behaviors are exhibited by individuals (Reference Dalal, Alaybek, Sheng, Holland and TomassettiDalal, Alaybek, Sheng, Holland, & Tomassetti, in press). Furthermore, the training focuses on explicit behaviors of reviewers, but does not target the implicit or unconscious biases that may exist in the system itself.
Bias in the dominant peer-review system
The current dominant peer review system in the fields of industrial and organizational psychology (I-O), organizational behavior (OB), and human resources (HR) requires authors to submit full manuscripts, including results, to a journal. There is good reason to believe that manuscript results can bias reviewers’ judgments toward the rest of the article. This, in turn, can undermine the efficacy of the focal article’s framework.
Research suggests that favoritism toward significant and novel findings in the organizational sciences is widespread (Antonakis, Reference Antonakis2017; Murphy & Aguinis, Reference Murphy and Aguinis2019; O’Boyle, Banks, Carter, Walter, & Yuan, Reference O’Boyle, Banks, Carter, Walter and Yuan2019; O’Boyle, Banks, & Gonzalez-Mulé, Reference O’Boyle, Banks and Gonzalez-Mulé2017), and null findings are typically undervalued (Coursol & Wagner, Reference Coursol and Wagner1986; Emerson, Warme, Wolf, Heckman, Brand, & Leopold, Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010). Emerson et al. (Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010) illustrated this bias, having conducted a randomized experiment where two versions of a fabricated manuscript were submitted to peer review at two different journals. The manuscripts were identical, with the only difference being that one version reported positive, statistically significant results while the other version reported no significant differences (aka, the null-outcome version). The authors also purposely placed five identical errors throughout the two versions. Reviewers at both journals favored the positive-outcome manuscript version, more frequently recommending it for publication than the null-outcome version (97% vs. 80%). Moreover, reviewers detected more of the errors in the null-outcome version than the positive-outcome manuscript. Furthermore, the null-outcome version received lower scores for methodological validity than the positive-outcome manuscript. These results suggest that reviewers often interpret statistically significant findings as a consequence of methodological rigor and are more likely to recommend positive-outcome manuscripts for publication (Emerson et al., Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010). This ultimately contributes to a publication bias in the literature, where knowledge is not entirely representative of research.
As the focal article suggests, training reviewers to look past the significance of findings and focus solely on the rigor of the research design could potentially reduce this bias. However, is a cautionary warning truly enough to eliminate implicit bias? It is our contention that blinding the research results of a manuscript under review will invariably reduce the bias caused by significant, positive, and novel results—as the results are not shown prior to the decision. This is akin, for example, to what was found in the sex-based hiring study conducted by Goldin and Rouse (Reference Goldin and Rouse2000). The researchers investigated systematic bias in the selection of symphony orchestras in the United States by implementing blind auditions, concealing candidates’ identities from the orchestra’s selection committee. The authors demonstrated that the blind audition reduced sex-based implicit biases in hiring decisions, increasing the probability of a women being selected to the orchestra. Blind auditioning promoted impartiality in the committees’ hiring decisions, increasing the proportion of women in the symphony orchestra by roughly 50%. Other examples of studies like this exist across the selection, performance appraisal, and decision-making literature that strengthen the case for changing the system to yield better and more accurate outcomes.
Results blind reviews
There are two stages in RBR. In Stage 1 of RBR, researchers send in a manuscript including an introduction, theoretical/conceptual rationale, relevant literature review, hypotheses, detailed methodology, and plan for analyses (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018). The actual results do not matter since compelling questions answered in appropriate ways is what should drive what is published, not the actual findings. On the basis of these criteria, reviewers may reject, invite for further revision, or recommend an in-principle acceptance to the researchers. An in-principle acceptance is a formal agreement stating that the research will be published as long as the procedures are followed and the proposed/agreed upon analyses are conducted as described. Overall, RBR is intended to alter the lens through which researchers and reviewers approach and evaluate research (Grand et al., Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018). Assuming the researchers fulfilled the promises indicated in their manuscript at Stage 1, their article would be accepted—regardless of supported hypotheses or significant findings.
In Stage 2, only upon receiving an in-principle acceptance, the researchers submit the full manuscript, including the results and discussion. The manuscript is assigned to the same reviewers from Stage 1. Assuming there are no unjustifiable deviations from the original proposal and all analyses are handled competently, the manuscript is accepted and published. Reviewers must focus on the quality and rationale of the study, evaluating on the basis of the appropriateness of research questions and hypotheses, the practical or theoretical importance, and the rigor of the methodology without knowledge of (or being biased by) results. It is important to note that RBR processes are highly akin to how we train our graduate students. Namely, students create thesis and dissertation proposals. The theoretical rational, research questions/hypotheses, and proposed methodology are highly scrutinized. But, once accepted by the committee (similar to an in-principle acceptance), the students’ passing of these milestones is not actually affected by what is found in the research.
There is another potential benefit of the RBR process. It could lead to authors being more transparent and honest in their reporting of results. The preference for significant results may incentivize researchers to engage in questionable research practices (QRPs), such as presenting post hoc findings as a priori, HARKing, selectively reporting fit indices, and strategically deleting outliers for the purpose of reaching statistical significance (Banks, Rogelberg, Woznyj, Landis, & Rupp, Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016). The QRPs mentioned above lead to a publication bias (i.e., file drawer problem) in academic journals, potentially hindering the advancement of knowledge by only displaying supportive findings or meaningful outcomes. The RBR process generally eliminates implicit or explicit incentives for researchers to engage in QRPs, instead encouraging scholars to submit manuscripts regardless of whether they found significant results (Button et al., Reference Button, Bal, Clark and Shipley2016).
Woznyj, Grenier, Ross, Banks, and Rogelberg (Reference Woznyj, Grenier, Ross, Banks and Rogelberg2018) explored the field’s reactions (including practitioners) to the RBR initiative and found that a majority of stakeholders in the organizational sciences (roughly 75%) believes the field would benefit from blind reviewing. Seventy-two percent of stakeholders believe the initiative would help advance cumulative knowledge, while 65% believe RBR would combat against questionable research practices. In regard to RBR benefiting the peer-review process, about one-fourth (22%) of stakeholders indicated blind results would shift how reviewers evaluate research. One respondent stated that the initiative would “focus reviewer attention on theory and the testing of theory rather than on statistical support,” while another respondent noted “reviewers and editors would probably focus a lot more on the quality of the arguments and hypothesis development than they currently do” (Woznyj et al., Reference Woznyj, Grenier, Ross, Banks and Rogelberg2018, p. 567).
With this all said, it is important to acknowledge some potential unknowns with the RBR process that warrant further examination. First, will reviewers, both academics and practitioners, like to review articles in this fashion? It is our contention that they will for a couple of reasons: (1) The Stage 1 manuscript is actually shorter than a full manuscript, thus it should take less time to read and review; and (2) RBR processes are basically identical to what is done for a thesis and dissertation process in graduate school. For example, if students effectively carry out their proposal (akin to a Stage 1 manuscript), they are given an in-principal acceptance—they just need to deliver on what was approved during the initial proposal meeting. Thus, the RBR process in many regards should feel comfortable and familiar. Second, does the RBR process require a longer reviewer commitment? This is certainly worthy of examination. We do not believe it will. Namely, the RBR process just changes the rhythms of reviewing. If a Stage 1 manuscript is given an in-principal acceptance, reviewer time with the Stage 2 manuscript should be absolutely minimal, as reviewers are just confirming the article was carried out as described in the Stage 1 manuscript and that the discussion section is appropriate and well-done. Typically, the Stage 2 review processes should be completed in one or, at most, two rounds.
Conclusion
The focal article develops a much-needed competency framework for training reviewers, offering concrete advice for how we can improve our reviewing skills. Training reviewers via the framework is a great step toward solving some of the issues with the peer-review process in the organizational sciences. This commentary extends the focal argument by suggesting a complementary solution, results-blind review, that addresses the scientific ecosystem at large. Using RBR, reviewers can more easily and successfully enact the competencies proposed in the focal article, ultimately advancing our science and improving the published research that represents our field.