Poor quality peer review is ubiquitous. Unfortunately, many reviewers lack the requisite domain-specific expertise to perform the requested review but also exhibit little apparent motivation to gain the expertise required of someone expected to provide such a review. We have all likely received reviews where the reviewers provided commentary but the content of the review itself exhibited a lack of expert knowledge and the paper/proposal was subsequently rejected based largely on opinion rather than for theoretically or methodologically derived reasons. Reviews provided by non-expert peersFootnote 1 hamper science and hurt careers.
Reading, understanding, integrating, and building on prior works is how science progresses. I can fully understand that we, as authors, need to detail previous works in our manuscripts and proposals. But reviewers must also have a similar or high level of expertise or be willing to gain that expertise for at least a meaningful proportion of the content of the reviewed work. As just one example, I recently received a review from a top-tier applied psychology journal where one reviewer stated that he/she did not know the core constructs used in the study, indicated that he/she was not going to read the cited works related to the core constructs, and then proceeded to provide a review (to make the situation more cringe-worthy, the core constructs were introduced in two manuscripts in the very journal for which the reviewer was reviewing). For a reviewer to indicate that he/she would not read cited works, remain willfully ignorant of their content, and then proceed to provide a review should never happen in our field. This, and similar failures, are common, as Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) note in the Focal Article.
How can a science build if we cannot rely on reviewers to understand prior works? It is glib to say “we cannot,” though the real answer is probably “very slowly.” Instead of having concise articles (especially compared to other sciences) our field has very long articles that start with extended literature reviews. Who are we speaking to with these extended reviews, and are they really necessary? If every article must start at the beginning of applied psychology research in order to educate reviewers and readers, then it becomes clearer as to why our science progresses so slowly (see for example Davis, Reference Davis2010).
So what do we do? The proposed competency framework, training, bootcamps, and collaborations with Consortium for the Advancement of Research Methods and Analysis (CARMA), Organizational Research Methods, and the Society for Industrial and Organizational Psychology (SIOP) are all a great start. But the combined fields of Industrial Organizational Psychology, Organizational Behavior, and Human Resources (IOOBHR) often grouped under the larger umbrellas of applied psychology and management, however, need to have serious conversations about what it means to be a peer, what our science looks like, and how our science should be consumed.
Consensus, identity, and science
IOOBHR tends to take a “big tent” perspective, meaning that we have an expectation that anyone with a PhD in IOOBHR should be able to pick up a journal, manuscript, or proposal and subsequently read it, understand it, and be able to provide cogent feedback on it. Yet, our field is often characterized as fractured or exhibiting a lack of a paradigm (Birkinshaw, Healey, Suddaby, & Weber, Reference Birkinshaw, Healey, Suddaby and Weber2014; Fisher & Aguinis, Reference Fisher and Aguinis2017; Pfeffer, Reference Pfeffer1993). These concerns are then used to argue that our field is immature, that we should seek consensus, and that our science cannot progress without consensus (Pfeffer, Reference Pfeffer1993). But what if we are not a low-paradigm field and, instead, are trying to fit too much under our big tent? While some might like to see “bigger, better, and more challenging questions compared to the orthodoxy in our management research” (Birkinshaw et al., Reference Birkinshaw, Healey, Suddaby and Weber2014, p. 38), that is an impossible task if we also insist on consensus across a wide range of domains within our fields as part of the review process. That is to say, exploring novel and big questions with new research techniques and designs will result in research that is rarely (when the stars align with the right editor and reviewers) published in top journals.
Instead of arguing that we are a low-paradigm field, I would, instead, argue that we are many high-paradigm fields forced to fight for limited resources (e.g., journal space, grant funding) because, we, ourselves, keep insisting that consensus across a diverse array of research specialties is the marker for good science. Instead of viewing ourselves then from this “big-tent,” consensus-necessary field, we should instead view ourselves as many specialized tents all at the same carnival. By better specializing, we would find that we are not nearly as fractured as currently suggested.
What do we do now?
I applaud the work of Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) but believe they did not spend adequate time with expertise. If we take the perspective of applied psychology and management being made of many tents at the same carnival, then this allows for a clearer definition of expertise. A peer, then, for the purposes of peer review is not just anyone with a PhD in a management or applied psychology discipline. Instead, a peer is someone who has clear and documentable experience in the domain. The editor, associate editor, or grant/proposal leader might not have this specific expertise, but those responsible for selecting the reviewers should become more cognizant of expertise, should develop some way to codify expertise, and then should be able to apply and document that system. (Imagine if the selection of peer reviewers needed to be as well documented and justified as the employee selection process outlined in the Uniform Guidelines!).
I recognize that it may not be possible to identify several experts to perform the review (following the narrower definition described above), but the non-expert reviewers who are asked to review a manuscript should then exhibit motivation for the review. This could come via a series of questions and actionable behaviors. First, the individuals asked to perform the review should ask, “Am I interested in this topic or am I reviewing this because I feel obligated to provide this service?” If the answer is driven primarily because of felt obligation, the requisite motivation to provide a quality review is likely too low (Deci & Ryan, Reference Deci and Ryan1985). In this case, the individual asked to perform the review should indicate to the editor that he/she cannot perform the review. This should come without pain or guilt on the part of the individual asked to perform the review or punitive action from the editorial team. Second, if the person is interested but lacks expertise, then the individual should ask, “Am I willing and do I have time to gain some expertise in this area?” Said more simply, the reviewer needs to assess whether or not he/she is willing to read the key works (all would be wonderful) cited by the authors of the manuscript in order to build familiarity with the domain. If the individual asked to perform the review is truly interested, then this is likely to result in more than just a passing level of expertise. If the individual responds in the affirmative, then it is time for him/her to start reading. If the individual instead realizes that he/she does not have the time to gain this expertise and/or is unwilling to gain the expertise, then that person needs to self-identify that he/she is not a peer for the purposes of reviewing the manuscript/proposal. Because he/she is not an expert and is unwilling to become an expert, then attempting to perform a review would subsequently not provide a service to the field (the journal, grant awarding institution, or the manuscripts’ authors). When this is the case, the individual asked to perform the review should decline the review.
Once reviews are returned to the editor, associate editor, or grant/proposal leader, then he/she needs to do more than count votes. Many responses that I have received from editors of manuscripts sent to journals have done a good job of trying to integrate the comments from the reviewers. Quite often they also weigh in by noting what they believe to be the more serious and/or addressable concerns with the manuscript. This work is appreciated and is immensely helpful. While some editors and other key decision makers may be doing these things already to some extent, editors and key decision makers need to heavily discount or disregard reviews that exhibit a lack of expertise. Weighing equally the feedback of experts and non-experts is a disservice not only to the authors of the manuscript in question but also to the expert reviewers. (I have been on the receiving end of more than one rejection where the expert reviewer’s response was in the affirmative but the non-expert’s response was in the negative with regard to resubmission, and the editor, based on a split decision, rejected the manuscript.) Why would an expert reviewer continue to offer his/her time to the review process and spend time providing expert information if he/she is going to subsequently be ignored? Experts who hold up progress in the field by failing to consider new paradigms may present a special challenge, but the failure to consider a new paradigm may indicate they are not really the right expert.
Conclusion
Our review system is broken. The proposed competency framework is a great suggestion, but it will only work if IOOBHR is willing to take additional steps. This means ditching the big tent, asking editors and grant/proposal leaders to make better choices on expertise, allowing editors and grant/proposal review leaders to reject poor quality reviews, weighing the feedback from experts more heavily in decisions than non-experts, and asking reviewers to check both their expertise and motivation before accepting a review. As noted by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020, p. 5), “… actual expertise in the topic, research method, and methodology are indispensable. Without it, reviews are likely to become a set of recommendations based on personal preferences or individual practices rather than an expression of applying superior research standards” (emphasis added). Unfortunately, this is where the field stands today, and it hampers our science. Several of the aforementioned suggestions could likely be integrated into the competency framework to make it a more robust and helpful guide to decision makers on what makes a peer an actual peer for the purposes of reviewing others’ work.
You have
Access
Poor quality peer review is ubiquitous. Unfortunately, many reviewers lack the requisite domain-specific expertise to perform the requested review but also exhibit little apparent motivation to gain the expertise required of someone expected to provide such a review. We have all likely received reviews where the reviewers provided commentary but the content of the review itself exhibited a lack of expert knowledge and the paper/proposal was subsequently rejected based largely on opinion rather than for theoretically or methodologically derived reasons. Reviews provided by non-expert peersFootnote 1 hamper science and hurt careers.
Reading, understanding, integrating, and building on prior works is how science progresses. I can fully understand that we, as authors, need to detail previous works in our manuscripts and proposals. But reviewers must also have a similar or high level of expertise or be willing to gain that expertise for at least a meaningful proportion of the content of the reviewed work. As just one example, I recently received a review from a top-tier applied psychology journal where one reviewer stated that he/she did not know the core constructs used in the study, indicated that he/she was not going to read the cited works related to the core constructs, and then proceeded to provide a review (to make the situation more cringe-worthy, the core constructs were introduced in two manuscripts in the very journal for which the reviewer was reviewing). For a reviewer to indicate that he/she would not read cited works, remain willfully ignorant of their content, and then proceed to provide a review should never happen in our field. This, and similar failures, are common, as Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) note in the Focal Article.
How can a science build if we cannot rely on reviewers to understand prior works? It is glib to say “we cannot,” though the real answer is probably “very slowly.” Instead of having concise articles (especially compared to other sciences) our field has very long articles that start with extended literature reviews. Who are we speaking to with these extended reviews, and are they really necessary? If every article must start at the beginning of applied psychology research in order to educate reviewers and readers, then it becomes clearer as to why our science progresses so slowly (see for example Davis, Reference Davis2010).
So what do we do? The proposed competency framework, training, bootcamps, and collaborations with Consortium for the Advancement of Research Methods and Analysis (CARMA), Organizational Research Methods, and the Society for Industrial and Organizational Psychology (SIOP) are all a great start. But the combined fields of Industrial Organizational Psychology, Organizational Behavior, and Human Resources (IOOBHR) often grouped under the larger umbrellas of applied psychology and management, however, need to have serious conversations about what it means to be a peer, what our science looks like, and how our science should be consumed.
Consensus, identity, and science
IOOBHR tends to take a “big tent” perspective, meaning that we have an expectation that anyone with a PhD in IOOBHR should be able to pick up a journal, manuscript, or proposal and subsequently read it, understand it, and be able to provide cogent feedback on it. Yet, our field is often characterized as fractured or exhibiting a lack of a paradigm (Birkinshaw, Healey, Suddaby, & Weber, Reference Birkinshaw, Healey, Suddaby and Weber2014; Fisher & Aguinis, Reference Fisher and Aguinis2017; Pfeffer, Reference Pfeffer1993). These concerns are then used to argue that our field is immature, that we should seek consensus, and that our science cannot progress without consensus (Pfeffer, Reference Pfeffer1993). But what if we are not a low-paradigm field and, instead, are trying to fit too much under our big tent? While some might like to see “bigger, better, and more challenging questions compared to the orthodoxy in our management research” (Birkinshaw et al., Reference Birkinshaw, Healey, Suddaby and Weber2014, p. 38), that is an impossible task if we also insist on consensus across a wide range of domains within our fields as part of the review process. That is to say, exploring novel and big questions with new research techniques and designs will result in research that is rarely (when the stars align with the right editor and reviewers) published in top journals.
Instead of arguing that we are a low-paradigm field, I would, instead, argue that we are many high-paradigm fields forced to fight for limited resources (e.g., journal space, grant funding) because, we, ourselves, keep insisting that consensus across a diverse array of research specialties is the marker for good science. Instead of viewing ourselves then from this “big-tent,” consensus-necessary field, we should instead view ourselves as many specialized tents all at the same carnival. By better specializing, we would find that we are not nearly as fractured as currently suggested.
What do we do now?
I applaud the work of Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) but believe they did not spend adequate time with expertise. If we take the perspective of applied psychology and management being made of many tents at the same carnival, then this allows for a clearer definition of expertise. A peer, then, for the purposes of peer review is not just anyone with a PhD in a management or applied psychology discipline. Instead, a peer is someone who has clear and documentable experience in the domain. The editor, associate editor, or grant/proposal leader might not have this specific expertise, but those responsible for selecting the reviewers should become more cognizant of expertise, should develop some way to codify expertise, and then should be able to apply and document that system. (Imagine if the selection of peer reviewers needed to be as well documented and justified as the employee selection process outlined in the Uniform Guidelines!).
I recognize that it may not be possible to identify several experts to perform the review (following the narrower definition described above), but the non-expert reviewers who are asked to review a manuscript should then exhibit motivation for the review. This could come via a series of questions and actionable behaviors. First, the individuals asked to perform the review should ask, “Am I interested in this topic or am I reviewing this because I feel obligated to provide this service?” If the answer is driven primarily because of felt obligation, the requisite motivation to provide a quality review is likely too low (Deci & Ryan, Reference Deci and Ryan1985). In this case, the individual asked to perform the review should indicate to the editor that he/she cannot perform the review. This should come without pain or guilt on the part of the individual asked to perform the review or punitive action from the editorial team. Second, if the person is interested but lacks expertise, then the individual should ask, “Am I willing and do I have time to gain some expertise in this area?” Said more simply, the reviewer needs to assess whether or not he/she is willing to read the key works (all would be wonderful) cited by the authors of the manuscript in order to build familiarity with the domain. If the individual asked to perform the review is truly interested, then this is likely to result in more than just a passing level of expertise. If the individual responds in the affirmative, then it is time for him/her to start reading. If the individual instead realizes that he/she does not have the time to gain this expertise and/or is unwilling to gain the expertise, then that person needs to self-identify that he/she is not a peer for the purposes of reviewing the manuscript/proposal. Because he/she is not an expert and is unwilling to become an expert, then attempting to perform a review would subsequently not provide a service to the field (the journal, grant awarding institution, or the manuscripts’ authors). When this is the case, the individual asked to perform the review should decline the review.
Once reviews are returned to the editor, associate editor, or grant/proposal leader, then he/she needs to do more than count votes. Many responses that I have received from editors of manuscripts sent to journals have done a good job of trying to integrate the comments from the reviewers. Quite often they also weigh in by noting what they believe to be the more serious and/or addressable concerns with the manuscript. This work is appreciated and is immensely helpful. While some editors and other key decision makers may be doing these things already to some extent, editors and key decision makers need to heavily discount or disregard reviews that exhibit a lack of expertise. Weighing equally the feedback of experts and non-experts is a disservice not only to the authors of the manuscript in question but also to the expert reviewers. (I have been on the receiving end of more than one rejection where the expert reviewer’s response was in the affirmative but the non-expert’s response was in the negative with regard to resubmission, and the editor, based on a split decision, rejected the manuscript.) Why would an expert reviewer continue to offer his/her time to the review process and spend time providing expert information if he/she is going to subsequently be ignored? Experts who hold up progress in the field by failing to consider new paradigms may present a special challenge, but the failure to consider a new paradigm may indicate they are not really the right expert.
Conclusion
Our review system is broken. The proposed competency framework is a great suggestion, but it will only work if IOOBHR is willing to take additional steps. This means ditching the big tent, asking editors and grant/proposal leaders to make better choices on expertise, allowing editors and grant/proposal review leaders to reject poor quality reviews, weighing the feedback from experts more heavily in decisions than non-experts, and asking reviewers to check both their expertise and motivation before accepting a review. As noted by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020, p. 5), “… actual expertise in the topic, research method, and methodology are indispensable. Without it, reviews are likely to become a set of recommendations based on personal preferences or individual practices rather than an expression of applying superior research standards” (emphasis added). Unfortunately, this is where the field stands today, and it hampers our science. Several of the aforementioned suggestions could likely be integrated into the competency framework to make it a more robust and helpful guide to decision makers on what makes a peer an actual peer for the purposes of reviewing others’ work.