The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) provided a much needed competency model for peer review. In addition to noting required competencies, their article provides a nice overview of the function and limitations of peer review. I will expand on some of their points from the perspective of three roles that peer reviewers might serve: gatekeeper, developmental editor, and coauthor. The situation in which peer review occurs determines which role is most appropriate for a given peer reviewer to take. I will focus largely on peer review in industrial and organizational (I-O) and management journals where the roles have become confounded, although the principles of maintaining clear role boundaries would hold in other peer review contexts, as well.
In their classic article “Split Roles in Performance Appraisal,” Meyer, Kay, and French Jr. (Reference Meyer, Kay and French1965) described research showing that mixing different purposes in an organizational endeavor (in their case performance appraisals) is often ineffective and can have unintended negative consequences. They showed that mixing a decision-making function (i.e., discussions of salary) with a feedback function designed to improve performance not only failed to improve performance, but it resulted in defensiveness and poor attitudes. Because split roles have become the norm in journal peer review, it is no wonder that most of us receive our reviews in such poor spirit.
As Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) correctly note, peer review provides several vital functions in our professional work, but it is not without problems. One source of the problem is that too often peer review mixes what should be separate roles. This produces role conflict for the reviewer, who is focused on more than one purpose, and for recipients who are torn among purposes of the review. Köhler et al. document well the problems that exist with peer review, and at least some of this can be attributed to mixing roles.
There are at least three distinct roles that a peer reviewer might assume, and in many contexts those roles are clearly distinguished. But in journal reviewing in particular, too often the boundaries among roles have become severely blurred.
Gatekeeper role
A gatekeeper either makes a decision (e.g., journal editor) or provides a recommendation to the decision-maker (journal reviewer). This is analogous to the situation in Meyer et al. (Reference Meyer, Kay and French1965) where the manager discusses an employee’s performance in the context of salary. The peer reviewer is in a position of power over the recipient because he/she controls something of value to the recipient—such as acceptance of a journal submission. Because the recipient has a lot riding on the outcome, he or she is under pressure to conform to requests by peer reviewers. Thus peer reviewers can be in a position to coerce recipients into accepting feedback with which they do not agree (Bedeian, Reference Bedeian2003).
The gatekeeper role is to evaluate the quality of the product being reviewed. This means listing the main strengths and weaknesses of the work, including the level of contribution and importance. Flaws in method and logic should be explained, and where presentation is unclear, it should be noted. The gatekeeper role is not to provide suggestions to the recipient about how to improve the work or ideas to do things differently—that is the developmental role.
Developmental role
A developmental editor is a writing expert, often hired by a book publisher, to assist a content expert in improving a work. Such an editor goes beyond mere copyediting (correction of grammar) to helping the author flesh out ideas in a way that is better organized and more clearly explained. The developmental role for a peer reviewer is similar. He or she assists a colleague in refining a work through improved writing, or by considering alternative ways of approaching a problem or analyzing data.
Often authors of journal articles and other works feel the need to get an outside perspective. They might ask an expert or a friend. Sometimes young faculty or professionals ask a mentor, such as their PhD advisor or a senior person in their department. The role of the reviewer in this case is to provide fresh eyes to be sure that the work accomplishes what is intended and to make some suggestions for improvement. Given that the developmental reviewer has no particular power over the recipient, that is, he or she does not help determine the fate of a particular work, recipients are free to accept or ignore suggestions. If the input is substantial and helps to shape the final product in significant ways, the developmental reviewer might be invited to become a coauthor.
The developmental role should be kept separate from the gatekeeper role. Prior to the 1980s, these roles were fairly distinct for journal peer review. Reviews were short and focused mainly on strengths and weaknesses. By the end of the 1980s there was a shift toward incorporating the developmental role into peer reviewing, which has had negative consequences for the journal peer review process because reviewers make suggestions that authors felt powerless to ignore (Bedeian, Reference Bedeian2003). If an untenured assistant professor is asked to engage in questionable research practices, such as HARKing or p-hacking, he or she will likely feel compelled to comply, assuming that chances are good that the reviewer will recommend acceptance in the next round. Developmental feedback of this type is a form of institutionalized coercion given the power differential between peer reviewers and recipients whose careers and livelihood depends on getting articles published.
Developmental feedback is important, but is most helpful to recipients when the developmental role is distinct from the gatekeeper role. Journal peer reviewers should limit themselves to the gatekeeper role, provide a detailed list of strengths and weaknesses, and avoid the temptation to make specific developmental suggestions. In my experience those sorts of suggestions are likely to be idiosyncratic to the reviewer (similar experiences were noted in an author survey by Bedeian, Reference Bedeian2003). I have found that too often after resubmission, Reviewer A did not like the changes suggested by Reviewer B, and/or Reviewer B did not like the changes suggested by Reviewer A. Given that recipients are in a low power position, they will likely accept all suggestions, perhaps to the detriment of the work.
Developmental feedback should occur when peer reviewers are only in the developmental role. This happens with friendly reviews when a recipient asks a colleague for feedback. Splitting the developmental role from the gatekeeper role allows the recipient to focus only on the feedback and not the outcome. Given the more equal power positions, recipients will be less likely to have negative feelings about the feedback, take a more constructive rather than adversarial approach, and feel empowered to accept or reject suggestions.
Author role
An author is an individual who makes a substantive contribution to a work. Often this means doing all or some of the writing, but it could mean helping to decide the content of a work without doing the writing itself. When a work is an empirical research report, an author might have input into methods, analyses, or interpretation. Peer review occurs among authors as they review one another’s contributions to a particular article. For example, the lead author might produce a draft that is reviewed by coauthors. Through a process of discussion and negotiation, coauthors come to consensus about each detail of the article, producing the final product.
Too often with journal peer review, reviewers cross the line from providing helpful developmental feedback to becoming what Bedeian (Reference Bedeian1996) referred to as ghost writers—individuals who make sufficient substantive contribution to a work to be coauthors, but are not recognized as such (i.e., they are ghost writers). There is an ethical conflict when the gatekeeper and author roles become confounded. An author of a journal submission will likely feel compelled to accept the “recommendations” of a reviewer whose input is so substantial that he or she should rightly be added to the list of authors. The article in a real sense is in part the reviewer’s own work. When such a reviewer adopts the author role, he or she has a conflict of interest when reviewing the revised version of the article where the advice was taken. He or she would be reviewing a work that is his or her own, even if that ownership is not publically acknowledged.
Reconciling the roles
Peer review is vitally important, but it works best when the three roles are kept distinct. Authors need to be empowered to create their works without being compelled by the review process to conform to the demands of peer reviewers. Of course, authors need feedback, but only in the context where they are free to accept or dismiss reviewer suggestions. This means that journals need to stop confounding the gatekeeping with the developmental and even the author roles.
It is clear that journal reviewers and editors must serve the gatekeeper role, but if they cannot also provide developmental feedback, how will that be done? There are at least two possibilities. First, we could leave it to authors to seek developmental feedback on their own. Of course, an unintended consequence might be that we all become bogged down with constant requests by friends and strangers to review their articles, and what about authors who do not have a network of colleagues and friends who can provide high quality feedback? Second, our journals could set up a developmental feedback system, where authors could submit articles, not for consideration of publication, but merely for developmental feedback in advance of publication submission. This might also be blind review but focus purely on feedback to improve the article. Since no acceptance decisions would be made, authors would be free to accept or reject feedback. The authors could then submit the article for publication consideration and receive a gatekeeper review from new reviewers who have provided no prior input into the paper. This review would most likely be short, and focus mainly on strengths and weaknesses, errors and omissions. Its purpose, as was the case a few decades past, would primarily be to advise the editor. It would be shared with the authors so they would understand the basis for decision—accept or reject. The reviewer would not provide suggestions just to improve the article, which is often largely idiosyncratic to the reviewer (Bedeian, Reference Bedeian2003).
Maybe it is just nostalgia, but I think our journal peer review system worked more smoothly a few decades ago when journal reviews were short, and roles were kept more distinct. It certainly made life easier for authors who did not feel coerced into making changes with which they did not agree, and it was easier for reviewers who did not feel compelled to write long and detailed reviews that included extensive developmental feedback. We all could be doing things that are more productive than going round after round in our adversarial peer review system.
%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
You have Access
The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) provided a much needed competency model for peer review. In addition to noting required competencies, their article provides a nice overview of the function and limitations of peer review. I will expand on some of their points from the perspective of three roles that peer reviewers might serve: gatekeeper, developmental editor, and coauthor. The situation in which peer review occurs determines which role is most appropriate for a given peer reviewer to take. I will focus largely on peer review in industrial and organizational (I-O) and management journals where the roles have become confounded, although the principles of maintaining clear role boundaries would hold in other peer review contexts, as well.
In their classic article “Split Roles in Performance Appraisal,” Meyer, Kay, and French Jr. (Reference Meyer, Kay and French1965) described research showing that mixing different purposes in an organizational endeavor (in their case performance appraisals) is often ineffective and can have unintended negative consequences. They showed that mixing a decision-making function (i.e., discussions of salary) with a feedback function designed to improve performance not only failed to improve performance, but it resulted in defensiveness and poor attitudes. Because split roles have become the norm in journal peer review, it is no wonder that most of us receive our reviews in such poor spirit.
As Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) correctly note, peer review provides several vital functions in our professional work, but it is not without problems. One source of the problem is that too often peer review mixes what should be separate roles. This produces role conflict for the reviewer, who is focused on more than one purpose, and for recipients who are torn among purposes of the review. Köhler et al. document well the problems that exist with peer review, and at least some of this can be attributed to mixing roles.
There are at least three distinct roles that a peer reviewer might assume, and in many contexts those roles are clearly distinguished. But in journal reviewing in particular, too often the boundaries among roles have become severely blurred.
Gatekeeper role
A gatekeeper either makes a decision (e.g., journal editor) or provides a recommendation to the decision-maker (journal reviewer). This is analogous to the situation in Meyer et al. (Reference Meyer, Kay and French1965) where the manager discusses an employee’s performance in the context of salary. The peer reviewer is in a position of power over the recipient because he/she controls something of value to the recipient—such as acceptance of a journal submission. Because the recipient has a lot riding on the outcome, he or she is under pressure to conform to requests by peer reviewers. Thus peer reviewers can be in a position to coerce recipients into accepting feedback with which they do not agree (Bedeian, Reference Bedeian2003).
The gatekeeper role is to evaluate the quality of the product being reviewed. This means listing the main strengths and weaknesses of the work, including the level of contribution and importance. Flaws in method and logic should be explained, and where presentation is unclear, it should be noted. The gatekeeper role is not to provide suggestions to the recipient about how to improve the work or ideas to do things differently—that is the developmental role.
Developmental role
A developmental editor is a writing expert, often hired by a book publisher, to assist a content expert in improving a work. Such an editor goes beyond mere copyediting (correction of grammar) to helping the author flesh out ideas in a way that is better organized and more clearly explained. The developmental role for a peer reviewer is similar. He or she assists a colleague in refining a work through improved writing, or by considering alternative ways of approaching a problem or analyzing data.
Often authors of journal articles and other works feel the need to get an outside perspective. They might ask an expert or a friend. Sometimes young faculty or professionals ask a mentor, such as their PhD advisor or a senior person in their department. The role of the reviewer in this case is to provide fresh eyes to be sure that the work accomplishes what is intended and to make some suggestions for improvement. Given that the developmental reviewer has no particular power over the recipient, that is, he or she does not help determine the fate of a particular work, recipients are free to accept or ignore suggestions. If the input is substantial and helps to shape the final product in significant ways, the developmental reviewer might be invited to become a coauthor.
The developmental role should be kept separate from the gatekeeper role. Prior to the 1980s, these roles were fairly distinct for journal peer review. Reviews were short and focused mainly on strengths and weaknesses. By the end of the 1980s there was a shift toward incorporating the developmental role into peer reviewing, which has had negative consequences for the journal peer review process because reviewers make suggestions that authors felt powerless to ignore (Bedeian, Reference Bedeian2003). If an untenured assistant professor is asked to engage in questionable research practices, such as HARKing or p-hacking, he or she will likely feel compelled to comply, assuming that chances are good that the reviewer will recommend acceptance in the next round. Developmental feedback of this type is a form of institutionalized coercion given the power differential between peer reviewers and recipients whose careers and livelihood depends on getting articles published.
Developmental feedback is important, but is most helpful to recipients when the developmental role is distinct from the gatekeeper role. Journal peer reviewers should limit themselves to the gatekeeper role, provide a detailed list of strengths and weaknesses, and avoid the temptation to make specific developmental suggestions. In my experience those sorts of suggestions are likely to be idiosyncratic to the reviewer (similar experiences were noted in an author survey by Bedeian, Reference Bedeian2003). I have found that too often after resubmission, Reviewer A did not like the changes suggested by Reviewer B, and/or Reviewer B did not like the changes suggested by Reviewer A. Given that recipients are in a low power position, they will likely accept all suggestions, perhaps to the detriment of the work.
Developmental feedback should occur when peer reviewers are only in the developmental role. This happens with friendly reviews when a recipient asks a colleague for feedback. Splitting the developmental role from the gatekeeper role allows the recipient to focus only on the feedback and not the outcome. Given the more equal power positions, recipients will be less likely to have negative feelings about the feedback, take a more constructive rather than adversarial approach, and feel empowered to accept or reject suggestions.
Author role
An author is an individual who makes a substantive contribution to a work. Often this means doing all or some of the writing, but it could mean helping to decide the content of a work without doing the writing itself. When a work is an empirical research report, an author might have input into methods, analyses, or interpretation. Peer review occurs among authors as they review one another’s contributions to a particular article. For example, the lead author might produce a draft that is reviewed by coauthors. Through a process of discussion and negotiation, coauthors come to consensus about each detail of the article, producing the final product.
Too often with journal peer review, reviewers cross the line from providing helpful developmental feedback to becoming what Bedeian (Reference Bedeian1996) referred to as ghost writers—individuals who make sufficient substantive contribution to a work to be coauthors, but are not recognized as such (i.e., they are ghost writers). There is an ethical conflict when the gatekeeper and author roles become confounded. An author of a journal submission will likely feel compelled to accept the “recommendations” of a reviewer whose input is so substantial that he or she should rightly be added to the list of authors. The article in a real sense is in part the reviewer’s own work. When such a reviewer adopts the author role, he or she has a conflict of interest when reviewing the revised version of the article where the advice was taken. He or she would be reviewing a work that is his or her own, even if that ownership is not publically acknowledged.
Reconciling the roles
Peer review is vitally important, but it works best when the three roles are kept distinct. Authors need to be empowered to create their works without being compelled by the review process to conform to the demands of peer reviewers. Of course, authors need feedback, but only in the context where they are free to accept or dismiss reviewer suggestions. This means that journals need to stop confounding the gatekeeping with the developmental and even the author roles.
It is clear that journal reviewers and editors must serve the gatekeeper role, but if they cannot also provide developmental feedback, how will that be done? There are at least two possibilities. First, we could leave it to authors to seek developmental feedback on their own. Of course, an unintended consequence might be that we all become bogged down with constant requests by friends and strangers to review their articles, and what about authors who do not have a network of colleagues and friends who can provide high quality feedback? Second, our journals could set up a developmental feedback system, where authors could submit articles, not for consideration of publication, but merely for developmental feedback in advance of publication submission. This might also be blind review but focus purely on feedback to improve the article. Since no acceptance decisions would be made, authors would be free to accept or reject feedback. The authors could then submit the article for publication consideration and receive a gatekeeper review from new reviewers who have provided no prior input into the paper. This review would most likely be short, and focus mainly on strengths and weaknesses, errors and omissions. Its purpose, as was the case a few decades past, would primarily be to advise the editor. It would be shared with the authors so they would understand the basis for decision—accept or reject. The reviewer would not provide suggestions just to improve the article, which is often largely idiosyncratic to the reviewer (Bedeian, Reference Bedeian2003).
Maybe it is just nostalgia, but I think our journal peer review system worked more smoothly a few decades ago when journal reviews were short, and roles were kept more distinct. It certainly made life easier for authors who did not feel coerced into making changes with which they did not agree, and it was easier for reviewers who did not feel compelled to write long and detailed reviews that included extensive developmental feedback. We all could be doing things that are more productive than going round after round in our adversarial peer review system.