Many argue that the peer review is broken—but it should not be. The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) comes at a time when many researchers and some editors are claiming the peer review process is broken with observations of the impact of human frailties (Rennie, Reference Rennie2016), that the peer review system is unsustainable (Kovanis, Porcher, Ravaud, & Trinquart, Reference Kovanis, Porcher, Ravaud and Trinquart2016), and that the system exploits academics (Fox & Petchy, Reference Fox and Petchey2010). While the focal article moves to address the human frailties issue, the second issue relates to the academic pool willing to complete peer reviews, and this is a problem that needs to be also resolved for the quality of reviews to have any effect. Indeed, if we are able to broaden the reviewing pool, this may make establishing quality standards even more important. The final issue, which relates to the exploitation of academic labor by publishing houses, while associated with the reduced reviewing pool, could be considered an economic or structural problem. In this response, I seek to address the problems that emerge from fewer academics engaging in the peer review process. I also offer solutions for what we can do about it and explain how this can be linked to the professionalism competency within the quality framework for peer review offered by Köhler et al.
Estimates of those participating in the peer review process vary. Kovanis et al. (Reference Kovanis, Porcher, Ravaud and Trinquart2016), in their discipline, estimate that approximately 90 percent of reviews are completed by just 20 percent of academics. Hazen and his colleagues (Reference Hazen, Fawcett, Ogden, Autry, Richey and Ellinger2016) note that editors can approach between 10 and 12 potential reviewers to get two peer reviews of a manuscript. There is a clear view, based on anecdotal evidence at conferences from editors and associate editors, that many journals have difficulty getting academics to agree to peer review submissions. This, in turn, has an impact on the turnaround time for reviews and the potential quality of the review, as each new reviewer approached may reduce the potential fit between the article to be reviewed and the reviewer’s knowledge. The dilemma for individual academics is explained by Smith (Reference Smith2016), who expresses, on reflection, a concern that he has published more papers than he has reviewed.
An argument could be mounted that there are positive aspects to declined reviews and a smaller pool of reviewers. One could argue that within this framework reviews are completed by experienced reviewers or reviewers who are regularly approached because of the quality of their reviews. That said, a reduced reviewing pool also contributes to a narrowing of expertise and less potential intellectual capital being applied in reviews, as well as an increased workload for a small cadre of reviewers. Certainly, the quality concern in an expanded reviewing pool may be ameliorated, particularly if reviewers adopt a common framework like the one offered in the focal article.
So, what is the solution to the small pool of academics who agree to be reviewers? A number of different avenues have been explored. Fox and Petchy (Reference Fox and Petchey2010) raise the potential for a system they call Pubcreds, where authors receive credits for reviewing work, which they can use to get their own work reviewed. While this is an interesting idea, it makes the call for a competency framework as described by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) even more important. Personally, I do not want my work reviewed by people who are only doing it to gain credit; I would prefer someone who is engaged in the research. Indeed, the Publons initiative (Smith, Reference Smith2016), a reviewer acknowledgment process, may be a move in the right direction in terms of academics receiving acknowledgment of the effort they expend on reviewing (Teixeira da Silva & Al-Khatib, Reference Teixeira da Silva and Al-Khatib2019). That said, research conducted by Squazzoni et al. (Reference Squazzoni, Bravo and Takács2013) suggests that incentives for reviewing do not work.
The focal article notes the importance of training for peer review. Although, I agree that this is an essential step to improving quality, the question is both about skills and about motivation (and expectations in the profession). There is clear evidence across a range of disciplines and professions that the more experience we have, the more our skills improve. So training is a first step, which then needs to be followed by regular practice requiring engagement from the academic community. Editors expect the vast bulk of reviewing to be done by academics. Editors also expect that a qualification most academics have to achieve to join the profession is a doctorate, which involves training. However, as Mitchell (Reference Mitchell2007) notes, doctoral training is generally narrow and focusses primarily on disciplinary expertise in research, often at the expense of all the other aspects of an academic career, including understanding the broader community we rely on and how academics can become good citizens in that community. Under this regime, reviewing research is seen as an ancillary task.
To counter this view, various professional and academic associations (e.g., SIOP, AOM, ANZAM, BAM) have included reviewing workshops as a way of reducing the skills gap and emphasizing the importance of reviewing to junior scholars. Although this may contribute to improving the quality of reviews, whether this stems the tide of declined reviews is yet to be established. As a community of scholars, we need to set up expectations that reviewing is a part of academic life and a contribution back to the discipline. Indeed, Hazen et al. (Reference Hazen, Fawcett, Ogden, Autry, Richey and Ellinger2016) consider that reviewing should move from a “professional courtesy” to a “professional obligation.” Just as in certain professions (e.g., psychology) members are expected to maintain their currency, so academics need to maintain their currency, and reviewing may be one avenue that could enable the maintenance of professional and disciplinary standards.
To this end, I consider there to be an additional competency that could potentially enhance the quality framework, that of engagement. Engagement could be included under the professionalism competency and be defined as engaging with the discipline by providing peer reviews. The competency may include factors such as (a) reviewing when requested or by offering alternative reviewers who may be able to contribute to reviewing the manuscript and (b) undertaking reviews based on expertise and not based solely on editorial board membership. Finally (c) participating in the reviewing process should be considered a standard contribution in any academic career. I hope my suggestions contribute to an ongoing debate about this essential element of maintaining the academic profession.
You have
Access
Many argue that the peer review is broken—but it should not be. The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) comes at a time when many researchers and some editors are claiming the peer review process is broken with observations of the impact of human frailties (Rennie, Reference Rennie2016), that the peer review system is unsustainable (Kovanis, Porcher, Ravaud, & Trinquart, Reference Kovanis, Porcher, Ravaud and Trinquart2016), and that the system exploits academics (Fox & Petchy, Reference Fox and Petchey2010). While the focal article moves to address the human frailties issue, the second issue relates to the academic pool willing to complete peer reviews, and this is a problem that needs to be also resolved for the quality of reviews to have any effect. Indeed, if we are able to broaden the reviewing pool, this may make establishing quality standards even more important. The final issue, which relates to the exploitation of academic labor by publishing houses, while associated with the reduced reviewing pool, could be considered an economic or structural problem. In this response, I seek to address the problems that emerge from fewer academics engaging in the peer review process. I also offer solutions for what we can do about it and explain how this can be linked to the professionalism competency within the quality framework for peer review offered by Köhler et al.
Estimates of those participating in the peer review process vary. Kovanis et al. (Reference Kovanis, Porcher, Ravaud and Trinquart2016), in their discipline, estimate that approximately 90 percent of reviews are completed by just 20 percent of academics. Hazen and his colleagues (Reference Hazen, Fawcett, Ogden, Autry, Richey and Ellinger2016) note that editors can approach between 10 and 12 potential reviewers to get two peer reviews of a manuscript. There is a clear view, based on anecdotal evidence at conferences from editors and associate editors, that many journals have difficulty getting academics to agree to peer review submissions. This, in turn, has an impact on the turnaround time for reviews and the potential quality of the review, as each new reviewer approached may reduce the potential fit between the article to be reviewed and the reviewer’s knowledge. The dilemma for individual academics is explained by Smith (Reference Smith2016), who expresses, on reflection, a concern that he has published more papers than he has reviewed.
An argument could be mounted that there are positive aspects to declined reviews and a smaller pool of reviewers. One could argue that within this framework reviews are completed by experienced reviewers or reviewers who are regularly approached because of the quality of their reviews. That said, a reduced reviewing pool also contributes to a narrowing of expertise and less potential intellectual capital being applied in reviews, as well as an increased workload for a small cadre of reviewers. Certainly, the quality concern in an expanded reviewing pool may be ameliorated, particularly if reviewers adopt a common framework like the one offered in the focal article.
So, what is the solution to the small pool of academics who agree to be reviewers? A number of different avenues have been explored. Fox and Petchy (Reference Fox and Petchey2010) raise the potential for a system they call Pubcreds, where authors receive credits for reviewing work, which they can use to get their own work reviewed. While this is an interesting idea, it makes the call for a competency framework as described by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) even more important. Personally, I do not want my work reviewed by people who are only doing it to gain credit; I would prefer someone who is engaged in the research. Indeed, the Publons initiative (Smith, Reference Smith2016), a reviewer acknowledgment process, may be a move in the right direction in terms of academics receiving acknowledgment of the effort they expend on reviewing (Teixeira da Silva & Al-Khatib, Reference Teixeira da Silva and Al-Khatib2019). That said, research conducted by Squazzoni et al. (Reference Squazzoni, Bravo and Takács2013) suggests that incentives for reviewing do not work.
The focal article notes the importance of training for peer review. Although, I agree that this is an essential step to improving quality, the question is both about skills and about motivation (and expectations in the profession). There is clear evidence across a range of disciplines and professions that the more experience we have, the more our skills improve. So training is a first step, which then needs to be followed by regular practice requiring engagement from the academic community. Editors expect the vast bulk of reviewing to be done by academics. Editors also expect that a qualification most academics have to achieve to join the profession is a doctorate, which involves training. However, as Mitchell (Reference Mitchell2007) notes, doctoral training is generally narrow and focusses primarily on disciplinary expertise in research, often at the expense of all the other aspects of an academic career, including understanding the broader community we rely on and how academics can become good citizens in that community. Under this regime, reviewing research is seen as an ancillary task.
To counter this view, various professional and academic associations (e.g., SIOP, AOM, ANZAM, BAM) have included reviewing workshops as a way of reducing the skills gap and emphasizing the importance of reviewing to junior scholars. Although this may contribute to improving the quality of reviews, whether this stems the tide of declined reviews is yet to be established. As a community of scholars, we need to set up expectations that reviewing is a part of academic life and a contribution back to the discipline. Indeed, Hazen et al. (Reference Hazen, Fawcett, Ogden, Autry, Richey and Ellinger2016) consider that reviewing should move from a “professional courtesy” to a “professional obligation.” Just as in certain professions (e.g., psychology) members are expected to maintain their currency, so academics need to maintain their currency, and reviewing may be one avenue that could enable the maintenance of professional and disciplinary standards.
To this end, I consider there to be an additional competency that could potentially enhance the quality framework, that of engagement. Engagement could be included under the professionalism competency and be defined as engaging with the discipline by providing peer reviews. The competency may include factors such as (a) reviewing when requested or by offering alternative reviewers who may be able to contribute to reviewing the manuscript and (b) undertaking reviews based on expertise and not based solely on editorial board membership. Finally (c) participating in the reviewing process should be considered a standard contribution in any academic career. I hope my suggestions contribute to an ongoing debate about this essential element of maintaining the academic profession.