Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) present a compelling argument for training and developing reviewers based on an application of the article’s proposed competency model. In this response we highlight how such training and development could be enhanced by taking the context and purpose of the review into account. We offer suggestions in three areas: (1) leveling the competency framework by proficiency based on the context of the review, (2) providing context-specific training and feedback mechanisms, and (3) empowering reviewers with the confidence to review successfully by clarifying expectations and providing reviewing aids.
Level the competency framework by proficiency and context
Well-written competency models use observable behaviors that are succinct, distinct, and unambiguous to clearly define expectations for performance. As noted in the focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), peer reviewing has numerous applications, both in the context of scientific reviews for academic papers, journal publications, or conference presentations and in practical reviews in organizations where evaluating and offering suggestions for improving others’ work is essential to delivering high-quality work products. Because the purposes of reviews in these contexts and the associated consequences are very different, different reviewer skill levels and focuses are required. By leveling performance expectations for peer reviewers based on the context, Köhler et al.’s proposed competency framework could be more readily applied to a variety of reviewing scenarios.
In the academic context, varying levels of rigor are required by different outlets for publication. These outlets range from conference submissions to book chapters to dissertations to manuscripts for publication in a top-tier journal. The consequences of reviews vary by context as well. A rejection of a conference poster based on poorly executed reviews may disappoint a graduate student but not hinder his or her career. On the other hand, a rejection based on poorly executed reviews of a submission in an academic journal may make an academic’s quest for tenure more difficult. Likewise, in practical applications there are varying contexts of reviews ranging from internal communications to company white papers to proposals written to win new business to high-priority client deliverables. These different contexts also have different levels of consequences. A poorly prepared internal document could negatively impact the writer’s performance review but not damage a client relationship, whereas a flawed client-facing document could damage a company’s reputation and lead to the loss of business.
With these differences of context in mind, we provide some illustrative examples of how the competency requirements, and the weighted importance of each requirement in terms of overall review quality, may differ by context:
Representing: The expectations and interests may be more apparent for a journal compared to a practitioner delivering a work product to a client. Whereas journals typically make their mission, scope, and values public, the interests of a client may not be explicitly stated and may vary significantly by client.
Integrity: The aspect of this competency involving an evaluation of the hypotheses and methodological rigor in testing those hypotheses likely takes a different form for practitioner reviews, where there may not be hypotheses specified in advance for some documents (e.g., white papers, technical manuals). For the practitioner context, the rigor may focus instead on the correct application of established principles to the client’s situation. In addition, practical constraints may place limitations on the level of methodological rigor (e.g., smaller client sample sizes) than is typical for peer-reviewed journal publications.
Expertise: The focus of expertise may vary depending on what the reviewer needs to evaluate. An academic submission may require the reviewer to judge the comprehensiveness of the author’s review of the existing literature or judge the unique contributions of the article. A client report may have either a very limited or no literature review. If a document is prepared for a specific client, there may be less attention on the work’s contribution to the field or community. Therefore, some elements of the expertise competency would not be applicable in this instance.
Thoroughness: There may be differences in the level of detailed review required by context. For example, accuracy is extremely important in reporting client technical results, and some practitioner reviews might involve verifying the accuracy of every number reported in a table as a form of quality assurance for the client deliverable. This is not typically done for a journal review, where reviewers generally trust that the reported results have already been reviewed for accuracy.
Professionalism competencies: While academic reviews are generally blind, when a practitioner reviews a work product, he/she typically knows and works with the author. This means the constructiveness, tone, and open-mindedness of the feedback could have significant consequences for the relationship between the author and the reviewer.
For these reasons, we suggest an expansion of the model to clearly describe behaviors at the level of proficiency required for each competency in different contexts. Leveling the behaviors according to context can help clarify expectations for reviewers in different settings by providing information for aspiring reviewers on what good looks like in the context relevant to them. In addition, leveling also provides a blueprint for developing specific behaviors needed to be an effective reviewer in different contexts. The levels of the competency model could be extended to capture the most basic context such as providing informal feedback to others up to the most high-stakes rigorous type of review. Not all reviews are equal, and this additional detail of requirement at different levels will help reviewers identify the right places to start building the skills they need to be successful.
Additionally, building out the detailed behaviors by proficiency in different contexts provides information for potential reviewers to self-evaluate their readiness to review in different situations, potentially leading to a better prequalified pool of available reviewers. Having these established standards in place will also enable professors or managers to evaluate the readiness of others to review in different situations and nominate or encourage peers or subordinates to volunteer as reviewers where appropriate.
Context-relevant training with feedback mechanisms
In past years, Society for Industrial and Organizational Psychology (SIOP) has made comments available from other reviewers of the same conference submission once the review is submitted, which can help reviewers calibrate their own ratings and comments by seeing how they compare to those of others and learning how comments may be improved in the future. This concept could be extended by offering training webinars tailored to different review contexts with a pool of experienced reviewers in that context, with the training varying in intensity depending on the complexity and stakes of the context. Participants could review and rate competencies in real time and discuss in order to calibrate ratings, share tips and guidance on how to identify major flaws to remain thorough but not nitpicky, and complete reviews in the given context efficiently. In an educational setting, it was shown that systematic training alongside discussion and interaction with peers contributed to improved articles due to the opportunity for reviewers to learn from each other using practical examples and scoring rubrics (Sanchez-Naranjo, Reference Sanchez-Naranjo2019). Practical examples tailored by context of the review is likely to add to the value of this collaborative process.
Similarly, within the Reviewer Center on the SIOP website, example review comments for different types of conference submissions are provided to help reviewers understand what is expected of them when reviewing conference submissions. While this example is very helpful to clarify expectations for the reviewers, the value could be extended even further by also making excerpts of the original submissions available with annotations specifically demonstrating how the competencies in the proposed peer-review competency model can be applied and how the content in the submission is evaluated. These examples can then be mapped to competencies to provide examples of ways in which they are effectively applied (e.g., here are effective ways of demonstrating professional tone or of providing constructive, actionable feedback to the author). Another avenue for providing feedback is encouraging editors of journals and content owners in practice to provide reviewers with qualitative feedback, based on the proposed peer-review competency model. If the proposed peer-review competency model will establish the best practices for our professional community, it will be helpful to embed the framework into review practices as widely as possible.
Ideally professional communities such as SIOP will lead the charge in offering training to support peer review (Ford, Reference Ford2019). Once these types of context-specific training resources are created, it will be important to make them readily available given that a recent study (Solberg & Porr, Reference Solberg and Porr2019) showed that practitioners are often unaware of valuable resources on the SIOP website. In addition to making them easy to access, notifications from different outlets (e.g., email, social media, SIOP homepage) can increase awareness about their availability and value, increasing visibility of opportunities to review and different applications of these skills for industrial and organizational (I-O) psychologists in a variety of contexts.
Clarifying expectations to empower reviewers
We agree with Köhler et al.’s (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) suggestion that reviews are often unreliable and biased, and we believe that establishing a competency framework for reviewers is an important step for addressing this issue. Beyond providing development resources for reviewers, we suggest that increasing transparency (for both authors and reviewers) regarding the criteria by which work is being evaluated is another important step. If rubrics or checklists are implemented as standard practice, the consumer of the content (e.g., publication outlets, organizations, dissertation chairs) can clearly specify expectations for authors so that they can tailor their work accordingly, creating a better and more fair experience for the author. When the same rubric is used by the reviewer, there is direct alignment of expectations, improving the ethics and integrity of the process for all parties involved. A well-developed rubric or checklist that is aligned with the competency model as well as other essential components such as mechanics and formatting will provide objective criteria against which to evaluate the mechanics and the elements of the content, ensuring that all important areas are covered. Providing a rubric to reviewers is likely to reduce subjectivity of the review and provide a better experience for reviewers as well, who may otherwise spend a lot of time critiquing aspects of the work that are not ultimately important to the consumer of the content. The level of specificity in the rubric is likely to vary by context and purpose of the document.
While rubrics and checklists have a process orientation and can lead to high levels of specificity and detail for some practical contexts, in publishing some disciplines are considering the concept of transparency praxis, which stresses the importance of considering that values can inform practices in the peer-review process (Ford, Reference Ford2019). Transparency praxis takes a broader perspective about clearly communicating the “whys and hows” of the publishing process and the rights and responsibilities of reviewers, rather than detailing the minutiae in a rubric or checklist. The shift on the focus from process to praxis entails increasing clarity of communication and clearly articulating the rights and responsibilities of all stakeholders.
Empowering potential reviewers by clarifying expectations and providing the tools they need to have confidence in their abilities and equip themselves to develop into highly effective reviewers is one possible strategy for addressing the shortage of practitioner reviewers. There are likely many practical constraints impacting the prevalence of practitioner reviewers, as one study found that 57% of invited reviewers declined to participate because they were too busy (McNair, Phyong, Cseri, & Szekely, Reference McNair, Phyong, Cseri and Szekely2019). However, in addition to the time involved with preparing a quality review, we also believe that volunteering to be a reviewer can be daunting, especially for first-time or inexperienced reviewers. More practitioners may volunteer to serve as reviewers when they feel adequately trained and empowered to do so. Likewise, this strategy may also be useful to encourage academic reviewers to engage in business- or practical-oriented review activities.
In summary, peer review is an essential role for I-O psychologists. The competency framework for peer reviewers clearly defines the expectations for reviewers, and the next step in this important initiative is to focus on training and empowering reviewers by extending the resources available to help develop reviewers to perform within the expectations of this framework. A critical aspect of this is expanding the framework to include proficiency by context, supported by training and education (tailored by context) that will increase transparency of the process.
You have
Access
Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) present a compelling argument for training and developing reviewers based on an application of the article’s proposed competency model. In this response we highlight how such training and development could be enhanced by taking the context and purpose of the review into account. We offer suggestions in three areas: (1) leveling the competency framework by proficiency based on the context of the review, (2) providing context-specific training and feedback mechanisms, and (3) empowering reviewers with the confidence to review successfully by clarifying expectations and providing reviewing aids.
Level the competency framework by proficiency and context
Well-written competency models use observable behaviors that are succinct, distinct, and unambiguous to clearly define expectations for performance. As noted in the focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), peer reviewing has numerous applications, both in the context of scientific reviews for academic papers, journal publications, or conference presentations and in practical reviews in organizations where evaluating and offering suggestions for improving others’ work is essential to delivering high-quality work products. Because the purposes of reviews in these contexts and the associated consequences are very different, different reviewer skill levels and focuses are required. By leveling performance expectations for peer reviewers based on the context, Köhler et al.’s proposed competency framework could be more readily applied to a variety of reviewing scenarios.
In the academic context, varying levels of rigor are required by different outlets for publication. These outlets range from conference submissions to book chapters to dissertations to manuscripts for publication in a top-tier journal. The consequences of reviews vary by context as well. A rejection of a conference poster based on poorly executed reviews may disappoint a graduate student but not hinder his or her career. On the other hand, a rejection based on poorly executed reviews of a submission in an academic journal may make an academic’s quest for tenure more difficult. Likewise, in practical applications there are varying contexts of reviews ranging from internal communications to company white papers to proposals written to win new business to high-priority client deliverables. These different contexts also have different levels of consequences. A poorly prepared internal document could negatively impact the writer’s performance review but not damage a client relationship, whereas a flawed client-facing document could damage a company’s reputation and lead to the loss of business.
With these differences of context in mind, we provide some illustrative examples of how the competency requirements, and the weighted importance of each requirement in terms of overall review quality, may differ by context:
Representing: The expectations and interests may be more apparent for a journal compared to a practitioner delivering a work product to a client. Whereas journals typically make their mission, scope, and values public, the interests of a client may not be explicitly stated and may vary significantly by client.
Integrity: The aspect of this competency involving an evaluation of the hypotheses and methodological rigor in testing those hypotheses likely takes a different form for practitioner reviews, where there may not be hypotheses specified in advance for some documents (e.g., white papers, technical manuals). For the practitioner context, the rigor may focus instead on the correct application of established principles to the client’s situation. In addition, practical constraints may place limitations on the level of methodological rigor (e.g., smaller client sample sizes) than is typical for peer-reviewed journal publications.
Expertise: The focus of expertise may vary depending on what the reviewer needs to evaluate. An academic submission may require the reviewer to judge the comprehensiveness of the author’s review of the existing literature or judge the unique contributions of the article. A client report may have either a very limited or no literature review. If a document is prepared for a specific client, there may be less attention on the work’s contribution to the field or community. Therefore, some elements of the expertise competency would not be applicable in this instance.
Thoroughness: There may be differences in the level of detailed review required by context. For example, accuracy is extremely important in reporting client technical results, and some practitioner reviews might involve verifying the accuracy of every number reported in a table as a form of quality assurance for the client deliverable. This is not typically done for a journal review, where reviewers generally trust that the reported results have already been reviewed for accuracy.
Professionalism competencies: While academic reviews are generally blind, when a practitioner reviews a work product, he/she typically knows and works with the author. This means the constructiveness, tone, and open-mindedness of the feedback could have significant consequences for the relationship between the author and the reviewer.
For these reasons, we suggest an expansion of the model to clearly describe behaviors at the level of proficiency required for each competency in different contexts. Leveling the behaviors according to context can help clarify expectations for reviewers in different settings by providing information for aspiring reviewers on what good looks like in the context relevant to them. In addition, leveling also provides a blueprint for developing specific behaviors needed to be an effective reviewer in different contexts. The levels of the competency model could be extended to capture the most basic context such as providing informal feedback to others up to the most high-stakes rigorous type of review. Not all reviews are equal, and this additional detail of requirement at different levels will help reviewers identify the right places to start building the skills they need to be successful.
Additionally, building out the detailed behaviors by proficiency in different contexts provides information for potential reviewers to self-evaluate their readiness to review in different situations, potentially leading to a better prequalified pool of available reviewers. Having these established standards in place will also enable professors or managers to evaluate the readiness of others to review in different situations and nominate or encourage peers or subordinates to volunteer as reviewers where appropriate.
Context-relevant training with feedback mechanisms
In past years, Society for Industrial and Organizational Psychology (SIOP) has made comments available from other reviewers of the same conference submission once the review is submitted, which can help reviewers calibrate their own ratings and comments by seeing how they compare to those of others and learning how comments may be improved in the future. This concept could be extended by offering training webinars tailored to different review contexts with a pool of experienced reviewers in that context, with the training varying in intensity depending on the complexity and stakes of the context. Participants could review and rate competencies in real time and discuss in order to calibrate ratings, share tips and guidance on how to identify major flaws to remain thorough but not nitpicky, and complete reviews in the given context efficiently. In an educational setting, it was shown that systematic training alongside discussion and interaction with peers contributed to improved articles due to the opportunity for reviewers to learn from each other using practical examples and scoring rubrics (Sanchez-Naranjo, Reference Sanchez-Naranjo2019). Practical examples tailored by context of the review is likely to add to the value of this collaborative process.
Similarly, within the Reviewer Center on the SIOP website, example review comments for different types of conference submissions are provided to help reviewers understand what is expected of them when reviewing conference submissions. While this example is very helpful to clarify expectations for the reviewers, the value could be extended even further by also making excerpts of the original submissions available with annotations specifically demonstrating how the competencies in the proposed peer-review competency model can be applied and how the content in the submission is evaluated. These examples can then be mapped to competencies to provide examples of ways in which they are effectively applied (e.g., here are effective ways of demonstrating professional tone or of providing constructive, actionable feedback to the author). Another avenue for providing feedback is encouraging editors of journals and content owners in practice to provide reviewers with qualitative feedback, based on the proposed peer-review competency model. If the proposed peer-review competency model will establish the best practices for our professional community, it will be helpful to embed the framework into review practices as widely as possible.
Ideally professional communities such as SIOP will lead the charge in offering training to support peer review (Ford, Reference Ford2019). Once these types of context-specific training resources are created, it will be important to make them readily available given that a recent study (Solberg & Porr, Reference Solberg and Porr2019) showed that practitioners are often unaware of valuable resources on the SIOP website. In addition to making them easy to access, notifications from different outlets (e.g., email, social media, SIOP homepage) can increase awareness about their availability and value, increasing visibility of opportunities to review and different applications of these skills for industrial and organizational (I-O) psychologists in a variety of contexts.
Clarifying expectations to empower reviewers
We agree with Köhler et al.’s (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) suggestion that reviews are often unreliable and biased, and we believe that establishing a competency framework for reviewers is an important step for addressing this issue. Beyond providing development resources for reviewers, we suggest that increasing transparency (for both authors and reviewers) regarding the criteria by which work is being evaluated is another important step. If rubrics or checklists are implemented as standard practice, the consumer of the content (e.g., publication outlets, organizations, dissertation chairs) can clearly specify expectations for authors so that they can tailor their work accordingly, creating a better and more fair experience for the author. When the same rubric is used by the reviewer, there is direct alignment of expectations, improving the ethics and integrity of the process for all parties involved. A well-developed rubric or checklist that is aligned with the competency model as well as other essential components such as mechanics and formatting will provide objective criteria against which to evaluate the mechanics and the elements of the content, ensuring that all important areas are covered. Providing a rubric to reviewers is likely to reduce subjectivity of the review and provide a better experience for reviewers as well, who may otherwise spend a lot of time critiquing aspects of the work that are not ultimately important to the consumer of the content. The level of specificity in the rubric is likely to vary by context and purpose of the document.
While rubrics and checklists have a process orientation and can lead to high levels of specificity and detail for some practical contexts, in publishing some disciplines are considering the concept of transparency praxis, which stresses the importance of considering that values can inform practices in the peer-review process (Ford, Reference Ford2019). Transparency praxis takes a broader perspective about clearly communicating the “whys and hows” of the publishing process and the rights and responsibilities of reviewers, rather than detailing the minutiae in a rubric or checklist. The shift on the focus from process to praxis entails increasing clarity of communication and clearly articulating the rights and responsibilities of all stakeholders.
Empowering potential reviewers by clarifying expectations and providing the tools they need to have confidence in their abilities and equip themselves to develop into highly effective reviewers is one possible strategy for addressing the shortage of practitioner reviewers. There are likely many practical constraints impacting the prevalence of practitioner reviewers, as one study found that 57% of invited reviewers declined to participate because they were too busy (McNair, Phyong, Cseri, & Szekely, Reference McNair, Phyong, Cseri and Szekely2019). However, in addition to the time involved with preparing a quality review, we also believe that volunteering to be a reviewer can be daunting, especially for first-time or inexperienced reviewers. More practitioners may volunteer to serve as reviewers when they feel adequately trained and empowered to do so. Likewise, this strategy may also be useful to encourage academic reviewers to engage in business- or practical-oriented review activities.
In summary, peer review is an essential role for I-O psychologists. The competency framework for peer reviewers clearly defines the expectations for reviewers, and the next step in this important initiative is to focus on training and empowering reviewers by extending the resources available to help develop reviewers to perform within the expectations of this framework. A critical aspect of this is expanding the framework to include proficiency by context, supported by training and education (tailored by context) that will increase transparency of the process.