In their focal article, Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) propose a competency framework for peer review, arguing that such a framework clarifies expectations toward reviewers and can be used to train reviewers, as well as to evaluate the quality and tone of reviews. As peer review is the cornerstone of the academic publishing system, we are delighted to see that more attention is being paid to issues aimed at professionalizing this activity. In this sense, we feel that the development of a competency framework is an important and crucial step. By the same token, however, such a competency framework should not be an end goal in itself but should be part of a larger, more comprehensive professionalization exercise. In such an exercise, a focus beyond the individual reviewer is needed because even the most competent reviewer will be unable to deliver high-quality reviews when the circumstances do not allow for the expression of these competencies. For example, unrealistic reviewing deadlines might severely undermine the thoroughness and comprehensiveness of the review, even in the case of a very conscientious and thorough reviewer. Drawing on this awareness, we call for complementing the focus on competencies of the individual reviewer with a focus on situational features that allow for the expression of those competencies.
In our discussion of situational features, we will draw on a well-known theoretical model that is specifically designed to better understand under which conditions traits result in the expression of trait-relevant behavior: trait activation theory (TAT; Tett & Burnett, Reference Tett and Burnett2003). According to TAT, traits only manifest as trait-relevant behaviors when the situation contains trait-relevant cues. This feature is referred to as situation trait relevance, which means that the situation should be thematically relevant to the trait in order for the trait to express itself. For example, a review process adopting a standardized, well-structured review format with comment boxes for each section of the article would speak to reviewers’ thoroughness and clarity. Whereas some situations might stimulate the expression of a trait, which was the case in our previous example, other situations might do the exact opposite and restrict relevant cues for trait expression, thus acting as a constraint. For example, stressing the importance of a very short turnaround time might put heavy time pressure on reviewers, thereby compromising thoroughness and clarity of the review. A second relevant feature in TAT is situation strength, which refers to the extent the situation provides cues for appropriate behavior. Such cues might come from the situation itself or from rewards associated with the situation. In weak trait-relevant situations—that is, when strong situational cues are lacking and extrinsic rewards are modest—the largest individual differences in trait-relevant behavior will be observed. In strong situations, however, individual differences in trait-relevant behavior will be smaller because the situation or the rewards associated with the behavior dictate how people should behave. For example, if reviewers knew that editors would thoroughly evaluate and rate their reviews on a public platform, they would be more inclined to work in an honorable, constructive, and thorough way.
Drawing on TAT, we will discuss five situational features that we believe are important for the expression of the competences discussed by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020). Two of those features stimulate the elicitation of trait-relevant behavior (i.e., structuring the review process and signing the review), two features constrain the elicitation of trait-relevant behavior (i.e., short review deadlines and access to material), and the final feature pertains to rewards associated with reviewing. Our overview is by no means meant to be exhaustive. We rather want to illustrate how looking at the reviewing process and reviewer competences through the lens of TAT might inspire us on how the process itself might need to be adapted to allow for a maximal expression of those competences. Moreover, because of space constraints, we focus on academic reviewing in our commentary article. However, the idea that the situation plays a role in the expression of the competences, and even some of our recommendations, are directly transferable to other contexts, such as peer review in organizations, universities, and classrooms.
Structuring the review process
Journals vary widely in the extent to which they structure the reviewing process. Some journals simply ask reviewers to write a review report without explicitly mentioning the topics that should be covered in the report. Other journals combine such an unstructured review process with a scoring system in which reviewers are asked to score the manuscript on a number of criteria (e.g., contribution, relevance, quality), typically using rating scales. Finally, a small number of journals use a tightly structured review process, in which reviewers are asked to give feedback on various predefined aspects of the manuscript. For example, the journal Frontiers uses such a system that asks reviewers to give both general feedback on strengths and limitations of the article and also specific feedback on particular aspects such as methods, results, and data interpretation.
According to TAT, the first situation (i.e., not explicitly mentioning which topics/themes should be covered) does not offer a lot of cues for appropriate behavior. In such a situation, whether the review will be thorough, clear, constructive, et cetera will largely depend on the competencies of the reviewer. That is, because in such system it is perfectly possible for a reviewer to focus on one aspect of the article (e.g., methods) without providing comments on the other parts of the article, the thoroughness of the review will very much depend on the thoroughness of the reviewer. Structuring the review process might provide a solution for this issue because by conveying which parts should be present in the review, the review system explicitly tells the reviewer which behavior is expected. In such strong situation, TAT holds that the behavior of the reviewers will depend less on reviewer idiosyncrasies, thereby making the reviews more homogeneous with regard to thoroughness.
Signing the review
In the field of industrial and organizational (I-O) psychology, double-blind review processes are the norm, meaning that reviewers and authors are blind to each other’s identity. However, when looking at the integrity, constructiveness, tone, and even thoroughness competences, such a process qualifies as a weak situation. Because the reviewer cannot be identified by the authors, there is little external motivation to behave in a kind, constructive, and thorough way, which implies that the tone and thoroughness of the review will very much depend on the extent to which the reviewer him/herself is considerate, constructive, and thorough. In their focal article, Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) briefly touch upon this issue, mentioning that reviewers should perform the review as if they were signing it. An alternative, and one that would strengthen the situation, would be to ask reviewers to disclose their identity to the authors. Although we are aware that this might be problematic for other reasons (e.g., fear of retaliation), the requirement to disclose would create a strong situation with regard to tone and thoroughness, causing the tone and thoroughness of the review to be less determined by individual differences in reviewer attitudes and skills, which would make for a more homogenous, high-quality review process. It is interesting to note that journals such as BMJ Open even go one step further by publishing the reviewers’ comments alongside the authors’ replies to those comments. This arguably serves as an extremely strong situation, and in such situation few reviewers will violate the norms of integrity, constructiveness, tone, and even thoroughness.
Review deadlines
During the last decade, journals have heavily invested in streamlining their publication processes. An important key performance indicator in the streamlining process is the time it takes the editor to make the first editorial decision on the article and the time to arrive at the final decision. Journals typically want to keep the turnaround time as short as possible because it makes their journal attractive for authors. The consequence is that review deadlines can be very sharp, with some journals expecting a review report already one week after accepting the review invitation. In addition, journals typically monitor those deadlines closely, sending regular reminders even before the deadlines have expired.
Such tight deadlines put a lot of pressure on reviewers, and, from a TAT perspective, this situational feature can be considered a constraint in the sense that it counteracts the expression of trait-relevant behavior, and particularly thoroughness, clarity, efficiency, and open-mindedness. This is supported by research which shows that under high time pressure, people do not think things through but rather engage in shallow information processing and inferior decision making, a phenomenon that is referred to as “closing of the mind” (Kruglanski & Freund, Reference Kruglanski and Freund1983). Thus, imposing tight deadlines on researchers might lead to low quality reviews because they oppose open-mindedness, particularly when those deadlines lead to perceptions of high time pressure.
Access to materials
As Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) rightfully argue, reviews should ideally cover all sections of the work, including the literature review, contributions made by the research, appropriateness of the methods, and accuracy of the findings. However, in a typical review process, reviewers only have access to the end product of the scientific process (i.e., the submitted manuscript). By giving them access to only the front end and not the back end of the work (such as raw data, analysis scripts, and materials such as questionnaires), reviewers might have a hard time putting their expertise and thoroughness to work. In that sense, limited access to materials might serve as a constraint for the expression of expertise and thoroughness. Therefore, asking authors to upload such material and giving reviewers access to it (e.g., via the Open Science Framework) would enable a better and more thorough review.
Rewards for reviewing
Academic peer reviewing is typically considered a volunteer service to academia. Reviewers do not get paid for performing reviews. Moreover, the importance of peer reviewing in one’s academic CV is usually rather low, if not near zero. Finally, reviewers rarely get feedback on their reviews. As these examples show, there are few extrinsic rewards associated with academic reviewing. In the absence of such rewards, motivational cues to behave in the desired way are lacking, and therefore the quality and tone of the review depend largely on the competencies of the individual reviewer.
According to TAT, providing extrinsic rewards might be one way to motivate reviewers to behave in a similar and professional way, regardless of their individual differences. Although being in the minority, some journals already offer extrinsic rewards to reviewers by for example publishing the reviewers’ names and affiliations along with the articles (e.g., BMJ Open), by nominating people for best reviewer awards (e.g., Academy of Management), or by allowing reviewers to decide a portion of the revenue a journal earns (e.g., Collabra: Psychology). However, ultimately, we feel that the strongest incentive would be that reviewing would be explicitly acknowledged and valued as part of one’s academic track record. Although recently online platforms have appeared that allow reviewers to record their review activity, thereby allowing them to use this as measurable research output (e.g., Publons), motivational cues to behave in the desired way will be low as long as universities, departments, funding agencies, and promotion and selection committees do not explicitly value this output.
Concluding thoughts
In our commentary, we argue that the creation of a competency framework for reviewers as suggested by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) triggers an associated question about how the expression of these competences can be facilitated by the reviewing system itself. Drawing on TAT, we argue that academia in general and publishers in particular might do well to rethink the reviewing system in such way that it maximally provides cues for the expression of the competencies identified by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020). Although we realize that some of the situational features discussed here might have consequences that go beyond the proposed competency framework (e.g., the problem of signing one’s review for early career researchers), we hope that by providing a theoretical framework that speaks to how situations can be relevant to the expression of competencies, we can contribute not only to enhancing the quality and professionalization of peer reviewing, but also to making peer review a more satisfying experience for the reviewers themselves.
You have
Access
In their focal article, Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) propose a competency framework for peer review, arguing that such a framework clarifies expectations toward reviewers and can be used to train reviewers, as well as to evaluate the quality and tone of reviews. As peer review is the cornerstone of the academic publishing system, we are delighted to see that more attention is being paid to issues aimed at professionalizing this activity. In this sense, we feel that the development of a competency framework is an important and crucial step. By the same token, however, such a competency framework should not be an end goal in itself but should be part of a larger, more comprehensive professionalization exercise. In such an exercise, a focus beyond the individual reviewer is needed because even the most competent reviewer will be unable to deliver high-quality reviews when the circumstances do not allow for the expression of these competencies. For example, unrealistic reviewing deadlines might severely undermine the thoroughness and comprehensiveness of the review, even in the case of a very conscientious and thorough reviewer. Drawing on this awareness, we call for complementing the focus on competencies of the individual reviewer with a focus on situational features that allow for the expression of those competencies.
In our discussion of situational features, we will draw on a well-known theoretical model that is specifically designed to better understand under which conditions traits result in the expression of trait-relevant behavior: trait activation theory (TAT; Tett & Burnett, Reference Tett and Burnett2003). According to TAT, traits only manifest as trait-relevant behaviors when the situation contains trait-relevant cues. This feature is referred to as situation trait relevance, which means that the situation should be thematically relevant to the trait in order for the trait to express itself. For example, a review process adopting a standardized, well-structured review format with comment boxes for each section of the article would speak to reviewers’ thoroughness and clarity. Whereas some situations might stimulate the expression of a trait, which was the case in our previous example, other situations might do the exact opposite and restrict relevant cues for trait expression, thus acting as a constraint. For example, stressing the importance of a very short turnaround time might put heavy time pressure on reviewers, thereby compromising thoroughness and clarity of the review. A second relevant feature in TAT is situation strength, which refers to the extent the situation provides cues for appropriate behavior. Such cues might come from the situation itself or from rewards associated with the situation. In weak trait-relevant situations—that is, when strong situational cues are lacking and extrinsic rewards are modest—the largest individual differences in trait-relevant behavior will be observed. In strong situations, however, individual differences in trait-relevant behavior will be smaller because the situation or the rewards associated with the behavior dictate how people should behave. For example, if reviewers knew that editors would thoroughly evaluate and rate their reviews on a public platform, they would be more inclined to work in an honorable, constructive, and thorough way.
Drawing on TAT, we will discuss five situational features that we believe are important for the expression of the competences discussed by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020). Two of those features stimulate the elicitation of trait-relevant behavior (i.e., structuring the review process and signing the review), two features constrain the elicitation of trait-relevant behavior (i.e., short review deadlines and access to material), and the final feature pertains to rewards associated with reviewing. Our overview is by no means meant to be exhaustive. We rather want to illustrate how looking at the reviewing process and reviewer competences through the lens of TAT might inspire us on how the process itself might need to be adapted to allow for a maximal expression of those competences. Moreover, because of space constraints, we focus on academic reviewing in our commentary article. However, the idea that the situation plays a role in the expression of the competences, and even some of our recommendations, are directly transferable to other contexts, such as peer review in organizations, universities, and classrooms.
Structuring the review process
Journals vary widely in the extent to which they structure the reviewing process. Some journals simply ask reviewers to write a review report without explicitly mentioning the topics that should be covered in the report. Other journals combine such an unstructured review process with a scoring system in which reviewers are asked to score the manuscript on a number of criteria (e.g., contribution, relevance, quality), typically using rating scales. Finally, a small number of journals use a tightly structured review process, in which reviewers are asked to give feedback on various predefined aspects of the manuscript. For example, the journal Frontiers uses such a system that asks reviewers to give both general feedback on strengths and limitations of the article and also specific feedback on particular aspects such as methods, results, and data interpretation.
According to TAT, the first situation (i.e., not explicitly mentioning which topics/themes should be covered) does not offer a lot of cues for appropriate behavior. In such a situation, whether the review will be thorough, clear, constructive, et cetera will largely depend on the competencies of the reviewer. That is, because in such system it is perfectly possible for a reviewer to focus on one aspect of the article (e.g., methods) without providing comments on the other parts of the article, the thoroughness of the review will very much depend on the thoroughness of the reviewer. Structuring the review process might provide a solution for this issue because by conveying which parts should be present in the review, the review system explicitly tells the reviewer which behavior is expected. In such strong situation, TAT holds that the behavior of the reviewers will depend less on reviewer idiosyncrasies, thereby making the reviews more homogeneous with regard to thoroughness.
Signing the review
In the field of industrial and organizational (I-O) psychology, double-blind review processes are the norm, meaning that reviewers and authors are blind to each other’s identity. However, when looking at the integrity, constructiveness, tone, and even thoroughness competences, such a process qualifies as a weak situation. Because the reviewer cannot be identified by the authors, there is little external motivation to behave in a kind, constructive, and thorough way, which implies that the tone and thoroughness of the review will very much depend on the extent to which the reviewer him/herself is considerate, constructive, and thorough. In their focal article, Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) briefly touch upon this issue, mentioning that reviewers should perform the review as if they were signing it. An alternative, and one that would strengthen the situation, would be to ask reviewers to disclose their identity to the authors. Although we are aware that this might be problematic for other reasons (e.g., fear of retaliation), the requirement to disclose would create a strong situation with regard to tone and thoroughness, causing the tone and thoroughness of the review to be less determined by individual differences in reviewer attitudes and skills, which would make for a more homogenous, high-quality review process. It is interesting to note that journals such as BMJ Open even go one step further by publishing the reviewers’ comments alongside the authors’ replies to those comments. This arguably serves as an extremely strong situation, and in such situation few reviewers will violate the norms of integrity, constructiveness, tone, and even thoroughness.
Review deadlines
During the last decade, journals have heavily invested in streamlining their publication processes. An important key performance indicator in the streamlining process is the time it takes the editor to make the first editorial decision on the article and the time to arrive at the final decision. Journals typically want to keep the turnaround time as short as possible because it makes their journal attractive for authors. The consequence is that review deadlines can be very sharp, with some journals expecting a review report already one week after accepting the review invitation. In addition, journals typically monitor those deadlines closely, sending regular reminders even before the deadlines have expired.
Such tight deadlines put a lot of pressure on reviewers, and, from a TAT perspective, this situational feature can be considered a constraint in the sense that it counteracts the expression of trait-relevant behavior, and particularly thoroughness, clarity, efficiency, and open-mindedness. This is supported by research which shows that under high time pressure, people do not think things through but rather engage in shallow information processing and inferior decision making, a phenomenon that is referred to as “closing of the mind” (Kruglanski & Freund, Reference Kruglanski and Freund1983). Thus, imposing tight deadlines on researchers might lead to low quality reviews because they oppose open-mindedness, particularly when those deadlines lead to perceptions of high time pressure.
Access to materials
As Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) rightfully argue, reviews should ideally cover all sections of the work, including the literature review, contributions made by the research, appropriateness of the methods, and accuracy of the findings. However, in a typical review process, reviewers only have access to the end product of the scientific process (i.e., the submitted manuscript). By giving them access to only the front end and not the back end of the work (such as raw data, analysis scripts, and materials such as questionnaires), reviewers might have a hard time putting their expertise and thoroughness to work. In that sense, limited access to materials might serve as a constraint for the expression of expertise and thoroughness. Therefore, asking authors to upload such material and giving reviewers access to it (e.g., via the Open Science Framework) would enable a better and more thorough review.
Rewards for reviewing
Academic peer reviewing is typically considered a volunteer service to academia. Reviewers do not get paid for performing reviews. Moreover, the importance of peer reviewing in one’s academic CV is usually rather low, if not near zero. Finally, reviewers rarely get feedback on their reviews. As these examples show, there are few extrinsic rewards associated with academic reviewing. In the absence of such rewards, motivational cues to behave in the desired way are lacking, and therefore the quality and tone of the review depend largely on the competencies of the individual reviewer.
According to TAT, providing extrinsic rewards might be one way to motivate reviewers to behave in a similar and professional way, regardless of their individual differences. Although being in the minority, some journals already offer extrinsic rewards to reviewers by for example publishing the reviewers’ names and affiliations along with the articles (e.g., BMJ Open), by nominating people for best reviewer awards (e.g., Academy of Management), or by allowing reviewers to decide a portion of the revenue a journal earns (e.g., Collabra: Psychology). However, ultimately, we feel that the strongest incentive would be that reviewing would be explicitly acknowledged and valued as part of one’s academic track record. Although recently online platforms have appeared that allow reviewers to record their review activity, thereby allowing them to use this as measurable research output (e.g., Publons), motivational cues to behave in the desired way will be low as long as universities, departments, funding agencies, and promotion and selection committees do not explicitly value this output.
Concluding thoughts
In our commentary, we argue that the creation of a competency framework for reviewers as suggested by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) triggers an associated question about how the expression of these competences can be facilitated by the reviewing system itself. Drawing on TAT, we argue that academia in general and publishers in particular might do well to rethink the reviewing system in such way that it maximally provides cues for the expression of the competencies identified by Köhler et al. (Reference Köhler, González‐Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020). Although we realize that some of the situational features discussed here might have consequences that go beyond the proposed competency framework (e.g., the problem of signing one’s review for early career researchers), we hope that by providing a theoretical framework that speaks to how situations can be relevant to the expression of competencies, we can contribute not only to enhancing the quality and professionalization of peer reviewing, but also to making peer review a more satisfying experience for the reviewers themselves.