In a recent IOP focal paper, Grand, Rogelberg, Allen et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) laid out six defining principles for fostering robust science in industrial and organizational (I-O) psychology. These are relevance, rigor, replication, accumulation and cumulation, transparency and openness, and theory orientation. As Grand, Rogelberg, Allen et al. further explain, upholding these principles and incorporating them in the scientific process is not the sole responsibility of the producers of research (e.g., authors, academics, consultants). Rather, the authors identify several other stakeholders who together need to practice applying these defining principles to create and contribute to a robust science. In this focal article, we want to focus on the role of reviewers.
Peer review fulfills a crucial role in creating a robust science based on the six defining principles that Grand, Rogelberg, Allen et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) outlined. Reviewers are responsible for judging the quality of research conducted and submitted for evaluation. They are instrumental in ensuring that research is relevant, conducted rigorously, and that it is transparent and theory-oriented. Furthermore, they are responsible for treating authors and their work with respect, in a supporting and developmental manner. We fully agree with this view.
However, we would like to go a step further and declare peer reviewing to be the cornerstone of our profession, on which we need to build all other endeavors. Peer review is pervasive throughout our profession. While many of us may immediately and exclusively think of the peer review process involved in publishing, peer review exists in organizations, university departments, classrooms, and many other work contexts. Colleagues may review each other’s work on a quarterly report or on a consulting bid. Course instructors may incorporate peer grading into their classes. Government funding agencies review grant proposals. Academics review each other’s work to determine promotion or hiring decisions. And as Grand, Rogelberg, Allen et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) suggest, it is an essential mechanism for upholding the standards of our profession and for creating a robust science.
But more than that, the role of peer review goes beyond its important contribution to the creation of the best possible research design or the most appropriate execution of research. Peer review is how we ensure the overall quality of our work, how we support each other, how we can push and constructively challenge each other to be truly innovative, how we can make sure that our work connects with and is relevant for others, and how we can drive impact and change that are appropriately targeted at the needs of different communities. Good peer reviewing ensures that the proposed work appropriately extends and tests prior work, but also that our application of research in practice is relevant and rigorous. An excellent peer review can help authors to challenge their more conventional thinking to achieve new insights or to refine their arguments and evidence base. Overall, peer review is essential in testing our ideas and ensuring that our subsequent efforts are well-directed.
Given its central role in our profession, it is curious that we do not have formalized review guidelines or standards and that most of us never receive formal training in peer reviewing. Consequently, we have seen many discussions in the academic literature and at professional conferences that bemoan the lack of quality and consistency of peer review. Some evidence exists that bad peer reviewing can contribute to detrimental research practices, such as encouraging authors to engage in hypothesizing after the results are known (HARKing) or dropping null results (Banks, Rogelberg, Woznyj, Landis, & Rupp, Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016b; Bedeian, Reference Bedeian2003; Emerson et al., Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010). In addition, bad peer reviews can have negative consequences for our careers. Therefore, as a profession we need to take steps to ensure the quality of the peer review process so that it can advance our science.
To support this endeavor, we are proposing a competency framework for peer review. The purpose of the competency framework is to provide a definition of excellent peer reviewing and guidelines to reviewers for which types of behaviors will lead to good peer reviews. By defining these competencies, we create clarity around expectations for peer review, standards for good peer reviews, and opportunities for training of the behaviors that are required to deliver good peer reviews.
In the following, we discuss in more detail the relevance of peer review for our profession as well as the issues and concerns raised about its reliability, rigor, and robustness. We then introduce the proposed framework and its competencies and describe how it can be used to improve peer reviewing. Finally, we discuss additional steps forward that involve suggestions for how several of the other stakeholders defined by Grand, Rogelberg, Allen et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) can get involved in fostering high-quality peer reviewing.
We hope that our focal article elicits commentaries from a diverse group of members of our profession. As part of our ongoing subcommittee work for the Society for Industrial and Organizational Psychology (SIOP), we seek to maximize the utility of the competency framework so that it can be used widely. As such, we hope to receive feedback on the competency framework from community members with a diverse range of positions and responsibilities. Furthermore, we would like to see academics and practitioners expand on our work to suggest how the framework can be used to train peer reviewers and facilitate a more useful, rigorous, and relevant peer review process. We acknowledge that the peer review process is embedded in a larger professional system, and as such, is likely not the only thing that needs to be altered to ensure a more robust science. Consequently, we also hope to see additional suggestions for ways forward or a discussion of potential obstacles that may hinder the improvement of the peer review process. We look forward to a lively debate.
The relevance of peer review as a cornerstone of our profession
Purpose and benefits of peer review
In this section, we discuss the purpose and benefits of peer review, especially in I-O psychology. We focus largely on the perspective of individual reviewers, including reviewers of academic articles (e.g., editorial board members, ad hoc reviewers, conference reviewers), reviewers of other research products (e.g., grant proposals, books, colleagues’ manuscripts), and reviewers of work products (e.g., white papers, organizational intellectual property, thought leadership pieces, consulting reports). Developmental reviews occur formally and informally when a colleague reads a piece of work (e.g., paper, grant proposal, class assignment, white paper, client deliverable) and provides constructive feedback during a review process to help cultivate and improve the work (Rupp, Reference Rupp2011). There are three key benefits to the peer-review process.
First, peer review serves to develop authors and ideas (Lee, Reference Lee1995). Many great thinkers and scientists had a reviewer or someone who critiqued their work and helped them shape it. A famous example of such a developmental review process includes the work of the Indian mathematician Srinivasa Ramanujan who corresponded with and sought feedback on his work from Godfrey Hardy in England in the early 1900s (Kanigel, Reference Kanigel2016). While Hardy acknowledged that Ramanujan was the better mathematician and was the source of many groundbreaking theorems, Hardy provided mentorship on how to create formal proofs for his ideas. As another example, Albert Einstein, Maurice Solovine, and Conrad Habicht formed a book discussion group known as the Olympia Academy in 1902 in order to discuss books on mathematics, science, and philosophy, which helped cultivate Einstein’s later scientific ideas (Highfield & Carter, Reference Highfield and Carter1994). Influential artists and writers, such as Pablo Picasso, Henri Matisse, F. Scott Fitzgerald, James Joyce, and Ernest Hemingway, gathered on Saturday nights in Gertrude Stein’s “Salon” on 27 rue de Fleurus in Paris to discuss ideas, critique each other’s work, network, and receive mentorship. Finally, Marty Ginsburg’s suggestion of the particular tax court case that then became their test case for gender-based discrimination, Moritz v. Commissioner of Internal Revenue, provided Ruth Bader Ginsburg with the opportunity to argue in front of a court that gender-based discrimination is unconstitutional (Smithsonian, 2019). Old debating societies, such as the Royal Academy, the Socrates School, or the Stratford-on-Odeon existed for a similar purpose, where theories and ideas were hotly debated in order to arrive at the best and most robust ones. Illustrative examples aside, developmental peer review has existed and been central to the scientific process for more than 300 years (Weller, Reference Weller2001).
Second, the peer review process can be intrinsically and extrinsically rewarding for reviewers. Annesley (Reference Annesley2012) suggests that those who serve as reviewers have the opportunity to continue gaining expertise in their field as well as partake in a process in which they can learn what constitutes rigorous research (or what fails to meet such expectations) and a significant theoretical, empirical, and/or practical contribution. As part of this process, reviewers can learn from other authors’ mistakes, which can further improve their own understanding and work quality. As a result, their own thought process, ideas, and work product may be developed and advanced more profoundly. Further, serving on a review board of any kind (e.g., a journal’s editorial board, a selection or hiring committee, the awards panel of a grant agency, or the judging panel of a consulting bid) is also recognized as a prestigious service role for one’s field, symbolizes one’s expertise on a given topic, and can provide some extrinsic benefits.
Third, peer review functions as a “vetting” process (Carpenter, Reference Carpenter2009) and consequently promotes the dissemination of high quality, relevant, and rigorous work products. Reviewers help evaluate what constitutes a theoretical, practical, or methodological contribution in research or a valuable and useful practical work product. Reviewers also help to ensure that rigorous research methods are used and transparently reported (Aguinis, Ramani, & Alabduljader, Reference Aguinis, Ramani and Alabduljader2018). In many ways, reviewers serve as a first line of defense against unethical or detrimental practices (Rupp, Reference Rupp2011). In addition, reviewers reinforce the mission and scope of the journal, funding scheme, or organization by evaluating the proposed work against a mission statement or set of defined organizational guidelines and standards. Having outlined the significant benefits, we now turn our attention to the issues previously identified with the existing peer review system.
Issues with the current peer review system
The peer review system is not without its detractors, many of whom raise quite valid concerns regarding its shortcomings for the social sciences in particular (e.g., Ioannidis, Reference Ioannidis2005; Kepes & McDaniel, Reference Kepes and McDaniel2013; Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012). These concerns include the rigor of the process (e.g., low interrater reliability, lack of formal standards or guidance), the encouragement of questionable research practices (e.g., rose-tinting results, dropping null results), the impinging of academic freedom (e.g., reviewers overstepping their roles and effectively ghost authoring papers), and the tone of feedback (e.g., the role anonymity plays in adopting an uncivil tone). Rather than take the Churchillian stance that peer review is the worst form of gatekeeping except for all the others that have been tried, we review the veracity of these concerns and their mitigation within and beyond the current model of knowledge creation and dissemination.
Reviews are unreliable
A frequent criticism of peer review is that it is not reliable (i.e., reviews may be inconsistent and lack agreement between reviewers). The lack of reliability is not in and of itself worrisome as it sometimes occurs for valid reasons. For example, decision-makers (i.e., editors, grant agencies, clients, employers) may intentionally select reviewers with diverging views in order to gain multiple perspectives. Similarly, reviewers with different areas of expertise may evaluate different aspects of the work. Reviewers for different journals may review the work with a slightly different set of criteria in mind, encapsulating the specific mission and scope of the journal or the strategic direction the editor has defined for the journal. In addition, before reviewers get to assess a piece of work, a pre-selection of work eligible for review may have already occurred, resulting, for example, in an assessment of work that meets some basic criteria.
However, low interrater reliability can also be caused by low-quality reviews (Starbuck, Reference Starbuck2003). For example, a disorganized review (i.e., consisting of generic criticisms of contribution and interestingness coupled with specific, but superficial, criticisms of typos and formatting) likely aligns poorly with a high-quality review or even another low-quality review focusing on a different, but equally arbitrary and superficial, set of complaints. Low-quality reviews can give authors the impression that the process is fraught with idiosyncrasies and random chance—an impression not entirely without merit. Numerous studies on the lack of quality in journal reviews have found reviewer agreement regarding specific aspects of the article such as the soundness of the methods and theoretical contribution as well as overall recommendations to be considerably lower than traditional thresholds of agreement and consistency (Cicchetti, Reference Cicchetti1980; Miller, Reference Miller2006; Scott, Reference Scott1974; Starbuck, Reference Starbuck2003).
The quality of a review depends on the knowledge, skills, ability, motivation, and professionalism of the reviewer. Regarding ability, the near total lack of formal peer review standards, guidelines, and training is a likely cause of the lack of skill mastery and detrimental reviewing practices. In addition, actual expertise in the topic, research method, and methodology are indispensable. Without it, reviews are likely to become a set of recommendations based on personal preferences or individual practices rather than an expression of applying superior research standards. A novice reviewer may learn the mechanical aspects of how to write a review by modeling their own reviews after those they received as an author. This can work as long as most reviews are of sufficient quality and as the novice correctly interprets the relevant points, tone, and boundaries. However, if the population of reviews is tainted by a non-negligible proportion of low-expertise, low-quality reviews or the novice misinterprets some crucial element of reviewing (e.g., “it is the reviewers job to criticize the article rather than identify ways to make it stronger”), then what is being modeled by novice reviewers are those practices that are likely to contribute to low interrater agreement.
As previously noted when describing the benefits for reviewers, reviewing can tap into both intrinsic and extrinsic aspects of motivation. Ideally, most of those asked to review do so out of a desire to serve the field, and if so, then the intrinsic rewards may be sufficient to provide the necessary motivation to conduct a high-quality evaluation of a manuscript. However, a desire to serve is not a universal sentiment, and there are very few extrinsic rewards associated with reviewing to supplement the pool of intrinsically motivated reviewers. Conference or field recognition, editorial board invitations, and even a sense of quid pro quo (i.e., giving back to the scholarly or professional community) may not be salient enough to elicit the necessary motivation to devote one’s time and resources to doing a quality review when not adequately intrinsically motivated. We will return to this point later to suggest how intrinsic and extrinsic motivation may be increased.
Reviews are biased and encourage bad science
Another frequent criticism is that the peer review process is susceptible to several common biases. Bias operates on multiple levels. First, there is a bias toward certain types of results, and some sub-optimal methodologies may serve as means to achieve desired results (Smaldino & McElreath, Reference Smaldino and McElreath2016). Novel findings, findings that support one’s own beliefs, findings that achieve some threshold of acceptability (e.g., statistical significance), or findings that support the desired direction a client would like to take are likely to be more acceptable than the alternative. At the same time, findings that contradict commonly held beliefs or established theories are often rejected by reviewers who then may question perfectly appropriate and rigorous methodological approaches to rationalize why the findings simply cannot be valid (e.g., Chan & Arvey, Reference Chan and Arvey2012). If reviewers are judging the rigor and relevance of the theory and methods of a piece of work based on the acceptability of its results, then it is the ends that are rewarded. Consequently, the means are ignored, which may allow for methodological shortcomings to be permitted or for rigorous designs to be given little to no credit (Grand, Rogelberg, Banks, Landis, & Tonidandel, Reference Grand, Rogelberg, Banks, Landis and Tonidandel2018).
There is evidence that reviewers are biased to some extent toward work that contains what is perceived as positive results—those that support the hypotheses (Antonakis, Reference Antonakis2017; Emerson et al., Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010) or a current practice despite methodological errors. When articles contain mixed results, then reviewers may be more inclined to recommend rejection or encourage authors to present a “cleaner” version of the work. This can entail dropping unsupported hypotheses, alterations to the theoretical model, changes to the analysis (e.g., subbing control variables in and out or excluding parts of the sample), or ignoring undesirable findings. When these changes are suggested not out of a desire for more robust findings, but rather out of a desire for more desirable findings, then this achieves the ends via corruption of the means. And with corrupted means, the ends are corrupted too.
Recent publications have shown that this is not a hypothetical scenario or one that only occurs in sister fields such as social psychology. For example, O’Boyle, Banks, and Gonzalez-Mulé (Reference O’Boyle, Banks and Gonzalez-Mulé2017) tracked defended dissertations to eventual journal publication. They found that after going through the peer review process the ratio of supported to unsupported hypotheses more than doubled and that this metamorphosis (labeled the chrysalis effect) was entirely attributable to the adding, dropping, and altering of hypotheses and the adding, dropping, and altering of data (i.e., questionable research practices, QRPs). Their findings suggest that either authors are making changes before submitting a manuscript in anticipation of reviewer expectations or that authors are acquiescing to reviewer demands. Their work and subsequent work (e.g., Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017; O’Boyle, Banks, Carter, Walter, & Yuan, Reference O’Boyle, Banks, Carter, Walter and Yuan2019) point to engagement in QRPs occurring primarily when their effects were perceived to increase the probability of publication (i.e., increased hypothesis support, statistical significance of interactions, mismatching degrees of freedom). Along similar lines, Banks et al. (Reference Banks, O’Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016a) reported that 40% of management scholars surveyed stated being asked by reviewers to selectively report hypotheses and 33% had been asked to engage in HARKing. Finally, Emerson et al. (Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010) conducted an experimental study in which the statistical significance of findings was manipulated. Reviewers were more likely to recommend publication when the results were significant and more likely to find methodological errors when the results were not statistically significant.
We do not know of equivalent research on reviewer bias in practice, for example, how many times findings or recommendations are altered to cater to the client’s expectations or goodwill, but we have no reason to believe that practitioner evaluators would operate any differently from academic ones. In fact, given that future funding (hiring consultants, receiving grant money, being assigned projects) often depends on (perceived) success on a previous piece of work, it is likely that practitioners feel similarly incentivized to selectively report positive outcomes. Furthermore, positive outcomes may protect internal jobs or organizational departments from downsizing or dissolution. Arguably, these very real threats to one’s livelihood probably weigh even heavier than not getting an article published.
Bias is also introduced when the blind review process is compromised. Blind reviews prevent reviewers from allowing an author’s previous success or lack thereof to bias their overall recommendation. Unfortunately, the easy access to conference proceedings and widespread use of personal, university, and commercial websites to advertise work in progress (e.g., ResearchGate) makes it far easier to “peek” and identify early versions of the work and the authorship team. Not only does this introduce unreliability and bias when there are some reviewers adhering to the blind review process and others that are not, but it also encourages authors to try to position their work to signal their past success or camouflage their lack thereof. For example, students being evaluated by peers may highlight their high grades on previous assignments to give the impression that they know what they are talking about. Likewise, authors of a grant proposal may allude to several pieces of prior work or prior funding to highlight their capability and past successes.
In the review process, a multiple hurdle system with theoretical novelty as the first hurdle may prevent manuscripts with rigorous and interesting findings to advance without a novel theoretical contribution. This places a great deal of weight on theoretical novelty. Similarly, methods sections tend to report certain statistics, such as fit indices for measurement models and coefficient alphas, to indicate the degree of measurement error. Authors may present, and reviewers may demand, statistics that surpass arbitrary thresholds (Greco, O’Boyle, Cockburn, & Yuan, Reference Greco, O’Boyle, Cockburn and Yuan2018; Williams, O’Boyle, & Yu, Reference Williams, O’Boyle and Yu2020). These hurdles in the review process put much weight on specific theoretical and methodological issues and may result in reviewers losing sight of other important features of a manuscript.
Reviewers overstep their roles
Sometimes peer review does not make the work better, just different. Authors experience mixed feelings when, despite publication or funding success, they believe the content of their original work was hijacked by reviewers and decision-makers (Feldman, Reference Feldman2005; Roth, Reference Roth2002). The peer review process contains an inherent power differential, and authors may feel compelled to bow to reviewer requests regardless of their perceived merit or relevance. Views differ on the nature of constructive feedback with the ultimate goal of improving and clarifying a work’s contribution and what crosses the line into ghost authoring the work. Bedeian (Reference Bedeian1996, p. 1189) recommends:
As forms of human expression, words are explanatory constructs that reflect ideologies. To tamper with these constructs or to color an author’s logic and rhetoric with the overly invasive demands of editors and reviewers denies the author full intellectual responsibility for his or her work and permits subrosa influences to be exerted on both a discipline’s current character and its future development.
Apart from tampering with the authors’ intellectual work, though, reviewers may not always be aware that an offhand suggestion to make a fundamental change to the work is often viewed by the author as a promise of success (i.e., publication, funding, promotion) if the author follows the suggestion (Roth, Reference Roth2002). Breaking this psychological contract by later rejecting an article that owes its structure, form, and ideas largely to the demands of the reviewers and editors is often perceived as a violation of the implicit psychological contract of the peer review process.
Although there is some gray area on what constitutes overstepping reviewers’ roles, there are certain behaviors that clearly do. For example, engaging in the practice of citation coercion (Köhler, Reference Köhler and Rogelberg2017): Reviewers may feel that there is no harm in insisting that tangentially relevant works be included that can help increase their own or the journal’s citation counts or their employer’s reputation. A large group of editors recently came together and endorsed a ban on such a practice (see https://editorethics.uncc.edu/). As another example, reviewers might pressure a researcher to take a writing or analytic approach that is based on personal preference, not empirical demonstration of its superiority. As such, reviewers may encourage questionable research practices either out of ignorance or out of a misplaced duty to the journal, to their client, or to the decision-maker. For example, reviewers may look for all p-values to be under the .05 threshold as a proxy of study quality. This may be a behavior that is driven by inadequate training and ignorance or by the conscious, but misdirected and often harmful, attitude of the reviewers for what constitutes good science.
Peer review encourages nastiness
Finally, the veil of anonymity allows reviewers to take on an aggressive and sometimes rude tone (Clair, Reference Clair2015; Souder, Reference Souder2011). Rather than focus on key concerns and providing constructive feedback, a reviewer instead attacks the work, engages in argumentum ad hominem, or tries to demonstrate his or her superior intellect and experience. Taking an approach more akin to an internet troll than a helpful colleague again gives authors the impression that the peer review process is unreliable, stifling, unfair, and mean-spirited.
Correcting the problem
Solving or at least mitigating these problems with peer reviewing is key to improving our science. We contend that each one of these problems can be addressed through training and clearer reviewing guidelines and criteria. However, we first must establish what good reviewing and good reviews consist of. For this, a competency framework for reviewing is necessary. In the following sections we introduce such a framework, outline its overarching purpose and competencies, and provide suggestions regarding how it might be implemented into our literature.
A competency framework for peer review
To address the issues with peer reviewing outlined above, our subcommittee was tasked by SIOP to create guidelines and resources that facilitate rigorous, robust, and reliable reviewing. As a result, we created a competency framework that is composed of nine core competencies for high-quality peer reviewing. We further grouped these nine competencies into three overarching categories related to the foundational knowledge reviewers need, the mechanical skill necessary to write a review, and the professionalism with which reviewers should approach a review. The purpose of the competency framework is to provide general guidelines for any type of peer reviewing (including academic journal reviewing, reviewing of grants, white papers, consulting reports, client presentations, student work, and any other kind of peer review) that highlight what competencies are needed to provide great peer reviews. The framework focuses on competencies to deliver a great review, not specific content in a review, and as such, it can be used universally to guide any type of peer review activity.
Creation of the competency framework
In order to create the competency framework, we first reviewed available resources on peer reviewing, aimed both at how to provide good peer reviews as well as at the challenges that prior work has identified with bad peer reviews. We accumulated a collection of resources from academic journals, websites, online blogs of professional organizations, and available reviewer resources from SIOP. We further used resources developed during the first meeting of the Alliance of Organizational Psychology (a collaboration between SIOP; the European Association of Work and Organizational Psychology, EAWOP; and the International Association of Applied Psychology, IAAP), which subsequently generated the Memorandum of Understanding on fostering ethical, rigorous, and relevant research (Grote, Reference Grote2016).
Next, we distilled from this collection of resources criteria for good reviewing and categorized them into different core characteristics that good reviews have. Based on these criteria, we then developed the nine competencies of peer review and their requisite behaviors. When we had arrived at a first draft of the competency framework, we elicited feedback on the model from subject matter experts, including journal editors and leaders of SIOP and other professional associations. We also sought feedback from a few academics not involved in the development of the model to gauge if they would find the model useful for teaching their students about peer reviewing. At the same time, we asked practitioners to react to the competency framework to provide feedback about the utility of the framework in applied settings. As a result of this process of external review, we made a few small alterations and additions to the competency framework.
The current framework is displayed in Table 1. The letters in brackets in the following sections refer to the corresponding position of the point in Table 1. The framework is composed of the following nine competencies in three overarching categories: expertise, representing, and integrity (foundational knowledge); thoroughness, clarity, and efficiency (mechanical skills); and constructiveness, tone, and open-mindedness (professionalism). We attempted for the competencies to be as comprehensive, inclusive, and all-encompassing as possible, while still maintaining parsimony and practicality, so that they would actually be used. The overarching categories are differentially important for producing a high-quality review, with foundational knowledge being the most critical, mechanical skills being the second-most important, and professionalism being the third-most important.
Competency framework for peer review
We will now provide a description of each category and its competencies. In the following, we refer to the content of a review as “the review.” We refer to the work that is being reviewed (e.g., manuscript, slide deck, report, student thesis or assignment, promotion application) as “the work.” Furthermore, the person providing the review is labeled “the reviewer” and the person receiving the review “the recipient.” The person making a final decision on the work (for example, a funder, a journal editor, a selection panel, a dissertation committee, or a CEO) is termed the “decision-maker.”
Foundational knowledge
The competency category foundational knowledge is composed of the competencies expertise, representing, and integrity. This is arguably the most critical category, as without these competencies the likelihood of writing a good review is slim. At the very minimum, a reviewer needs to have the expertise to review a piece of work. This expertise can entail knowledge of the topic and domain of the work, knowledge of the core theories and of the context in which the work is embedded, and methodological knowledge (i.e., ontology and epistemology in which the work is embedded, knowledge of the data collection and analysis techniques). In addition, a reviewer needs to understand the mission and scope of the journal, agency, company, or other outlet/decision-maker for which they review. Without a thorough understanding of how a reviewer needs to represent the outlet, a reviewer cannot appropriately determine fit to the mission and scope. Finally, a reviewer needs foundational knowledge about the ethical and professional standards of the outlet, agency, or client they review for (i.e., integrity) in order to evaluate whether the work has been carried out appropriately and ethically. We will now explain each competency in more detail.
Expertise
Mainly, a decision-maker will ask a reviewer to review a piece of work to lend his or her expertise on the matter. That expertise is brought in to evaluate the currency of the research topic and its treatment as well as the quality of the work. In order to provide advice on the former, reviewers should ensure that the literature review or conceptual foundation of the work is current and comprehensive (a). Similarly, reviewers need to evaluate the currency and importance of the topic of the work (e.g., research topic, topic of the government report, etc.) (b). More generally, reviewers are asked to assess the work’s contribution to the field, community, or debate of interest (c). Lending one’s expertise can also include an evaluation of the appropriateness of the research design, methods, training design, evaluation criteria, or other methodological foundation used (relative to the work’s purpose and its epistemological and ontological grounding) (d). Said evaluation might involve checking the accuracy of the applied methodology and transparency in reporting (e.g., check the authors’ use of measures and methods, including checking whether the use of a particular scale is in line with its original purpose and design, check degrees of freedom, intercorrelations, effect size reporting, the coding scheme, the training evaluation, application of selection and marking criteria, and other aspects relative to the method used) (e).
As noted above, expertise is the most critical competency that prevents reviewers from giving bad or wrong advice. Many of the concerns about the peer review process that we highlighted above can be resolved if decision-makers choose reviewers with a high level of expertise. We acknowledge that it is difficult to find reviewers who are equally versed in both content and methods expertise. For editors, it is often easier to determine the content knowledge of a reviewer than their specific level of expertise in a given methodology. Furthermore, only few reviewers may keep up with the methodological advancements in the field and keep their methods training current. Arguably, though, judging the quality with which a piece of work was carried out is crucial for putting any stock in its findings. Consequently, some journals, such as The Leadership Quarterly or Journal of Management, have begun to employ both traditional reviewers as well as a designated methods reviewer. The editorial policy ensures that all limitations are clearly discussed and there are no flaws in the research design or in the analysis. Having this diversity in expertise can provide additional benefits to authors as they receive more accurate and applicable suggestions for the revision of their manuscript. We provide further advice below about how decision-makers can ensure high reviewer expertise, but also acknowledge many of the limitations that decision-makers face when choosing and inviting experts to review for their outlet, agency, or organization.
Representing
Reviewers represent the interests and vision of the organization or outlet for which they review (e.g., the journal, the funding or government agency, the university, the company, their employer). As such, when they evaluate the work, they need to consider the interest of the organization or outlet and its clients or readership (a). This also means that they need to ensure that the work is aligned with the organization’s or outlet’s missions, scope, and values (b). In addition, when the intended outcome of the review is a decision about how to proceed with the work (e.g., revise and resubmit, reject, promote, lay-off, put on probation), reviewers should include comments to the decision-maker (but not the authors; see the competency “Tone” for more detail) about their assessment of the work, i.e., how likely is a successful development of the work (c). In that way, reviewers help the decision-maker understand their overall evaluation of the work and aid the decision-maker in his or her job.
Integrity
When assessing the appropriateness of the work, reviewers need to keep in mind applicable ethics codes (for reviewers, for the conduct of research), but also other codes of conduct (such as legal requirements, employment laws, confidentiality agreements, intellectual property contracts) (a). Reviewers need to refrain from questionable research practices and from encouraging authors to engage in them at all times (i.e., they should not suggest HARKing, p-hacking, and other post-hoc changes to the data and hypotheses; recommend dropping null results simply because they are null; disregard the rights and legitimate treatment of co-workers and employees; or engage in behaviors that challenge professional conduct, workplace regulations, and laws) (b). This includes, for example, not pushing coercive citations on the recipient and only suggesting additional literature and resources that matter (c).
If a peer review is set up to be double-blind (i.e., when neither the recipient nor the reviewer know each other’s identity) or partially blind (i.e., when the recipients do not know who the reviewer is), reviewers should uphold the spirit of the blind review and not try to find out who the recipients are (d). In addition, reviewers should be timely (e) in delivering their reviews to not hold up the progress of the recipient.
We further need to acknowledge that a reviewer cannot be an expert in everything, and decision-makers may often ask reviewers with different areas of expertise to contribute to a review. So, not being able to comment comprehensively on all aspects of the work is fine (see also our point about thoroughness relative to one’s expertise). What reviewers should do is acknowledge if there are aspects of the work that exceed their area of expertise (f). They can do that in their personal comments to the decision-maker, but also in their comments to the recipient. Furthermore, reviewers should try to not have their assessment of different parts of the work be tainted by an overall impression or halo effect (g). For example, when assessing the strength of a manuscript, reviewers should distinguish their evaluation of the theoretical/practical contribution of hypotheses and research questions from the rigor of the methodology and from what is learned in the results, discussion, and recommendations sections. When assessing an employment application, for example, selection panel members should comment on different areas of competence, such as “has extensive training, but lacks the depth of real work experience.” When judging a consulting bid, reviewers may comment that the method in the presentation was suitable, but the purpose and implications for the client (the “what is in it for me” - WIIFM) was not strong enough. Overall, reviewers should try to assess and evaluate different components of the application or the project deliverable independently as well as holistically to provide a fair evaluation. This aspect of integrity relates in part to the open-mindedness dimension that we describe below but goes further in stating that reviewers should not go beyond their expertise and should try not to pass imbalanced or biased judgment.
Mechanical skills
The second most important category represents the mechanical skills necessary to write a good review. This category is composed of the competencies of thoroughness, clarity, and efficiency (i.e., the competencies that result in making a review useful and easily digestible for the reader). Mechanical skills are easier to train and acquire than expertise, but they need to be practiced to become routine behaviors that reviewers carry out in their reviews.
Thoroughness
Reviewers should strive for thoroughness in their review. Given the length of some pieces of work (e.g., doctoral dissertations, promotion or job applications, detailed government reports, academic books), it is sometimes hard to maintain an equal level of focus and depth in one’s review. Nevertheless, it is important to review all sections of the work, including all chapters, tables, figures, the abstract, the executive summary, the limitations, supplementary (online) materials, or the footnotes (a). Furthermore, when raising a point purportedly missing from the work (e.g., important literature that is overlooked, a missing Appendix announced in the article, a particular caveat or limitation that needs to be addressed), reviewers should ensure they have not overlooked the treatment of said “missing” point (b). Recipients who perceive that reviewers did not read their work properly can quickly reject all of the reviewers’ suggestions as unimportant and untrustworthy. To encourage recipients to intellectually engage with the feedback, we need them to feel that the reviewers have paid appropriate attention to their work. Ultimately, being thorough likely increases the chance that recipients heed the advice, improve their work appropriately, and see it communicated to a wider audience (i.e., through publication or presentation). It is important to clarify that being thorough does not mean that a reviewer is required to evaluate every single aspect of the submitted work if that would mean going beyond his/her expertise. Careful attention to all details of the submitted work is the key; yet, reviewers should also clearly recognize and communicate the boundaries (or limitations) of their knowledge when commenting on the aspects of the work that fall outside of their expertise (this relates to the integrity competency above).
Clarity
Clear communication of the issues raised and which parts of the work they address are a necessary condition for allowing the recipient to understand the comments and revise the relevant part of their work. As a first aspect of clarity in the review, reviewers should use quotes from the work to highlight what specific aspects or sections they find challenging. They should also use page numbers or other specific indications of locations in the work to which their comments are referring (a). In addition, reviewers should number their own comments and suggestions to the recipient (b). This allows the decision-maker to refer to the reviewer’s suggestions and comments and integrate them appropriately with the comments of other reviewers, providing a revision roadmap for the recipient.
Beyond ensuring that the recipient can find the relevant parts of the work for which the reviewer provides feedback, the provided comments should also be worded in a way that minimizes misunderstanding or ambiguity (c). For example, if a reviewer suggests the evaluation of a different approach or a different theoretical lens, they should provide enough detail for the author to understand what they have in mind, even if the author may have never heard of this approach or lens.
Finally, reviewers should ensure that their comments to the recipient and their comments to the decision-maker align (d). For example, if a reviewer suggests in their personal communication to the decision-maker that the work is too flawed to be developed further in a revision process, and hence recommends rejection of the work, then the substantive comments to the authors should reflect the seriousness of the concerns on which the reviewer is basing his or her recommendation to the decision-maker. Note, as discussed below (under “Tone”), reviewers should not state their recommendation to the editor in their letter to the recipient. Most journals and grant agencies provide tick boxes in online forms so that reviewers can indicate their outcome recommendation, in addition to a way to communicate confidential comments about the manuscript to the editor. Rather, what we refer to here is that a reviewer should not, for example, congratulate the authors for their great work and provide comments that only highlight the positive aspects of the work, if their actual perception, and communication to the editor, is that the work is too flawed to be revised. While maintaining a supportive, developmental tone, as described above, reviewers need to communicate the limitations of the work clearly and concisely. This ensures that recipients can appropriately process the challenges perceived in their work and take necessary actions to improve it. As a final example, a reviewer may not perceive that a manuscript is a good fit for the journal, either because the contribution is too narrow for that specific journal or the content is not a good fit. There may be an overall contribution, but that contribution may not align with the journal. Reviewers should recognize that such a comment is akin to recommending rejection. Assuming reviewers cannot make any suggestions to address such misalignment, reviewers should at least clearly articulate the reason for their position.
Efficiency
Balancing constructiveness with pragmatism, it is important that a review efficiently communicates the most important concerns. Nobody will ever create the perfect product. So, we need to communicate realistically what needs to be improved and what can be improved in future work or in a next step. In order to ensure efficiency, reviewers should separate major concerns/issues from minor concerns/issues (a). In other words, reviewers should help recipients understand what is essential to address versus what is suggested. Such an approach also aids decision-makers in their decision-making process. In addition, there is no predetermined number of feedback points that a review needs to contain. While some outlets might provide some guidelines, for example, some journals might tell their reviewers to strive for six to eight main points in their reviews, while others recommend two to three main points, there is no convention that prescribes a certain number of comments. Consequently, reviewers should avoid writing feedback to reach a predetermined number of comments (b). Rather than providing feedback on every minute limitation of the work, it is more efficient to focus on all of the major concerns and some of the more important minor concerns to allow the recipients to address what is most crucial in their revision of the work. This subsequently also makes the length of the response to reviewer letter much more manageable. In order to further improve efficiency, the comments provided should be organized in a logical way (c). What is most logical might depend on the issues and the format as well as the structure of the original work. Most importantly, reviewers should clearly communicate the logic of their comments to the recipient. This ensures that the recipient is not just paying attention to separate issues raised in the review, but also gets a feel for the overall spirit in which the review is written and the overarching themes for the necessary improvements.
Professionalism
Finally, the last overarching category, “professionalism,” captures how reviewers should approach a review (i.e., their attitude, motivations, and general etiquette). This category includes the competencies constructiveness, tone, and open-mindedness. We note that a recipient could probably get high-quality advice that is very gruff and maybe even unkind. As such, the competencies in this overarching category are not as essential as the previous two categories. However, as we elaborate in our explanation of this category below, a recipient will find it much easier to read and digest a review that is written following a high level of professionalism. As such, the positive impact of the review on the recipient may be higher and thus more consequential for the overall improvement of the science in our field.
Constructiveness
Reviewers need to strive for providing constructive feedback, or feedback that allows the recipient to understand how he or she can overcome the stated limitations and achieve the suggested improvements. This includes giving actionable advice or solutions for the issues identified in the review (a) and providing supporting citations or resources where recipients can get further information about the suggested improvements (b). For example, if a reviewer deems a particular data analysis method or chosen training approach to be inappropriate in the recipient’s research or applied context, the reviewer should explain why it is inappropriate and suggest which alternative approach would be more appropriate. The reviewer should then also make the recipient aware of important references and resources where they can learn how to apply the suggested approach to their work. This is not to suggest that reviewers should be heavily educating authors. Rather, by providing enough rationale, reviewers give authors useful direction. In addition, having to provide rationale prevents reviewers from dismissing authors’ ideas too quickly or simply in favor of their own preferred approach or theory, and often without fully assessing the applicability of their suggestion. To achieve constructiveness in a review, pointing out a flaw should always be accompanied by specific suggestions for viable alternatives. In some cases, this can also entail suggesting alternative outlets that may suit the topic of the work better (e.g., different journals, different presentation formats, or different grants) (c).
When providing constructive comments, reviewers should attempt to balance breadth (i.e., big-picture comments like “What is new here?”) and depth (i.e., nuts and bolts comments like “That is a six-item scale; why did you drop two of the items?”) (d). Providing comments that are too specific and often too numerous may result in the recipient getting lost in aspects of the work that may represent relatively minor issues. At the same time, focusing only on the big-ticket items may create the impression that the work suffers from such fundamental, overarching issues that there is nothing tangible that can be improved about it. Reviewers need to find a balance between big-ticket and small-ticket items.
Finally, reviewers should acknowledge that recipients come from different backgrounds, operate with different levels of expertise and experience, and may struggle with different parts of the research process. For example, reviewers should be patient with non-native English speakers, novices in the profession, and recipients who might have been trained in different traditions or disciplines (e).
Tone
Tone refers to how the overall message of the review is packaged and delivered to the recipient. When choosing a tone for a review, the reviewer should keep in mind that the recipient will have an intellectual and an emotional reaction to the review. As such, reviewers should attempt to minimize potential negative emotions that the recipient might experience as a result of the review and maximize the intellectual engagement with the feedback provided. Recipients will be better able to hear and process the feedback if it is packaged in a way that is less identity-threatening and more developmental.
For peer reviewing, this means more specifically that reviewers may want to start with listing the strengths of the work, going further than just offering some superficial introductory comments (a). Instead, reviewers should consider highlighting what is innovative, interesting, or engaging about the work. The review will then usually continue with a discussion of limitations of the work that the reviewer would like the recipient to address in a revision of the work. Reviewers should discuss these limitations in a tactful and respectful way. We know from previous work on feedback effectiveness that when feedback focuses attention and comments on the author’s intent or intellect, it debilitates motivation and performance (Kanfer & Ackerman, Reference Kanfer and Ackerman1989; Kluger & DeNisi, Reference Kluger and DeNisi1996). Thus, reviewers should always refer to the content of the work and the precise ways of developing the work (goal specification) rather than focusing on the abilities and inabilities of the author in meeting those goals (b). Recipients as authors of the work have usually expanded a lot of effort to create the work. Reviewers should remind themselves about this as they read the work and write the review. By acknowledging this, reviewers provide feedback assuming that the authors have done their best to produce a piece of work that they think is worthwhile to be reviewed by a panel of experts.
Along similar lines, reviewers may notice inconsistencies in the work. For instance, analyses do not match the theoretical model built in the literature review, the use of measures is reported incorrectly, such that scale items were deleted or reworded without telling the reader, degrees of freedom do not line up with the information provided in the methods section, rerunning the analyses using a correlation table does not result in the same results, the description of the data collection or coding process in qualitative work indicates that the chosen approach may be leading or too confirmatory, a chosen selection or training approach has serious flaws, or information seems to be plagiarized. In these and similar cases, reviewers should refrain from immediately concluding academic misconduct. However, these are serious issues that reviewers need to weigh heavily in their recommendation about the work. Given grave methodological flaws, a piece of work might communicate, in the best case, meaningless findings, and, in the worst case, wrong findings, which may then get perpetuated as the work gets cited (e.g., DeSimone, Köhler, & Schoen, Reference DeSimone, Köhler and Schoen2019). As a first step, reviewers should probe the author for answers (c) (e.g., “There are three missing degrees of freedom in the CFA. What were these estimated paths and what was the model fit before they were freed?”). If methodological inconsistencies amount, then rejection of the piece of work is likely the way to go, especially if the observed issues would be difficult or impossible to fix without essentially redoing the whole study. More candor about potential QRPs can be given in the private comments to the editor (e.g., “Given the marginal statistical support, the author’s decision to use a 94% confidence interval to test their indirect effects seems suspiciously post hoc.”).
Most estimates suggest that outright research fraud and malfeasance in the social sciences are relatively rare (for a systematic review, see Banks et al., Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016b). Much more common are a lack of expertise and bad habits, or sometimes different conventions and accepted practices. Consequently, in the spirit of continuous improvement, reviewers have an opportunity to educate review recipients, encourage them to adopt better research habits for their future work, or make them aware of different discipline-specific practices and conventions. If reviewers have concerns about malintent and detrimental research practices (DRPs), they should note them to the relevant decision-maker (e.g., editor, head of a selection panel, program director of a grant agency, PhD program director) in the private comments section or a personal communication. It is the decision-maker’s responsibility to follow up with the recipients and take the necessary steps to ensure that the principles of robust science and professionalism are upheld.
Related to this point, editors and decision-makers often instruct reviewers not to include their recommendation for the ultimate decision about the work (i.e., revise, reject, accept) in their comments to the recipient (d). Doing so may, on the one hand, artificially constrain the decision-maker’s decision. On the other hand, reviewers sometimes differ in their final assessment of the work, both from one another but also from the editor or decision-maker. This can create confusion for the authors, which should be avoided. Making a final decision on the work (e.g., if the work gets rejected or offered an opportunity for revision; if an application will receive funding; or whether a pitch is going to be presented to a client) is the decision-maker’s prerogative, and there might be many reasons why a decision-maker might choose not to follow a reviewer’s recommendation while still valuing his or her substantive comments.
Ultimately, reviewers need to balance their dual roles as gatekeepers (i.e., highlighting problems and upholding the guiding principles of robust science) with the generative and developmental role of peers (i.e., highlighting contributions and continually improving the profession and its practices) (e). Overall, this means that reviewers should write their review as if they were to sign their name under the review (f). If reviewers choose a tone that would make them apprehensive about the recipient potentially finding out who they are, then the tone needs to be further improved. Along similar lines, reviewers should take the perspective of the authors and empathize with them. If reviewers give feedback in a tone they would have a hard time reading if it was directed at their own work, then they should reconsider the tone of the review. In summary, ensuring an appropriate tone for the review enables recipients to better process the suggestions and recommendations, develops them as authors of professional work, and generally improves our personal interactions with each other as members of a common professional community.
Open-mindedness
Given that much work is aimed at raising novel or controversial issues, reviewers should keep an open mind to new topics and techniques (both quantitative and qualitative; deductive and abductive/inductive) as well as submissions that revisit old topics (a). Acknowledging differences in training and diversity in research traditions that might be employed in the work, reviewers should assess research within its proposed paradigm, epistemology, and ontology (b). Furthermore, they should recognize that excellent work can come in many forms and that all research has strengths and limitations. Reviewers need to work on the basis that no research project is perfect. Their role is to provide feedback on the strengths of the chosen approach and assess how authors have managed the limitations, as well as what influence those limitations may have on the interpretation of the data and conclusions drawn from the findings (c). This, for example, also means that reviewers should avoid making qualitative research quantitative (d). Being a reviewer may require doing additional outside research to better position oneself to evaluate the work (e). In the end, reviewers should not eliminate the contribution and voice of the authors by making them compose the work they would have liked them to compose. Instead, reviewers should support recipients by improving the work they wanted to compose (f). Being open-minded in assessing a piece of work increases the likelihood of furthering innovation and cutting-edge scientific endeavors.
An important caveat regarding the open-mindedness competency is that reviewers need to assess authors’ work in relation to the scope and mission of the outlet or client (see also the “Representing” competency above). For example, journals usually bound their topic scope and sometimes the research methods they consider appropriate to studying the topics of interest in their mission statement. Grant agencies often define topics of interest or preferred research approaches in their calls for research proposals. As such, reviewers need to have an awareness of the boundaries of the outlet or preferences of the client against which they evaluate the authors’ work. Nevertheless, the spirit of this competency is to maintain openness toward different research techniques, methodologies, epistemologies, or theoretical foundations to foster diversity in and inclusivity of research approaches.
How can the competency framework be used to reduce issues with reviewing?
Having introduced the competency framework, we now provide some suggestions for how it can be incorporated into our daily practice for providing feedback to others. As we discussed above, the competency framework is broad enough to apply it to a wide range of reviewing activities. As such, we offer a few concrete examples for using the framework to improve reviewing practices and skills.
First, the competency framework can be used to create a clear set of guidelines for reviewing. For example, when course instructors would like to introduce peer evaluations in their classes or when PhD program directors decide to introduce peer reviewing among a cohort of PhD students, they could use the framework to guide novice reviewers in defining what they should pay attention to, what their review should touch upon, how they should word their review, what kind of advice they need to provide, and how they should overall go about peer reviewing. Additionally, more seasoned reviewers may find these guidelines helpful in encouraging them to dig a bit deeper in their reviewing, identifying substantive issues, rather than simply finding the most obvious issues and writing a superficial (i.e., path of least resistance for the reviewer) review.
As an extension of this, advisors of PhD students, who would like their students to become involved in peer reviewing for journals, could use the competency framework to help students understand what is required for a good journal review. The faculty advisor could, for example, take a journal review that they have previously completed and explain to the student, using the competency framework, how their review displays the different required behaviors as described in the framework. They could then work together with the student on a new journal review request (seeking permission from the journal editor to share the submitted article with the student for this purpose), allow the student to independently do a review of the new journal submission, and then work with the student to improve it. Going a step further, the advisor may even ask the journal to include the student as a reviewer for the article, so that the student can get a reviewer rating and may slowly be introduced more formally into the journal review process.
The competency framework could also be used to improve the collegial peer review process in organizations. For example, in order to provide useful feedback on organizational documents, such as quarterly reports, internal competency frameworks, or strategy documents, relevant colleagues (e.g., team members, HR Directors, internal communication specialists, steering committee members) could use the competency framework to structure their review of these documents. By using the competency framework as a guideline, they could provide feedback to their colleagues that is more structured, relevant, and valuable. Furthermore, the feedback would use a tone and style that likely improves the author’s receptiveness to it, thus improving the utility and effectiveness of the feedback process. Similarly, when preparing a client-oriented product, such as a proposal or client deliverable, the competency framework could be incorporated as part of the final internal review process prior to submission or presentation to the client. Third, the proposed competency framework would also be a valuable addition to the internal review process completed before dissemination of externally oriented documents, such as organizationally branded white papers/thought leadership publications; publications in professional outlets (e.g., blogs, business magazines, books); or an organization’s own proprietary frameworks, tools, and methodologies. Many organizations and practitioners seek to differentiate themselves on their thought leadership and branded frameworks and methodologies. Using the proposed competency framework would support professional practitioners to provide balanced and thorough feedback that encourages the publication of ideas and practices that are theoretically, methodologically, and practically sound.
Finally, the framework could also be used to evaluate new concepts, tools, and frameworks introduced from outside of the organization. For example, when new ideas are introduced by senior leaders or other stakeholders, there is a need to review the quality and applicability of these ideas (Rotolo et al., Reference Rotolo, Church, Adler, Smither, Colquitt, Shull and Foster2018). I-O psychologists often act as gatekeepers who can offer balanced, rigorous, and thorough evaluations of published research of highly credible professional outlets or content from popular business magazines or books. The competency framework can be used to guide quality reviews of the underlying constructs and empirical evidence available to support the pitch or to discourage the sale and/or implementation of the next new “interesting” idea.
As a caveat to the utility of and likelihood of engaging in constructive peer review in organizations, we note that organizational performance management systems build on a competitive culture and individual-based incentive structures (which arguably make for a significant proportion of I-O practitioner jobs) may not be conducive to the kind of peer review activities described here. In organizations in which employees are encouraged to compete with each other, the motivation to engage in constructive review for others is likely low. In addition, recipients could never be sure if the provided advice is actually in their best interest or if it is given in a spirit to be detrimental for their performance. The proposed framework for reviewing in organizational and practitioner settings is thus especially useful when used in highly collaborative, learning-oriented or purpose-driven cultures where team-based incentives, collaboration among peers, and psychological safety are explicitly recognized and valued by the organization and aligned with its performance management practices or reward structures. Similarly, the competencies outlined here can be used by senior managers as a guide for providing feedback to junior colleagues or mentees, who are not in direct competition with them and whose subsequent success may, in fact, be part of the senior manager’s success and key performance indicators. To further explore the role of constructive peer review in organizations, especially in the form suggested in this article, we invite practitioners to provide commentaries on how peer review and reviewer competencies may become useful in organizational settings.
A similar objective could be the basis for providing peer reviews for grant or funding proposals. When providing a funding proposal to a colleague who may read it with a critical eye, the colleague could use the competency framework to identify aspects of the proposal that may need improvement. That way, a funding proposal may be revised, taking a reviewer lens, before being submitted to a funding body in order to increase chances of acceptance. In addition, funding bodies could use the competency framework to review their own reviewing criteria to evaluate whether their review process could be improved, both in regard to the content of the reviews as well as the tone and constructiveness of the feedback provided to the submitting authors. In that way, feedback from the decision-maker and reviewers might become more useful for the recipient of the feedback, in order to provide a better proposal next time.
Another avenue for using the competency framework is to create specific reviewer training programs or guidelines for professional associations, practitioner publications, or organizations. For example, the framework could be shared with practitioner publication outlets to complement their existing processes (e.g., HBR, Bersin by Deloitte, Gartner, Forbes) or HR professional associations (e.g., SHRM) to encourage more widespread use in practice. The competency framework could also be a good accompaniment to “quality assurance” steps often taken in organizations. For example, the framework principles could be included as one of the organizational tools provided to project directors/managers to support their review of work products as part of their quality assurance role.
Training reviewers for academic work, SIOP’s subcommittee for Rigorous, Robust, and Reliable Reviewing has created reviewer trainings for conferences (e.g., the annual conferences of SIOP, AOM, EAWOP, between 2017 and 2019), during which participants are introduced to the competency framework and interact with experienced reviewers to learn how to apply it to their own reviewing activities. Along similar lines, in August 2018, the subcommittee collaborated with the Consortium for the Advancement of Research Methods and Analysis (CARMA) to create a reviewing bootcamp, which introduces the competency framework and provides guidelines for good reviewing to novice reviewers. The bootcamp and supplemental certification test can be accessed via the following link: http://carmarmep.org/siop-carma-reviewer-series/. Future initiatives might include creating training opportunities using situational judgment tests, in which participants can respond to brief scenarios to hone their skills. Furthermore, we envision creating mock review assignments that trainees can do to then compare their review to a template review created using the competency framework.
Furthermore, we envision creating online repositories for reviewing resources that further explain and elaborate on the competencies and associated behaviors in the framework, offering concrete advice for how reviewers can improve their reviewing skills. Such resources might entail, for example, checklists for the application and interpretation of specific research methods, statistical techniques, or coding approaches. Authors and reviewers could use such checklists to improve their use of a method, but also their assessment of how others have employed the method in their work. CARMA and the journal Organizational Research Methods are currently collaborating on a set of special issues targeted at advanced reviewer training for common research methods. These special issues will also include the creation of such checklists. In addition, other resources to be made available online could include editorials and guidelines for improving the tone and professionalism of peer review. Having explained how different stakeholders in our profession could use the competency framework to improve peer reviewing practices and skills, we now discuss some additional ideas for steps forward to more firmly integrate excellent peer review practices in our professional practice.
Steps forward
As we proceeded to develop the competency framework, other ideas for improving the review process and experience emerged. Some of these are reflected in the literature (e.g. Epstein, Reference Epstein1995), while others are reflected in how some outlets and organizations describe their review process and what they expect from decision-makers, authors, and reviewers. In this section, we reflect on some of these ideas while considering what can be done by authors, journal editors, publishers, and practitioners to extend the improvement of the peer review process to other stakeholders identified in Grand, Rogelberg, Allen et al.’s (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018) piece in IOP.
Authors
For authors to meet the expectations of reviewers and editors, it may be instructive to review general best practices that are often shared with novice academics and practitioners. First, authors expecting to receive a great review should ensure that their work is the very best they can make it. In other words, review, revise, and edit one or a few times more than what might feel comfortable. Reviewers and decision-makers can tell when a piece of work is presented prematurely. In their revision process, authors should consider asking for a friendly review from a colleague not tied to the project and make the changes they recommend. Perhaps as an alternative or in conjunction with having a colleague review the manuscript, consider submitting the work to a conference, workshop, or mock presentation for additional review and peer commentary.
Second, always take reviewer feedback seriously (informal or formal). It is easy to discard reviewer ideas and comments when the feedback is not agreeable to what the work currently contains. Reviewers are supposed to acknowledge the authors’ expertise in the topic presented, and so authors should also assume the reviewers’ expertise to provide relevant and meaningful feedback. Authors are well-advised to adopt a growth mindset and take a learning orientation approach when receiving and responding to feedback. Many successful authors we know have commented how much they have learned and benefitted from the peer review process because they have taken the feedback as an opportunity to make the work better, not as a threat to the worth of their work.
Third, as we alluded to earlier, journals, grant agencies, and clients have different missions, scopes, and objectives that inform reviewers’ and editors’ expectations of relevant content and appropriate methodological rigor. In addition, journal editors often publish editorials that further clarify journal preferences for submissions, while grant agencies may provide additional information in calls for proposals regarding preferred approaches or methods. Authors are advised to explore these expectations to direct their work to a suitable journal or funding sources, as fit with the mission and scope increases the likelihood of a successful outcome of the review process. Along the same lines, it is helpful to keep in mind that reviewers may (and often should) raise concerns and make suggestions that are specific to the particular journal, grant agency, or client (see the reviewer competency “Representing”). Recognizing this, authors should try to discern which comments are outlet-specific versus universal (see Tay & Diener, Reference Tay, Diener and Sternberg2018, for further discussions).
Finally, we want to acknowledge that authors may very often only read and cite from the mainstream literature, assuming validation of the quality of the work by having been published in the top journals of their discipline. Work published elsewhere, such as in practitioner or lower ranked journals, may be seen as “inferior” and not worth citing. Some of these trends seem to be discipline-specific. Starbuck (Reference Starbuck2005) showed that authors in management and psychology over time have been increasing their citations of work published in lower-tier journals. Authors may have realized that due to the here-described shortcomings of the peer review process, much good work ends up being published in lower-tier journals and much work published in top-tier journals may be of low quality. However, given that these findings were published almost 15 years ago, we are not sure what current citation trends look like. From rising impact factors for the top journals, it certainly seems that they are getting most of the citation attention from authors. Nevertheless, important contextualization, practical application, and adaptation of core constructs might find publication in practitioner outlets. In addition, much relevant work on a particular phenomenon or construct might be published in other disciplines. As such, we strongly encourage authors to read broadly any relevant work that has been carried out rigorously, without dismissing whole categories of outlets based on predetermined notions of their standing. Similarly, we expect the same of reviewers when they criticize authors for including sources other than top-tier journal articles purely based on where they were published and not on their actual quality.
Editors and other decision-makers
The most important responsibility of editors and decision-makers for their evaluation of a piece of work is to weigh the quality of the feedback provided by the reviewers and to make appropriate decisions based on that. In many ways, the competencies and guidelines discussed in our framework are equally relevant for editors and decision-makers, as their lack of competence could amplify the consequences of bad reviews. Given that editors and decision-makers hold almost all the power in the decision-making process, they must be both good reviewers as well as the gatekeeper of good reviews. This means that if reviewers provide bad feedback or make inappropriate suggestions (e.g., wrong methodology, coercive citations or theorizing, suggest HARKing, or engage in other reviewer misconduct), editors and decision-makers need to recognize that the advice is bad and steer the authors away from following it. This means that editors and decision-makers should write decision letters that provide appropriate direction to the authors, even if the piece of work gets rejected.
In addition to weighing the quality of the feedback provided by reviewers, editors and decision-makers should weigh the quality of the piece of work before sending it out for review in the first place. In the instance of grant agencies, a precondition for the evaluation of the work by a reviewing panel is often that the grant application meets a set of predetermined standards and passes a set of completeness checks. Otherwise, the application will not get sent out. Some grant agencies may even ask authors to indicate whether they have received friendly reviews of the work before submitting it. Journal editors can differ widely in the percentage of desk rejection decisions they make. At some journals, this may mean that work that was submitted prematurely and may not meet the journal’s own standards gets sent out to reviewers. This can be problematic, as reviewers may feel that the editor is wasting their time. In the long run, they may stop reviewing for the journal if they feel that the journal only sends them substandard manuscripts that could have been processed more efficiently with a desk rejection decision. As such, decision-makers who desk-reject more (using relevant and rigorous criteria) can signal to reviewers that they value their time and will only send them work to review that meets a set of standards. In turn, decision-makers can request that reviewers actually agree to a review invitation and provide thoughtful and thorough reviews as outlined above.
Another interesting idea emerged as we reviewed the literature and prepared the competency model: Reviewers seldom receive feedback on the quality of their reviews. Notably, award-winning reviewers generally learn that they are doing well. However, most reviewers do not receive awards, and thus do not know if they are good, bad, or simply good enough. In addition, given the high demands on reviewer time and the fact that reviewing is a service and rarely considered in academic or practitioner promotions or other reward allocation, it may be tempting to be “good enough” rather than really good at reviewing. Thus, we suggest that one thing that could be done by editors and decision-makers is to provide feedback to reviewers. Initially, we thought that perhaps this could be done one-on-one as the reviews occur. Although a meaningful way to provide feedback, the time needed to do that effectively may prove problematic. Additionally, editors and decision-makers often struggle to identify appropriate reviewers in the first place. Providing them with critical feedback may create fear of losing reviewers. Therefore, we suggest a few possible ideas here:
First, editors and decision-makers may consider providing feedback to reviewers either one-on-one or perhaps in summary form for the entire reviewer group. The latter provides a way to point out problems that are relatively common, highlight the positive behaviors they want to encourage, and create change among the more proactive reviewers. Second, editors and decision-makers could consider providing examples of particularly great reviews. Note, this would require permission from the reviewers and the authors, but it seems like a reasonable way to model good reviews and reviewer expectations. Third, journal editors and decision-makers could consider building reviewer shadowing/mentoring opportunities for advanced doctoral candidates or other inexperienced reviewers. The structure of such a mentoring process could vary, but it may allow for unique and useful competency-building experiences. In doing so, we would suggest that using the suggested competency framework should prove very helpful in communicating reviewing expectations and good reviewing practices. Along these lines, using Reviewer 3 strategically to build reviewing competence might be an option for editors and other decision-makers to extend and train their pool of capable reviewers. As such, choosing as Reviewer 3 a more junior reviewer or a reviewer who has not yet reviewed for the journal or grant may be a way to assess the reviewer’s reviewing competence and expertise. In addition, by including the person in the review process, he or she learns valuable information about the review process at said outlet by receiving copies of the other reviewers’ evaluations and the editor’s or decision-maker’s decision letter.
Editors and decision-makers may consider a few other ways to adjust the review process and experience. First, they could consider incentivizing great reviewing through meaningful reviewing acknowledgment. Explicit recognition of good work might be key here. Reviewers, like authors, sometimes take the path of least resistance in their reviews, meaning they identify a few easily identifiable issues and then proceed with a more superficial review, especially if putting in the extra effort of delivering a high-quality review never gets recognized. As such, when reviewers repeatedly provide clear, constructive, respectful, and thorough reviews, editors and decision-makers should send them a personalized note acknowledging their effort. These notes can be added to tenure and promotion applications and may be especially valuable for early career professionals.
For more public acknowledgments, journals may recognize the reviewing activities of all their reviewers in the final issue of the year or during their editorial board meetings at professional conferences (many journals already do so). For example, the Consulting Psychology Journal: Practice and Research publicly recognizes reviewers during the Society of Consulting Psychology conference (Division 13). Similarly, grant agencies may publicize the names of their reviewing panels, and departments may choose to acknowledge who served on selection and promotion panels during the year. In addition, these different outlets and organizations could consider providing opportunities to their best reviewers to write best practice recommendations, editorial pieces, or invited commentaries or articles.
Finally, more transparency on the selection and assignment of roles in the review process (e.g., editorial board membership, heading a judging panel, or participation in grant evaluations) would provide reviewers with a better idea of what is needed to be acknowledged as an expert reviewer. Even though these reviewer roles are usually not paid positions, the reputation and prestige associated with them can be rewarding and incentivizing, making reviewers keener to deliver high-quality reviews in a timely manner. Given the complex role of editors and decision-makers to ensure the efficiency and quality of the review process, we would like to encourage readers to consider writing a commentary to this piece on the editor’s or decision-maker’s role.
Publishers
In the academic peer review and publication process, researchers serve as the primary or sole producer (authors), quality control (reviewers and editors), and customer base (university libraries). Yet, the role of the publisher is often overlooked. However, publishers are arguably the most direct financial beneficiaries of the peer review process, and thus, this stakeholder merits some attention. The chief responsibilities of publishers are physical production of the publication of the research, maintenance of online material, and promotion and marketing. Each aspect of this plays a role in the peer review process. For example, copyediting is the final quality control before physical production, and like any profession, there is variance in the quality and level of support offered by copyediting services. This level of service is commensurate with cost. The additional care that goes into ensuring language and visual accuracy; consistency across text, tables, and figures; and conformity with appropriate style guidelines can be key differentiators from alternative outlets such as open access and self-publishing. We encourage publishers to recognize this competitive advantage and devote the appropriate resources to maintaining it. Likewise, ensuring that authors have the ability to include supplemental online materials, such as appendices and syntax files, and that readers have easy access to these materials is another value-added feature that publishers should provide. Our final recommendation to publishers is that when marketing their journal(s), consider whether they are doing enough to recognize and promote the reviewers and editors.
Practitioner–academic collaborations
If there are few formal incentives for academics to review, it is not hard to imagine why practitioners have slowly disappeared from the editorial boards of our applied journals (e.g., Journal of Applied Psychology, Personnel Psychology) (Silzer & Parson, Reference Silzer and Parson2012). In general, in the last decade several academics and practitioners have warned us about the perils of the lack of practitioner participation in academic publishing, both writing and reviewing. It is suggested that the science–practitioner gap exists because practitioners do not read or apply the scientific literature, and academics do not do the research that answers the questions that emerge in the field. In sum, there is a lag between the trends and needs of practice and the interests and inquiries of academia. According to Briner and Rousseau’s IOP focal article back in Reference Briner and Rousseau2011, it is necessary to “pursue ways in which evidence can better inform practice,” as they suggested an “evidence-based practice” model to address this problem (p. 4).
Bartlett and Francis-Smythe (Reference Bartlett and Francis-Smythe2016) surveyed 163 work and organizational psychologists to explore how evidence is applied in practice. They found that the barriers were mainly practical in nature, some related to managing the client expectations and interest, but others related to the cost of accessing, finding, and reading the scientific evidence (e.g., lengthy academic reports, expensive pay-per-read articles). They suggest that scientific evidence should be delivered in a more easily accessible, cost-effective, readable, and brief format, with reporting standards that facilitate the application of the evidence. At the same time, we believe that effective scientist–practitioner collaborations require practitioners to express and show an active interest and involvement in academic research that generates the kind of evidence they look for when creating organizational interventions.
Having more practitioners as reviewers of academic publishing would provide several benefits: First of all, it would provide free access behind the pay-walls of publishers, as journal reviewers are usually given free access to journals and books from the editorial group of the journal they are reviewing for. This would help bring evidenced-based research to practitioners, which they, in turn, could take to market, and give them more opportunities to apply findings in their context. Second, it would provide practitioners with an important voice to help craft academic publications in a way that can be more easily read and applied in practice. For example, we know that the practical implications of academic articles are often written as an afterthought without much relevance for real applications in organizations. Practitioners could review articles as experts on the topic, the method, or the application. They could ask pertinent questions targeted at linking research to practice in all sections of the manuscript: “So what?”, “What does this mean in practice?”, “What is the crucial phenomenon and why is it important?”, “How would this theory play out in a real context?”, “What does it mean for my organizational practices, processes, systems?”, and “How might I modify the way I engage with people or a process based on this insight?” This reviewer–author dialogue, in turn, can help inspire more relevant future research questions and scientific lines of inquiry that match the applied nature of our profession. Third, being in touch with the latest research findings may inspire practitioners to write up their own insights, implications, and recommendations in practitioner-oriented outlets (e.g., HBR; SHRM; HR Magazine; Fortune, etc.) or their own internally developed and marketed thought papers, frameworks or methodologies. This benefits the field of I-O and helps create greater exposure to the science (and academic authors and institutions) behind the ideas.
How can this be done successfully? More practitioner representation as reviewers and members of editorial boards necessitates that practitioners become motivated to contribute to and shape academic output. A first step to highlighting the mutual benefit of contributing to academic output through reviewing might be that researchers include practitioners in conference submissions (such as at SIOP). Chairing, presenting, or discussing academic work can create tangible links between scientists and practitioners and visibility for those practitioners who would like to contribute to academic output on a continuing basis—for their own personal interests or to increase exposure of their organizations, or both. On the other hand, we note that not all reviewers need to be experts on all aspects of a piece of work. As such, journals could be explicit about what practitioners might be expected to review and how those reviews should be conducted given the specific expertise, constraints, and motivations of practitioner reviewers.
Another avenue to encourage practitioner involvement in reviews would be if journals created space in their publications for practitioner commentaries or contributions, for example, inquiring, evaluating, or adding examples or challenges of the specific utility of various theories or methods in applied settings. Through this, practitioner reviewers may increase their profile and credibility as contributing authors and thought leaders in the field in their own right. Last, where thought leadership is a key part of a practitioner’s role and expected contribution (e.g., contributing to the development of new frameworks or methodologies; publishing books or publishing in professional outlets; public speaking at professional events or conferences), there might be an opportunity to include peer reviewing for journals as a suggested or required element of their performance success criteria. As these suggestions indicate, for practitioners to join editorial boards and teams and to increase the visibility and importance of the applied nature of I-O psychology, we need a change in the system. We encourage commentaries to this focal article that suggest creative ways to increase the participation of practitioners in the review process of academic journals.
Similarly, academics can review for applied purposes. For example, when a practitioner is developing a new framework or methodology (e.g., performance management framework, performance management system, succession planning approach), writing a thought paper/white paper/business magazine article, developing a client research report, designing a leadership program, or creating a response to a request for proposal (RFP) or grant application, soliciting feedback from an academic peer can add significant rigor and depth to their approach. Drawing on their expertise in a particular area, academics can share relevant research evidence for the practitioner’s applied problem. Furthermore, academics can provide advice on the design of a new methodology and suggest specific ways to collect data that will provide strong evidence of success to the client. These are just a few examples of the reciprocal value that can come from greater academic–practitioner collaboration on reviewing core outputs in each other’s domains.
What can be done to encourage academics to review for practice purposes (e.g., grant proposals, consulting bids, new methodologies)? Some academics may ask for remuneration in exchange for reviewing a potentially profitable proposal (interestingly, this is something that academics do for free for journals, and maybe a commentary on this is warranted). Beyond financial incentives, academic motivation may increase if reviewing for practitioners increased their awareness of the issues of interest to organizations and consultants. By reviewing, they can be inspired to delve deeper into lines of scientific inquiry that would not have occurred to them. Paired with the possibility to create opportunities to engage in partnerships to do applied research in the field (e.g., practice-oriented research projects, community-engaged scholarship), not only would this engagement support their academic careers, but it would benefit our profession by narrowing the research–practice gap and by creating real impact through individual, organizational, and at times societal change. Given the rapid growth of I-O psychology as an applied profession, and the number of I-O graduate students that we train, creating I-O training programs that involve partnerships or direct work for or with practitioners would be a synergistic way of facilitating collaborations that involve students, academics, and practitioners, providing constructive feedback to each other in the different stages of applied practice, theses, and research grants.
Professional associations
Last, the advancement of the peer review process can and will continue to benefit from collaborations between professional academic and practitioner associations. As previously mentioned, SIOP recently partnered with CARMA to record video panels as part of an introductory reviewer training. CARMA is also partnering with Organizational Research Methods to promote advanced reviewer development, including video lectures and the publication of special feature topics. Professional development workshops (PDWs) hosted at the Academy of Management’s annual conference and new doctoral student consortia provide excellent vehicles for collaborating to train new and experienced organizational science scholars. Nonprofit organizations, such as the Center for Open Science (COS) or Publons, provide educational resources as well as needed infrastructure to support collaboration. Even publishers have recently started to create reviewer training opportunities (e.g., Springer and Sage). In addition, formal collaborations between professional academic and practitioner associations (such as the Society for Human Resources Management, Human Capital Institute, Australian HR Institute) could be extended to include explicit feedback processes.
We have also observed tremendous advancement of peer-review standards as a result of collaboration, for instance, when creating Editor Ethics 2.0 (https://editorethics.uncc.edu/). In this initiative, a group of editors from I-O psychology and management assembled to establish a voluntary code of conduct. With almost 5,000 journal and professional society signatories (including almost all major publishers), the Transparency and Openness Promotion (TOP) guidelines have been another very successful set of collaborative standards. TOP guidelines focus on, in part, providing suggestions for improving the peer-review process (Nosek et al., Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler and Yarkoni2015). Standards such as these are valuable, as they cut across fields and require collaboration for implementation and evaluation.
The COS and individual research collaborations continue to provide needed meta-science research evaluating the effectiveness of such standards for the peer-review process. These initiatives include evaluating the effectiveness of publication badges (Kidwell et al., Reference Kidwell, Lazarević, Baranski, Hardwicke, Piechowski, Falkenberg and Hess-Holden2016), the potential for bias in the peer review process (Emerson et al., Reference Emerson, Warme, Wolf, Heckman, Brand and Leopold2010), evaluating journal policies related to peer review (Banks et al., Reference Banks, O’Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016a), and compliance with those policies (Hardwicke et al., Reference Hardwicke, Mathur, MacDonald, Nilsonne, Banks, Kidwell and Tessler2018). Further work is needed to continue to evaluate the effectiveness of alternative paths to publication, such as results-blind reviews (Findley, Jensen, Malesky, & Pepinsky, Reference Findley, Jensen, Malesky and Pepinsky2016; Grand et al., Reference Grand, Rogelberg, Banks, Landis and Tonidandel2018b; Woznyj, Grenier, Ross, Banks, & Rogelberg, in press); registered reports (Chambers, Feredoes, Muthukumaraswamy, & Etchells, Reference Chambers, Feredoes, Muthukumaraswamy and Etchells2014); and preprint services launched for psychology (http://psyarxiv.org/), the broader social sciences (http://socarxiv.org/), and earth sciences (http://eartharxiv.org/).
Conclusion
As these suggestions show, it will take the whole professional I-O psychology community to improve our science through excellent peer review. In the current focal article, we take a first step by providing a competency framework for reviewing that defines the critical behaviors that are necessary for good reviewing. Employing this framework, we can initiate more formal reviewer trainings and improve reviewing practice by providing more targeted feedback to reviewers. In addition, by following the guidelines of the competency framework, we can ultimately improve the research that gets published and presented, propelling our science forward. In that way, reviewing can become the cornerstone of our profession on which we build our scientific advancements.
