The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) provides a useful framework for promoting robust, rigorous, and reliable reviewing by developing a competency framework for reviewers. We applaud important efforts such as this, aimed at enhancing the peer review process and creating common peer review expectations. It is also useful that Köhler et al. tackled this complex issue from the vantage point of multiple stakeholders (e.g., academia, organizations, granting agencies). In this commentary, we focus specifically on the peer review process in academic journals, given our reviewing and editorial experience. The author team’s experience includes 157 collective years reviewing for peer-reviewed academic journals, membership on 33 different editorial boards, serving as associate editors for 15 different journals, and leading 7 journals as editor-in-chief, guest editor, or special issue editor.
As outlined by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), clear standards for what constitutes an effective review have potential benefits for authors (e.g., more developmental and actionable feedback), reviewers (e.g., clearer reviewing expectations, improved knowledge and skills for reviews), and journal editors and associate editors (henceforth referred to as “editors”; e.g., greater efficiency in processing reviews and reaching decisions), as well as the field as a whole (e.g., improved scientific rigor, reliability, and reproducibility). One of the approaches proposed in the focal article was training on the competency framework (1) in academic classes that discuss peer reviewing, (2) when providing feedback to PhD students who prepare reviews, and (3) at conferences and through the Consortium for the Advancement of Research Methods and Analysis (CARMA).
As important and useful as training-based approaches are, such efforts tend to have a relatively limited reach, relative to the number of authors and reviewers (and editors) for which such training would be beneficial. More specifically, although there is overlap in membership, The Society for Industrial and Organizational Psychology has over 10,000 members, the Human Resource Management and Organizational Behavior Divisions of the Academy of Management have over 9,000 members, and the European Association of Work and Organizational Psychology has approximately 2,000 members. Thus, providing formal training for industrial and organizational (I-O) psychologist reviewers on the large-and-growing scale seems formidable or perhaps even prohibitive. Reviewing is also a volunteer activity, so it may be unrealistic to expect individuals to invest a great deal of time on reviewer training given many superseding obligations (e.g., for faculty, this may include teaching, conducting research, mentoring students, applying for grants, and university as well as external professional service). Efforts aimed at training graduate students in classroom settings and through faculty feedback may also pose challenges, because this requires faculty familiarity with, acceptance of, and demonstrated proficiency in all or most of the features found in Köhler et al.’s (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) competency framework. Moreover, unless required by academic journal policies, reviewer training may become a practice in “preaching to the converted,” meaning that those who select into training might be the most competent and/or eager to improve their reviewing in the first place. Considering all of these factors, it seems as though widespread dissemination using training-based approaches will take considerable time.
The current commentary offers a complementary strategy that has the potential for quicker and perhaps more widespread change in the peer review process. We propose the development and use of methodological reporting checklists by peer-reviewed academic journals. In so doing, we address one of the many issues that are raised in the focal article. Although much narrower in scope than the competency framework proposed by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), our focus on methodological reporting is consistent with several of the defining principles for a robust science of I-O psychology as proposed by Grand et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018): rigor, reproducibility, replication, and transparency/openness. Ideally, a methodological checklist would serve as a guide when preparing (in the “guide to authors” section of journal websites), reviewing (sent to reviewers alongside the manuscript), and accepting (used by editors when processing conditionally accepted manuscripts) empirical manuscripts. In recommending methodological reporting checklists, we realize that to have broad reach, the use of such checklists must be strongly encouraged (or required) by journals. The checklists themselves must be practical as well as tailored to the specific types of articles typically published in a particular journal. When created with these features in mind, a methodological checklist will be easier to use and a more customized complement to various comprehensive reporting guidelines that already exist (e.g., APA, 2008; Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu, & Rao, Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018).
What methodological reporting guidelines already exist and are we using them?
Perhaps the most coordinated effort related to methodological reporting guidelines for the psychological sciences is the Working Group on Journal Article Reporting Standards (JARS). This resulted in an initial set of recommendations (APA, 2008), which has been recently updated (Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018) and now includes a separate set of reporting recommendations for qualitative research (Levitt, Bamberg, Creswell, Frost, Josselson, & Suárez-Orozco, Reference Levitt, Bamberg, Creswell, Frost, Josselson and Suárez-Orozco2018). Despite the fact that these standards have been published for over a decade, there is still a lack of consistent adherence to them in I-O/OB publications (e.g., Banks, Rogelberg, Woznyj, Landis, & Rupp, Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016; Bosco, Aguinis, Field, Pierce, & Dalton, Reference Bosco, Aguinis, Field, Pierce and Dalton2015). Although there is likely a multitude of factors contributing to non-adherence, we argue that one reason is because the JARS standards are very extensive, as they were developed to cover all types of research across all areas of psychology. As a consequence, the guidelines may be so broad and comprehensive as to be overwhelming for authors, reviewers, and editors. In fact, Appelbaum et al.’s (Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018) revised standards include over 11 pages of tables that outline methodological reporting standards.
We contend that the development and use of I-O/OB-relevant checklists that focus solely on methodology can help overcome some of the aforementioned issues. Such checklists would be less overwhelming; more practical, user-friendly, and tailored; and we hope more conducive to widespread adoption as a result. In turn, as others have noted (e.g., Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012), checklists should contribute to principles of robust research by defining a common set of standards for disseminating research, helping to catch reporting errors, improving transparency, and codifying best practices and methodological standards within our discipline (while realizing they are subject to change based on new insights and innovations).
Are I-O/OB journals already providing guidance about methodological reporting?
Banks, Rogelberg et al. (Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016) recommended that I-O/OB journals be more explicit about which research practices and methodological information should be reported. In an effort to understand whether journals have heeded this advice, we reviewed the submission websites of 11 mainstream I-O/OB journals: Academy of Management Journal, Journal of Applied Psychology, Journal of Business and Psychology, Journal of Management, Journal of Occupational Health Psychology, Journal of Occupational and Organizational Psychology, Journal of Vocational Behavior, Leadership Quarterly, Organizational Behavior and Human Decision Processes, Organizational Research Methods, and Personnel Psychology. The majority of these journals provided no information to authors about specific expectations for methodological reporting. A handful of journals (e.g., Journal of Applied Psychology, Journal of Occupational and Organizational Psychology, Personnel Psychology) reference existing standards, including APA’s (2008) Journal Article Reporting Standards (JARS), as well as other standards with seemingly little relevance to I-O/OB (e.g., FORCE11’s Recommended Reporting Guidelines for Life Science Resources; Hooijmans, Leenaars, and Ritskes-Hoitinga’s (Reference Hooijmans, Leenaars and Ritskes-Hoitinga2010) gold standard publication checklist for animal studies). We found that Leadership Quarterly and Journal of Management are the sole journals that provide explicit methodological guidelines on the manuscript submission site; the guidelines include items such as reporting effect sizes, describing all data-related decisions such as dealing with missing data, including correlation matrices, and disclosing data cleaning processes. None of the other journals appear to provide specific and methodological guidelines to authors. Thus, to date there does not seem to be current widespread acceptance of Banks, Rogelberg et al.’s (Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016) suggestions, which indicates that the time is ripe for a push toward the development and use of methodological reporting checklists in I-O/OB research.
What might be included in a methodological reporting checklist?
As a starting point, Table 1 presents a list of methodological elements that can be easily checked and identified in manuscripts by authors, reviewers, and editors. We attest that methodological reporting checklists can increase rigor, replicability, and transparency/openness in the research process. Note that all elements contribute to transparency/openness simply by providing more information about the research process. Certain elements are also applicable to rigor and/or replicability, as indicated in the table.
Table 1.Methodological checklist for improving research quality and reporting consistency
Before describing the elements in this methodological reporting checklist, a few key caveats are in order. First, the material presented in Table 1 is not intended to be exhaustive. Rather, we have identified some (but not all) important methodological elements that are broadly applicable and easily reported. Second, the checklist items in Table 1 are not intended to apply to all empirical studies. Any methodological checklist will require judgment on the part of the researcher, reviewer, and editor as to which elements are relevant and appropriate to include in a research article. Third, we focused our efforts on quantitative methods used in primary studies published in mainstream I-O/OB journals (e.g., Academy of Management Journal, Journal of Applied Psychology, Personnel Psychology, Journal of Management, Journal of Vocational Behavior, Journal of Occupational Health Psychology). Certainly, other checklists can be helpful to I-O/OB research (qualitative data, mixed-methods data, and meta-analyses) but that is not our focus here.
The first four items in Table 1 call for a clear description of the research sample. A complete description of the data should include the sampling plan and recruiting practices (Item 1). Even when research involves a convenience sample or a non-random sample (e.g., company employees, undergraduate psychology students), the population should be clearly defined, along with a description of any systematic sampling strategies applied and the conditions under which participants were recruited and screened. Information about recruiting practices provides critical information to assess generalizability and potential selection effects (e.g., participants who are on social media are much more likely respond to social media requests), as well as factors that might affect participant motivation or create demand characteristics (cf., Highhouse & Gillespie, Reference Highhouse and Gillespie2009).
Researchers should also be explicit about decisions regarding which participants were included and/or excluded in each analysis (Items 2 & 3). Participants might be excluded from the analysis for a variety of reasons, such as missing data, inattentive responding, outlier analyses, and/or failed manipulation checks (also see Item 8). Often the specific criteria for exclusion and the number of participants removed for each reason are not provided in current study reporting. Of particular concern is the practice of excluding participants post hoc, based on the results of the analysis (e.g., removal of outliers). This practice has been rightfully dubbed a questionable research practice and identified as a threat to scientific replication (Banks, O’Boyle et al., Reference Banks, O’Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016). Including final sample sizes for each analysis is also important for the accumulation of knowledge vis-à-vis meta-analysis. Last, a thorough description of research procedures is also important. Including descriptive data on the sample (Item 4) provides important information regarding the populations to which the findings may (or may not) generalize. It is also important for subsequent meta-analytic research, as sociodemographic factors are often examined in meta-analytic research as focal variables and moderators (e.g., sex, race/ethnicity, age, income).
Another important component, one that is key to research rigor, is the description of the measures used to operationalize study variables (Items 5, 6, and 7). Overall, researchers should provide sufficient descriptive information, descriptive statistics (e.g., M, SD, skew when appropriate), and reliability information (e.g., coefficient alpha, test-retest reliability), so that the reader can appropriately understand the construct validity, psychometric quality, and similarity of a study’s measures to other scales used in related research (Item 5). When modifications are made to existing scales (e.g., adapting item content to reflect a different perspective or translating a survey into a different language), the details and rationale for the modification should be clearly explained (Item 6; see Heggestad, Scheaf, Banks, Hausfeld, Tonidandel, & Williams, Reference Heggestad, Scheaf, Banks, Hausfeld, Tonidandel and Williams2019). Where possible, access to the actual items should be provided, in an appendix or online supplement, or through citations to publications that include all original items (Item 7). When proprietary restrictions prevent publishing established measures, sample items should be provided, and sufficient information should be provided for readers to locate the original source. To the extent that measures are not transparently known, journals and readers cannot understand exactly how measure-based research findings were established.
Item 8 asks authors to report manipulation checks and also how the authors treated participants who failed to pass those checks. This item is linked to a larger concern about complete reporting of study designs. Manipulation checks are typically short items, scales, or cognitive tests (e.g., attention tests) that are designed to measure a variable that is presumably closely related to the latent variable that the experimental manipulation targeted. The use of manipulation checks, their reliability, and their exact nature is a controversial topic (Sigall & Mills, Reference Sigall and Mills1998), and we do not suggest that there is a “right” way to deal with failed manipulations. The key point here is that it is difficult to adequately evaluate study findings in the absence of manipulation check information. Authors should report all manipulation checks that were conducted, manipulation check pass rates, and provide full disclosure when no manipulation checks were used.
The last set of elements focuses on reporting and interpretation of analyses. Correlation matrices should include all variables, including control variables (Becker, Atinc, Breaugh, Carlson, Edwards, & Spector, Reference Becker, Atinc, Breaugh, Carlson, Edwards and Spector2016) and sociodemographic information (Item 9). In addition to providing key information about potential multicollinearity and an initial examination of study predictions, some statistical analyses can be replicated using the correlation or covariance matrix (which can be estimated using the Ms and SDs requested in Item 5). For example, such information may be sufficient to run/replicate basic SEM (e.g., Lievens & Conway, Reference Lievens and Conway2001) or linear regression analyses, enabling readers to double-check the reproducibility and plausibility of more complex analyses reported in the manuscript (e.g., where correlations involving interactions and multiplicative terms might also need to be included). Another benefit of reporting full correlation matrices is that meta-analytic reviews relying on correlational studies can frequently include studies that reported a correlation between two variables that were not the central focus of the study.
Item 10 focuses on transparency in model testing. One important practice in this context is to report some measure of uncertainty of the parameter estimates. This estimate can either be the standard errors, the 95% CI, or the test statistic of the parameter estimates. Item 10 also includes clear reporting on each of the statistical models that are actually tested and in what order. Many statistical model evaluation procedures include testing a sequence of nested models that start either with a basic model that increases in complexity at each step or start with a complex model that gets more restricted at each step. Therefore, to evaluate the results of such a sequence of model testing steps fully, it is necessary to include the results of each of the steps in the sequence and the information necessary to recalculate the steps by, for instance, reporting all degrees of freedom correctly (e.g., Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017).
The next item in the checklist (Item 11) suggests that authors provide some effect size estimate. Effect sizes are essential for the interpretation of evidence, help in meta-analytic work, and inform the planning (i.e., power analyses) of future studies. For some more complex models, it can be challenging to report effect sizes. For instance, multilevel models can be described using a variety of different R 2 measures (e.g., Lahuis, Hartman, Hakoyama, & Clark, Reference LaHuis, Hartman, Hakoyama and Clark2014), and the interpretation of coefficients depends on centering decisions (Hofmann & Gavin, Reference Hofmann and Gavin1998). A pragmatic approach in this case may be to report likelihood-based R 2s (e.g., Lang, Bliese, & Runge, Reference Lang, Bliese and Rungein press) and/or a version of the model with all IVs and DVs standardized (e.g., Hox, Reference Hox2002). Effect size measures do not always need to be in the familiar r/d/R 2 metric, especially for methods that are less commonly used in I-O/OB. Odds ratios and hazard ratios are examples; as another example, equations for glms with a logit or probit link can relatively easily be translated into predicted probabilities.
The final item of the checklist (Item 12) encourages authors to be precise when reporting interaction effects. A common practice is to report simple slopes at “higher” and “lower” levels of the moderator (e.g., one SD above and below the mean). Given the variety of samples used in I-O psychology, “high” in one sample can very well be equal to “low” in another sample, and it is accordingly crucial to be precise about the meaning of those tests. In addition, often measures are either positively skewed (e.g., self-reported job performance) or negatively skewed (e.g., abusive supervision). In such cases, it is important to not only report skewness statistics but also to use relative descriptors (e.g., “lower job performance” or “higher abusive supervision”) rather than absolute descriptors (e.g., “low job performance” or “high abusive supervision”), because these descriptors have substantively different meanings and implications.
Potential drawbacks and unintended negative outcomes
Before discussing possible drawbacks, it should be noted that our ideas and recommendations about the use of the methodological checklist and its content are based on our extensive experience as authors, reviewers, and editors. This commentary does not reflect the formal policy of any specific journal (e.g., Journal of Applied Psychology) or professional association (e.g., American Psychological Association), but rather we seek to contribute to the larger conversation concerned with improving journal reviewing and reporting practices.
Although developing methodological reporting checklists is a means toward those ends and has many advantages for reviewers and authors (e.g., reduced uncertainty, improved transparency), the use of such checklists may also engender some drawbacks. One such drawback is that checklists may artificially narrow authors’ approach to analyzing their data and reporting results. Although a checklist highlights the information that needs to be presented, this is not necessarily the only information that could or should be presented. However, authors may merely report what appears to be minimally sufficient, based on the criteria included in the checklist, not digging deeper into other needed reporting or additional exploratory work that might follow the original work as proposed. In fact, authors, reviewers, and even editors may come to view the checklist as an implicit contract, where authors assume that the checklist fulfills the analysis and reporting requirements for the journal review process. Authors holding this assumption may then react with surprise (or even anger) if they are requested to report additional analyses. Methodological checklists may similarly narrow how reviewers and editors approach manuscripts if it leads to a rote practice of simply scanning for the relevant information in order to check the boxes, rather than deeper processing of the meaning and implications of the reported information. For example, in the laudable effort of locating all effect size, confidence interval, and model-fit information found in a checklist, reviewers or editors may neglect another important goal: to consider whether the results presented are substantively interpretable and practically significant. To avoid these pitfalls, checklists must remain a means or a tool for improving journal reviewing and reporting and not come to be seen as end in and of itself.
Concluding thoughts
We are hopeful that more consistent and transparent methodological reporting presents an opportunity to improve the science behind I-O/OB research, based on Munafò et al.’s (Reference Munafò, Nosek, Bishop, Button, Chambers, Du Sert and Ioannidis2017) conclusion that the use of reporting standards in other fields has been highly effective. For example, Stevens et al. (Reference Stevens, Shamseer, Weinstein, Yazdi, Turner, Thielman and Schulz2014) reported higher quality reporting in journals that explicitly endorsed the CONSORT (Consolidated Standards of Reporting Trials) guidelines in medical research compared to those journals that did not endorse those guidelines. Our goal in this commentary was to discuss how the development and use of methodological reporting checklists, such as the one offered, might improve the peer review process and ultimately support a more robust science of I-O psychology. Such a checklist could be used at all stages of the research article lifecycle: by authors when preparing submissions, reviewers when evaluating manuscripts, and editors when examining conditionally accepted manuscripts for publication.
You have
Access
The focal article by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) provides a useful framework for promoting robust, rigorous, and reliable reviewing by developing a competency framework for reviewers. We applaud important efforts such as this, aimed at enhancing the peer review process and creating common peer review expectations. It is also useful that Köhler et al. tackled this complex issue from the vantage point of multiple stakeholders (e.g., academia, organizations, granting agencies). In this commentary, we focus specifically on the peer review process in academic journals, given our reviewing and editorial experience. The author team’s experience includes 157 collective years reviewing for peer-reviewed academic journals, membership on 33 different editorial boards, serving as associate editors for 15 different journals, and leading 7 journals as editor-in-chief, guest editor, or special issue editor.
As outlined by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), clear standards for what constitutes an effective review have potential benefits for authors (e.g., more developmental and actionable feedback), reviewers (e.g., clearer reviewing expectations, improved knowledge and skills for reviews), and journal editors and associate editors (henceforth referred to as “editors”; e.g., greater efficiency in processing reviews and reaching decisions), as well as the field as a whole (e.g., improved scientific rigor, reliability, and reproducibility). One of the approaches proposed in the focal article was training on the competency framework (1) in academic classes that discuss peer reviewing, (2) when providing feedback to PhD students who prepare reviews, and (3) at conferences and through the Consortium for the Advancement of Research Methods and Analysis (CARMA).
As important and useful as training-based approaches are, such efforts tend to have a relatively limited reach, relative to the number of authors and reviewers (and editors) for which such training would be beneficial. More specifically, although there is overlap in membership, The Society for Industrial and Organizational Psychology has over 10,000 members, the Human Resource Management and Organizational Behavior Divisions of the Academy of Management have over 9,000 members, and the European Association of Work and Organizational Psychology has approximately 2,000 members. Thus, providing formal training for industrial and organizational (I-O) psychologist reviewers on the large-and-growing scale seems formidable or perhaps even prohibitive. Reviewing is also a volunteer activity, so it may be unrealistic to expect individuals to invest a great deal of time on reviewer training given many superseding obligations (e.g., for faculty, this may include teaching, conducting research, mentoring students, applying for grants, and university as well as external professional service). Efforts aimed at training graduate students in classroom settings and through faculty feedback may also pose challenges, because this requires faculty familiarity with, acceptance of, and demonstrated proficiency in all or most of the features found in Köhler et al.’s (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020) competency framework. Moreover, unless required by academic journal policies, reviewer training may become a practice in “preaching to the converted,” meaning that those who select into training might be the most competent and/or eager to improve their reviewing in the first place. Considering all of these factors, it seems as though widespread dissemination using training-based approaches will take considerable time.
The current commentary offers a complementary strategy that has the potential for quicker and perhaps more widespread change in the peer review process. We propose the development and use of methodological reporting checklists by peer-reviewed academic journals. In so doing, we address one of the many issues that are raised in the focal article. Although much narrower in scope than the competency framework proposed by Köhler et al. (Reference Köhler, González-Morales, Banks, O’Boyle, Allen, Sinha and Gulick2020), our focus on methodological reporting is consistent with several of the defining principles for a robust science of I-O psychology as proposed by Grand et al. (Reference Grand, Rogelberg, Allen, Landis, Reynolds, Scott and Truxillo2018): rigor, reproducibility, replication, and transparency/openness. Ideally, a methodological checklist would serve as a guide when preparing (in the “guide to authors” section of journal websites), reviewing (sent to reviewers alongside the manuscript), and accepting (used by editors when processing conditionally accepted manuscripts) empirical manuscripts. In recommending methodological reporting checklists, we realize that to have broad reach, the use of such checklists must be strongly encouraged (or required) by journals. The checklists themselves must be practical as well as tailored to the specific types of articles typically published in a particular journal. When created with these features in mind, a methodological checklist will be easier to use and a more customized complement to various comprehensive reporting guidelines that already exist (e.g., APA, 2008; Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu, & Rao, Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018).
What methodological reporting guidelines already exist and are we using them?
Perhaps the most coordinated effort related to methodological reporting guidelines for the psychological sciences is the Working Group on Journal Article Reporting Standards (JARS). This resulted in an initial set of recommendations (APA, 2008), which has been recently updated (Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018) and now includes a separate set of reporting recommendations for qualitative research (Levitt, Bamberg, Creswell, Frost, Josselson, & Suárez-Orozco, Reference Levitt, Bamberg, Creswell, Frost, Josselson and Suárez-Orozco2018). Despite the fact that these standards have been published for over a decade, there is still a lack of consistent adherence to them in I-O/OB publications (e.g., Banks, Rogelberg, Woznyj, Landis, & Rupp, Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016; Bosco, Aguinis, Field, Pierce, & Dalton, Reference Bosco, Aguinis, Field, Pierce and Dalton2015). Although there is likely a multitude of factors contributing to non-adherence, we argue that one reason is because the JARS standards are very extensive, as they were developed to cover all types of research across all areas of psychology. As a consequence, the guidelines may be so broad and comprehensive as to be overwhelming for authors, reviewers, and editors. In fact, Appelbaum et al.’s (Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018) revised standards include over 11 pages of tables that outline methodological reporting standards.
We contend that the development and use of I-O/OB-relevant checklists that focus solely on methodology can help overcome some of the aforementioned issues. Such checklists would be less overwhelming; more practical, user-friendly, and tailored; and we hope more conducive to widespread adoption as a result. In turn, as others have noted (e.g., Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012), checklists should contribute to principles of robust research by defining a common set of standards for disseminating research, helping to catch reporting errors, improving transparency, and codifying best practices and methodological standards within our discipline (while realizing they are subject to change based on new insights and innovations).
Are I-O/OB journals already providing guidance about methodological reporting?
Banks, Rogelberg et al. (Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016) recommended that I-O/OB journals be more explicit about which research practices and methodological information should be reported. In an effort to understand whether journals have heeded this advice, we reviewed the submission websites of 11 mainstream I-O/OB journals: Academy of Management Journal, Journal of Applied Psychology, Journal of Business and Psychology, Journal of Management, Journal of Occupational Health Psychology, Journal of Occupational and Organizational Psychology, Journal of Vocational Behavior, Leadership Quarterly, Organizational Behavior and Human Decision Processes, Organizational Research Methods, and Personnel Psychology. The majority of these journals provided no information to authors about specific expectations for methodological reporting. A handful of journals (e.g., Journal of Applied Psychology, Journal of Occupational and Organizational Psychology, Personnel Psychology) reference existing standards, including APA’s (2008) Journal Article Reporting Standards (JARS), as well as other standards with seemingly little relevance to I-O/OB (e.g., FORCE11’s Recommended Reporting Guidelines for Life Science Resources; Hooijmans, Leenaars, and Ritskes-Hoitinga’s (Reference Hooijmans, Leenaars and Ritskes-Hoitinga2010) gold standard publication checklist for animal studies). We found that Leadership Quarterly and Journal of Management are the sole journals that provide explicit methodological guidelines on the manuscript submission site; the guidelines include items such as reporting effect sizes, describing all data-related decisions such as dealing with missing data, including correlation matrices, and disclosing data cleaning processes. None of the other journals appear to provide specific and methodological guidelines to authors. Thus, to date there does not seem to be current widespread acceptance of Banks, Rogelberg et al.’s (Reference Banks, Rogelberg, Woznyj, Landis and Rupp2016) suggestions, which indicates that the time is ripe for a push toward the development and use of methodological reporting checklists in I-O/OB research.
What might be included in a methodological reporting checklist?
As a starting point, Table 1 presents a list of methodological elements that can be easily checked and identified in manuscripts by authors, reviewers, and editors. We attest that methodological reporting checklists can increase rigor, replicability, and transparency/openness in the research process. Note that all elements contribute to transparency/openness simply by providing more information about the research process. Certain elements are also applicable to rigor and/or replicability, as indicated in the table.
Methodological checklist for improving research quality and reporting consistency
Before describing the elements in this methodological reporting checklist, a few key caveats are in order. First, the material presented in Table 1 is not intended to be exhaustive. Rather, we have identified some (but not all) important methodological elements that are broadly applicable and easily reported. Second, the checklist items in Table 1 are not intended to apply to all empirical studies. Any methodological checklist will require judgment on the part of the researcher, reviewer, and editor as to which elements are relevant and appropriate to include in a research article. Third, we focused our efforts on quantitative methods used in primary studies published in mainstream I-O/OB journals (e.g., Academy of Management Journal, Journal of Applied Psychology, Personnel Psychology, Journal of Management, Journal of Vocational Behavior, Journal of Occupational Health Psychology). Certainly, other checklists can be helpful to I-O/OB research (qualitative data, mixed-methods data, and meta-analyses) but that is not our focus here.
The first four items in Table 1 call for a clear description of the research sample. A complete description of the data should include the sampling plan and recruiting practices (Item 1). Even when research involves a convenience sample or a non-random sample (e.g., company employees, undergraduate psychology students), the population should be clearly defined, along with a description of any systematic sampling strategies applied and the conditions under which participants were recruited and screened. Information about recruiting practices provides critical information to assess generalizability and potential selection effects (e.g., participants who are on social media are much more likely respond to social media requests), as well as factors that might affect participant motivation or create demand characteristics (cf., Highhouse & Gillespie, Reference Highhouse and Gillespie2009).
Researchers should also be explicit about decisions regarding which participants were included and/or excluded in each analysis (Items 2 & 3). Participants might be excluded from the analysis for a variety of reasons, such as missing data, inattentive responding, outlier analyses, and/or failed manipulation checks (also see Item 8). Often the specific criteria for exclusion and the number of participants removed for each reason are not provided in current study reporting. Of particular concern is the practice of excluding participants post hoc, based on the results of the analysis (e.g., removal of outliers). This practice has been rightfully dubbed a questionable research practice and identified as a threat to scientific replication (Banks, O’Boyle et al., Reference Banks, O’Boyle, Pollack, White, Batchelor, Whelpley and Adkins2016). Including final sample sizes for each analysis is also important for the accumulation of knowledge vis-à-vis meta-analysis. Last, a thorough description of research procedures is also important. Including descriptive data on the sample (Item 4) provides important information regarding the populations to which the findings may (or may not) generalize. It is also important for subsequent meta-analytic research, as sociodemographic factors are often examined in meta-analytic research as focal variables and moderators (e.g., sex, race/ethnicity, age, income).
Another important component, one that is key to research rigor, is the description of the measures used to operationalize study variables (Items 5, 6, and 7). Overall, researchers should provide sufficient descriptive information, descriptive statistics (e.g., M, SD, skew when appropriate), and reliability information (e.g., coefficient alpha, test-retest reliability), so that the reader can appropriately understand the construct validity, psychometric quality, and similarity of a study’s measures to other scales used in related research (Item 5). When modifications are made to existing scales (e.g., adapting item content to reflect a different perspective or translating a survey into a different language), the details and rationale for the modification should be clearly explained (Item 6; see Heggestad, Scheaf, Banks, Hausfeld, Tonidandel, & Williams, Reference Heggestad, Scheaf, Banks, Hausfeld, Tonidandel and Williams2019). Where possible, access to the actual items should be provided, in an appendix or online supplement, or through citations to publications that include all original items (Item 7). When proprietary restrictions prevent publishing established measures, sample items should be provided, and sufficient information should be provided for readers to locate the original source. To the extent that measures are not transparently known, journals and readers cannot understand exactly how measure-based research findings were established.
Item 8 asks authors to report manipulation checks and also how the authors treated participants who failed to pass those checks. This item is linked to a larger concern about complete reporting of study designs. Manipulation checks are typically short items, scales, or cognitive tests (e.g., attention tests) that are designed to measure a variable that is presumably closely related to the latent variable that the experimental manipulation targeted. The use of manipulation checks, their reliability, and their exact nature is a controversial topic (Sigall & Mills, Reference Sigall and Mills1998), and we do not suggest that there is a “right” way to deal with failed manipulations. The key point here is that it is difficult to adequately evaluate study findings in the absence of manipulation check information. Authors should report all manipulation checks that were conducted, manipulation check pass rates, and provide full disclosure when no manipulation checks were used.
The last set of elements focuses on reporting and interpretation of analyses. Correlation matrices should include all variables, including control variables (Becker, Atinc, Breaugh, Carlson, Edwards, & Spector, Reference Becker, Atinc, Breaugh, Carlson, Edwards and Spector2016) and sociodemographic information (Item 9). In addition to providing key information about potential multicollinearity and an initial examination of study predictions, some statistical analyses can be replicated using the correlation or covariance matrix (which can be estimated using the Ms and SDs requested in Item 5). For example, such information may be sufficient to run/replicate basic SEM (e.g., Lievens & Conway, Reference Lievens and Conway2001) or linear regression analyses, enabling readers to double-check the reproducibility and plausibility of more complex analyses reported in the manuscript (e.g., where correlations involving interactions and multiplicative terms might also need to be included). Another benefit of reporting full correlation matrices is that meta-analytic reviews relying on correlational studies can frequently include studies that reported a correlation between two variables that were not the central focus of the study.
Item 10 focuses on transparency in model testing. One important practice in this context is to report some measure of uncertainty of the parameter estimates. This estimate can either be the standard errors, the 95% CI, or the test statistic of the parameter estimates. Item 10 also includes clear reporting on each of the statistical models that are actually tested and in what order. Many statistical model evaluation procedures include testing a sequence of nested models that start either with a basic model that increases in complexity at each step or start with a complex model that gets more restricted at each step. Therefore, to evaluate the results of such a sequence of model testing steps fully, it is necessary to include the results of each of the steps in the sequence and the information necessary to recalculate the steps by, for instance, reporting all degrees of freedom correctly (e.g., Cortina, Green, Keeler, & Vandenberg, Reference Cortina, Green, Keeler and Vandenberg2017).
The next item in the checklist (Item 11) suggests that authors provide some effect size estimate. Effect sizes are essential for the interpretation of evidence, help in meta-analytic work, and inform the planning (i.e., power analyses) of future studies. For some more complex models, it can be challenging to report effect sizes. For instance, multilevel models can be described using a variety of different R 2 measures (e.g., Lahuis, Hartman, Hakoyama, & Clark, Reference LaHuis, Hartman, Hakoyama and Clark2014), and the interpretation of coefficients depends on centering decisions (Hofmann & Gavin, Reference Hofmann and Gavin1998). A pragmatic approach in this case may be to report likelihood-based R 2s (e.g., Lang, Bliese, & Runge, Reference Lang, Bliese and Rungein press) and/or a version of the model with all IVs and DVs standardized (e.g., Hox, Reference Hox2002). Effect size measures do not always need to be in the familiar r/d/R 2 metric, especially for methods that are less commonly used in I-O/OB. Odds ratios and hazard ratios are examples; as another example, equations for glms with a logit or probit link can relatively easily be translated into predicted probabilities.
The final item of the checklist (Item 12) encourages authors to be precise when reporting interaction effects. A common practice is to report simple slopes at “higher” and “lower” levels of the moderator (e.g., one SD above and below the mean). Given the variety of samples used in I-O psychology, “high” in one sample can very well be equal to “low” in another sample, and it is accordingly crucial to be precise about the meaning of those tests. In addition, often measures are either positively skewed (e.g., self-reported job performance) or negatively skewed (e.g., abusive supervision). In such cases, it is important to not only report skewness statistics but also to use relative descriptors (e.g., “lower job performance” or “higher abusive supervision”) rather than absolute descriptors (e.g., “low job performance” or “high abusive supervision”), because these descriptors have substantively different meanings and implications.
Potential drawbacks and unintended negative outcomes
Before discussing possible drawbacks, it should be noted that our ideas and recommendations about the use of the methodological checklist and its content are based on our extensive experience as authors, reviewers, and editors. This commentary does not reflect the formal policy of any specific journal (e.g., Journal of Applied Psychology) or professional association (e.g., American Psychological Association), but rather we seek to contribute to the larger conversation concerned with improving journal reviewing and reporting practices.
Although developing methodological reporting checklists is a means toward those ends and has many advantages for reviewers and authors (e.g., reduced uncertainty, improved transparency), the use of such checklists may also engender some drawbacks. One such drawback is that checklists may artificially narrow authors’ approach to analyzing their data and reporting results. Although a checklist highlights the information that needs to be presented, this is not necessarily the only information that could or should be presented. However, authors may merely report what appears to be minimally sufficient, based on the criteria included in the checklist, not digging deeper into other needed reporting or additional exploratory work that might follow the original work as proposed. In fact, authors, reviewers, and even editors may come to view the checklist as an implicit contract, where authors assume that the checklist fulfills the analysis and reporting requirements for the journal review process. Authors holding this assumption may then react with surprise (or even anger) if they are requested to report additional analyses. Methodological checklists may similarly narrow how reviewers and editors approach manuscripts if it leads to a rote practice of simply scanning for the relevant information in order to check the boxes, rather than deeper processing of the meaning and implications of the reported information. For example, in the laudable effort of locating all effect size, confidence interval, and model-fit information found in a checklist, reviewers or editors may neglect another important goal: to consider whether the results presented are substantively interpretable and practically significant. To avoid these pitfalls, checklists must remain a means or a tool for improving journal reviewing and reporting and not come to be seen as end in and of itself.
Concluding thoughts
We are hopeful that more consistent and transparent methodological reporting presents an opportunity to improve the science behind I-O/OB research, based on Munafò et al.’s (Reference Munafò, Nosek, Bishop, Button, Chambers, Du Sert and Ioannidis2017) conclusion that the use of reporting standards in other fields has been highly effective. For example, Stevens et al. (Reference Stevens, Shamseer, Weinstein, Yazdi, Turner, Thielman and Schulz2014) reported higher quality reporting in journals that explicitly endorsed the CONSORT (Consolidated Standards of Reporting Trials) guidelines in medical research compared to those journals that did not endorse those guidelines. Our goal in this commentary was to discuss how the development and use of methodological reporting checklists, such as the one offered, might improve the peer review process and ultimately support a more robust science of I-O psychology. Such a checklist could be used at all stages of the research article lifecycle: by authors when preparing submissions, reviewers when evaluating manuscripts, and editors when examining conditionally accepted manuscripts for publication.