In his focal article, Murphy (Reference Murphy2021) highlighted the noteworthy issue of descriptive statistics being largely overlooked in our field. Indeed, we agree with Murphy’s assessment that descriptive statistics have considerable untapped value. We further propose that in addition to the advantages outlined by Murphy, a central—and timely—advantage of descriptive statistics is their potential to facilitate greater attention to issues of diversity in our samples, and therefore in our research and extant literature as a whole. Historically, research samples, including those in industrial-organizational (I-O) psychology, have been limited by oversampling from WEIRD—Western, educated, industrialized, rich, and democratic—populations (Henrich et al., Reference Henrich, Heine and Norenzayan2010). This, in turn, has led researchers—and, often, the field as a whole—to overattribute generalizability of findings to populations beyond those on which sufficient data have been collected, and to which findings therefore may not apply. Although some may argue that in certain cases the ramifications of generalizing findings to populations beyond those sampled may arguably be quite benign, assuring appropriately bounded generalizability is nonetheless critical not only for reasons of responsible science but also because, in some instances, the ramifications are indeed nonnegligible (Criado Perez, Reference Criado Perez2021). Moreover, treating WEIRD samples and experiences as default also exacerbates the extent to which systematic bias is built into—and continually reinforced—in our processes, systems, and, for our part, research.
In this commentary, we discuss the implications of the lack of diversity in samples in psychological research, noting how an emphasis on detailed descriptions of samples is a necessary first step in addressing this issue for researchers and practitioners alike. We then draw on these discussions to offer suggestions for researchers as we as a field work toward remedying this situation.
Untapped potential of Table 1 in diversity considerations
There is an increasing recognition in our field that there exists a lack of diversity in our research methodologies and samples. To this end, Ruggs et al. (Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013) highlighted that a great majority of research in I-O psychology fails to take into account differential experiences between minority groups, too often comparing, for example, Whites versus non-Whites. Complementary to this are recent findings from Roberts et al.’s (Reference Roberts, Bareket-Shavit, Dollins, Goldie and Mortenson2020) meta-analysis of more than 26,000 empirical articles published between 1974 and 2018 in top-tier journals across various fields of psychology. Roberts et al. found that psychological publications rarely highlight the diversity of their participants, and too frequently dichotomize the breakdown of racial demographics of their samples (e.g., White vs. non-White). Together, these considerations raise multiple related issues:
Collecting the demographic data
The first step is the necessity of collecting relevant demographic data in the first place. All too often, researchers stop at collecting minimal demographic data—sufficient enough to appease journals, yet with less regard for actually thoroughly considering demographics of import and/or the added value that analyzing such demographic data could contribute to their study, its findings, and its implications. We suggest that, in addition to race, gender, and age of participants, whenever possible authors should also include as standard practice other descriptors of the sample’s demographic identities and diversity of experience. Depending on the focus of the research, this might include demographics such as marital status, parent status, education level, socioeconomic status, sexual orientation, and nationality, to name a few. Doing so will offer researchers, readers, and practitioners alike a more comprehensive understanding of the sample informing the conclusions, thereby ensuring that consumers of the research understand the extent to which the findings are responsibly generalizable, as well as clear indications of where and for whom further research and wider sampling is warranted.
Using the demographic data
Indeed, collecting the demographic data is a necessary first step. Yet, so too is analyzing it in sufficient detail and with enough clarity so as to offer some insight insofar as how the constructs and relationships of interest may function differentially by certain demographics. In some cases, this may be as simple as testing for mean differences and comparing patterns; in other cases, it could be explicit tests of model functioning by different demographic groups (e.g., Mills et al., Reference Mills, Matthews, Henning and Woo2014). Either way, offering this additional information can produce a useful story from the data for practitioners and academics alike, making it more actionable and illuminating patterns and information in the data that would otherwise be left uncovered.
Collapsing across minority demographic categories
Nevertheless, a caution is warranted here concerning the responsible analysis of such descriptive and demographic data. Collapsing across minorities to make generalized statements (e.g., White vs. non-White; Ruggs et al., Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013) as a collective risks leading to the assumption that all minority employees face similar challenges or are otherwise dissimilar from one another in important ways. However, distinct minority groups face notably different issues and challenges in the workplace and beyond; without identifying and analyzing them as distinct groups—or even collecting that information to begin with—we effectively lose the ability to identify and subsequently combat such issues from an optimally informed perspective. In other words, we leave relevant information on the cutting room floor.
Relevance to the research–practice gap
From a usage and implementation perspective, overlooking the value of descriptive statistics in favor of more complex analytic methods risks widening the scientist–practitioner gap, potentially alienating those in practice who may see more obvious value in greater understanding of the wealth of information offered by descriptive statistics. Indeed, it has been suggested (e.g., Islam et al., Reference Islam, Chetta, Martins, van Govan, Kozikowski and Needhammer2018) that academic research is all too often perceived as having limited practical relevance in the applied world. Highhouse et al. (Reference Highhouse, Zickar and Melick2020) recently echoed this contention, arguing that I-O practitioners do not find research in the field to be relevant and believe that it neglects pressing concerns within organizations. Affording greater emphasis to descriptive statistics will not only go toward addressing the scientist–practitioner gap in this way but will also help practitioners better understand the experiences of diverse employees, ultimately better informing organizations’ diversity, equity, and inclusion (DEI) practices.
Recommendations
Much has been made of practical recommendations to facilitate DEI in organizations, and there is no shortage of white papers and other such outreach touting the value of practical strategies to “manage diversity.” Such outreach is crucially important and much needed. Yet, it is also necessary that we take a step back periodically, seeing the forest through the trees, to critically evaluate the science driving such recommendations and ensure that they are indeed grounded in sufficiently diverse research samples at the outset. As such, we offer recommendations to precede such practical recommendations; specifically, we offer four key ways that researchers can work to ensure that they are more fully capitalizing on the potential of descriptives in setting the stage for a more diversity-conscious research landscape in I-O psychology:
Report detailed diversity of samples
Too many studies fail to report (or even collect) sufficient demographics of their samples, or they report simplified dichotomies (e.g., White vs. non-White; Ruggs et al., Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013). We encourage researchers to use Table 1 to report a detailed picture of the demographic makeup of their sample as standard practice. For instance, reporting the breakdown of racial demographics of samples (e.g., 65% White/Caucasian, 10% Black/African American, 7% Asian/Pacific Islander, 6% Indigenous/Native Peoples, 5% Hispanic/Latino, 4% Multiracial, 3% Prefer not to respondFootnote 1) is not only more transparent, but also facilitates comparisons across studies (DeJesus et al., Reference DeJesus, Callanan, Solis and Gelman2019; Roberts et al., Reference Roberts, Bareket-Shavit, Dollins, Goldie and Mortenson2020) and offers valuable information about both likely generalizability as well as future needed research directions in replicating and extending the research to different populations.
Claim reasonable generalizability of the findings
Similar to justifying the sample size of their studies, researchers should justify their sample demographics (Rowley & Camacho, Reference Rowley and Camacho2015). Furthermore, authors should discuss the constraints on generalizability of their findings based on the representativeness of their sample. In so doing, research needs and oversights are responsibly outlined, thereby (a) avoiding the dangerous ramifications of overgeneralizing one’s data to overlooked demographics (e.g., Criado Perez, Reference Criado Perez2021), and (b) providing a clear path forward for future research to replicate and extend the study with particular demographic groups prior to generalizing such findings.
Consider differential experiences and perceptions by demographics
Murphy (Reference Murphy2021) recommended calculating means and standard deviations to investigate whether a treatment or intervention led to a change in the dependent variable. We suggest a similar approach to investigate whether different demographic groups evidenced meaningfully different means and standard deviations on the measures administered. We recommend that research leverage the information in (an expanded) Table 1 to be mindful of differential experiences and perceptions across groups before testing hypotheses. That said, researchers must embark on this thoughtfully and responsibly, considering not only the objective metrics but also other relevant (e.g., structural) considerations that may have played a part in informing quantitative or seemingly objective group differences (e.g., Matarazzo & Wiens, Reference Matarazzo and Wiens1977; Williams, Reference Williams1972). Likewise, researchers must consider where such comparisons make sense (e.g., experiences, perceptions) versus where they are unwarranted or may be inflammatory (e.g., individual differences). Doing so will ensure that the value in the data is sufficiently unpacked, but in a responsible manner.
Collaborate with diverse coauthors
In addition to recruiting diverse participants, we recommend that researchers collaborate with a diverse set of colleagues to ensure a broader range of perspectives informing their research, and a greater breath of experience and knowledge of relevant topics. This inclusive approach will likewise help researchers more fully consider the experiences of a broader range of workers, and think more critically about their findings and even the early methodology of their studies. Notably, however, following through on this final recommendation will require intentionality on the part of researchers in expanding their networks. It will require reaching outside of our comfort zones, our informal networks, and our most immediate colleagues, and investing in—fully committing to—a broader and more diverse set of collaborators with whom to develop and refine our research.
Conclusion
Overall, we encourage researchers to hold themselves accountable by intentionally collecting, explicitly reporting, and responsibly analyzing sufficient demographic descriptives of their samples. Doing so begins with paying much greater attention to the largely untapped potential of Table 1, and giving more credence—and a greater critical eye—to the information contained therein. In so doing, both researchers and practitioners alike can glean from a dataset important information that would otherwise remain overlooked yet can provide crucial insight into how and to whom the findings can be applied, as well as where future research is needed in order to avoid overgeneralizing to insufficiently sampled populations.
You have
Access
In his focal article, Murphy (Reference Murphy2021) highlighted the noteworthy issue of descriptive statistics being largely overlooked in our field. Indeed, we agree with Murphy’s assessment that descriptive statistics have considerable untapped value. We further propose that in addition to the advantages outlined by Murphy, a central—and timely—advantage of descriptive statistics is their potential to facilitate greater attention to issues of diversity in our samples, and therefore in our research and extant literature as a whole. Historically, research samples, including those in industrial-organizational (I-O) psychology, have been limited by oversampling from WEIRD—Western, educated, industrialized, rich, and democratic—populations (Henrich et al., Reference Henrich, Heine and Norenzayan2010). This, in turn, has led researchers—and, often, the field as a whole—to overattribute generalizability of findings to populations beyond those on which sufficient data have been collected, and to which findings therefore may not apply. Although some may argue that in certain cases the ramifications of generalizing findings to populations beyond those sampled may arguably be quite benign, assuring appropriately bounded generalizability is nonetheless critical not only for reasons of responsible science but also because, in some instances, the ramifications are indeed nonnegligible (Criado Perez, Reference Criado Perez2021). Moreover, treating WEIRD samples and experiences as default also exacerbates the extent to which systematic bias is built into—and continually reinforced—in our processes, systems, and, for our part, research.
In this commentary, we discuss the implications of the lack of diversity in samples in psychological research, noting how an emphasis on detailed descriptions of samples is a necessary first step in addressing this issue for researchers and practitioners alike. We then draw on these discussions to offer suggestions for researchers as we as a field work toward remedying this situation.
Untapped potential of Table 1 in diversity considerations
There is an increasing recognition in our field that there exists a lack of diversity in our research methodologies and samples. To this end, Ruggs et al. (Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013) highlighted that a great majority of research in I-O psychology fails to take into account differential experiences between minority groups, too often comparing, for example, Whites versus non-Whites. Complementary to this are recent findings from Roberts et al.’s (Reference Roberts, Bareket-Shavit, Dollins, Goldie and Mortenson2020) meta-analysis of more than 26,000 empirical articles published between 1974 and 2018 in top-tier journals across various fields of psychology. Roberts et al. found that psychological publications rarely highlight the diversity of their participants, and too frequently dichotomize the breakdown of racial demographics of their samples (e.g., White vs. non-White). Together, these considerations raise multiple related issues:
Collecting the demographic data
The first step is the necessity of collecting relevant demographic data in the first place. All too often, researchers stop at collecting minimal demographic data—sufficient enough to appease journals, yet with less regard for actually thoroughly considering demographics of import and/or the added value that analyzing such demographic data could contribute to their study, its findings, and its implications. We suggest that, in addition to race, gender, and age of participants, whenever possible authors should also include as standard practice other descriptors of the sample’s demographic identities and diversity of experience. Depending on the focus of the research, this might include demographics such as marital status, parent status, education level, socioeconomic status, sexual orientation, and nationality, to name a few. Doing so will offer researchers, readers, and practitioners alike a more comprehensive understanding of the sample informing the conclusions, thereby ensuring that consumers of the research understand the extent to which the findings are responsibly generalizable, as well as clear indications of where and for whom further research and wider sampling is warranted.
Using the demographic data
Indeed, collecting the demographic data is a necessary first step. Yet, so too is analyzing it in sufficient detail and with enough clarity so as to offer some insight insofar as how the constructs and relationships of interest may function differentially by certain demographics. In some cases, this may be as simple as testing for mean differences and comparing patterns; in other cases, it could be explicit tests of model functioning by different demographic groups (e.g., Mills et al., Reference Mills, Matthews, Henning and Woo2014). Either way, offering this additional information can produce a useful story from the data for practitioners and academics alike, making it more actionable and illuminating patterns and information in the data that would otherwise be left uncovered.
Collapsing across minority demographic categories
Nevertheless, a caution is warranted here concerning the responsible analysis of such descriptive and demographic data. Collapsing across minorities to make generalized statements (e.g., White vs. non-White; Ruggs et al., Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013) as a collective risks leading to the assumption that all minority employees face similar challenges or are otherwise dissimilar from one another in important ways. However, distinct minority groups face notably different issues and challenges in the workplace and beyond; without identifying and analyzing them as distinct groups—or even collecting that information to begin with—we effectively lose the ability to identify and subsequently combat such issues from an optimally informed perspective. In other words, we leave relevant information on the cutting room floor.
Relevance to the research–practice gap
From a usage and implementation perspective, overlooking the value of descriptive statistics in favor of more complex analytic methods risks widening the scientist–practitioner gap, potentially alienating those in practice who may see more obvious value in greater understanding of the wealth of information offered by descriptive statistics. Indeed, it has been suggested (e.g., Islam et al., Reference Islam, Chetta, Martins, van Govan, Kozikowski and Needhammer2018) that academic research is all too often perceived as having limited practical relevance in the applied world. Highhouse et al. (Reference Highhouse, Zickar and Melick2020) recently echoed this contention, arguing that I-O practitioners do not find research in the field to be relevant and believe that it neglects pressing concerns within organizations. Affording greater emphasis to descriptive statistics will not only go toward addressing the scientist–practitioner gap in this way but will also help practitioners better understand the experiences of diverse employees, ultimately better informing organizations’ diversity, equity, and inclusion (DEI) practices.
Recommendations
Much has been made of practical recommendations to facilitate DEI in organizations, and there is no shortage of white papers and other such outreach touting the value of practical strategies to “manage diversity.” Such outreach is crucially important and much needed. Yet, it is also necessary that we take a step back periodically, seeing the forest through the trees, to critically evaluate the science driving such recommendations and ensure that they are indeed grounded in sufficiently diverse research samples at the outset. As such, we offer recommendations to precede such practical recommendations; specifically, we offer four key ways that researchers can work to ensure that they are more fully capitalizing on the potential of descriptives in setting the stage for a more diversity-conscious research landscape in I-O psychology:
Report detailed diversity of samples
Too many studies fail to report (or even collect) sufficient demographics of their samples, or they report simplified dichotomies (e.g., White vs. non-White; Ruggs et al., Reference Ruggs, Hebl, Law, Cox, Roehling and Wiener2013). We encourage researchers to use Table 1 to report a detailed picture of the demographic makeup of their sample as standard practice. For instance, reporting the breakdown of racial demographics of samples (e.g., 65% White/Caucasian, 10% Black/African American, 7% Asian/Pacific Islander, 6% Indigenous/Native Peoples, 5% Hispanic/Latino, 4% Multiracial, 3% Prefer not to respondFootnote 1) is not only more transparent, but also facilitates comparisons across studies (DeJesus et al., Reference DeJesus, Callanan, Solis and Gelman2019; Roberts et al., Reference Roberts, Bareket-Shavit, Dollins, Goldie and Mortenson2020) and offers valuable information about both likely generalizability as well as future needed research directions in replicating and extending the research to different populations.
Claim reasonable generalizability of the findings
Similar to justifying the sample size of their studies, researchers should justify their sample demographics (Rowley & Camacho, Reference Rowley and Camacho2015). Furthermore, authors should discuss the constraints on generalizability of their findings based on the representativeness of their sample. In so doing, research needs and oversights are responsibly outlined, thereby (a) avoiding the dangerous ramifications of overgeneralizing one’s data to overlooked demographics (e.g., Criado Perez, Reference Criado Perez2021), and (b) providing a clear path forward for future research to replicate and extend the study with particular demographic groups prior to generalizing such findings.
Consider differential experiences and perceptions by demographics
Murphy (Reference Murphy2021) recommended calculating means and standard deviations to investigate whether a treatment or intervention led to a change in the dependent variable. We suggest a similar approach to investigate whether different demographic groups evidenced meaningfully different means and standard deviations on the measures administered. We recommend that research leverage the information in (an expanded) Table 1 to be mindful of differential experiences and perceptions across groups before testing hypotheses. That said, researchers must embark on this thoughtfully and responsibly, considering not only the objective metrics but also other relevant (e.g., structural) considerations that may have played a part in informing quantitative or seemingly objective group differences (e.g., Matarazzo & Wiens, Reference Matarazzo and Wiens1977; Williams, Reference Williams1972). Likewise, researchers must consider where such comparisons make sense (e.g., experiences, perceptions) versus where they are unwarranted or may be inflammatory (e.g., individual differences). Doing so will ensure that the value in the data is sufficiently unpacked, but in a responsible manner.
Collaborate with diverse coauthors
In addition to recruiting diverse participants, we recommend that researchers collaborate with a diverse set of colleagues to ensure a broader range of perspectives informing their research, and a greater breath of experience and knowledge of relevant topics. This inclusive approach will likewise help researchers more fully consider the experiences of a broader range of workers, and think more critically about their findings and even the early methodology of their studies. Notably, however, following through on this final recommendation will require intentionality on the part of researchers in expanding their networks. It will require reaching outside of our comfort zones, our informal networks, and our most immediate colleagues, and investing in—fully committing to—a broader and more diverse set of collaborators with whom to develop and refine our research.
Conclusion
Overall, we encourage researchers to hold themselves accountable by intentionally collecting, explicitly reporting, and responsibly analyzing sufficient demographic descriptives of their samples. Doing so begins with paying much greater attention to the largely untapped potential of Table 1, and giving more credence—and a greater critical eye—to the information contained therein. In so doing, both researchers and practitioners alike can glean from a dataset important information that would otherwise remain overlooked yet can provide crucial insight into how and to whom the findings can be applied, as well as where future research is needed in order to avoid overgeneralizing to insufficiently sampled populations.