How Culturally Unrepresentative Is I-O Research?

To better understand the national (un)representativeness of recent I-O research samples (beyond the general laments of overreliance on Western samples), I examined the nationality of participants in 399 study samples published in 2014 in the five consensus top journals for I-O psychology identified by Bergman and Jean: Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Journal of Organizational Behavior, and Personnel Psychology.Footnote ² Of the 399 samples (including all samples from articles that contained multiple studies), 100 were excluded as meta-analyses, organization-level studies, simulations, reviews, or theoretical articles, leaving 299 studies that were coded for the culture of each sample (displayed in Figure 1, aggregated at the region level). As revealed in Figure 1, this analysis demonstrated that I-O and OB research—at least the research published in these top-tier, U.S.-based journals—does appear to favor samples drawn from North America (136 study samples) and to underrepresent samples from other parts of the world. At the national level, study participants from the United States were by far the most common (appearing in 127 samplesFootnote ³ ), followed by participants from China (32 samples) and Germany (13 samples). No other country reached double-digit representation in the samples. Interestingly, 65 study samples provided no information about the nationality of participants, and an additional 7 study samples noted that participants came from various countries or were employees of a global organization without providing a complete list of participants’ nationalities.

Figure 1.

Nationality of samples from five top industrial–organizational journals (Academy of Management Journal, Journal of Applied Psychology, Journal of Management, Journal of Organizational Behavior, and Personnel Psychology) in 2014, aggregated by United Nations (UN) geoscheme regions. Samples including participants from multiple countries are counted once in each relevant region. Samples with no description of participants’ nationality are counted in the “Not Described” category; samples described as “international” or “from various countries” but without complete country information are counted in the “Multi-National” category.

These relative sample frequencies suggest that I-O research is substantially out of alignment with the makeup of the global workforce. Comparing these sample data with worldwide labor statistics reveals that although U.S. participants appeared in 54.3% of I-O research samples in 2014 (out of the 234 samples that made some mention of the nationality of their participants), they represented only 4.8% of the global labor force in 2013 (The World Bank, 2015). By comparison, individuals from India represented 14.6% of the global labor force (The World Bank, 2015) but appeared in only four study samples (1.7% of samples). Moreover, despite the continued growth of multinational organizations and global work networks, only 15 study samples reported including participants from more than one country. Of those, nine were explicitly cultural studies (e.g., examining the motivation and adjustment of expatriates to international work assignments; Firth, Chen, Kirkman, & Kim, Reference Firth, Chen, Kirkman and Kim2014), and only about half (8 of the 15 samples) noted all of the specific countries from which participants were drawn.

Why Does Cultural Underrepresentation Matter?

The underrepresentation of various national cultures, and of multicultural settings, in I-O research samples is problematic for a variety of reasons. One concern is that culture intersects other sampling biases in the I-O and OB literatures, including the bias toward studying managers and other professionals. As Bergman and Jean note in their review of occupational data, countries vary widely in the percentage of their labor force represented by professionals and managers relative to the percentage of nonprofessional workers, and so issues that disproportionately impact workers likely also disproportionately impact members of certain national cultures (and not others). For instance, occupational death and injury rates (which are overrepresented by individuals in “worker” occupations) vary greatly across nations, ranging from 4.0 fatal injuries (per 100,000 employees in 2008) in Italy to 1.1 in Switzerland and 17.9 in the Dominican Republic (International Labour Organization, 2010). Differing workplace issues observed across occupations are thus likely entwined with differences in nationality/culture and vice versa. Without adequate sampling of different nationalities and cultures, these important organizational issues remain conceptually underspecified, leading researchers to overlook key boundaries or processes that might influence adverse outcomes such as workplace accidents or fatalities.

At the same time, culture influences individuals’ workplace experience even within a single occupational group, as evident by the large (and growing) body of knowledge regarding the effects of culture on management and leadership. A review of this literature is beyond scope here, but scholars have long established that leadership varies across cultures, such that many of the world's cultures do not share the independent, “heroic” leader prototypes characteristic of the United States and other Western nations (e.g., Gerstner & Day, Reference Gerstner and Day1994) and that individuals respond to particular leadership behaviors or tactics differently according to their culture (e.g., Jung & Avolio, Reference Jung and Avolio1999). Indeed, this literature has advanced the study of leadership by looking across cultures, examining the boundary conditions of leader behaviors for workplaces outside the United States (e.g., examining transformational behaviors among Chinese leaders with Chinese followers; Fu, Tsui, Liu, & Li, Reference Fu, Tsui, Liu and Li2010), but also by moving toward a perspective that looks at leadership between cultures, examining how leadership unfolds in multicultural settings with leaders and followers from different countries (e.g., how expatriate leaders from the United States encourage participation among followers in China; Chen & Tjosvold, Reference Chen and Tjosvold2006).

It is this latter emphasis—on multicultural work settings—that brings to the fore perhaps the most pressing challenge of underrepresenting national culture in I-O research samples. Facing an increasingly globalized world of work (enabled in part by the expansion of transnational companies and advancements in remote working technologies; Society for Human Resource Management, 2015), organizational decisions are more likely to impact individuals across multiple countries, and employees are more likely to interact with coworkers from other cultures. In this kind of work environment, nearly all of the organizational practices and individual characteristics on which I-O research focuses are imbued with cultural influences. For instance, the basic choice of what language to use as a lingua franca in a multinational organization differentially affects employees from different cultures, providing some with an unearned boost in status or prestige (by virtue of being a native speaker; Neeley & Dumas, Reference Neeley and Dumas2015) and others with a sense of insecurity and status loss (Neeley, Reference Neeley2013). Similarly, implementing various human resource or development strategies, such as 360° performance feedback, may result in differential engagement and quality of feedback from peers, superiors, and subordinates of different cultures or may induce psychological pressure on particular individuals as they have to “retool” to engage in a form of feedback that is inconsistent with their own cultural values (see Molinsky, Reference Molinsky2013).

If our field aspires to be conceptually precise and practically relevant, it is imperative that our samples be representative of this multicultural work environment. More specifically, if I-O research samples are to accurately reflect a transnational, global work population, where people from multiple countries and cultures work together in a single organization, cultural representativeness cannot be addressed only at the field level (i.e., ensuring representativeness across the many samples published in organizational research) but must also be addressed at the level of the individual study. Simply adding studies situated in another country to the existing literature is necessary (though potentially difficult, given widely perceived editorial biases against replication studies) but insufficient. Future research needs to prioritize truly multicultural samples that better approximate the makeup of global organizations.

More than just an issue of sample-population fidelity, basing future organizational research on multicultural samples will allow leaders in global organizations to make better-informed decisions on how to translate research findings into action. For example, just as a study of the antecedents of training motivation in a sample of U.S. employees may not be fully informative for a training manager in a Peruvian company, it is likely also less informative for a manager of a company consisting of 60% American and 40% Peruvian employees. Aspects of the findings may be universally applicable, but certain culture-specific elements may not. Multicultural studies (particularly when the sample is well-documented) can help managers locate research findings that more closely approximate their real-world organizational setting or at the very least help them make a more informed evaluation and application of the study's findings (e.g., potentially applying findings only to a portion of employees).

What Can Be Done?

This example highlights one critical first step in addressing the cultural misrepresentation of I-O research: better documentation of study samples. As noted earlier, of the 15 multinational study samples published in top I-O journals in 2014, only about half stated all of the countries from which the participants were drawn. The other studies stated simply that it was a multinational sample, provided only broad regions or examples of sampled countries, or noted that some percentage of participants came from the United States while omitting the nationality of the remainder of the sample. In order for managers and other researchers to make informed judgments about the representativeness of a finding, the sample must be more fully described, noting the number of participants from each country sampled, as well as potentially including a table of descriptive statistics by nationality (for an example, see Table 1 of Reiche et al., Reference Reiche, Cardona, Lee, Canela, Akinnukawe, Briscoe and Wilkinson2014).

At a more general level, the findings of the brief analysis presented here reveal a fundamental need for more research drawing from multinational settings. The relative lack of multinational research may stem from the difficulties of collecting such data (see Spector, Liu, & Sanchez, Reference Spector, Liu and Sanchez2015) but may also reflect an inherent bias on the part of researchers, reviewers, and journals in favor of research based in the United States. Indeed, many studies conducted outside of the United States carry a disclaimer in their discussion of limitations that the results may not generalize due to the national setting of the study, but the majority of studies using only U.S. samples seemingly do not carry this disclaimer (despite having similar concerns of generalizability to the global work environment). As Bergman and Jean note with respect to worker occupations, this creates a situation where the use of a U.S. sample is unquestioned but where the use of any other type of sample has to be justified and any differing findings (from the U.S. “norm”) have to be explained. Greater awareness and attention to the need for field- and study-level cultural representativeness can help scholars—whether in the role of researcher or reviewer—appreciate and evaluate the role of data collected from all over the world in building a valuable body of I-O findings.

Many of the traditional barriers to collecting multinational data are coming down, with online survey tools and data panels providing access to a broad range of participants from a number of differing countries and cultures (see Buhrmester, Kwang, & Gosling, Reference Buhrmester, Kwang and Gosling2011; Landers & Behrend, Reference Landers and Behrend2015). However, employees in some countries, and particularly in lower-wage occupations, may still lack access to computers (Bergman & Jean). One opportunity for scholars going forward is to explore the use of mobile devices for conducting multinational research across a variety of occupation levels. Despite lower computer adoption rates, cell phone adoption rates are quite high around the world, with a median of 84% cell phone ownership even in emerging and developing nations, and access to Internet-capable smartphones is also growing (Pew Research Center, 2015). Research that utilizes text messaging, for instance, to engage participants might enable greater access to a broad range of workers across the globe.

Whatever methods are chosen to combat it, cultural underrepresentation in research samples raises significant challenges for research in I-O psychology, contributing to the potential misspecification of constructs in organizational research as well as to missed opportunities for applying findings to practice. Concerted effort is needed to ensure our samples are representative of the differing nationalities and cultures that make up the global labor force lest we erode the validity and utility of our scholarship for the world of work.