Levels of Analysis: Theorizing about Individuals and Organizations

A primary purpose of organizational research is to estimate theoretically interesting parameters (e.g., the relationship between personality and counterproductive work behavior [CWB] or the relationship between job satisfaction and work withdrawal). Of note, organizational phenomena can reside at different levels of analysis, including the within-group and the between-group levels.

Thorndike (Reference Thorndike1939) and Robinson (Reference Robinson1950) classically demonstrated that within-group correlations and between-group correlations need not be similar (also see discussions by Dansereau, Alluto, & Yammarino, Reference Dansereau, Alluto and Yammarino1984; Ostroff, Reference Ostroff1993; Simpson, Reference Simpson1951; Yule, Reference Yule1903). This is illustrated in Figure 1, where we see that the within-group correlation between X and Y (r _within) can be different from the between-group correlation between X and Y (r _between). What is true within-groups (e.g., each oval has a positive slope, indicating a positive within-group correlation) need not be true between groups (e.g., the slope across ovals is negative, indicating a negative between-group correlation).

Figure 1.

Illustration that the Between-Group X-Y Relationship Can Differ from the Within-Group X-Y Relationship Note. Each oval represents one group. The total correlation, r _total, is the correlation that would have been found if one had ignored group membership, and it is a weighted average of r _within and r _between (see Ostroff, Reference Ostroff1993). (Color online)

As a hypothetical example, suppose we are interested in the relationship between individual-level job satisfaction (X) and individual turnover intentions (Y). We might find that the individual-level relationship between satisfaction and turnover intentions (e.g., the slope of the X-Y line within an oval, or r _within; cf. Figure 1) is negative. When individuals within an organization are more satisfied, they are less likely to plan to quit. At the same time, it is entirely possible that at the organizational level of analysis (e.g., the slope of the X-Y line between different ovals, r _between), the satisfaction–turnover intention relationship could be zero or even positive. That is, it is plausible that entire organizations with high organizational-mean levels of satisfaction might experience higher organizational-mean levels of turnover. One theory potentially implying this is the frame-of-reference effect (Hernandez, Newman, & Jeon, in press; Hulin, Reference Hulin1966), in which organizations in poor/less affluent areas experience higher job satisfaction due to the social comparisons being made by workers to others in the local community. If these same organizations in less affluent geographic areas have workers who intend to leave the poor community for various reasons, then we might observe a positive relationship between job satisfaction and turnover intentions, at the organizational level of analysis (due to the fact that both organization-level satisfaction and organization-level turnover intentions are enhanced by low socioeconomic status conditions). Although this example is contrived, it makes the point that individual-level correlations do not imply group-level correlations of the same magnitude. Previous work has labeled inappropriate generalizations from the organizational level to the individual level as “ecological fallacies,” whereas inappropriate generalizations from the individual level to the organizational level have been labeled “atomistic fallacies” (see Kozlowski & Klein, Reference Kozlowski, Klein, Klein and Kozlowski2000).

External Validity

Equipped with an understanding of levels of analysis, we can now articulate a particular, commonly used notion of external validity. That is, we assert that many organizational researchers who use the term “external validity” are essentially defining external validity as the extent to which the magnitude of an individual-level relationship observed within one or more groups (e.g., r _within) generalizes to other groups. For example, does a published research finding from a single-organization sample generalize to other companies? This definition treats external validity as the absence of group-level moderators of individual-level effects. In Figure 1, external validity would imply that all ovals in a specified population have roughly the same slope (i.e., if the X-Y relationship, r _within, is sampled in one oval, the slope identified in this oval would be similar to that observed in other ovals). Although our current interpretation of external validity is not the only definition of external validity available (see the excellent discussion of external validity definitions by Landers and Behrend), we assert that it is one of the most common notions of external validity used in industrial–organizational (I-O) psychology (i.e., Do the results from this study generalize to other organizations?).

Multi-Organization Samples and External Validity

For organizational researchers who seek to study individual-level phenomena within organizations (e.g., the relationship between justice perceptions and CWB), there is a choice to make about which type of sample to use: single-organization samples versus multi-organization samples. According to Landers and Behrend's (Reference Landers and Behrend2015) typology of convenience samples, organizational samples are usually single-organization samples: “the most typical convenience sample found in I-O psychology journals involves a single organization with which the researcher has some prior relationship” (p. 19). These are the samples that Landers and Behrend characterize as most esteemed by organizational researchers. In contrast, crowdsourced (MTurk) samples, college student samples, online panels, and snowball samples (all of which are less esteemed by organizational researchers) tend to be multi-organization samples, in which the respondents are affiliated with/employed by many different organizations. For example, in an MTurk sample of 200 respondents, it is likely that nearly 200 different organizations are represented (i.e., it is unlikely that a sizeable portion of MTurk respondents work for the same group or organization). As such, any research conducted on MTurk respondents will inherently be estimating group-level phenomena (between groups) rather than within-group/individual-level phenomena (e.g., see Figure 1, but imagine there is only one respondent per group [one respondent per oval]; in such cases, the intraclass correlation [ICC(1)] ≈ 1.00, and all variance is group-level variance [Bliese, Reference Bliese, Klein and Kozlowski2000]; thus, the observed correlations are r _between correlations).

Why does it matter whether one's sample is a single-organization sample versus a multi-organization sample? These two types of samples tend to address two fundamentally different sets of phenomena. On the one hand, a single-organization sample does a good job of estimating individual-level relationships (r _within) but is unable to assess the existence of group-level moderators (i.e., is unable to assess external validity of r _within across groups). On the other hand, a multi-organization sample tends to do a poor job of estimating individual-level relationships (r _within), because almost all of the individual-level variance is confounded with group-level (between-groups) variance (i.e., because there is approximately one individual per group). As such, MTurk researchers are in the unfortunate position of studying r _between while assuming that they are studying r _within (thus committing an ecological fallacy).

Although Landers and Behrend appropriately acknowledge that any use of a convenience sample should include a thoughtful discussion of the external validity of that sample, these authors do not acknowledge that such discussions must recognize that an ecological fallacy has likely been committed in the case of multi-organization convenience samples. Further, with so few respondents per organization, it is typically impossible for researchers with multi-organization convenience samples to meaningfully assess external validity (i.e., to assess the absence of moderators of r _within across groups). Thus MTurk researchers (and researchers using other multi-organization convenience samples) should carefully question whether an MTurk sample is an appropriate sample to use for investigating an individual-level phenomenon, and this issue should be discussed in the article itself.

Therefore, we assert that Landers and Behrend's conclusion, “for studies with the goal of generalizing to the global worker pool, MTurk may be ideal” (2015, p. 18; which implies that the diversity of MTurk samples is beneficial for external validity), requires a caveat. MTurk inferences involving the “global worker pool” should often be limited to inferences about r _between (or perhaps about r _total, which is a combination of r _between and r _within), but if there are differences across organizations in the variables being studied, MTurk data can only support confounded inferences about what happens within organizations (r _within). As such, we suggest that the multi-organizational diversity of MTurk samples can actually harm a common type of external validity inference—because when only one participant from each organization is sampled, it is nearly impossible to infer the extent to which the observed r _between relationship represents the targeted r _within relationship (i.e., r _within relationships are often the focus of I-O psychology, but it is difficult to know whether multi-organization results provide relevant estimates of what happens within organizations).