Over the years, many in the field of organizational psychology have claimed that meta-analytic tests for moderators provide evidence for validity generalization (Schmidt & Hunter, Reference Schmidt and Hunter1977), a term first used in the middle of the last century (Mosier, Reference Mosier1950). In response, Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017) advise caution when it comes to our inclination toward generalizing findings across workplaces/domains and urge precision in attaching meaning to the statistic we are generalizing. Their focal article was insightful and offers important recommendations for researchers regarding certain statistical indicators of unexplained variability, such as SDρ. In this commentary, we would like to make a different point about SDρ, namely that it, and other statistics based on residual variance, will be deflated due to the lack of variance in moderators. It is this lack of between-study variance, as much as anything else, that leads to misguided conclusions about validity generalization.
First, we would amplify the authors’ position that validity generalization has been overstated and that as a field we should look for moderators even when some of these residual variance indicators are consistent with a sampling error explanation. Consider the example of Table 1, loosely based on the socioeconomic status-police brutality example from Hunter and Schmidt (Reference Schmidt and Hunter2004). Culture researchers describe differences between cultures with respect to teamwork and cooperation. Wagner (Reference Wagner1995) and others have hypothesized that there is a stronger relationship between identifiability and cooperation in individualistic cultures than in collectivistic cultures because norms in collectivistic cultures ensure that cooperation occurs irrespective of identifiability. Let us suppose that a meta-analysis were conducted, with results such as those in Table 1. The variance among these correlations (.015) is roughly equal to the variance that one would expect by chance given the sample sizes. Thus, the pattern of correlation is consistent with a sampling error explanation. This could lead researchers to conclude that there is no reason to pursue cultural moderator variables. However, the pattern is also consistent with Wagner's (Reference Wagner1995) hypothesis that the correlation would be weaker in collectivist cultures. Therefore, even though there is not “residual variance,” this does not actually indicate a lack of moderation. This example demonstrates the fact that residual variance and its derivatives (e.g., SDρ) are insufficient for deciding whether one should search for moderators.
Table 1.
Hypothetical Correlations Between Accountability and Cooperation in Collectivistic Versus Individualistic Cultures
The primary point of our commentary, however, is that whichever approach one uses in searching for moderators, those searches are likely to be fruitless due to lack of between-study variance in moderators, particularly those moderators that are substantive rather than methodological. Such moderators tend to vary far more within studies than between studies. As a result, meta-analytic results that appear to support validity generalization may instead mask situation specificity because of a lack of moderator variance.
This issue is addressed in Markell, Lei, and Foroughi (Reference Markell, Lei and Foroughi2017), in which we estimate the amount of variance in meta-analytic moderators, whether it varies by type of moderator, and the degree to which the amount of variance is linked to success in finding an interaction. Thus far, we have coded more than 700 moderators from dozens of meta-analyses.
Our preliminary results support our hypothesis that restricted between-studies variance is the key culprit behind low moderator detection rates. Although we have conducted many analyses, two findings are most relevant here. First, we found that between-study variance in moderators is typically much smaller than it could be. For example, whereas the optimal variance of a dichotomous moderator is .25, we found that the average variance of a categorical moderator was .14, which is less than 60% of the possible variance. Furthermore, one-third of the variances are less than half the possible maximum value, that is to say, 33% of the values fall in the range of 0 to .12. We found that moderators that were found to be significant had significantly larger variances than moderators that were nonsignificant (p < .000).
This lack of between-study variance in moderators would explain some of the findings from Cortina (Reference Cortina2003). Unlike Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017), Cortina's (Reference Cortina2003) review of 59 meta-analyses showed little tendency for residual variance and related statistics to decrease as a function of specificity. Cortina concluded that “breaking studies down by potential moderators does little in the way of increasing our confidence in the notion that meta-analyses typically isolate those studies producing values sampled from a single population” (2003, p. 426). Thus, we have no way of knowing from meta-analyses whether validity generalization exists or not because we lack the between-study variance in moderators across which validities might (or might not) generalize. Meta-analytic oversampling of studies with extreme values of moderator variables (e.g., primary studies using all-female samples and studies using all-male samples) does not seem feasible. What would help would be for authors of primary studies to include in their correlation matrices products of predictors and common moderators, even if those interactions are not hypothesized. This would allow the meta-analysis of relationships between residualized products and outcomes, of regression weights of products, and so on that reflect within-study (rather than between-study) variance in moderators.
You have
Access
Over the years, many in the field of organizational psychology have claimed that meta-analytic tests for moderators provide evidence for validity generalization (Schmidt & Hunter, Reference Schmidt and Hunter1977), a term first used in the middle of the last century (Mosier, Reference Mosier1950). In response, Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017) advise caution when it comes to our inclination toward generalizing findings across workplaces/domains and urge precision in attaching meaning to the statistic we are generalizing. Their focal article was insightful and offers important recommendations for researchers regarding certain statistical indicators of unexplained variability, such as SDρ. In this commentary, we would like to make a different point about SDρ, namely that it, and other statistics based on residual variance, will be deflated due to the lack of variance in moderators. It is this lack of between-study variance, as much as anything else, that leads to misguided conclusions about validity generalization.
First, we would amplify the authors’ position that validity generalization has been overstated and that as a field we should look for moderators even when some of these residual variance indicators are consistent with a sampling error explanation. Consider the example of Table 1, loosely based on the socioeconomic status-police brutality example from Hunter and Schmidt (Reference Schmidt and Hunter2004). Culture researchers describe differences between cultures with respect to teamwork and cooperation. Wagner (Reference Wagner1995) and others have hypothesized that there is a stronger relationship between identifiability and cooperation in individualistic cultures than in collectivistic cultures because norms in collectivistic cultures ensure that cooperation occurs irrespective of identifiability. Let us suppose that a meta-analysis were conducted, with results such as those in Table 1. The variance among these correlations (.015) is roughly equal to the variance that one would expect by chance given the sample sizes. Thus, the pattern of correlation is consistent with a sampling error explanation. This could lead researchers to conclude that there is no reason to pursue cultural moderator variables. However, the pattern is also consistent with Wagner's (Reference Wagner1995) hypothesis that the correlation would be weaker in collectivist cultures. Therefore, even though there is not “residual variance,” this does not actually indicate a lack of moderation. This example demonstrates the fact that residual variance and its derivatives (e.g., SDρ) are insufficient for deciding whether one should search for moderators.
Hypothetical Correlations Between Accountability and Cooperation in Collectivistic Versus Individualistic Cultures
The primary point of our commentary, however, is that whichever approach one uses in searching for moderators, those searches are likely to be fruitless due to lack of between-study variance in moderators, particularly those moderators that are substantive rather than methodological. Such moderators tend to vary far more within studies than between studies. As a result, meta-analytic results that appear to support validity generalization may instead mask situation specificity because of a lack of moderator variance.
This issue is addressed in Markell, Lei, and Foroughi (Reference Markell, Lei and Foroughi2017), in which we estimate the amount of variance in meta-analytic moderators, whether it varies by type of moderator, and the degree to which the amount of variance is linked to success in finding an interaction. Thus far, we have coded more than 700 moderators from dozens of meta-analyses.
Our preliminary results support our hypothesis that restricted between-studies variance is the key culprit behind low moderator detection rates. Although we have conducted many analyses, two findings are most relevant here. First, we found that between-study variance in moderators is typically much smaller than it could be. For example, whereas the optimal variance of a dichotomous moderator is .25, we found that the average variance of a categorical moderator was .14, which is less than 60% of the possible variance. Furthermore, one-third of the variances are less than half the possible maximum value, that is to say, 33% of the values fall in the range of 0 to .12. We found that moderators that were found to be significant had significantly larger variances than moderators that were nonsignificant (p < .000).
This lack of between-study variance in moderators would explain some of the findings from Cortina (Reference Cortina2003). Unlike Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017), Cortina's (Reference Cortina2003) review of 59 meta-analyses showed little tendency for residual variance and related statistics to decrease as a function of specificity. Cortina concluded that “breaking studies down by potential moderators does little in the way of increasing our confidence in the notion that meta-analyses typically isolate those studies producing values sampled from a single population” (2003, p. 426). Thus, we have no way of knowing from meta-analyses whether validity generalization exists or not because we lack the between-study variance in moderators across which validities might (or might not) generalize. Meta-analytic oversampling of studies with extreme values of moderator variables (e.g., primary studies using all-female samples and studies using all-male samples) does not seem feasible. What would help would be for authors of primary studies to include in their correlation matrices products of predictors and common moderators, even if those interactions are not hypothesized. This would allow the meta-analysis of relationships between residualized products and outcomes, of regression weights of products, and so on that reflect within-study (rather than between-study) variance in moderators.