Validity Distributions Are Generalized in VG, Not Mean Corrected Validity

Tett et al. (Reference Tett, Hundley and Christiansen2017) argue that if the overall ρ distribution has a SDρ that they regard as too large for generalizing the mean rho, and if subgrouping moderator analyses indicate that some or all of the subgroup $\bar{\rho}$ s are different from the overall $\bar{\rho}$ and the corresponding subgroup SDρ values are smaller than the overall SDρ, then the overall ρ distribution (in Tett et al.’s terms, the overall $\bar{\rho}$ ) cannot be the basis for VG because there are moderators within the overall ρ distribution (in Tett et al.’s terms, subgroup $\bar{\rho}$ s different from the overall $\bar{\rho}$ and each other). This conclusion is false.

The initial distribution of ρ still allows VG, because what is generalized is the ρ distribution, not just $\bar{\rho}$ , as we have clarified above. The user can generalize across the moderators; they fall within the credibility interval because they are included in the validity distribution, and the overall distribution indicates that validity (and validity generalization) is present regardless of whether moderators exist or not. Although it is true that the larger SDρ values often found in the overall ρ distribution mean less certainty of the actual value of ρ in any user application, the distribution still indicates the presence of practical validity. This is what is important in determining whether a procedure can be used with positive utility. This is what VG is.

It is important to note that the same principle applies to the subgroup moderator analyses: Although the uncertainty may be reduced somewhat due to a smaller SDρ, there is still considerable uncertainty (as discussed later in our section below on second order sampling error). So there is no qualitative difference between the overall distribution and the moderator-level distributions.

Standards for Generalization and Indices for Precision

Tett et al. (Reference Tett, Hundley and Christiansen2017) state that VG in the sense described here sets a standard for generalization that is too low. This is an arbitrary judgment on their part. Further, their judgment is based on the false assumption that what is generalized in VG is $\bar{\rho}$ , as noted above. Throughout the target article, the authors state that SDρ is the measure of the precision of the estimate of $\bar{\rho}$ . This is not true. The measure of the precision of the estimate of $\bar{\rho}$ is the standard error of $\bar{\rho}$ , (SE $\bar{\rho}$ ), which they do not calculate or present anywhere in their article. It is also missing from their figure 1.Footnote ¹

Although Tett et al. (Reference Tett, Hundley and Christiansen2017) mistakenly use SDρ as the measure of the precision of the $\bar{\rho}$ estimate, they also include a measure of what they call uncertainty in the estimate of the mean observed r ( $\bar{r}$ ) (p. 12); this is the SE of the $\bar{r}$ , symbolized by them as SEr_xy (their symbol omits the usual bar [representing the mean] over r that indicates the SE applies to the $\bar{r}$ ). What should be used here is again the SE of $\bar{\rho}$ . They justify use of the SE of $\bar{r}$ in footnote 15 (p. 449), which states “Practical applications call for generalizability of mean r, not mean rho.” This is incorrect for two reasons. First, what is relevant in VG is ρ, operational validity in selection research; given the focus on mean rho here, it is the mean observed validity corrected for measurement error in the criterion measure and applicable range restriction). It is not r or $\bar{r}$ . Second, it is the distribution of ρ that is generalized not the single point in the distribution that is the $\bar{\rho}$ estimate.

Based on their assumption that it is $\bar{\rho}$ that is generalized in VG, the article by Tett et al. (Reference Tett, Hundley and Christiansen2017) sets up arbitrary standards for generalizability; that is, their standards for generalizing $\bar{\rho}$ . Using a select set of 24 industrial and organizational (I-O) psychology meta-analyses from the literature, they show that when the overall ρ distribution is broken down into subgroup analyses based on potential moderators, the subgroup $\bar{\rho}$ estimates are often different from the $\bar{\rho}$ estimate in the overall ρ distribution, and the SD_ρ estimates are often smaller than the overall SD_ρ estimate. Further, the percent variance accounted for is often larger. These findings are not surprising and are expected. Tett et al.’s article also shows that the SE of the $\bar{r}$ is larger in the subgroup analyses. This means that the estimated $\bar{r}$ is less precise in the subgroup analyses. (Again, SE of $\bar{r}$ is not the relevant statistic; SE of $\bar{\rho}$ should have been used.) This is also expected, because each subgroup analysis is based on fewer studies (k) and a smaller total N. Their “finding” that estimates of $\bar{r}$ have a larger SE when the number of studies (k) is reduced in subgroup analysis has been known for years; it is not new. (But again, the relevant statistic is the SE of $\bar{\rho}$ [which will also be larger in the subgroups], not the SE of $\bar{r}$ .) When they evaluate the results of all meta-analyses (overall analyses and moderator/subgroup analyses) against their arbitrary standards for generalizability of $\bar{\rho}$ , they conclude that these standards are met only in about 4% of the analyses. However, not only are their generalizability standards arbitrary, but the basis for these standards is the false assumption that it is the estimate of $\bar{\rho}$ , and not the distribution of ρ, that is generalized. Hence, Tett et al.’s conclusion that generalizability is rare in the meta-analytic literature is not correct. VG should be interpreted as a matter of degree, not mechanically as a matter of dichotomy (VG or not).

The article by Tett et al. (Reference Tett, Hundley and Christiansen2017) does acknowledge that some artifacts that cause variance in SD_ρ estimates in meta-analytic study results are typically not corrected for, meaning that estimates of SD_ρ are almost always overestimates and thus exaggerate the amount of uncertainty in the ρ distribution. However, they do not make any allowance for this fact when they set up their arbitrary standards for generalizability. This is an important omission, because such considerations are important. In those cases in which it is possible to correct for the effects of artifacts that are not corrected for in other meta-analyses, the results show that such corrections greatly reduce the size of the SD_ρ estimates. For example, in the meta-analytic study by Schmidt et al. (Reference Schmidt, Law, Hunter, Rothstein, Pearlman and McDaniel1993), simply removing non-Pearson rs (which have larger sampling error variances than Pearson rs) led to a large increase in percent variance accounted for and to smaller SD_ρ values. This article lists six other artifacts that the authors were unable to correct for, suggesting that actual SD_ρ values were even smaller than they reported.

Some of the “moderators” listed in Tett et al.’s (Reference Tett, Hundley and Christiansen2017) table 1 are actually artifacts whose effects should be corrected for, not real moderators. Examples include measurement length—multi-item vs. single item, and independent variable and dependent variable share common bias. Also, it is possible that in creating their overall VG analyses, some of the 24 studies included may have violated the requirements of meta-analysis (Schmidt & Hunter, Reference Schmidt and Hunter2015) by including completely different independent and/or dependent variable constructs.Footnote ² An example would be a meta-analysis that included as dependent variable measures of both job performance and counterproductive work behaviors. Another example would be a personality meta-analysis that included several personality constructs (e.g., Conscientiousness, Ambition, and Traditionalism) in the same overall VG analysis. Although it has long been known that only the same or very similar constructs should be included in a meta-analysis, it is not uncommon to see this requirement violated. In such cases, the subsequent subgroup analyses will appear to indicate moderator effects (perhaps large moderator effects) that are not real moderator effects but rather construct effects. This could account for some of the reductions in SD_ρ and increases in percent variance accounted for observed by Tett et al. in moving from the overall VG distribution to moderator subgroupings (i.e., mixing apples and oranges first in the overall meta-analysis and then separating apples from oranges in the subsequent subgroup analyses; Cortina, Reference Cortina2003).

On the Importance of Considering Second-Order Sampling Error in VG Research

Tett et al.’s (Reference Tett, Hundley and Christiansen2017) article also discusses second-order sampling error and publication bias, but does so only in passing. When the goal is to understand the actual distribution of validities, second-order sampling error cannot be ignored. When the number of independent samples contributing to a meta-analysis (k) is small for each subgroup, seemingly different subgroup $\bar{\rho}$ s with associated SD_ρ values (often smaller than the SD_ρ from the overall analysis) are likely to represent an artifactual finding that is due to second-order sampling error, not real moderating effects (see examples in Schmidt & Oh, Reference Schmidt and Oh2013). To the extent that the number of independent samples included in a meta-analysis are small, there is a random chance element involved in uncovering the real distribution of validities (i.e., second order sampling error; Borenstein, Hedges, Higgins, & Rothstein, Reference Borenstein, Hedges, Higgins and Rothstein2009; Cooper & Koenka, Reference Cooper and Koenka2012; Hunter & Schmidt, Reference Hunter and Schmidt2004; Schmidt & Hunter, Reference Schmidt and Hunter2015). Further, publication (availability) bias traditionally conceived, if present, tends to overestimate $\bar{\rho}$ and to underestimate SD_ρ (assuming $\bar{\rho}$ is positive in the population).Footnote ³ Thus, second-order sampling error and publication bias have important implications for VG. Methods for second-order sampling error are covered in detail in Schmidt and Oh (Reference Schmidt and Oh2013).Footnote ⁴ Publication bias has been thoroughly discussed in many places in the literature; for an overview, see Schmidt and Hunter (Reference Schmidt and Hunter2015, chapter 13). Ones, Viswesvaran, and Schmidt (Reference Ones, Viswesvaran and Schmidt2017) provide concrete, valuable recommendations about ensuring comprehensive databases in meta-analytic research. We refer the reader to these sources. Tett et al.’s minimal attendance to potential second-order sampling error in their subgroup analyses is at best naïve and could lead to erroneous conclusions about the actual distributions of validities when comparing results across different subgroupings.

Calling Into Question Previous Research Findings in I-O Psychology

Tett et al.’s (Reference Tett, Hundley and Christiansen2017) article is an example of a recent pattern in the I-O psychology literature, namely attempts to reopen a settled question in I-O psychology with the goal of overturning well established research findings. We have no qualms about overturning established theories and empirical findings based on accumulating data and new evidence. However, we find such attempts to be hollow when (a) there is little or no new empirical evidence that is brought to bear by the newer publications, or when (b) techniques and statistical corrections used are inappropriate and inadequate. An early example is Tett, Jackson, and Rothstein's (Reference Tett, Jackson and Rothstein1991) personality–job performance meta-analysis (see Ones, Mount, Barrick, and Hunter [Reference Ones, Mount, Barrick and Hunter1994] for a methodological critique of this study). Recent examples include the challenges to established findings regarding the lack of differential validity by race of cognitive ability tests (original research by Hunter, Schmidt, and Hunter [1979]), validity of integrity tests (original research by Ones, Viswesvaran, and Schmidt [Reference Ones, Viswesvaran and Schmidt1993]), and measurement error in job performance ratings (original research by Viswesvaran, Ones, and Schmidt [Reference Viswesvaran, Ones and Schmidt1996]). In all these cases, statistical and psychometric misunderstandings led to misinformed analyses and/or inaccurate results. Significant resources and subsequent articles had to be devoted to correcting the scientific record (e.g., Harris et al., Reference Harris, Jones, Klion, Arnold, Camara and Cunningham2012; Ones, Viswesvaran, & Schmidt, Reference Ones, Viswesvaran and Schmidt2012; Roth et al., Reference Roth, Le, Oh, Van Iddekinge, Buster, Robbins and Campion2014; Roth, Le, Oh, Van Iddekinge, & Robbins, Reference Roth, Le, Oh, Van Iddekinge and Robbins2017; Viswesvaran, Ones, & Schmidt, Reference Viswesvaran, Ones and Schmidt2016; and Viswesvaran, Ones, Schmidt, Le, & Oh, Reference Viswesvaran, Ones, Schmidt, Le and Oh2014).

The focal article by Tett et al. (Reference Tett, Hundley and Christiansen2017) attempts to challenge the established principles of meta-analytic VG, contending that VG is “a myth.” Our commentary explains that this attempt is based on a central underlying assumption that is false. The entire validity distribution is generalized in VG, not just mean corrected validity. Tett et al. describe arbitrary and erroneous standards for VG and rely on incorrect indices for precision. Their inadequate attention to second-order sampling error ignores an important potential threat to the veracity of meta-analytic inferences. As a result of these errors, their attempt to challenge established VG principles and findings fails.