Being familiar with their earlier work investigating the factor structures of the Armed Services Vocational Aptitude Battery and the Air Force Officer Qualifying Test, we read with interest Ree, Carretta, and Teachout's (Reference Ree, Carretta and Teachout2015) proposal to extend the idea of a dominant general factor (DGF) beyond the realm of cognitive abilities to other areas of research and practice in industrial–organizational (I-O) psychology. We found their ideas intriguing and their arguments compelling, but we stumbled on a reference to an article of one of the present authors (Lance, Teachout, & Donnelly, Reference Lance, Teachout and Donnelly1992) and Ree et al.’s claim that Lance et al. (Reference Lance, Teachout and Donnelly1992) had found a DGF that accounted for 59% of the variance in job performance ratings in a military job. They did not. Rather, Lance et al. reported a hierarchical confirmatory factor analysis (CFA) model that explained parsimoniously the correlations among 15 job performance first-order factors in terms of four Job Proficiency and four other Measurement Source second-order factors. We reasoned that Ree et al. must have conducted some secondary analysis on the results presented by Lance et al., and indeed we replicated their claim by finding that the first unrotated principal component accounted for 59% of the variance in correlations among the four Proficiency second-order factors reported in Lance et al.’s Table 6.
Skeptics that we are, we sought to determine whether the other studies cited in Ree et al.’s Table 1 to support their claim of ubiquitous DGF effects had been reported as original findings or, rather, as in the case of the Lance et al. study, whether secondary analyses were undertaken to develop the authors’ case for DGFs. We were able to obtain original copies of 20 of the 37 studies cited by Ree et al. The remainder consisted of unpublished data sets, unpublished theses and dissertations, proprietary publications, book chapters, military technical reports, or documents otherwise unavailable to us. We will discuss our insights from our review of these studies, but first we consider Ree et al.’s criteria for a DGF.
Table 1.Summaries of Studies in Which a DGF Was Reported in Original Findings
Criteria for a DGF
Ree et al.’s first criterion for a DGF is that it “must come from the largest source of reliable variance” (p. xx): That is, if it accounts for “a plurality of the variance, it satisfies the first criterion to be considered dominant.” Ree et al. recommend the use of principal components analysis (PCA), which “posits no model but is an orthogonal decomposition that will reproduce the correlation/covariance matrix perfectly.” The first, unrotated principal component is to be regarded as the DGF. As is well known, the first principal component extracted from a square symmetric matrix will always represent the maximum variance explained relative to that explained by orthogonal components extracted subsequently (Harris, Reference Harris1985), by design, except for zero or identity matrices (or, e.g., c*I where c is a constant). That's the way PCA works. By this definition, a DGF is guaranteed to emerge from an orthogonal PCA decomposition of any correlation matrix, and so, in this sense, this criterion is tautologous.
The second criterion for a DGF is that “every variable measuring the construct must be influenced by the DGF” (Ree et al., p. xx); “if the common factor . . . influenced all of the variables, then it satisfies the second criterion to be considered general” (p. xx).Footnote 1 By definition, each of m variables will load on each of the m principal components extracted from a positive definite m × m correlation matrix because PCA is an unrestricted multivariate technique (vs., e.g., CFA, which permits restrictions on model parameter estimates). What we assume Ree et al. meant was that all variables should have nontrivial loadings on a DGF. But what is “nontrivial?” One widely cited cutoff for a nontrivial factor loading is |.4| (Ford, MacCallum, & Tait, Reference Ford, MacCallum and Tait1986). Field, Miles, and Field (Reference Field, Miles and Field2012) generally concur but recommend that “if you think that a loading of .4 will be interesting, you should use a lower cutoff (say, .3), because you don't want to miss a loading of .39” (p. 788). But Stevens (Reference Stevens2002) suggests that what is considered to be a nontrivial factor loading should vary with sample size from |.70| for N = 50 to |.162| for N = 1,000. Clearly, judging what is a “nontrivial” factor loading is a subjective enterprise.
Does the Finding of a DGF Necessarily Mean That a General Factor Dominates the Data?
No. As Ree et al. note, PCA “posits no model” (p. xx). Could some (other) theoretically motivated model serve just as well, or perhaps even better, to explain the data? Of course (Vandenberg & Grelle, Reference Vandenberg, Grelle, Lance and Vandenberg2009). To illustrate this point we return to the Lance et al. (Reference Lance, Teachout and Donnelly1992) study. We conducted a PCA on the matrix of correlations among 15 multitrait, multisource performance first-order factors presented in their appendix. The first principal component explained 42% of the variance and component loadings that ranged between .48 and .76, suggesting, according to Ree et al.’s criteria, the presence of a DGF. In fact, Lance et al. also tested for a general factor using hierarchical CFA. They found that the standardized second-order factor loadings ranged between .23 and .98, and all were greater than three times their standard errors, also suggesting a DGF. However, fit for this model was quite poor by conventional standards, χ2(1814, N = 261) = 3,709.96, p < .001; standardized root-mean-squared residual (SRMSR) = .107, Tucker-Lewis Index (TLI) = .821, relative normed fit index (RNI) = .700. In contrast, a theoretically based model that contained latent Proficiency and Measurement Source second-order factors provided a much better fit to the data, χ2(1787, N = 261) = 2,851.72, p < .001; SRMSR = .067, TLI = .898, RNI = .981. The moral is that although an alleged DGF may be derived from (nearly any) data, it is not a unique explanation, and (perhaps many) other theoretically motivated models may provide a (perhaps much) more complete explanation of the data. That is, armed with relevant theory, one will often achieve a more complete and meaningful explanation of the data at hand.
Our Review of Studies Cited by Ree et al.
Studies in Which Percentage of Variance Accounted for by a DGF Was Reported in Original Findings
Our summary of these studies, of which there were only five, is shown in Table 1. Three of these are from the original body of literature that emanated from the United States Air Force research that concluded there was “not much more than g” (e.g., Ree & Earles, Reference Ree and Earles1991) in some of the Air Force's operational measures. We assumed that these studies analyzed and reported their data correctly, and so we did not attempt replication.
Two of the studies in Table 1 used PCA, three used CFA to model the DGF, and all five studies’ results met the first criterion for demonstrating a DGF (however, see above). Two of the five studies also met the second DGF criterion of all variables having nontrivial loadings on the DGF; one very much did not (Carretta & Ree, Reference Carretta and Ree1997), and it could not be ascertained from the remaining two studies whether the second DGF criterion was met. As such, we do not see that either the identity or the ubiquity of DGFs was established by these five studies.
Studies in Which Percentage of Variance Accounted for by a DGF Was Not Reported in Original Findings
Our summary of these studies is shown in Table 2. In none of these studies was an estimate of variance accounted for by a DGF reported originally, so we attempted to replicate Ree et al.’s estimates of the percentage of variance accounted for by a DGF using data presented in the original article that we best guessed were the data that Ree et al. must have also reanalyzed. The presumed sources of these data are listed in the third column of Table 2. Our first finding was that we substantially replicated their results. In several cases, however (Barbuto & Wheeler, Reference Barbuto and Wheeler2006; Castro, Scandura, & Williams, Reference Castro, Scandura and Williams2004; Chan & Drasgow, Reference Chan and Drasgow2001; Fiori & Antonakis, Reference Fiori and Antonakis2011; Liden & Maslyn, Reference Liden and Maslyn1998; Rahim & Magner, Reference Rahim and Magner1995; Rushton & Irwing, Reference Rushton and Irwing2008), multiple candidate correlation matrices were reported in the original studies, whereas Ree et al. reported only one DGF variance-accounted-for estimate. Interestingly, in every case but one, Ree et al. chose to report the largest of the available estimates (for the Liden & Maslyn study, Ree et al. reported 61%, whereas PCA indicated 64% variance accounted for in the other correlation matrix reported in the same Table 3).
Table 2.Attempted Replications of Results From Studies That Did Not Originally Report DGF Findings
Table 3.DGFs Derived From Random Data
Second, we noted that data were reported at various levels of analysis in these studies, including correlations among item parcels (Fiori & Antonakis, Reference Fiori and Antonakis2011), subscale scores (e.g., Ahmetoglu, Leutner, & Chamorro-Premuzic, Reference Ahmetoglu, Leutner and Chamorro-Premuzic2011; Barbuto & Wheeler, Reference Barbuto and Wheeler2006; Chan & Drasgow, Reference Chan and Drasgow2001), total scale/test scores (e.g., Judge, Van Vianen, & De Pater, Reference Judge, Van Vianen and De Pater2004; Rushton & Irwing, Reference Rushton and Irwing2008), experimental task scores (Ackerman & Cianciolo, Reference Ackerman and Cianciolo2000), first-order factors (Castro et al., Reference Castro, Scandura and Williams2004; Parker, Keefer, & Wood, Reference Parker, Keefer and Wood2011; Rahim & Magner, Reference Rahim and Magner1995), and second-order factors (e.g., Lance et al., Reference Lance, Teachout and Donnelly1992). Of course there is nothing in Ree et al.’s criteria that precludes the discovery of DGFs at various levels of analysis, but this finding leads us to question whether, for example, a DGF derived from correlations among item parcels (Fiori & Antonakis, Reference Fiori and Antonakis2011) affords the same interpretation as a job performance third-order factor, because factors that are deeper in nomological networks are ordinarily attributed much broader explanatory power (Cronbach & Meehl, Reference Cronbach and Meehl1955).
Third, we noted that most of the correlation matrices that Ree et al. apparently reanalyzed were rather small. Figure 1 shows a frequency plot of matrix sizes with the Stauffer, Ree, and Carretta (Reference Stauffer, Ree and Carretta1996) outlier (25 tests) removed. The median number of elements analyzed was four. The chances of satisfying both of Ree et al.’s DGF criteria are increased by conducting PCA on correlations among a small number of at least moderately correlated variables; larger numbers of variables increase the likelihood of a multidimensional solution (Mardia, Kent, & Bibby, Reference Mardia, Kent and Bibby1979) and failure of Ree et al.’s second DGF criterion. Also, with a small number of moderately correlated variables, a DGF is virtually guaranteed to emerge regardless of whether it is meaningful theoretically. For example, we conducted a PCA on the correlations among the four Measurement Source second-order factors reported in Lance et al.’s (Reference Lance, Teachout and Donnelly1992) Table 6 and, not surprisingly, a DGF emerged, accounting for 42% of the variance and principal component loadings ranging between .48 and .76. Correlations among these second-order factors are sensical and are easily interpretable as indicating degree of intersource agreement, but how is a third-order DGF to be interpreted in this case?
Figure 1.Number of elements in correlation matrices analyzed in our replication of Ree et al.’s findings.
We present evidence of DGFs for four other data sets in Table 3. In each case the first principal component explains a plurality of the variance and, with the exception of one loading in Study 4, all principal component loadings meet conventional (if slightly liberal) criteria for nontrivial principal component loadings (see above). However, the data are entirely random; results in Table 3 are from a PCA of 50 observations eachFootnote 2 on four independent (i.e., uncorrelated), normally distributed random variables.Footnote 3 Thus, even when a correlational structure arises entirely from sampling error, PCA still does its job of extracting the maximum variance in the first principal component, possibly creating the illusion of order out of noise.
Summary Points and Conclusions
Ree et al.’s definition of a DGF is tautologous and subjective. By design, the first unrotated principal component will always explain a plurality of the variance in a correlation matrix. Whether the variables’ loadings on the first principal component are nontrivial is, to some extent, in the eyes of the beholder. The definition is operational, not theoretical, though we do appreciate Ree et al.’s speculations as to possible etiologies of DGFs. Still, as presented by Ree et al., DGFs are mathematical rather than theoretical constructions. As such, we question whether DGFs can be afforded equivalent theoretical, psychological interpretations across different construct domains and different levels of analysis. For example, general mental ability qua DGF can be considered a key individual difference characteristic that has biological, educational, nutritive, and other cultural determinants and is itself (in part) a determinant of more specific mental abilities and a wide variety of important life outcomes (Herrnstein & Murray, Reference Herrnstein and Murray1994). In a different domain, a general factor in assessment center performance is probably best and most parsimoniously understood as the tendency to perform generally well (or not so well) across all exercises and dimensions, itself influenced by factors such as General Mental Ability, Job Knowledge and Experience, Conscientiousness, and so on (Hoffman, Kennedy, LoPilato, Monahan, & Lance, Reference Hoffman, Kennedy, LoPilato, Monahan and Lance2015; Meriac, Hoffman, & Woehr, Reference Meriac, Hoffman and Woehr2014). As such, DGFs in different construct domains necessarily have different conceptualizations and different nomological networks. Likewise, DGFs deeper in nomological networks (i.e., higher order factors such as general mental ability) will have broader explanatory power than will those that are closer to observed data (e.g., spatial ability). In brief, DGFs are easily found but, at least so far, not as well understood.
PCA is, by design, prone to discover DGFs and especially in small matrices of moderate to high correlations even if, as we showed earlier, there is no theoretical reason to expect one. PCA is an atheoretical variance decomposition technique that neither provides a unique representation of any data structure nor invites comparisons of alternative structures. Use of more modern modeling methods, such as generalizability theory (G-theory) and CFA, allows much more power and flexibility in this regard. For example, Putka and Hoffman (Reference Putka and Hoffman2013) and Hoffman, Melchers, Blair, Kleinmann, and Ladd (Reference Hoffman, Melchers, Blair, Kleinmann and Ladd2011) have used G-theory and CFA, respectively, to test theoretically derived structures of assessment center data that include a general factor. These are just two examples in which theory of covariance structures can be translated into testable models of data—a component that is missing entirely in the analytic paradigm proposed by Ree et al.
Finally there is the issue of multidimensionality. As Ree et al. showed (and we replicated), a DGF can be found at the top of many multidimensional constructs’ structures, though this is often not the focus of the original research. For q = 3 oblique first-order factors, it is mathematically arbitrary whether they are structured as simply correlated or dependent on a single second-order factor. For q≥4, these alternative structures represent alternative hypotheses that are testable using techniques such as hierarchical CFA but not PCA of the first-order factor correlations. A DGF may be supported as a second-order factor in many hierarchical CFAs but not in others. Either way, multidimensionality will continue to be ubiquitous at least at the level of first-order factors for many constructs.
In summary, (a) the criteria proposed for DGFs and the procedures proposed to detect them set a low bar for demonstrating the ubiquity of DGFs; (b) theory should be viewed as a necessary prerequisite guide to the analysis of covariance structures, including tests of the viability of DGFs; and (c) multidimensionality still reigns, at least at the level of first-order factors. Ree et al.’s proposal to extend the idea of a DGF beyond the cognitive realm is an intriguing concept but one that is not (yet) poised to be dominant in I-O psychology.
%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Being familiar with their earlier work investigating the factor structures of the Armed Services Vocational Aptitude Battery and the Air Force Officer Qualifying Test, we read with interest Ree, Carretta, and Teachout's (Reference Ree, Carretta and Teachout2015) proposal to extend the idea of a dominant general factor (DGF) beyond the realm of cognitive abilities to other areas of research and practice in industrial–organizational (I-O) psychology. We found their ideas intriguing and their arguments compelling, but we stumbled on a reference to an article of one of the present authors (Lance, Teachout, & Donnelly, Reference Lance, Teachout and Donnelly1992) and Ree et al.’s claim that Lance et al. (Reference Lance, Teachout and Donnelly1992) had found a DGF that accounted for 59% of the variance in job performance ratings in a military job. They did not. Rather, Lance et al. reported a hierarchical confirmatory factor analysis (CFA) model that explained parsimoniously the correlations among 15 job performance first-order factors in terms of four Job Proficiency and four other Measurement Source second-order factors. We reasoned that Ree et al. must have conducted some secondary analysis on the results presented by Lance et al., and indeed we replicated their claim by finding that the first unrotated principal component accounted for 59% of the variance in correlations among the four Proficiency second-order factors reported in Lance et al.’s Table 6.
Skeptics that we are, we sought to determine whether the other studies cited in Ree et al.’s Table 1 to support their claim of ubiquitous DGF effects had been reported as original findings or, rather, as in the case of the Lance et al. study, whether secondary analyses were undertaken to develop the authors’ case for DGFs. We were able to obtain original copies of 20 of the 37 studies cited by Ree et al. The remainder consisted of unpublished data sets, unpublished theses and dissertations, proprietary publications, book chapters, military technical reports, or documents otherwise unavailable to us. We will discuss our insights from our review of these studies, but first we consider Ree et al.’s criteria for a DGF.
Summaries of Studies in Which a DGF Was Reported in Original Findings
Note. DGF = dominant general factor; PC = principal component; AFQT = Air Force Officer Qualifying Test; CFA = confirmatory factor analysis; PCA = principal components analysis.
Criteria for a DGF
Ree et al.’s first criterion for a DGF is that it “must come from the largest source of reliable variance” (p. xx): That is, if it accounts for “a plurality of the variance, it satisfies the first criterion to be considered dominant.” Ree et al. recommend the use of principal components analysis (PCA), which “posits no model but is an orthogonal decomposition that will reproduce the correlation/covariance matrix perfectly.” The first, unrotated principal component is to be regarded as the DGF. As is well known, the first principal component extracted from a square symmetric matrix will always represent the maximum variance explained relative to that explained by orthogonal components extracted subsequently (Harris, Reference Harris1985), by design, except for zero or identity matrices (or, e.g., c*I where c is a constant). That's the way PCA works. By this definition, a DGF is guaranteed to emerge from an orthogonal PCA decomposition of any correlation matrix, and so, in this sense, this criterion is tautologous.
The second criterion for a DGF is that “every variable measuring the construct must be influenced by the DGF” (Ree et al., p. xx); “if the common factor . . . influenced all of the variables, then it satisfies the second criterion to be considered general” (p. xx).Footnote 1 By definition, each of m variables will load on each of the m principal components extracted from a positive definite m × m correlation matrix because PCA is an unrestricted multivariate technique (vs., e.g., CFA, which permits restrictions on model parameter estimates). What we assume Ree et al. meant was that all variables should have nontrivial loadings on a DGF. But what is “nontrivial?” One widely cited cutoff for a nontrivial factor loading is |.4| (Ford, MacCallum, & Tait, Reference Ford, MacCallum and Tait1986). Field, Miles, and Field (Reference Field, Miles and Field2012) generally concur but recommend that “if you think that a loading of .4 will be interesting, you should use a lower cutoff (say, .3), because you don't want to miss a loading of .39” (p. 788). But Stevens (Reference Stevens2002) suggests that what is considered to be a nontrivial factor loading should vary with sample size from |.70| for N = 50 to |.162| for N = 1,000. Clearly, judging what is a “nontrivial” factor loading is a subjective enterprise.
Does the Finding of a DGF Necessarily Mean That a General Factor Dominates the Data?
No. As Ree et al. note, PCA “posits no model” (p. xx). Could some (other) theoretically motivated model serve just as well, or perhaps even better, to explain the data? Of course (Vandenberg & Grelle, Reference Vandenberg, Grelle, Lance and Vandenberg2009). To illustrate this point we return to the Lance et al. (Reference Lance, Teachout and Donnelly1992) study. We conducted a PCA on the matrix of correlations among 15 multitrait, multisource performance first-order factors presented in their appendix. The first principal component explained 42% of the variance and component loadings that ranged between .48 and .76, suggesting, according to Ree et al.’s criteria, the presence of a DGF. In fact, Lance et al. also tested for a general factor using hierarchical CFA. They found that the standardized second-order factor loadings ranged between .23 and .98, and all were greater than three times their standard errors, also suggesting a DGF. However, fit for this model was quite poor by conventional standards, χ2(1814, N = 261) = 3,709.96, p < .001; standardized root-mean-squared residual (SRMSR) = .107, Tucker-Lewis Index (TLI) = .821, relative normed fit index (RNI) = .700. In contrast, a theoretically based model that contained latent Proficiency and Measurement Source second-order factors provided a much better fit to the data, χ2(1787, N = 261) = 2,851.72, p < .001; SRMSR = .067, TLI = .898, RNI = .981. The moral is that although an alleged DGF may be derived from (nearly any) data, it is not a unique explanation, and (perhaps many) other theoretically motivated models may provide a (perhaps much) more complete explanation of the data. That is, armed with relevant theory, one will often achieve a more complete and meaningful explanation of the data at hand.
Our Review of Studies Cited by Ree et al.
Studies in Which Percentage of Variance Accounted for by a DGF Was Reported in Original Findings
Our summary of these studies, of which there were only five, is shown in Table 1. Three of these are from the original body of literature that emanated from the United States Air Force research that concluded there was “not much more than g” (e.g., Ree & Earles, Reference Ree and Earles1991) in some of the Air Force's operational measures. We assumed that these studies analyzed and reported their data correctly, and so we did not attempt replication.
Two of the studies in Table 1 used PCA, three used CFA to model the DGF, and all five studies’ results met the first criterion for demonstrating a DGF (however, see above). Two of the five studies also met the second DGF criterion of all variables having nontrivial loadings on the DGF; one very much did not (Carretta & Ree, Reference Carretta and Ree1997), and it could not be ascertained from the remaining two studies whether the second DGF criterion was met. As such, we do not see that either the identity or the ubiquity of DGFs was established by these five studies.
Studies in Which Percentage of Variance Accounted for by a DGF Was Not Reported in Original Findings
Our summary of these studies is shown in Table 2. In none of these studies was an estimate of variance accounted for by a DGF reported originally, so we attempted to replicate Ree et al.’s estimates of the percentage of variance accounted for by a DGF using data presented in the original article that we best guessed were the data that Ree et al. must have also reanalyzed. The presumed sources of these data are listed in the third column of Table 2. Our first finding was that we substantially replicated their results. In several cases, however (Barbuto & Wheeler, Reference Barbuto and Wheeler2006; Castro, Scandura, & Williams, Reference Castro, Scandura and Williams2004; Chan & Drasgow, Reference Chan and Drasgow2001; Fiori & Antonakis, Reference Fiori and Antonakis2011; Liden & Maslyn, Reference Liden and Maslyn1998; Rahim & Magner, Reference Rahim and Magner1995; Rushton & Irwing, Reference Rushton and Irwing2008), multiple candidate correlation matrices were reported in the original studies, whereas Ree et al. reported only one DGF variance-accounted-for estimate. Interestingly, in every case but one, Ree et al. chose to report the largest of the available estimates (for the Liden & Maslyn study, Ree et al. reported 61%, whereas PCA indicated 64% variance accounted for in the other correlation matrix reported in the same Table 3).
Attempted Replications of Results From Studies That Did Not Originally Report DGF Findings
Note. DGF = dominant general factor; PCA = principal components analysis; CFA = confirmatory factor analysis; MIs = manifest indicators; LV = latent variable; SOF = second-order factor; HCFA = hierarchical CFA; FOF = first-order factor; JDI = Job Descriptive Index.
DGFs Derived From Random Data
Note. DGFs = dominant general factors; VAR = variance; PC = principal component.
Second, we noted that data were reported at various levels of analysis in these studies, including correlations among item parcels (Fiori & Antonakis, Reference Fiori and Antonakis2011), subscale scores (e.g., Ahmetoglu, Leutner, & Chamorro-Premuzic, Reference Ahmetoglu, Leutner and Chamorro-Premuzic2011; Barbuto & Wheeler, Reference Barbuto and Wheeler2006; Chan & Drasgow, Reference Chan and Drasgow2001), total scale/test scores (e.g., Judge, Van Vianen, & De Pater, Reference Judge, Van Vianen and De Pater2004; Rushton & Irwing, Reference Rushton and Irwing2008), experimental task scores (Ackerman & Cianciolo, Reference Ackerman and Cianciolo2000), first-order factors (Castro et al., Reference Castro, Scandura and Williams2004; Parker, Keefer, & Wood, Reference Parker, Keefer and Wood2011; Rahim & Magner, Reference Rahim and Magner1995), and second-order factors (e.g., Lance et al., Reference Lance, Teachout and Donnelly1992). Of course there is nothing in Ree et al.’s criteria that precludes the discovery of DGFs at various levels of analysis, but this finding leads us to question whether, for example, a DGF derived from correlations among item parcels (Fiori & Antonakis, Reference Fiori and Antonakis2011) affords the same interpretation as a job performance third-order factor, because factors that are deeper in nomological networks are ordinarily attributed much broader explanatory power (Cronbach & Meehl, Reference Cronbach and Meehl1955).
Third, we noted that most of the correlation matrices that Ree et al. apparently reanalyzed were rather small. Figure 1 shows a frequency plot of matrix sizes with the Stauffer, Ree, and Carretta (Reference Stauffer, Ree and Carretta1996) outlier (25 tests) removed. The median number of elements analyzed was four. The chances of satisfying both of Ree et al.’s DGF criteria are increased by conducting PCA on correlations among a small number of at least moderately correlated variables; larger numbers of variables increase the likelihood of a multidimensional solution (Mardia, Kent, & Bibby, Reference Mardia, Kent and Bibby1979) and failure of Ree et al.’s second DGF criterion. Also, with a small number of moderately correlated variables, a DGF is virtually guaranteed to emerge regardless of whether it is meaningful theoretically. For example, we conducted a PCA on the correlations among the four Measurement Source second-order factors reported in Lance et al.’s (Reference Lance, Teachout and Donnelly1992) Table 6 and, not surprisingly, a DGF emerged, accounting for 42% of the variance and principal component loadings ranging between .48 and .76. Correlations among these second-order factors are sensical and are easily interpretable as indicating degree of intersource agreement, but how is a third-order DGF to be interpreted in this case?
Number of elements in correlation matrices analyzed in our replication of Ree et al.’s findings.
We present evidence of DGFs for four other data sets in Table 3. In each case the first principal component explains a plurality of the variance and, with the exception of one loading in Study 4, all principal component loadings meet conventional (if slightly liberal) criteria for nontrivial principal component loadings (see above). However, the data are entirely random; results in Table 3 are from a PCA of 50 observations eachFootnote 2 on four independent (i.e., uncorrelated), normally distributed random variables.Footnote 3 Thus, even when a correlational structure arises entirely from sampling error, PCA still does its job of extracting the maximum variance in the first principal component, possibly creating the illusion of order out of noise.
Summary Points and Conclusions
Ree et al.’s definition of a DGF is tautologous and subjective. By design, the first unrotated principal component will always explain a plurality of the variance in a correlation matrix. Whether the variables’ loadings on the first principal component are nontrivial is, to some extent, in the eyes of the beholder. The definition is operational, not theoretical, though we do appreciate Ree et al.’s speculations as to possible etiologies of DGFs. Still, as presented by Ree et al., DGFs are mathematical rather than theoretical constructions. As such, we question whether DGFs can be afforded equivalent theoretical, psychological interpretations across different construct domains and different levels of analysis. For example, general mental ability qua DGF can be considered a key individual difference characteristic that has biological, educational, nutritive, and other cultural determinants and is itself (in part) a determinant of more specific mental abilities and a wide variety of important life outcomes (Herrnstein & Murray, Reference Herrnstein and Murray1994). In a different domain, a general factor in assessment center performance is probably best and most parsimoniously understood as the tendency to perform generally well (or not so well) across all exercises and dimensions, itself influenced by factors such as General Mental Ability, Job Knowledge and Experience, Conscientiousness, and so on (Hoffman, Kennedy, LoPilato, Monahan, & Lance, Reference Hoffman, Kennedy, LoPilato, Monahan and Lance2015; Meriac, Hoffman, & Woehr, Reference Meriac, Hoffman and Woehr2014). As such, DGFs in different construct domains necessarily have different conceptualizations and different nomological networks. Likewise, DGFs deeper in nomological networks (i.e., higher order factors such as general mental ability) will have broader explanatory power than will those that are closer to observed data (e.g., spatial ability). In brief, DGFs are easily found but, at least so far, not as well understood.
PCA is, by design, prone to discover DGFs and especially in small matrices of moderate to high correlations even if, as we showed earlier, there is no theoretical reason to expect one. PCA is an atheoretical variance decomposition technique that neither provides a unique representation of any data structure nor invites comparisons of alternative structures. Use of more modern modeling methods, such as generalizability theory (G-theory) and CFA, allows much more power and flexibility in this regard. For example, Putka and Hoffman (Reference Putka and Hoffman2013) and Hoffman, Melchers, Blair, Kleinmann, and Ladd (Reference Hoffman, Melchers, Blair, Kleinmann and Ladd2011) have used G-theory and CFA, respectively, to test theoretically derived structures of assessment center data that include a general factor. These are just two examples in which theory of covariance structures can be translated into testable models of data—a component that is missing entirely in the analytic paradigm proposed by Ree et al.
Finally there is the issue of multidimensionality. As Ree et al. showed (and we replicated), a DGF can be found at the top of many multidimensional constructs’ structures, though this is often not the focus of the original research. For q = 3 oblique first-order factors, it is mathematically arbitrary whether they are structured as simply correlated or dependent on a single second-order factor. For q≥4, these alternative structures represent alternative hypotheses that are testable using techniques such as hierarchical CFA but not PCA of the first-order factor correlations. A DGF may be supported as a second-order factor in many hierarchical CFAs but not in others. Either way, multidimensionality will continue to be ubiquitous at least at the level of first-order factors for many constructs.
In summary, (a) the criteria proposed for DGFs and the procedures proposed to detect them set a low bar for demonstrating the ubiquity of DGFs; (b) theory should be viewed as a necessary prerequisite guide to the analysis of covariance structures, including tests of the viability of DGFs; and (c) multidimensionality still reigns, at least at the level of first-order factors. Ree et al.’s proposal to extend the idea of a DGF beyond the cognitive realm is an intriguing concept but one that is not (yet) poised to be dominant in I-O psychology.