Partitioning Variance in Multifaceted Variables

For starters, consider item-level data from two types of measures mentioned by Ree et al. (i.e., cognitive ability tests and assessment centers). When decomposing variance in a simple cognitive ability test with PCA (e.g., a basic test of mathematical reasoning), the variables being analyzed typically differ along a single facet of measurement, namely, items (Cronbach, Gleser, Nanda, & Rajaratnam, Reference Cronbach, Gleser, Nanda and Rajaratnam1972). In contrast, when decomposing variance in item-level data from other, more complex types of measures, such as assessment centers (ACs), interviews, job performance ratings, multisource feedback instruments, or situational judgment tests, the variables being analyzed are often multifaceted in nature (Putka & Sackett, Reference Putka, Sackett, Farr and Tippins2010). For example, in the case of AC data, within-exercise dimension ratings (WEDRs, i.e., ratings on a given dimension within a given exercise) have been framed as item-level variables that reflect combinations of a given dimension (e.g., interpersonal skill) and given exercise (e.g., role play; Kuncel & Sackett, Reference Kuncel and Sackett2014). As another example, variables in a multisource rating data set might reflect combinations of a given rating dimension and given rating source (e.g., peers, subordinates, supervisors; Scullen, Mount, & Goff, Reference Scullen, Mount and Goff2000). As we demonstrate below, applying PCA to multifaceted variables can give a misleading picture of the percentage of variance that is attributable to a general factor because it ignores the multifaceted nature of the variance underlying the first principal component.

Conceptually, the problem with using PCA to estimate the magnitude of a general factor among multifaceted variables is that variance associated with the first principal component fails to distinguish between what Ree et al. describe as a general factor and group factors. One of the primary distinctions between general and group factors is that a general factor influences each observed variable, whereas group factors do not influence all observed variables—only those that are part of the group in question. In the context of the WEDR-level variables common in AC data sets, we know that there will be group factors that manifest in the covariation among WEDRs that share a given dimension and group factors that manifest in the covariation among WEDRs that share a given exercise (e.g., Arthur, Woehr, & Maldegan, Reference Arthur, Woehr and Maldegan2000; Bowler & Woehr, Reference Bowler and Woehr2006; Putka & Hoffman, Reference Putka and Hoffman2013). More generally, group factors will manifest in the covariation among items that share a level of a measurement facet in common (e.g., a specific dimension, specific exercise, specific rating source).

When applying PCA to decompose variance in a set of variables, researchers have long recognized that the variance associated with the first principal component may reflect not only a general source of variance but also group-specific sources of variance and error (e.g., Fabrigar, Wegener, MacCallum, & Strahan, Reference Fabrigar, Wegener, MacCallum and Strahan1999; Floyd, Shands, Rafael, Bergeron, & McGrew, Reference Floyd, Shands, Rafael, Bergeron and McGrew2009; Widaman, Reference Widaman, Cudeck and MacCallum2007). This happens because PCA is not model-based and makes no attempt to distinguish distinct sources of variance underlying any given component. It simply attempts to maximally reproduce observed variance regardless of whether that variance is common (e.g., general, group-specific) or unique (e.g., error). Simply put, a PCA's first component reflects a linear combination of observed variables, each of which reflect multiple sources of variance (Fabrigar et al., Reference Fabrigar, Wegener, MacCallum and Strahan1999). This means that the resulting linear combination of those variables will also be a function of multiple sources of variance, only one of which may be a general factor. As such, it is hazardous to equate variance associated with the first principal component as solely reflecting variance attributable to a general factor. The first principal component and a general factor are two different things. Failure to make this distinction can lead to unwarranted conclusions regarding the magnitude of variance attributable to a general factor based on PCA. As we discuss next, this can be particularly troublesome when analyzing multifaceted variables using PCA, where group-specific variance associated with measurement facets may be particularly strong (e.g., Hoffman, Lance, Bynum, & Gentry, Reference Hoffman, Lance, Bynum and Gentry2010; Lance, Reference Lance2008).

An Alternative to PCA

Given the observations above, an alternative to PCA would be to use a model that is sensitive to the multifaceted variables being analyzed and that allows for better differentiation between general and group-specific variance. The random effects model underlying applications of generalizability theory provides such a model and facilitates providing an empirical illustration of the limitation of PCA outlined above (Cronbach et al., Reference Cronbach, Gleser, Nanda and Rajaratnam1972).

From the perspective of generalizability theory, the closest analogues to the effects of a “general factor” are person main effects (Cronbach et al., Reference Cronbach, Gleser, Nanda and Rajaratnam1972; Shavelson & Webb, Reference Shavelson, Webb, Green, Camilli and Elmore2005). Person main effects imply that some people generally perform better than others do on the measures of interest, regardless of any particular dimension, exercise, rating source, or other design features involved—by definition, they reflect an effect based on all observed variables. To properly estimate variance attributable to person main effects, the random effects model used to decompose observed variance must appropriately reflect the multifaceted nature of the variables being analyzed. For example, in the context of AC data, if there is variance in observed assessment scores specific to a given dimension or exercise, then terms corresponding to dimension- and exercise-related effects should be included in the model used to decompose variance. Failure to include such terms will lead to artificially inflated estimates of variance attributable to person main effects, as covariation among indicators attributable to those group effects will be partially manifest in person main effect variance. In contrast, including such terms would do little harm if dimension- or exercise-related effects happened to be weak or nonexistent. The problem that the model misspecification described above creates for estimating variance attributable to person main effects is analogous to the problem with interpreting the first component in PCA as a general factor. In both cases, the misspecified random effects model and PCA offer no means for differentiating general and group sources of variance. In the case of PCA, this results in a first component that can reflect more than simply general variance. To date, the problem with applying PCA to multifaceted variables may have gone unappreciated by researchers because previous comparisons of PCA to alternative variance partitioning techniques have largely focused on the cognitive ability test domain, where single-facet variables (e.g., simple test items) often comprise the measures being analyzed (e.g., Floyd et al., Reference Floyd, Shands, Rafael, Bergeron and McGrew2009; Widaman, Reference Widaman, Cudeck and MacCallum2007).

Empirical Illustration of the Problem

To empirically illustrate the issues with PCA noted above, we reanalyzed AC data from Putka and Hoffman (Reference Putka and Hoffman2013). Specifically, we evaluated the magnitude of variance accounted for by the first principle component from a PCA and variance accounted for by person main effects from two different random effects models applied to WEDR variables from that study (i.e., variables defined by unique dimension-exercise combinations). The first random effects model was fully specified and included terms that indexed dimension- and exercise-related effects (i.e., the structure of the model reflected the multifaceted nature of the variables being analyzed; Woehr, Putka, & Bowler, Reference Woehr, Putka and Bowler2012). That model provided the following decomposition of observed variance:

(1)$$\begin{equation} \sigma ^2 _{\,\,\,\,\rm observed} \;{\rm = }\;\sigma ^{\rm 2} _{\,\,\,\rm p} \;{\rm + }\;\sigma ^{\rm 2} _{{\,\,\,\rm pd}} \;{\rm + }\;\sigma ^{\rm 2} _{{\,\,\,\rm pe}} \;{\rm + }\;\sigma ^{\rm 2} _{{\,\,\,\rm residual}}\end{equation}$$

In Equation 1, σ²_p reflects the variance due to person main effects, σ²_pd reflects the variance due to dimension-specific effects, σ²_pe reflects variance due to exercise-specific effects, and σ²_residual reflects residual variance.

The second random effects model ignored the multifaceted nature of the data (akin to PCA) and simply treated each WEDR variable as a single faceted item. This second, misspecified random effects model provided the following decomposition of observed variance:

(2)$$\begin{equation} \sigma ^2 _{{\,\,\,\rm observed}} \;{\rm = }\;\sigma ^{\rm 2} _{\,\,\,\rm p} \;{\rm + }\;\sigma ^{\rm 2} _{{\,\,\,\rm residual}}\end{equation}$$

Next, we examined the results from the PCA and random effects model analyses to determine what they implied about the magnitude of a general factor underlying WEDRs. As Table 1 shows across the three samples of AC data from Putka and Hoffman (Reference Putka and Hoffman2013), results of the PCA suggested that the first principal component accounted for between 32% and 36% of the variance in the WEDRs. In contrast, the fully specified random effects model revealed that person main effects accounted for 15% to 20% of the variance in the WEDRs—much less variance than the first principal component. Furthermore, given the person main effect variance component may reflect the contribution of a general factor and covariance among group factors (Woehr et al., Reference Woehr, Putka and Bowler2012), the estimates above can be viewed as upper bounds on the magnitude of variance in WEDRs attributable to a general factor.

Table 1.

Comparison of PCA and Random Effects-Based Estimates

Note. Principal components analysis (PCA) = percentage of within-exercise dimension ratings (WEDR) variance accounted for by the first principal component from the PCA. σ²_{p, M1} = percentage of observed WEDR variance accounted for by person main effects based on the fully specified random effects model. σ²_{p, M2} = percentage of observed WEDR variance accounted for by person main effects based on the misspecified random effects model. Analyses based on samples from Putka and Hoffman (Reference Putka and Hoffman2013).

Examination of the results for the deliberately misspecified random effects model helps explain the differences in estimates above. On the basis of the misspecified random effects model, we find that person main effects now appear to account for 28% to 32% of variance in the WEDRs. Note the estimates from this misspecified model are far closer to the variance accounted for by the first principal component from the PCA, and the estimates demonstrate that failing to account for group-specific variance stemming from specific dimensions and exercises artificially inflates the magnitude of variance attributable to the person main effect.Footnote ¹

To help further illustrate how results regarding the first principal component can be misleading when it comes to informing the presence and dominance of a general factor, we also examined the percentage of observed variance in WEDRs attributable to other components in the fully specified random effects model. This examination revealed that person main effect variance wasn't even the “dominant” source of variance, but rather, group-specific variance attributable to each exercise underlying the WEDRs was dominant. In all samples examined, the percentages of observed variance attributable to exercise-specific effects (σ²_pe) were greater than were those for person main effects (39.4% vs. 20.9% in Sample 1, 48.5% vs. 15.7% in Sample 2, and 39.6% vs. 20.4% in Sample 3). Given simple application of PCA to these data, one would have had no way to realize the findings above and might have come to a very different conclusion regarding the presence and magnitude of a general factor underlying the AC data.

Summary

The moral of our commentary is that it is hazardous to equate the first component of a PCA with a general factor and that this may become particularly problematic when applying PCA to multifaceted variables. Although the first point is evident based on the mathematics underlying PCA (e.g., Widaman, Reference Widaman, Cudeck and MacCallum2007), the latter point has arguably been underappreciated given the common evaluation of PCA within the cognitive abilities domain, where variables are often defined along a single facet of measurement. The variance associated with the first principal component can be viewed as reflecting multiple sources of variance, only one of which corresponds to a general factor. PCA does not help to determine the contribution of a general factor to variance underlying that first component, and interpreting it as such can lead to faulty conclusions regarding the dominance of a general factor, particularly when analyzing multifaceted variables. Random effects models underlying generalizability theory may be useful when confronted with partitioning variance in multifaceted variables and can help provide an upper bound estimate on the contribution of a general factor to observed variance in multifaceted assessment data.