SKDA Defined

Work by Kraut (Reference Kraut and Kraut2006), Lundby and Johnson (Reference Lundby, Johnson and Kraut2006), Rasch et al. (Reference Rasch, Kowske and Herman2009), and Leonard and Lieberman (Reference Leonard and Lieberman2006) provides the framework and reasoning behind SKDA. For the purposes of our review, we are focused on these authors’ operationalization of SKDA. This analysis identifies a set of survey items that has the greatest impact on either a single outcome item or an index score (computed from several outcome items). The criterion score represents a key indicator of organizational effectiveness such as overall job satisfaction or turnover intention. The set of items is seen as “driving” the results for the outcome measure and is meant to provide organizational stakeholders a means of focusing limited resources on issues that will yield the greatest improvements in organizational effectiveness. Survey key driver items are identified through analyses based on the general linear model, such as correlation coefficients (including zero-order, partial, or semipartial correlation coefficients) and regression analyses (including stepwise and multiple regression or relative weights analysis). Individual employees’ responses are used as the cases, or unit of analysis. A unique set of survey key driver items is computed for each organizational entity, with the implicit assumption that drivers are specific to each entity. Although SKDA could be conducted using scale scores as predictors, we focus our focal article on the use of item scores as predictors as this approach is more prevalent. We also focus on the use of a single criterion item, as it is what we have observed most frequently in practice.

Issue 2: SKDA Capitalizes on Chance

SKDA examines the relationship between multiple items with a criterion item and then selects those items that best predict the criterion item. Essentially SKDA resorts to dustbowl empiricism. In our experience, a priori hypotheses are not made before the SKDA is conducted. Furthermore, sometimes SKDA uses stepwise regression, a practice that is strongly criticized in psychometrics literature because its true Type I error rate grossly exceeds its nominal Type I error rate (Cureton, Reference Cureton1950; Nunnally & Bernstein, Reference Nunnally and Bernstein1994). Psychometricians emphasize the importance of cross-validation whenever stepwise regression is conducted; yet, SKDA rarely takes this into account. Using the federal dataset, we found that shrinkage occurs when stepwise regression is used to select groups of five or 10 items, mainly when the data come from small organizations (defined as those organizations whose sample sizes were in the bottom third). After triple cross-validation, the validity coefficients for the predicted criterion scores dropped slightly from .81 to .74 when five items were selected and from .83 to .74 when 10 items were selected. The median sample size for the small organizations was 31; we suspect that shrinkage would be even higher at the work-unit level.

Issue 3: Rational and Even Random Item Selection Performs as Well as SKDA

Even when stepwise regression is not used, SKDA relies on a dustbowl empirical approach to selecting item as drivers; no a priori hypotheses are made concerning which items will or will not predict the criterion items. In contrast, psychometric theorists often recommend a rational approach to item development, taking into account past research on constructs measured by the items and the criterion. Even in instances where empirical methods are used in psychometrics and personnel selection, the importance of a priori and rational hypotheses regarding items is paramount. Church and Waclawski (Reference Church and Waclawski1998) made similar arguments in the context of linkage analysis (which is similar to SKDA except that an external criterion, such as turnover or profit, is used). They reinforce the importance of “logically” linking survey items to the criterion and recommend “a thorough review of the relevant literature regarding the variables to be linked and a solid research design” (p. 269). These arguments suggest that a rational approach to identifying survey key driver items is best.

We compared the validity of empirical and rational SKDA using the federal survey dataset. Three of the authors rationally rated each item with respect to its relationship with the criterion item and whether change can be actionable at the suborganization, organizational, or governmentwide level. The consistency of the ratings was measured using the intraclass correlation (3, k); a value of .611 (95% CI: .517 to .689) was obtained. We chose the five and 10 best items based on the rational key and compared it with the five and 10 item sets from the stepwise regression analysis. The five-item stepwise regression empirical method yielded a triple cross-validity of .83, which was only slightly higher than that from the rational approach (r=.79). When 10 items were selected, stepwise regression yielded a triple cross-validity of .82, compared with a validity of .81 from the rational key. Thus, rational ratings reasonably approximated the results of SKDA using stepwise regression.

In the spirit of Cureton's (Reference Cureton1950) classic article “Validity, Reliability, and Baloney,” we also came up with a rather bold idea for an analysis that yielded a shocking result. We selected five and 10 items from the federal survey using a random number generator and computed the validities. Selecting five and 10 items at random yielded validities (.74, in both cases) that were only slightly lower than empirical SKDA and that were similar to rational SKDA.

The success of empirical item selection is dependent on sample size, and many practitioners are in a position to conduct SKDA for small organizational units. Thus, we replicated this analysis using data from small organizational units, which were defined in the section covering Issue 2. After cross-validation, randomly selected items had the same validity as empirically selected ones when data from small organizational units were used. Thus, when stepwise regression is used, SKDA is little more effective than pulling the items out of a hat, especially for smaller samples. The same can be said for the correlational SKDA methodology. When we selected the items with the top five and 10 correlations with the criterion, the resulting criterion-related validities of the unit-weighted sum of these items were only slightly higher (the largest difference was .10 correlational units) than randomly chosen items.

Issue 5: SKDA Assumes Nonexistent Situational Specificity

SKDA is often performed for each organization that administers a survey; this analysis is performed even when identical item content is used across the different organizations. Many survey vendors now include SKDA as part of their default analysis packages for surveys, conducting them for each organization that completes the survey. This practice implies that the analysis is situationally specific, and that survey key driver items would be expected to vary across organizations. In essence, differences among sets of survey key driver items could be attributed to organizational variables, necessitating the use of SKDA for each organization that takes a survey.

This practice seems at odds with what we have learned from the psychometrics literature. A psychometric approach would posit that the survey items represent universal constructs, such as job satisfaction, and that the relationships between potential driver items and the criterion item should be generalizable across organizations. Although the scores on the survey items may vary across organizations, the construct measured by the items should be constant. Our argument parallels Schmidt and Hunter's (Reference Schmidt and Hunter1977) validity generalization (VG), whereby the validity of a test is not situationally specific and differences in validity coefficients are due to sampling error or established moderators. Note also that under VG, the validity of a test can be invariant across different situations; however, the mean test and criterion scores may differ significantly.

Indeed, when we examined the federal survey data, we found a lack of situational specificity. We compared the adjusted R and cross-validities of predicted values from suborganization, organization, and governmentwide regression equations. These results indicated that survey key driver items created at the governmentwide level have very similar validity (r=.84) to those created at the organization (r=.85) and suborganization levels (r=.85). Similar results were obtained when we computed the triple cross-validities for the stepwise regression SKDAs. The results for the five-item stepwise regressions were .83, .83, and .82 for the governmentwide, organization, and suborganization analyses. For the 10-item stepwise regressions, the triple cross-validities were .82, .82, and .81, respectively.

Further analyses indicated that some items are frequently identified as drivers almost regardless of the type and method of analysis used. Examples of the common driver items include “My work gives me a feeling of personal accomplishment,” “How satisfied are you with your involvement in decisions that affect your work?” and “Considering everything, how satisfied are you with your pay?” Overall, we found that the practice of conducting SKDA for different organizations is not empirically supported. Instead, it appears to be a waste of resources both on the side of those conducting the analysis and the side of those sponsoring the analysis (in terms of cost and time spent interpreting the results). If a survey vendor or provider wishes to use SKDA, the best approach would be to conduct the analysis one time using all available data, rather than for each client organization's administration. Although there were differences in the mean scores on the survey items across different organizations and suborganizations, the SKDA results tended to be quite similar. That said, a potential caveat to our findings is the use of data from only the U.S. federal government. Data from a more diverse database (e.g., survey responses from multiple countries and public and private sector organizations) might yield different findings.

Issue 6: SKDA Assumes Nonexistent Temporal Specificity

An SKDA is often performed each time a survey is administered to a single organization, suggesting that survey key driver items are temporally specific (i.e., not stable across time). Temporal specificity of survey key driver items is essentially an extension of the situational specificity issue described above; however, years are used as situations, whereas organizations were used as situations in the previous question. There is an intuitive appeal to temporal specificity as employee surveys measures attitudes and opinions, which may change over time.

Looking at the federal survey data, we did not find evidence of temporal specificity. We obtained survey responses for four years of survey administrations and conducted SKDA for each year. We then placed the key driver metrics into a new dataset and examined the interrelationships among the survey key drivers from year to year. At the governmentwide level, the bivariate and β-weight regression survey key driver metrics correlated across years at values approaching unity (rs=.99 and .98, respectively). In addition, we computed correlation coefficients between the dichotomous status (0, 1) of each item with respect to the top five and 10 items, and we conducted χ² tests. Our results suggest strong correlations and evidence of contingency, especially for the governmentwide analyses. For the five-item stepwise regression, the correlation coefficient was 1.00 and χ² was 65 (df=1, p <.01). For the 10-item stepwise regression, the correlation coefficient was r=.82 and χ² was 39 (df=1, p <.01). In general, the survey key driver items were significantly correlated across years, indicating a degree of temporal generalizability. Thus, there is little empirical evidence to support the practice of conducting SKDA each year that a survey is administered. However, we recognize that follow-up research looking at longer time periods or at nonfederal organizations might yield slightly different results.

Issue 7: Multivariate Analyses Are Suspect When Variables Outnumber Observations

So far we have discussed the use of SKDA for data from large groups of employees. Additional problems can arise when the analysis is conducted for smaller groups, such as teams of workers. A recent article on SKDA suggested that one best practice proposed by SKDA users is to conduct the analysis at the lowest levels within an organization (i.e., the “team” level, Alonso & Wang, Reference Alonso and Wang2013, p. 93). In our view, organizations that conduct SKDA at the team level will likely have too few data points to obtain meaningful results. In our experience, most work units have at most 10–15 employees reporting to one supervisor. Consider the 71-item federal survey used in our examples; when completing this survey, the 10–15 employees would respond to a total of 71 predictor items and the criterion. The ratio of respondents to variables (i.e., 10–15 to 71) presents five additional psychometric issues for SKDA.

First, traditional parametric statistics (e.g., Pearson's r) cannot be used effectively with fewer than 30 cases; instead, nonparametric statistics (e.g., Spearman's ρ) should be used. A second issue is statistical power. Even if a work unit was sufficiently large (e.g., at least 30 survey respondents) for parametric statistics, the power for detecting a significant correlation is low; one would need 50 survey respondents in order to detect a significant correlation of .48 (the mean observed correlation in our results) with 95% power. An even greater number of cases are required for sufficient power for determining whether one survey driver correlation is significantly different than another. Using the average interitem correlation from the 2008 dataset (.40) and the average correlation with the criterion (.48), to achieve 95% power, totals of (a) 1,824, (b) 200, and (c) 72 survey respondents would be required to detect (a) small (q=.10), (b) medium (q=.30), and (c) large (q=.50) effect sizes for the differences between the two correlation coefficients (using q as described in Cohen, Reference Cohen1992). (We note that nonparametric statistics further exacerbate the issues with low power since they have less power than parametric statistics.)

Third, in small samples the confidence intervals around a point estimate of a correlation are quite large. With 30 cases and a true correlation of .48, the values within the 95% confidence interval would range from .15 to .72. Fourth, regression-based analyses are problematic when there are more variables (i.e., the 71 items) than cases (i.e., the 10–30 survey respondents per work unit). Both least squares multiple regression and RWA require the inversion of the correlation matrix between the item variables (i.e., the matrix algebra equivalent to taking the reciprocal of a number). Consequently, the correlation matrix in this situation (i.e., 71 variables and 10–30 cases) would be a singular matrix (i.e., the matrix algebra equivalent to zero) and could not be inverted (i.e., the matrix algebra equivalent of dividing by zero). Fifth, even if the number of items was reduced (e.g., to 12) at the work unit level, the ratio of group sample size to survey items entered as predictors (n:k) would yield uninterpretable regression coefficients, if any are estimated (Cohen & Cohen, Reference Cohen and Cohen1983). In fact, a regression analysis conducted on a set of randomly generated data with n cases, one criterion, and n − 2 predictors (which is just barely enough to avoid singularity) will yield a nonsignificant R ² ≈ 1.0.

Issue 8: Correlation ≠ Causation

The ultimate aim of SKDA is to identify areas (represented by items) in which an organization can focus action-planning efforts to improve overall job satisfaction or employee engagement. Essentially, SKDA identifies items that are driving job satisfaction or employee engagement up or down within an organization. This interpretation is also consistent with the use of the word “driver,” which is defined as “something that makes other things progress, develop, or grow stronger” in the Cambridge Dictionary (Cambridge University Press, 2016). Although we realize that not all practitioners who use SKDA believe that the driver items reflect underlying causal issues, the word “drive” and the emphasis on addressing driver items in action planning might imply causation to stakeholders.

No doubt, the vast majority of I-O psychologists are familiar with the tenet “correlation is not causation,” yet the practice of conducting a SKDA and assuming that the driver items are causing job satisfaction or employee engagement seems to contradict this tenet. Unfortunately, causation cannot be established using the observational and correlational approaches inherent in SKDA. If a survey vendor decides to use SKDA, despite the issues we identified above, they should always caveat the findings with a disclaimer that the results are correlational not causational. If we as a field want to provide organizational leaders with the causes of job satisfaction or employee engagement, then we need to conduct experiments (with random assignment) and employ the scientific method to establish causation. Such applied research could prove fruitful for I-O psychologists, who could partner with organizations to conduct the experiments or summarize existing experiments using meta-analysis.

Should I-O Psychologists Conduct SKDA?

SKDA is seductive because it provides the illusion of an easy answer to questions about the causes of an organization's overall level of job satisfaction or employee engagement. In our discussion, we identified eight issues concerning the use of SKDA. Many of these issues incorporate fundamental findings in the psychometrics literature. It remains to be seen whether these issues can be addressed to make SKDA a more issue-free practice. We hope that our discussion will raise the awareness of those using SKDA of the issues stemming from its use. Many may decide to halt use of SKDA, as we have done ourselves. Those who continue to use SKDA should consider its serious limitations and accordingly frame their presentations of survey results to organizational leaders.

Over the course of history, new analysis techniques have evolved. Consider the emergence of item response theory and CFA. Ordinarily when new analyses are developed, they undergo rigorous testing (e.g., Monte Carlo simulations, formulaic derivations) before being used. Ideally, published empirical, theoretical, and simulation papers will appear in the peer-reviewed literature before a new analytic tool is put into practice. In conducting our literature searches for this focal article, we found surprisingly few examples of this work. The review article by Scherbaum et al. (Reference Scherbaum, Putka, Naidoo, Youssefnia and Albrecht2010) stands out as an exception. Despite the relatively small and insufficient research base to support SKDA, I-O psychologists have been putting SKDA into practice. In our opinion, SKDA is an example of practice moving ahead of—and beyond—science. Also, as suggested by an anonymous reviewer, SKDA may tempt organizational leaders to accept quick and easy answers rather than having to dig deeper and engage in discussions with their employees about the underlying issues impacting them.

What Other Approaches Exist?

Given the issues with SKDA, practitioners likely will need to turn to other approaches for guiding the development of action plans. Several existing approaches have been described in the literature and have enjoyed extensive use. The Office of Personnel Management (Berry, Reference Berry2012) has prepared a public-domain document that provides concise guidance on interpreting survey results, conducting follow-up analyses (including the use of focus groups and open-ended surveys), and performing action planning. Though geared toward executives, this document has much to offer I-O practitioners. Other approaches include holding strategic meetings with senior leaders (Church & Oliver, Reference Church, Oliver and Kraut2006; Wiley, Reference Wiley2010) and asking employees to rate the importance of different issues on their job satisfaction (Church & Waclawski, Reference Church and Waclawski1998; Society for Human Resource Management, 2012). These approaches can be used in addition to, or as a replacement for, SKDA.