Navigating with SKDA and GPS Requires Forethought, Caution, and Responsibility

We acknowledge the methodological issues common to SKDA mentioned by Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017). As with any research design, methodological approach, or data analysis, there are fundamental assumptions and limitations with SKDA, and thus the approach should be used with caution. In addition to the cautions raised by Cucina et al., we note a few more. For example, SKDA results are impacted by the distribution and reliability of drivers and outcomes. There are at least three common checks researchers should conduct when it comes to distribution assumptions. First, do the drivers and outcomes have a similar skewness and kurtosis? The less similar the distributions of a driver and an outcome are, the more a statistical relationship is reduced (Nunnally & Bernstein, Reference Nunnally and Bernstein1994). If the distributions are not similar, the researcher should determine if such differences are reflections of the population parameters or due to measurement error. Steps should be taken to mitigate that latter (e.g., look for socially desirable wording, double-barreled items, etc.). Second, are there outliers in the data that could impact relationships in either direction (Cohen, Cohen, West, & Aiken, Reference Cohen, Cohen, West and Aiken2003)? Last, researchers should examine whether the hypothesized and observed relationships are indeed linear. SKDA utilizes multivariate analyses (such as correlation and variations of regression) that assume a linear relationship between predictors and criterion. However, underpredication can result if nonlinear relationships are present (Goodwin & Leech, Reference Goodwin and Leech2006). Nonlinear relationships are not inherently bad; however, practitioners must be vigilant and account for them appropriately.

Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) point out that latent factor structures should be examined—we agree. The academic literature provides tremendous guidance in terms of the antecedents, relationships, and construction of key constructs. However, a key point not acknowledged in the focal article is that SKDA should be guided by theory but led by the organization's strategic goals. These business priorities provide the context and direction for applied researchers to better understand the results and make appropriate recommendations. For example, retaining an item simply because it adds to an alpha coefficient or adding an item to a broader category solely based on its factor loading without regard to the organizational context will only serve to reduce the face validity of the analyses and potentially the entire survey effort in the eyes of the organizational leaders. We believe that many of these measurement concerns can be mitigated by conducting a small sample survey prior to any broad survey data-collection effort. Conducting reliability tests of measures is key—if the items of a measure do not consistently measure the construct of interest, it is important to catch these flaws prior to the broad survey effort and adjust as needed. Moreover, it is equally important to ensure that any adjustments remain relevant to the client, which may result in greater debate and effort to achieve a better fit than what otherwise would result in a strictly theoretical or statistical approach.

We should also keep in mind that the methodological concerns that Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) outline stem from analyses that are based on the general linear model. This is certainly the most common approach to SKDA. However, there are other approaches to SKDA (which are beyond the scope of this commentary to review) that mitigate many of these concerns. For example, Ridge regression and Shapley value regression are approaches used when multicollinearity is a concern. Principle components regression, structural equation modeling (SEM), and partial least squares regression are three other approaches that, although distinct from one another, employ composite variables (components, factors, or latent variables) in place of the original item-level variables. SEM in particular provides a great deal of flexibility in modeling. Other more recent advances in SKDA use approaches such as hierarchical Bayes regression to create separate models for each respondent before aggregating them to an organization level. The resulting coefficients tend to be more accurate than what we get from a single regression analysis at an organizational level.

Last, it is important to keep in mind that SKDA is a snapshot in time and expecting the same drivers to maintain the same impact on organizational outcomes is misguided. As organizations and the environments in which they operate evolve, the impact and relevancy that drivers have on key business outcomes change. Practitioners and researchers must constantly evaluate the relevancy and consistency of constructs measured.

SKDA and GPS Have Practical Advantages Despite Their Flaws

One of the many challenges we face as I-O professionals is the translation of complex theory and statistical analyses into simplified models of human predictability for our clients. In a practical sense, SKDA is one of the more intuitive analyses we do that business leaders understand and can readily utilize for positive change. SKDA engages business leaders to think about the impact that certain actions or work-related characteristics (i.e., “key drivers”) have on key business outcomes. This fact alone has made SKDA an invaluable tool in elevating and shaping business leaders’ perceived value of people and exactly how they influence business outcomes. Almost 2 decades ago, Rucci, Kirn, and Quinn (Reference Rucci, Kirn and Quinn1998) demonstrated how data linkage analysis can tie employee attitudes and human resource practices to business outcomes. Since then, business leaders have a better appreciation and understanding that some of the softer constructs we measure (e.g., engagement) can have real business consequences. SKDA helps business leaders think about what drives some of these softer organizational constructs.

SKDA also helps our clients understand a set of variables’ relative contribution (explained variance) on the outcomes of interest. For example, clients are often interested in which drivers or areas of action are most likely to yield the most impact on an outcome of interest (e.g., engagement). SKDA allows us to answer that question relative to other drivers, thus helping clients to prioritize action for meaningful change. However, because SKDA involves statistical modeling, I-O professionals must assume the mantel of responsibility to prevent any inferences clients have about causality—a topic that Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) raise and a topic on which we agree. Educating business leaders on proper inferences from statistics is equally as important as delivering the insights. Therefore, our view is that SKDA must be used responsibly by I-O psychologists, meaning that we need to select the most appropriate approach given the context, interpret the data appropriately, and continually relay the strengths and limitations of our analyses to our clients. Quite often we must do this in the face of organizational pressures to see results. It is the job of I-O scientist–practitioners to push back when theoretical or statistical lines are crossed.

There are other benefits of SKDA that are not acknowledged by Cucina et al., one of which being that SKDA helps to lessen the cognitive burden of data. Organizing drivers and outcomes, and analyzing the relationship between the two in a meaningful way helps to make data and insights much more digestible and usable than a strict report out of all items (even if clustered in some way). Another benefit of SKDA is that of relevancy, whereby SKDA can help leaders focus on areas that are not only within their control for action planning, but also important for advancing their business outcomes of interest. This ultimately increases the likelihood that leaders will do something with the results, and is a more useful technique than establishing aspirational but vague targets (e.g. “improve the culture around here”) and leaving the business to “just figure it out.” For example, without some direction, clients will often target the following:

• • The lowest scoring item (or dimension)

• • The biggest increasing or declining item(s) over time

• • The results they think matter based on their previous experience, personal motivation, or inherent bias

• • The results a client feels they can impact (e.g., the “easiest” but in actuality least impactful action)

If drivers are not provided based on some type of statistical model, clients will hypothesize relationships on their own (based on anecdotal experience or personal bias) and decide what outcomes need to be solved and how to solve them—often resulting in misguided or misaligned actions that can increase the degree of frustration between stakeholders and among employees.

You Should Never Follow SKDA or GPS Blindly

More broadly, the scientist–practitioner model is a balance, and we believe that there are more practical implications that supersede the psychometric points raised by Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017). First and foremost, survey content should be driven by the business problem at hand and secondarily by theory and research hypotheses. Solving all of the psychometric issues raised by Cucina et al. but not having the content relevant to the business issue at hand results in outcomes that are irrelevant and unusable to the business.

We should also keep in mind that not all items included in an employee survey are meant to be “drivers” of anything (e.g., safety, engagement). Survey content may include items for the sole purpose of communicating company values, principles, or vision, or as a mechanism for evaluating or tracking other business concerns (e.g., potential ethics or safety concerns, or perception of product quality). These types of items may provide meaningful data for specific stakeholders (e.g., labor relations team or health and safety department) regardless of whether or not these additional items are conceptually related to engagement or satisfaction. In these cases, the response characteristics may not be ideal according to Cucina et al., but such high scores or range restriction would be expected given their purpose. In circumstances such as these, sound theory, proper evaluation of statistics, and the organizational context would guide us to not use certain items in key driver analyses.

Our clients look to us as I-O professionals to measure, analyze, and deliver insights in a smart and meaningful way. Decisions regarding what items are included in SKDA and how results are reported is part of our role in translating complex findings into simple, actionable insights. As I-O professionals, we must carefully balance data analytics with identifying the insights and crafting the story that leads to action.

However, regardless of how well-crafted our survey story might be, it is still a single source of data collection. We must keep in mind that the story becomes much more powerful when we can augment the survey results with other data (e.g., qualitative) and other outcome measures (e.g., customer satisfaction, business performance, or turnover). Again, this is where our clients rely on I-O professionals to find and present meaningful linkages that are understandable, relevant, and help the organization achieve its goals.

Don't Throw Out SKDA or GPS Just Because It Once Recommended a Poor Route

We disagree with Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) recommendation to discontinue SKDA. In our opinion, doing so would be “throwing the baby out with the bath water,” or in keeping with the GPS analogy, it would be akin to stopping use of your GPS all together because it has the potential to guide you down the wrong street. In addition to the practical advantages outlined above, we believe there are many reasons why SKDA should remain in the practitioner toolkit, namely these:

1. 1. Many of the issues cited by Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) as reasons to abandon the technique are not challenges exclusive to SKDA—rather, they are methodological challenges inherent in many types of multivariate analyses (e.g., discriminant analysis, analysis of variance, etc.) regardless of purpose (applied or academic). Therefore, to discourage the use of an analytical tool based on the premise that there it has methodological limitations and potential for misuse would by implication severely limit the scientific tools we have available as scientists and practitioners. Using this logic, we would have to call into question any research that uses regression, correlation, and the like.

2. 2. SKDA is commonly used in other fields such as market research. For example, market researchers may use it to determine which attributes of a consumer product factor into a consumer's decision to buy it. The field of market research has debated the challenges of SKDA fairly extensively (e.g., Gray, Reference Gray2014) but still continues to see it as a viable tool when used with caution. If another field interested in predicting and understanding human behavior finds the method useful despite its challenges, why should I-O psychology reject it outright? This is not an issue of “everyone else is doing it, so should we.” The point here is that the field of market research is just as, if not more, rigorous than I-O (e.g., it has a formal code of conduct, standards for conducting research, and codes for data analytics). If a field similar to ours in terms of rigor and methods finds ways to recognize the flaws and mitigate them appropriately, why can't we?

3. 3. SKDA helps us understand what variables are related to key constructs of interest. These relationships are typically not able to be identified by simply asking employees what is important to them. Stated differently, SKDA distinguishes derived importance from stated importance. For example, employees might say that flexible work options are very important to their commitment with a company, yet statistical models might show that they play a very small role in predicting retention relative to other factors such as pay, career opportunity, and so forth.

4. 4. Cucina et al.’s proposed approach to replace SKDA is not sufficient. It does not provide the degree of rigor that SKDA provides (e.g., providing derived importance of variables), and the process they outline is essentially the classic survey-feedback-action planning process (Kraut, Reference Kraut2006), which often incorporates SKDA.

We Should Keep Using GPS and SKDA, Just More Intelligently and Thoughtfully

Our view is that SKDA should continue to be in the researcher's toolkit, if applied thoughtfully while striving to provide organizations with practical implications. To do so, we have two recommendations for survey researchers.

1. 1. Follow your I-O training when conducting SKDA. Our training as I-O psychologists remains central to our success as scientists, practitioners, and professionals. It is our responsibility to research and conduct analyses in a thoughtful, thorough way to ensure we are pointing organizations in a direction that is well supported by data. Understanding the problem to solve and how best to approach the questions posed by our clients are critical first steps in identifying the constructs and hypotheses to test. Survey items should be designed with the purpose in mind (e.g., to measure a larger construct, to provide feedback on an organizational initiative), and multi-item constructs should be supported by theory and pilot tested to understand their psychometric properties. We must choose the right SKDA analytical approach based on the context, data, and research goals. Last, we must educate the business on how to interpret results with the limitations in mind (e.g., violation of assumptions, causality).

2. 2. Do not let SKDA be the only tool in the toolkit. SKDA is one of many views when examining survey results, and using it as the only analytical approach limits our understanding of the data. Internal (e.g., year-over-year trends, item comparisons) and external (e.g., consortium benchmarks) comparisons are other measures that can help practitioners understand how employee attitudes are changing over time or how they relate to other organizations. Also, linking survey data to other sources of data (e.g., performance ratings, talent movement data, regional economic data) can help add to the results and provide greater insights in the drivers and outcomes. Qualitative methods (e.g., survey comments, interviews, focus groups) can also help us confirm (or disconfirm) survey results and add more understanding behind the survey findings.

In summary, we believe that SKDA plays a critical role in understanding important organizational outcomes. Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) cite a number of statistical and methodological challenges associated with SKDA, suggesting that the limitations inherent with this approach should overshadow any practical utility of enacting this approach. Our view is that there are methodological limitations to any measurement or analytical approach in our I-O toolkit. Instead of abandoning these approaches altogether as Cucina et al. recommend, we need to (a) choose the right approach for the situation, (b) understand the strengths and limitations of that approach, (c) work to mitigate the vulnerabilities throughout the data analysis, and (d) ensure we convey to our clients the strengths and limitations of the approach taken.