Exploring SKDA With a Broader Set of Data

Most of Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) findings are based on data from 212,223 U.S. federal employees who participated in a government-wide survey in 2008. Although this is a large sample, it is possible that there is something unique about this population or this survey instrument that threatens the external validity of the authors’ findings.

Each year, we survey approximately one million employees working in small, medium, and large organizations around the world. To examine the validity of the authors’ claims in a broader context, we analyzed the survey data of a diverse sample of 20 organizations (N = 423,351) from our research database, replicating many of the authors’ analyses. These organizations varied in size, industry, and geographical location. Two government agencies were also included to determine if Cucina et al.’s findings are a function of sector (public versus private). An engagement index served as the criterion, comprising items that assessed pride in working for the organization, motivation to go above and beyond expectations, willingness to recommend the organization as a place to work, willingness to stay with the organization, and overall satisfaction. A total of 1,110 survey items was analyzed; specific survey items varied across organizations, though many were the same (drawn from our normative item bank) and covered common themes related to engagement such as leadership, immediate manager, and teamwork. Items were positively worded, with the exception of one reverse-scaled item.

Situational and Temporal Specificities Do Exist

Taking a one-size-fits-all approach to SKDA ignores important differences between and within organizations, based on the assumption that the variance within and between their levels of analyses (government-wide, organization, and suborganization) is greater than variance between organizations that may operate under completely different laws, social, and cultural environments, and whose workforces may diverge on a multitude of demographic characteristics. These differences may underlie the aforementioned disparate distributions. The authors offer a list of common drivers they had found, including satisfaction with pay, which we will use to illustrate that organizational differences do in fact exist. Across the 20 organizations in our research sample, satisfaction with pay was never identified as a driver of the criterion. In fact, its correlation typically ranked toward the middle-bottom of items within an organization, with an average percentile rank of 25 for the 16 organizations that ask about pay.

Regarding temporal specificity, we examined the same two organizations (global corporation and federal agency) described above. Across 3 years of data, there was some evidence of temporal stability: More than half of the drivers remained the same from one year to the next. But the assumption of temporal stability of drivers hinges on organizational stability. It would be irresponsible to simply assume that the same drivers are applicable before and after a large organizational change. In the wake of change and interventions, we found that some new key drivers emerged.

Random Items Do Not Perform as Well as SKDA

We selected the government agency included in our analysis to examine whether or not a random selection of survey items performs as well as SKDA. Whereas Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) used triple cross-validation, we chose to run relative weights analysis (RWA) against 10 different sets of five randomly selected items. R ² was .75 for the empirically derived drivers. For the 10 random sets, R² ranged from .46 to .73 (M = .56, SD = .09). The top two R ² values were .73 and .68, but they contained three and two of the drivers, respectively. Based on these results, it is clear that randomly selected items account for less variance than SKDA.

Conducting SKDA in a Rigorous Way

Although our findings do not replicate Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017), we do agree that SKDA can be misused. Based on our experience, we have found that four fundamental analytic guidelines can prevent many of the concerns the authors raise.

First, the authors point out that the underlying factor structure is often ignored while performing a SKDA. We believe that this is a cardinal statistical sin and recommend the use of exploratory factor analysis as the first step in any SKDA. Following this process will allow the researcher to remove items that may be double loading with predictor and criterion measures. We have found that this is most often seen with distal, high-level items such as “I am confident in the future of company XYZ,” which often functions as a separate outcome of employee workplace experiences that is independent but related to employee engagement/satisfaction.

Second, the authors point out the importance of utilizing rational hypotheses when conducting statistical analyses. We agree, and leverage theory and research when designing all our surveys. Based on our client's goals and objectives, we design strategic surveys (Schiemann & Morgan, Reference Schiemann, Morgan and Kraut2006) that are informed by relevant nomological networks (Cronbach & Meehl, Reference Cronbach and Meehl1955) and are supported by academic research. We then use an empirical method—such as relative weights analysis—to compile a set of items that is highly predictive of the criterion. We review the key driver items that emerge and note any that are theoretically surprising (i.e., not theoretically related). When we do find such items, we explore the quantitative and qualitative data for possible reasons and discuss with key stakeholders (e.g., employees, managers) within the client system to gather their insights and perspectives. This allows us to bring both an inductive and deductive approach to our analytical work.

Third, the authors remind us that “correlation is not causation.” We agree: In both the applied and academic world, correlations are often overstated as absolute proof of causation. But we believe this tendency to conflate the two can be addressed in multiple ways. First, any correlational findings should be presented as evidence of prediction and not causality. Second, time-lagged analysis can be used to explore causal relationships. When our clients want greater confidence in the causal relationships in their data, we measure individual-level employee engagement in Year 2, and regress this onto employee perceptions measured in Year 1. Although this alone does not prove causation, it improves our understanding and confidence in the temporal relationships between employee perceptions and employee engagement. To genuinely establish a causal relationship, the steps described by Antonakis, Bendahan, Jacquart, and Lalive (Reference Antonakis, Bendahan, Jacquart and Lalive2010) using two-stage least squares can be followed. By clarifying the distinction between prediction and causality, and adding methodological rigor to the SKDA process, we have found that confusion about causation and correlation subsides.

Finally, Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) caution against conducting SKDA on very small subgroups and teams. We agree: When SKDA is conducted on small groups, spurious relationships emerge. We do not conduct SKDA on any group with less than 250 respondents. We have found this threshold satisfies various variable-to-observation rules of thumb for ordinary least squares (OLS) regression analysis (e.g., six observations to one predictor, 100 + number of predictors, etc.). Although more modern regression techniques such as ridge regression and least absolute shrinkage and selection operator (LASSO) have less restrictive variable-to-observation requirements, these methods remain relatively underused compared to OLS regression techniques for SKDA.

Utilizing SKDA in a Responsible Way

In addition to conducting SKDA in a rigorous way, we also think it is critical that organizational scientists and practitioners utilize SKDA results in a responsible way. Influenced by the principles of action research, we believe that the survey feedback process works best when leaders, managers, and employees are fully involved in identifying organizational issues and developing solutions and action plans.

With that in mind, we do not consider key driver results to be an exclusive or exalted diagnostic of organizational health and effectiveness, and we do not counsel our clients to automatically take action on low scoring key driver items. Instead, we encourage our clients to view these items as one of many points of consideration for review, discussion, interpretation, and action. We believe that key drivers should be used as a conversation starter—as a way to help a client system start making sense of relationships in the data. But key drivers should not be seen as a set of magic bullets or a prescription for change delivered from scientists on high. By framing key drivers as a starting point for organizational reflection, we seek to combine the strengths of diagnostic and dialogic OD (see Bushe & Marshak, Reference Bushe and Marshak2015). We have found that by providing our clients with data-based insights and allowing organizational members to interpret those results in context of their strategic goals, cultural tendencies, competitive challenges, and people initiatives, they are more likely to engage with their data, explore organizational dynamics, and commit to meaningful action.

In their focal article, Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) outline a four-step intervention plan for improving job satisfaction and employee engagement. Their plan is a good one, albeit idealistic as they admit. But at the core of this plan there appears to be an implicit assumption that once an intervention has been tested and shown to be effective, it could and should be implemented across teams, departments, and organizations. Although the authors do note that organizational context should be considered when planning for change, they clearly place more faith in off-the-shelf interventions than client-specific SKDA.

We think this approach runs the risk of oversimplifying the problems that most organizations are struggling with these days. Toward the end of the 20th century, Heifetz (Reference Heifetz1998) made a distinction between two types of challenges that leaders face: technical challenges, which are easy to identify and can be solved with best practices, and adaptive challenges, which are complex and require deep changes in values, beliefs, behaviors, and practices. Based on our experience, most of the challenges we observe in organizations are more adaptive than technical. Therefore, they require complex and customized interventions.

That is not to say that predesigned interventions are not effective. We have seen job crafting workshops (Berg, Dutton, & Wrzesniewski, Reference Berg, Dutton and Wrzesniewski2013) create powerful changes in organizations. But we have also seen this intervention backfire without buy-in from leaders, managers, and employees. We have found that the best way to create postsurvey change in an organization is by doing the following:

• • Making sure organizational members thoroughly understand their data, including their SKDA

• • Providing them with a range of possible interventions

• • Encouraging them to come up with their own possible solutions

• • Empowering them to select the intervention(s) that they think will be most effective

This approach is based on foundational organization development (OD) concepts. As Burke (Reference Burke1992), in his review of Argyris's three criteria for effective intervention, notes, “The client is provided alternatives for action. No particular or specified action is automatic, preordained, or imposed” (p. 141). For anyone applying Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) four-step plan, we encourage you to keep in mind the importance of employee voice, choice, and buy-in when selecting an intervention.

Final Thoughts

At the heart of Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) focal article is a call for more responsible science and more evidence-based decision making within organizations. We acknowledge the need for both. Based on our experience, we think that SKDA can be conducted in a scientifically responsible way that produces evidence-based insights for leaders, managers, and employees seeking to improve their organizations.