A fair reading of Cucina, Walmsley, Gast, Martin, and Curtin's (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) astute analysis of survey key driver analysis (SKDA) presents a good case that such analyses are driving down the wrong road and should slow down or come to a halt. In large part, the SKDA they describe seems to be a misleading set of exercises.

But how can this be? After all, most of the industrial and organizational (I-O) practitioners doing survey work are well trained and conscientious, right? How did we get to this state? Surely we knew that correlation is not causation, even if we did not fully appreciate the outsized role that larger standard deviations have in creating stronger correlations with an overall “outcome” measure.

Based on my own experience and observations, I think the unfortunate use of SKDA may partly be the result of a discomfort in developing diagnoses and solutions based on organization development methods. It leads to an easy arrangement between survey vendors/consultants and their clients, which suits their mutual interests. This “arrangement” is not necessarily overt or even recognized. Using SKDA, the survey researcher appears to give a scientific solution that explains what is causing employee satisfaction or engagement, and the client gets to avoid the hard work of actually talking to the survey respondents about what is behind their answers and what they think should be done about the issues that are important to them.

Most survey firms are quite good at statistical analyses and at marketing their services. If they are typical I-O practitioners though, they often lack the skills of organization development (OD). Although definitions of OD differ, all varieties would have a survey practitioner start by “contracting” with the client to specify the purpose and expectations for the entire survey process, which includes data analysis, diagnosis and reporting, and the design of appropriate actions, or “interventions,” that respond to the findings.

The OD process explicitly recognizes the need to react and prepare for the likelihood of organizational change. It also means working with the client organization at several levels to define reasonable actions and help to carry them out.

Most survey practitioners have never been trained in such skills, nor do most I-O psychologists come by them naturally. The same is true for many of our clients. In fact, many clients are uncomfortable about even talking to their staff members in meetings about survey results. These management fears about unscripted feelings and unexpected suggestions from employees make the clients all too eager to get scientific-looking analyses that obviate the need to talk to survey respondents.

I have had clients who do not even want to discuss what will be done after a survey is administered, saying, “Let's see what the results look like before we decide.” That is a recipe for no progress. An example is the case described in The Washington Post after a terrible set of survey results from the federal government's Transportation Security Agency. Afterwards, TSA hired a consultant to look at the data, but the resulting study was put in a drawer. This process was then repeated three more times, costing more than $2 million, without any action being taken (Markon, Reference Markon2015).

To overcome a falsely comforting use of inappropriate tools, like the SKDA, survey practitioners need to focus instead on using OD methods and OD-skilled staff to get the survey data properly understood and used by appropriate units and levels. Fortunately, there is a wealth of knowledge and practice available to us if we want to improve. The early work by Nadler (Reference Nadler1977) on the energizing power of survey data has been advanced by more recent work by Church and Waclawski (Reference Church and Waclawski1998) and Waclawski and Church (Reference Waclawski and Church2002). Another resource is Borg and Mastrangelo (Reference Borg and Mastrangelo2008), who detail how to “position” a survey, to set up expectations, with both management clients and employee respondents, and how to run an employee survey workshop afterward to report and analyze survey data. Hinrichs (Reference Hinrichs and Kraut1996) also explains the steps in running a feedback meeting to generate actions and follow-through.

These activities often require training of unit managers so they are prepared and skilled at conducting feedback and action planning sessions. Many clients seem reluctant to contract and pay for this training because they see it as an unneeded “extra.” From an OD perspective, this is not an extra at all, and clients should be made aware of this at the outset. Another explanation may be that the survey firms are uncomfortable proposing or doing such training and do not suggest it.

Based on these observations, survey feedback may end with no more than an organization newsletter summarizing results. (Scott Adams's satirical cartoon series, Dilbert, has more than one acerbic illustration of such nonresponsive behavior.) Unfortunately, solid research by Church and Oliver (Reference Church, Oliver and Kraut2006) shows that providing survey feedback to employees without taking action has a more negative impact on employee attitudes than not giving feedback at all. It is action-taking that is the real secret sauce. To the field's credit, there are many well-executed surveys being done, using OD methods. Such surveys are energized at top management levels and empower action at surprisingly low levels of management.

Ironically, Cucina et al.’s (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) advice to avoid repeated SKDA analyses in the search for temporal or situational specificity may be correct for most psychometric topics, but this caution is not right for analyzing survey results. Attitudinal differences over time and among organizations (and subunits) are common and real. Yes, meaningful differences may not be found by the typical SKDA, but such differences will be seen by careful comparisons across time and across units, with the aid of norm data. Such evaluations do not require sophisticated statistics—just straightforward t-tests at best—to assess significant differences.

These comparisons will also reveal that, on average, some items, such as those asking about pay and promotion opportunity, are almost always less favorable than typically higher-rated items such as “like my job” and “like my coworkers.” Usually, higher rated items have a more skewed distribution than lower rated items, so that unfavorable items will have a larger standard deviation (SD). Because of the larger SDs, SKDA may inadvertently be pointing to lower rated items as being more important to overall employee satisfaction, and that might be true, or not. Only candid conversations with respondents will reveal what is truly important and actionable.

I believe that the ideas above explain why key driver analysis has become so popular. I-O survey practitioners, well trained and enamored of sophisticated analyses, have succumbed to wishful thinking that they are explaining what drives overall employee satisfaction. The SKDA analyses in most cases suit the clients’ needs and their lower statistical sophistication, and keep them from having to do uncomfortable or unpleasant things, like actually talking to survey respondents about what is important to them.

I believe this convenient arrangement, albeit unvoiced or even unrecognized, keeps survey results from being used to their full potential. Better use of survey data would improve practice, and science too. (Some of the steps in Cucina et al.’s “Proposed New Approach” overlap with the suggestions made here, though their focus on psychometrics ignores much of organization development.)

To make a real difference, practitioners must improve their OD skills and use them in guiding clients engaged in employee surveys. Of course, they should temper the use of SKDA. These steps would help both practitioners and their clients to truly improve their employees’ work lives and organizational performance. In a world of increasing competitiveness, we all have to step up our game.