In their focal article, Cucina, Walmsley, Gast, Martin, and Curtin (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) raise an important and timely issue, but go too far in arguing that practitioners should abandon survey key driver analysis (SKDA) altogether. There are many valuable benefits to using SKDA, and rather than discontinuing the practice entirely, we suggest that a better approach is to focus on ways to make it even more valuable to organizations. It is true that this area is in need of further research, but instead of asking whether we are driving down the right road, perhaps the right question is, “Are we there yet?”
We will organize our commentary into three sections. First, we provide a short summary of concerns with the primary study sample and conclusions. Second, we dig into the purported methodological issues with SKDA. Finally, we will summarize the reasons why we believe the focal article overstates the flaws of SKDA, and instead we put forth an argument for the practical utility and benefit of such analyses.
A Short Critique of the Primary Study Sample
Reliable scientific conclusions are based on solid research and consistently replicable effects. Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) instead make sweeping generalizations about potential flaws intrinsic to SKDA and call for a halt to its practice based on evidence from a single study using an arguably limited sample. For several reasons, we do not believe that findings from the primary study are reliable justification for the focal authors’ broad conclusions. First, the sample itself is archival data from 2008—nearly a decade old—and includes only U.S. federal government employees. We question the generalizability of this population to employees in other industries and (particularly) other time periods. Second, the response rate for this survey data was relatively low—only 51%. By way of contrast, the average response rate for 400-plus IBM Kenexa clients is approximately 80% (see the Appendix for further information). Here again, it is fair to question the representativeness of the sample data and whether the results were affected by nonresponse bias.
Replication is in order before entirely abandoning a potentially useful methodology such as SKDA. We set out to replicate the findings of the focal study using multiple large-scale data sets representative of their respective populations. We found different results that do not support the focal authors’ conclusions, and more research is clearly necessary.
Clarifying and Exposing the Purported Methodological Issues With SKDA
In the following subsections, we carefully consider several of the methodological issues raised by the focal article authors. These issues include the potential impact of item standard deviations, the importance of a priori hypotheses, the existence of situational and temporal specificities, and the thorny issue of causation.
Are SKDA Results Impacted by Item Standard Deviations (SDs)?
In order to better understand the relationship between item standard deviations and their bivariate correlations to criterion item(s), we attempted to replicate a key finding of the focal study using our own data. We leveraged our global normative employee survey database, which is among the largest of such databases in existence. It is maintained as a 3-year rolling average of survey responses and includes nearly 500 survey items. For the current replication, we examined data from 318 of our most common survey items with robust normative data (over 14 million respondents across more than 400 organizations worldwide). The 318 items measure 18 areas common to employee surveys (e.g., communication, growth, inclusion, leadership, recognition), and all survey items were positively scaled using the same 1–5 Likert-type agreement scale.
Consistent with the findings provided by Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017), we found our items to have an average SD of .96 (range .70 to 1.23) and an average item-criterion correlation of .53 (range .34 to .74). In stark contrast to their findings, however, we found a correlation of just .12 between the SD of an item and its correlation to our four-item employee engagement index—our most common criterion in employee engagement surveys (the index measures satisfaction, pride, advocacy, and commitment). This is significantly and substantially lower than the estimate of .62 provided in the focal article. The correlation we found in our own data would be considered small according to both Cohen's (Reference Cohen1992) guidelines and the newer, lower threshold guidelines set forth by Bosco, Aguinis, Singh, Field, and Pierce (Reference Bosco, Aguinis, Singh, Field and Pierce2015). Thus, our findings suggest that item SDs do not impact SKDA results to the extent suggested in the focal article, and concerns about items with larger SDs undermining the ability to properly identify the right targets for action are overstated.
Are A Priori Hypotheses Needed for SKDA?
Although formal a priori hypotheses are not always explicitly stated in organizational survey research, the entire survey design should be based on theory or an identified model regarding things that matter most. For example, our survey methodology is based on a hypothesized high performance engagement model. The model is supported by behavioral science theories, previous research, and “live” data from organizations. We also customize our surveys to address key issues and hypotheses identified by the organization. Before building a survey, we typically spend significant amounts of time with stakeholders identifying what is most important, and then we create the survey with this in mind.
With regard to analyzing survey data, modern methods of SKDA such as dominance analysis (Budescu, Reference Budescu1993) and relative weights analysis (Johnson, Reference Johnson2000) presuppose that an appropriate model has already been identified (LeBreton, Hargis, Griepentrog, Oswald, & Ployhart, Reference LeBreton, Hargis, Griepentrog, Oswald and Ployhart2007). Operating without prior hypotheses before conducting SKDA may occur in some cases and for some practitioners, but it is not a methodological issue with SKDA.
As a way to test the focal article's argument that random items perform as well as SKDA, we examined the difference between the average correlation of an item to our four-item employee engagement index for key drivers versus items that were selected at random. Here, key drivers were included in the pool of survey items available for random selection. That is, key driver items were not removed from the sample prior to randomly selecting items for analysis, thus ensuring a truly random sample of items.
We compared key drivers versus random items across two distinct, large, global client data sets (see Appendix). In both cases, the correlations of key drivers to the criterion were approximately .10 higher than correlations of items pulled at random. The average item–criterion correlation for key drivers versus items pulled at random was .69 compared to .59 in the first data set and .58 vs. .50 in the second. This means that in our data, key driver items showed about a 14% significantly higher correlation to the criterion than items pulled at random. This is certainly an improvement upon chance. Importantly, we went out to 10 driver items for this analysis (had we focused only on the top five, the difference in correlation strength would have been even more pronounced, an approximate difference of .15 correlation units, or 25% higher correlation to the criterion). Finally, in contrast to the final statement in this section provided by the focal article authors, we found the largest difference in correlation strength to be at least .30 correlational units—well above the .10 largest difference stated in the focal article. In other words, .10 was the average difference we found in our data—not the largest.
Observed differences in correlation strength are admittedly an insufficient way to fully justify the importance of focusing on key drivers. A better approach would be to provide evidence that focusing on key drivers leads to greater improvement in the criterion compared to focusing on nondrivers (or items identified at random). Fortunately, we can provide such evidence. We recently examined action plans for over 1,000 managers within a diversified health care company with over 80,000 employees. Data from this 2011 study showed that managers who focused on key driver items had a 13% greater impact on employee engagement the following year than managers who action planned on nondriver items. More research is needed in this area, but this result suggests a clear benefit to focusing on driver items identified through SKDA.
Are There Situational and Temporal Specificities for SKDA?
The focal article argues that SKDA results should be similar across different organizations at different times. However, this argument ignores the potential impact of background factors identified in the psychometric literature. Organizations vary in background factors, such as climate, setting, context, relevant history, and time, and these factors can moderate relationships between predictor and criterion (Johns, Reference Johns2006), and thus influence SKDA results.
We not only assume situational specificity but also count on it in helping to focus action. Good practice includes forming hypotheses about important factors specific to an organization and constructing or adapting items to test those hypotheses. In short, we see different key driver items appear across business units in the same organization (and especially from one organization to another). There is often similarity and overlap in key driver items but also plenty of differences from one group to the next that allow for prescriptive advice and targeted suggestions. Differences across industries, countries, and other factors commonly contribute to observed differences for which key drivers are identified.
With regard to temporal differences, there are many examples in which we have found major differences in key drivers identified from one year to the next within the same organization. We do find that key drivers are fairly stable over time, but it is impractical to simply ignore the possibility that new drivers can emerge across time. Major organizational changes, financial success or decline, M&A activity, and several other factors can impact the relative importance of key driver issues from one time period to the next.
An anonymous reviewer pointed out the possibility that differences attributed to situational and temporal specificity could be due to sampling artifacts in the form of measurement error. However, it is important to note that our census survey programs elicit feedback from entire populations of employees. Because we are dealing with population data and we typically see very high response rates, potential concerns about sampling error affecting SKDA results are greatly reduced.
Does SKDA Test Causation?
SKDA tends to provide information about the extent to which each predictor contributes to the overall R2 relative to other predictors. Well-trained researchers and consultants won't confuse the results of SKDA with causation. In our own practice, we refer to “key drivers” as “priority items” and carefully choose our language to ensure that we avoid implying causation. For executive presentations at the company or division level, we often conduct more advanced statistical analyses including regression, structural equation modeling, relative weights analyses, and factor analyses in order to provide stronger evidence for our conclusions. Although we are careful not to imply that issues underlying items that correlate highly with the criterion actually cause changes in the criterion, perhaps not all practitioners appreciate this nuance. Misinterpretation or misrepresentation of SKDA by some practitioners does not reflect a methodological problem with SKDA itself.
Making a Case for the Practical Value of SKDA
Identification of key drivers is about providing practical guidance and focus for managers who are tasked with understanding, communicating, and action planning based on survey results. Simple correlations such as those used in SKDA are easily understood by consultants and clients alike, and SKDA provides a starting point for discussions about where to focus attention when implementing organizational change. Not conducting analyses at all can leave practitioners blindly guessing about what drives employee engagement. Thus, there is practical utility in SKDA methodology over and above the identification of chance or random factors, and identifying priority items using SKDA can also help organizations avoid the temptation to focus solely on the lowest absolute scores from employee surveys.
As new research methodologies and technologies enter our field, there is a clear need for additional research into ways to leverage these capabilities. Future research areas include potential uses for continuous listening approaches to measure employee attitudes, as well as the role of big data, cognitive (machine learning) technology, and predictive analytics. In addition, we acknowledge the utility of alternatives to statistics-based SKDA. Alternative organizational development methods might include focus groups, stakeholder interviews, simple polling, qualitative research, or action planning in support of strategic organizational priorities. These can be used as standalone methods or in conjunction with statistics-based SKDA. Using evidence from multiple methods to triangulate priorities can provide even stronger arguments for taking action on what matters most. The appropriate approach and best research methodology truly depend on the unique needs and specific goals of the organization.
Practitioners often get a bad reputation for not adhering to the research and theory behind psychometrics and test development. But this reputation is often unwarranted. For example, survey design is very much influenced by theory and hypotheses. We conduct stakeholder interviews and focus groups to form hypotheses about the items most likely to impact engagement. When building surveys, we select the items that are the best drivers across clients to reflect our hypotheses. Moreover, in our practice, we avoid conducting SKDA with groups of fewer than 30 people. Even with smaller groups of 30–50 people, we urge caution in interpreting the results (e.g., encouraging managers to have conversations with their teams to validate the relative importance of factors identified as priority items). In addition, we disagree that most criterion measures are composed of a single item. That is not our approach, and it is also not the approach of any of our closest competitors in this space.
Although we appreciate the nod to a proposed “new” approach, we see it as an ideal state that is quite rare in practice. Given the time, money, and other resources desired, we too would prefer to approach organizational research in a manner more closely aligned with the scientific method. However, when an organization signs on to get actionable data from an employee survey, there is little interest in conducting a series of experiments or interventions prior to launch. Clients rely upon consultants’ expertise and experience, and have come to expect that we will begin the survey process with a good understanding of what drives employee engagement. Experiments or interventions that contribute to the understanding of drivers can be conducted behind the scenes.
Concluding Thoughts
When used appropriately, SKDA provides valuable information for practitioners who are tasked with interpreting, communicating, and acting upon survey results. We feel our use of SKDA strikes a balance between theory-driven methodological rigor and practical application for organizations needing quick direction that is easy to understand.
The issues identified in the focal article raise awareness of potential inappropriate use of SKDA and some important theoretical concerns. Further exploration and research are clearly needed, but this does not mean that SKDA should be abandoned altogether. Instead we encourage appropriate knowledge and use of the method, along with more research and empirical support to refine the process and advance the approach. Indeed, we are driving down the right road, but we have not yet reached our destination.
%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
In their focal article, Cucina, Walmsley, Gast, Martin, and Curtin (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) raise an important and timely issue, but go too far in arguing that practitioners should abandon survey key driver analysis (SKDA) altogether. There are many valuable benefits to using SKDA, and rather than discontinuing the practice entirely, we suggest that a better approach is to focus on ways to make it even more valuable to organizations. It is true that this area is in need of further research, but instead of asking whether we are driving down the right road, perhaps the right question is, “Are we there yet?”
We will organize our commentary into three sections. First, we provide a short summary of concerns with the primary study sample and conclusions. Second, we dig into the purported methodological issues with SKDA. Finally, we will summarize the reasons why we believe the focal article overstates the flaws of SKDA, and instead we put forth an argument for the practical utility and benefit of such analyses.
A Short Critique of the Primary Study Sample
Reliable scientific conclusions are based on solid research and consistently replicable effects. Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017) instead make sweeping generalizations about potential flaws intrinsic to SKDA and call for a halt to its practice based on evidence from a single study using an arguably limited sample. For several reasons, we do not believe that findings from the primary study are reliable justification for the focal authors’ broad conclusions. First, the sample itself is archival data from 2008—nearly a decade old—and includes only U.S. federal government employees. We question the generalizability of this population to employees in other industries and (particularly) other time periods. Second, the response rate for this survey data was relatively low—only 51%. By way of contrast, the average response rate for 400-plus IBM Kenexa clients is approximately 80% (see the Appendix for further information). Here again, it is fair to question the representativeness of the sample data and whether the results were affected by nonresponse bias.
Replication is in order before entirely abandoning a potentially useful methodology such as SKDA. We set out to replicate the findings of the focal study using multiple large-scale data sets representative of their respective populations. We found different results that do not support the focal authors’ conclusions, and more research is clearly necessary.
Clarifying and Exposing the Purported Methodological Issues With SKDA
In the following subsections, we carefully consider several of the methodological issues raised by the focal article authors. These issues include the potential impact of item standard deviations, the importance of a priori hypotheses, the existence of situational and temporal specificities, and the thorny issue of causation.
Are SKDA Results Impacted by Item Standard Deviations (SDs)?
In order to better understand the relationship between item standard deviations and their bivariate correlations to criterion item(s), we attempted to replicate a key finding of the focal study using our own data. We leveraged our global normative employee survey database, which is among the largest of such databases in existence. It is maintained as a 3-year rolling average of survey responses and includes nearly 500 survey items. For the current replication, we examined data from 318 of our most common survey items with robust normative data (over 14 million respondents across more than 400 organizations worldwide). The 318 items measure 18 areas common to employee surveys (e.g., communication, growth, inclusion, leadership, recognition), and all survey items were positively scaled using the same 1–5 Likert-type agreement scale.
Consistent with the findings provided by Cucina et al. (Reference Cucina, Walmsley, Gast, Martin and Curtin2017), we found our items to have an average SD of .96 (range .70 to 1.23) and an average item-criterion correlation of .53 (range .34 to .74). In stark contrast to their findings, however, we found a correlation of just .12 between the SD of an item and its correlation to our four-item employee engagement index—our most common criterion in employee engagement surveys (the index measures satisfaction, pride, advocacy, and commitment). This is significantly and substantially lower than the estimate of .62 provided in the focal article. The correlation we found in our own data would be considered small according to both Cohen's (Reference Cohen1992) guidelines and the newer, lower threshold guidelines set forth by Bosco, Aguinis, Singh, Field, and Pierce (Reference Bosco, Aguinis, Singh, Field and Pierce2015). Thus, our findings suggest that item SDs do not impact SKDA results to the extent suggested in the focal article, and concerns about items with larger SDs undermining the ability to properly identify the right targets for action are overstated.
Are A Priori Hypotheses Needed for SKDA?
Although formal a priori hypotheses are not always explicitly stated in organizational survey research, the entire survey design should be based on theory or an identified model regarding things that matter most. For example, our survey methodology is based on a hypothesized high performance engagement model. The model is supported by behavioral science theories, previous research, and “live” data from organizations. We also customize our surveys to address key issues and hypotheses identified by the organization. Before building a survey, we typically spend significant amounts of time with stakeholders identifying what is most important, and then we create the survey with this in mind.
With regard to analyzing survey data, modern methods of SKDA such as dominance analysis (Budescu, Reference Budescu1993) and relative weights analysis (Johnson, Reference Johnson2000) presuppose that an appropriate model has already been identified (LeBreton, Hargis, Griepentrog, Oswald, & Ployhart, Reference LeBreton, Hargis, Griepentrog, Oswald and Ployhart2007). Operating without prior hypotheses before conducting SKDA may occur in some cases and for some practitioners, but it is not a methodological issue with SKDA.
As a way to test the focal article's argument that random items perform as well as SKDA, we examined the difference between the average correlation of an item to our four-item employee engagement index for key drivers versus items that were selected at random. Here, key drivers were included in the pool of survey items available for random selection. That is, key driver items were not removed from the sample prior to randomly selecting items for analysis, thus ensuring a truly random sample of items.
We compared key drivers versus random items across two distinct, large, global client data sets (see Appendix). In both cases, the correlations of key drivers to the criterion were approximately .10 higher than correlations of items pulled at random. The average item–criterion correlation for key drivers versus items pulled at random was .69 compared to .59 in the first data set and .58 vs. .50 in the second. This means that in our data, key driver items showed about a 14% significantly higher correlation to the criterion than items pulled at random. This is certainly an improvement upon chance. Importantly, we went out to 10 driver items for this analysis (had we focused only on the top five, the difference in correlation strength would have been even more pronounced, an approximate difference of .15 correlation units, or 25% higher correlation to the criterion). Finally, in contrast to the final statement in this section provided by the focal article authors, we found the largest difference in correlation strength to be at least .30 correlational units—well above the .10 largest difference stated in the focal article. In other words, .10 was the average difference we found in our data—not the largest.
Observed differences in correlation strength are admittedly an insufficient way to fully justify the importance of focusing on key drivers. A better approach would be to provide evidence that focusing on key drivers leads to greater improvement in the criterion compared to focusing on nondrivers (or items identified at random). Fortunately, we can provide such evidence. We recently examined action plans for over 1,000 managers within a diversified health care company with over 80,000 employees. Data from this 2011 study showed that managers who focused on key driver items had a 13% greater impact on employee engagement the following year than managers who action planned on nondriver items. More research is needed in this area, but this result suggests a clear benefit to focusing on driver items identified through SKDA.
Are There Situational and Temporal Specificities for SKDA?
The focal article argues that SKDA results should be similar across different organizations at different times. However, this argument ignores the potential impact of background factors identified in the psychometric literature. Organizations vary in background factors, such as climate, setting, context, relevant history, and time, and these factors can moderate relationships between predictor and criterion (Johns, Reference Johns2006), and thus influence SKDA results.
We not only assume situational specificity but also count on it in helping to focus action. Good practice includes forming hypotheses about important factors specific to an organization and constructing or adapting items to test those hypotheses. In short, we see different key driver items appear across business units in the same organization (and especially from one organization to another). There is often similarity and overlap in key driver items but also plenty of differences from one group to the next that allow for prescriptive advice and targeted suggestions. Differences across industries, countries, and other factors commonly contribute to observed differences for which key drivers are identified.
With regard to temporal differences, there are many examples in which we have found major differences in key drivers identified from one year to the next within the same organization. We do find that key drivers are fairly stable over time, but it is impractical to simply ignore the possibility that new drivers can emerge across time. Major organizational changes, financial success or decline, M&A activity, and several other factors can impact the relative importance of key driver issues from one time period to the next.
An anonymous reviewer pointed out the possibility that differences attributed to situational and temporal specificity could be due to sampling artifacts in the form of measurement error. However, it is important to note that our census survey programs elicit feedback from entire populations of employees. Because we are dealing with population data and we typically see very high response rates, potential concerns about sampling error affecting SKDA results are greatly reduced.
Does SKDA Test Causation?
SKDA tends to provide information about the extent to which each predictor contributes to the overall R2 relative to other predictors. Well-trained researchers and consultants won't confuse the results of SKDA with causation. In our own practice, we refer to “key drivers” as “priority items” and carefully choose our language to ensure that we avoid implying causation. For executive presentations at the company or division level, we often conduct more advanced statistical analyses including regression, structural equation modeling, relative weights analyses, and factor analyses in order to provide stronger evidence for our conclusions. Although we are careful not to imply that issues underlying items that correlate highly with the criterion actually cause changes in the criterion, perhaps not all practitioners appreciate this nuance. Misinterpretation or misrepresentation of SKDA by some practitioners does not reflect a methodological problem with SKDA itself.
Making a Case for the Practical Value of SKDA
Identification of key drivers is about providing practical guidance and focus for managers who are tasked with understanding, communicating, and action planning based on survey results. Simple correlations such as those used in SKDA are easily understood by consultants and clients alike, and SKDA provides a starting point for discussions about where to focus attention when implementing organizational change. Not conducting analyses at all can leave practitioners blindly guessing about what drives employee engagement. Thus, there is practical utility in SKDA methodology over and above the identification of chance or random factors, and identifying priority items using SKDA can also help organizations avoid the temptation to focus solely on the lowest absolute scores from employee surveys.
As new research methodologies and technologies enter our field, there is a clear need for additional research into ways to leverage these capabilities. Future research areas include potential uses for continuous listening approaches to measure employee attitudes, as well as the role of big data, cognitive (machine learning) technology, and predictive analytics. In addition, we acknowledge the utility of alternatives to statistics-based SKDA. Alternative organizational development methods might include focus groups, stakeholder interviews, simple polling, qualitative research, or action planning in support of strategic organizational priorities. These can be used as standalone methods or in conjunction with statistics-based SKDA. Using evidence from multiple methods to triangulate priorities can provide even stronger arguments for taking action on what matters most. The appropriate approach and best research methodology truly depend on the unique needs and specific goals of the organization.
Practitioners often get a bad reputation for not adhering to the research and theory behind psychometrics and test development. But this reputation is often unwarranted. For example, survey design is very much influenced by theory and hypotheses. We conduct stakeholder interviews and focus groups to form hypotheses about the items most likely to impact engagement. When building surveys, we select the items that are the best drivers across clients to reflect our hypotheses. Moreover, in our practice, we avoid conducting SKDA with groups of fewer than 30 people. Even with smaller groups of 30–50 people, we urge caution in interpreting the results (e.g., encouraging managers to have conversations with their teams to validate the relative importance of factors identified as priority items). In addition, we disagree that most criterion measures are composed of a single item. That is not our approach, and it is also not the approach of any of our closest competitors in this space.
Although we appreciate the nod to a proposed “new” approach, we see it as an ideal state that is quite rare in practice. Given the time, money, and other resources desired, we too would prefer to approach organizational research in a manner more closely aligned with the scientific method. However, when an organization signs on to get actionable data from an employee survey, there is little interest in conducting a series of experiments or interventions prior to launch. Clients rely upon consultants’ expertise and experience, and have come to expect that we will begin the survey process with a good understanding of what drives employee engagement. Experiments or interventions that contribute to the understanding of drivers can be conducted behind the scenes.
Concluding Thoughts
When used appropriately, SKDA provides valuable information for practitioners who are tasked with interpreting, communicating, and acting upon survey results. We feel our use of SKDA strikes a balance between theory-driven methodological rigor and practical application for organizations needing quick direction that is easy to understand.
The issues identified in the focal article raise awareness of potential inappropriate use of SKDA and some important theoretical concerns. Further exploration and research are clearly needed, but this does not mean that SKDA should be abandoned altogether. Instead we encourage appropriate knowledge and use of the method, along with more research and empirical support to refine the process and advance the approach. Indeed, we are driving down the right road, but we have not yet reached our destination.
Appendix
IBM Kenexa WorldNorms™ is our premier benchmarking database. The WorldNorms database is one of the most extensive and up-to-date repositories of employee engagement survey data in the marketplace. Our database includes a total of 825 survey projects from 404 clients across more than 20 industry sectors. It is maintained as a 3-year rolling average of survey responses. Survey data have been gathered from more than 200 countries and in companies ranging in size from several hundred employees to 340,000. Our database currently contains more than 16.4 million employee respondents, and we have received over 850 million survey item responses in the last 3 years.
In addition to our benchmarking database, we utilized three distinct, large, global client data sets to replicate analyses from the focal article. We leveraged two data sets to illustrate the correlations of key driver items to the criterion compared to items pulled at random (including key driver items). The first data set came from a 2016 survey of a global software and human capital management company. The headcount was over 50,000 employees, and the company achieved a response rate of 88%. The second data set came from a 2016 survey of a global engineering, procurement, and construction services company. This company had a headcount of nearly 9,000 employees and a response rate of 76%. A third data set was used to illustrate the impact of action planning on key driver items, over and above the impact seen from focusing on nondriver items or items identified at random. In this case, we made use of data from over 1,000 managers within a diversified healthcare company with over 80,000 employees.