Considering the increasing complexity of the statistical analyses that are used in organizational research, Murphy (Reference Murphy2021) highlights the need for our research findings to be readily interpretable and applicable in practice contexts. The paper encourages the use of effect-size estimates to reveal whether statistically significant findings are also practically significant. Murphy’s paper focuses primarily on communicating research findings within the organizational researcher/practitioner community among those who are trained in understanding statistical analyses. The concerns raised by the paper are amplified when the research results must be communicated to those with only a rudimentary understanding of statistics. Many human resources (HR) professionals and other organizational decision makers fall into this category of having a limited understanding of statistical methods. To advocate for change based on research findings, it is critical for these individuals to understand the findings and to see the practical importance of them for their organization. This commentary explores the challenges that are associated with explaining the practical importance of the results in the particular case of validation studies. Suggestions on how to communicate validation results more clearly are also discussed.
Validation studies on selection processes are a common form of research in organizations. Organizational leaders seek to ensure that selection procedures are both legally defensible and useful in predicting who will perform well on job-related criteria that are important to the organization’s success. The performance criterion may be supervisor ratings or objective metrics such as total sales, average call handle time, or the number of items assembled in a work shift. Along-employed effect size estimate used in validation studies, which employ correlation and multiple regression analyses, is R 2 (Ozer, Reference Ozer1985). However, three issues may make R 2 an ineffectual and potentially confusing way to communicate the effectiveness (i.e., effect size) of a selection process to HR professionals and other organizational decision makers.
First, the concept that R 2 represents the “proportion of the variability of the dependent variable that is predictable from a set of variables” (Fritz et al., Reference Fritz, Morris and Richler2012, p. 11) is well entrenched in the minds of organizational researchers and even in the minds of some HR professionals. But, those outside the organizational research community may not fully understand what variance represents. Variance, being the average of the squared distances from the mean, is not an intuitive concept. When those with a casual understanding of statistics hear the term variance of the performance measure, they may interpret it as how much the measure varies between the top and bottom performers (i.e., the range of the performance scores). When they hear that R 2 is the proportion of variance that is predicted by the selection process, they may think of it as the proportion of the range of performance scores that is predicted. This could lead to the true practical effect of the selection process being grossly underrepresented. For example, a correlation of .21 represents the 50th percentile for predictive relationships between job performance and knowledge, skills, and abilities (Bosco et al., Reference Bosco, Aguinis, Singh, Field and Pierce2015). The R 2 value for this correlation is .0441. If a decision maker who is evaluating a selection procedure is told that it will predict 4.4% of the variance in employee performance and they interpret that as 4.4% of the difference in employee performance, they are likely to be unimpressed.
Second, the units of variance are typically nonsensical when it comes to interpreting them in terms of employee performance. Variance is in terms of squared measurement units. If the performance criterion in a validation study for a sales role was the total dollar amount of sales the individual made in a year, then the unit associated with the variance would be $2. Discussing effect size in squared units is contrary to the APA Publication Manual’s (APA, 2020) observation that effect sizes are most easily understood when they are presented in the original units (p. 89). Expanding on the example above, if an organizational consultant were to tell an HR professional that the proposed selection process predicted 4.4% of the variance in $2, they would likely be met with a confused look.
Third, the parabolic, nonlinear relationship between R and R 2 is likely to cause confusion for those who are not familiar with the details of mathematical relationships. Suppose a new selection procedure resulted in the validity coefficient increasing .1 from .1 to .2. This .1 increase in the validity coefficient would increase the proportion of variance explained from 1% to 4%, a gain of 3%. If further refinement to the selection process resulted in another gain of .1 leading to a validity coefficient of .3, the variance explained would be 9%, a gain of 5%. The same increase in the validity coefficient leads to a different gain in the percentage of variance explained. This inconsistent gain could be confusing to some.
So, when communicating with HR professionals and other organizational decision makers about the results of validation studies, how can we heed Murphy’s (Reference Murphy2021) admonition to make our results understandable and applicable? Three possibilities are discussed here. One approach is to discuss the practical importance directly in terms of R rather than R 2. Although this may still be somewhat abstract for some HR professionals, there are resources available to benchmark what is a good validity coefficient in a particular context. If a validation study shows that a measure of some psychological characteristics has a validity coefficient of .31 in predicting job performance, the organization can be shown that this value is in the top range that can be expected for this type of predictor/criterion combination (Bosco et al., Reference Bosco, Aguinis, Singh, Field and Pierce2015).
A second approach is to use the regression equation to show the expected gain in performance for a certain point gain on the selection procedure. This allows the effectiveness of the selection process to be viewed directly in terms of the performance criterion. The results can be presented in terms such as “on average, for each 10-point gain a person has on the assessment score, you can expect to see a $30,000 gain in annual sales.” However, although research of the past 25 years within the field of industrial-organizational psychology has leaned negative on the endorsement of utility analyses with organizational decision makers, Winkler et al. (Reference Winkler, König and Kleinmann2010) suggested a modified approach that focuses on a causal chain analysis rather than a single-outcome variable analysis that may provide more applicable and simple methods to communicate the value of selection processes.
The third approach is to create theoretical expectancy tables based on the validity coefficient. Scores on both the assessment process and on the performance criterion can be divided into groups based on percentiles. For example, scores on the assessment process could simply be divided into low, medium, and high zones that represent the lower, middle, and upper 33.3rd percentiles. Then a table showing what percentage of people in each assessment score zone is expected to be in the lower, middle, and upper performance zones can be constructed. The information in the table can also be represented in a chart to provide a visual aid for explaining the expected outcomes. This allows HR practitioners to easily see how using the selection process to hire the higher scorers will lead to having a higher percentage of top performers. Cucina et al. (Reference Cucina, Berger and Busciglio2017) provided a new approach for creating expectancy charts, which uses the bivariate-normal distribution, which has overcome many of the limitations of older methods of creating theoretical expectancy tables. Cucina et al. provide Rcode that allows the easy creation of theorical expectancy tables with customized numbers of predictor and performance groups. Avariation on the expectancy table, which aligns with Murphy’s (Reference Murphy2021) call to make better use of descriptive statistics, is to calculate the expected means on the performance metric for each of the assessment score groups.
The effectiveness of these three approaches in communicating the practical significance of results from validation studies can be enhanced if practitioners also heed the recommendations from Latham (Reference Latham2007). Latham encouraged organizational researchers to ensure they’re “bilingual” in communicating results using vocabulary that nonresearchers can easily understand (e.g., instead of describing the need for a control group, explain to management that you have to compare those who did and did not use an experimental HR program). For a validation study, this may mean avoiding the term regression coefficient and instead referring to the influence of the various predictors.
Clearly communicating the practical significance of validation (and other) research results to organizational decision makers who are not familiar with the details of statistical methods can be a challenge. But, if our field’s insights and understanding of selecting those with the best chance of performing well in a job are to be implemented, we need to exert the effort to apply Murphy’s (Reference Murphy2021) advice of making research results clear and applicable to our communications with HR professionals and other organizational decision makers.
You have
Access
Considering the increasing complexity of the statistical analyses that are used in organizational research, Murphy (Reference Murphy2021) highlights the need for our research findings to be readily interpretable and applicable in practice contexts. The paper encourages the use of effect-size estimates to reveal whether statistically significant findings are also practically significant. Murphy’s paper focuses primarily on communicating research findings within the organizational researcher/practitioner community among those who are trained in understanding statistical analyses. The concerns raised by the paper are amplified when the research results must be communicated to those with only a rudimentary understanding of statistics. Many human resources (HR) professionals and other organizational decision makers fall into this category of having a limited understanding of statistical methods. To advocate for change based on research findings, it is critical for these individuals to understand the findings and to see the practical importance of them for their organization. This commentary explores the challenges that are associated with explaining the practical importance of the results in the particular case of validation studies. Suggestions on how to communicate validation results more clearly are also discussed.
Validation studies on selection processes are a common form of research in organizations. Organizational leaders seek to ensure that selection procedures are both legally defensible and useful in predicting who will perform well on job-related criteria that are important to the organization’s success. The performance criterion may be supervisor ratings or objective metrics such as total sales, average call handle time, or the number of items assembled in a work shift. Along-employed effect size estimate used in validation studies, which employ correlation and multiple regression analyses, is R 2 (Ozer, Reference Ozer1985). However, three issues may make R 2 an ineffectual and potentially confusing way to communicate the effectiveness (i.e., effect size) of a selection process to HR professionals and other organizational decision makers.
First, the concept that R 2 represents the “proportion of the variability of the dependent variable that is predictable from a set of variables” (Fritz et al., Reference Fritz, Morris and Richler2012, p. 11) is well entrenched in the minds of organizational researchers and even in the minds of some HR professionals. But, those outside the organizational research community may not fully understand what variance represents. Variance, being the average of the squared distances from the mean, is not an intuitive concept. When those with a casual understanding of statistics hear the term variance of the performance measure, they may interpret it as how much the measure varies between the top and bottom performers (i.e., the range of the performance scores). When they hear that R 2 is the proportion of variance that is predicted by the selection process, they may think of it as the proportion of the range of performance scores that is predicted. This could lead to the true practical effect of the selection process being grossly underrepresented. For example, a correlation of .21 represents the 50th percentile for predictive relationships between job performance and knowledge, skills, and abilities (Bosco et al., Reference Bosco, Aguinis, Singh, Field and Pierce2015). The R 2 value for this correlation is .0441. If a decision maker who is evaluating a selection procedure is told that it will predict 4.4% of the variance in employee performance and they interpret that as 4.4% of the difference in employee performance, they are likely to be unimpressed.
Second, the units of variance are typically nonsensical when it comes to interpreting them in terms of employee performance. Variance is in terms of squared measurement units. If the performance criterion in a validation study for a sales role was the total dollar amount of sales the individual made in a year, then the unit associated with the variance would be $2. Discussing effect size in squared units is contrary to the APA Publication Manual’s (APA, 2020) observation that effect sizes are most easily understood when they are presented in the original units (p. 89). Expanding on the example above, if an organizational consultant were to tell an HR professional that the proposed selection process predicted 4.4% of the variance in $2, they would likely be met with a confused look.
Third, the parabolic, nonlinear relationship between R and R 2 is likely to cause confusion for those who are not familiar with the details of mathematical relationships. Suppose a new selection procedure resulted in the validity coefficient increasing .1 from .1 to .2. This .1 increase in the validity coefficient would increase the proportion of variance explained from 1% to 4%, a gain of 3%. If further refinement to the selection process resulted in another gain of .1 leading to a validity coefficient of .3, the variance explained would be 9%, a gain of 5%. The same increase in the validity coefficient leads to a different gain in the percentage of variance explained. This inconsistent gain could be confusing to some.
So, when communicating with HR professionals and other organizational decision makers about the results of validation studies, how can we heed Murphy’s (Reference Murphy2021) admonition to make our results understandable and applicable? Three possibilities are discussed here. One approach is to discuss the practical importance directly in terms of R rather than R 2. Although this may still be somewhat abstract for some HR professionals, there are resources available to benchmark what is a good validity coefficient in a particular context. If a validation study shows that a measure of some psychological characteristics has a validity coefficient of .31 in predicting job performance, the organization can be shown that this value is in the top range that can be expected for this type of predictor/criterion combination (Bosco et al., Reference Bosco, Aguinis, Singh, Field and Pierce2015).
A second approach is to use the regression equation to show the expected gain in performance for a certain point gain on the selection procedure. This allows the effectiveness of the selection process to be viewed directly in terms of the performance criterion. The results can be presented in terms such as “on average, for each 10-point gain a person has on the assessment score, you can expect to see a $30,000 gain in annual sales.” However, although research of the past 25 years within the field of industrial-organizational psychology has leaned negative on the endorsement of utility analyses with organizational decision makers, Winkler et al. (Reference Winkler, König and Kleinmann2010) suggested a modified approach that focuses on a causal chain analysis rather than a single-outcome variable analysis that may provide more applicable and simple methods to communicate the value of selection processes.
The third approach is to create theoretical expectancy tables based on the validity coefficient. Scores on both the assessment process and on the performance criterion can be divided into groups based on percentiles. For example, scores on the assessment process could simply be divided into low, medium, and high zones that represent the lower, middle, and upper 33.3rd percentiles. Then a table showing what percentage of people in each assessment score zone is expected to be in the lower, middle, and upper performance zones can be constructed. The information in the table can also be represented in a chart to provide a visual aid for explaining the expected outcomes. This allows HR practitioners to easily see how using the selection process to hire the higher scorers will lead to having a higher percentage of top performers. Cucina et al. (Reference Cucina, Berger and Busciglio2017) provided a new approach for creating expectancy charts, which uses the bivariate-normal distribution, which has overcome many of the limitations of older methods of creating theoretical expectancy tables. Cucina et al. provide Rcode that allows the easy creation of theorical expectancy tables with customized numbers of predictor and performance groups. Avariation on the expectancy table, which aligns with Murphy’s (Reference Murphy2021) call to make better use of descriptive statistics, is to calculate the expected means on the performance metric for each of the assessment score groups.
The effectiveness of these three approaches in communicating the practical significance of results from validation studies can be enhanced if practitioners also heed the recommendations from Latham (Reference Latham2007). Latham encouraged organizational researchers to ensure they’re “bilingual” in communicating results using vocabulary that nonresearchers can easily understand (e.g., instead of describing the need for a control group, explain to management that you have to compare those who did and did not use an experimental HR program). For a validation study, this may mean avoiding the term regression coefficient and instead referring to the influence of the various predictors.
Clearly communicating the practical significance of validation (and other) research results to organizational decision makers who are not familiar with the details of statistical methods can be a challenge. But, if our field’s insights and understanding of selecting those with the best chance of performing well in a job are to be implemented, we need to exert the effort to apply Murphy’s (Reference Murphy2021) advice of making research results clear and applicable to our communications with HR professionals and other organizational decision makers.