In his focal article, Murphy (Reference Murphy2021) raised legitimate concerns regarding the need to comprehensively report the contents of Table 1. This sentiment is grounded in the idea that complex analyses isolate people in the organizational sciences and the increased usage of correlations should help bridge the gap between overly complicated procedures and sound science. However, a critical topic that was overlooked by the author pertains to the limitations that are inherent to an overreliance on correlations and semipartial correlations for evaluating the feasibility of hypotheses. For this reason, I propose an alternative method for evaluating hypotheses: dominance analysis (Azen & Budescu, Reference Azen and Budescu2003; Budescu, Reference Budescu1993). This method goes beyond bivariate correlations yet is far less complicated than traditional structural equation modeling (SEM) and hierarchical linear modeling (HLM) techniques.
In this commentary, I recommend that researchers report the relative importance of predictors by using dominance analysis because it leverages the correlation matrix that is available in Table 1 and provides more reliable information for evaluating hypotheses. Dominance analysis information is intuitive to researchers and practitioners because it simply rank orders predictors (Azen & Budescu, Reference Azen and Budescu2003). Ostensibly, providing dominance analysis results adds value to studies regardless of model complexity, whether null hypotheses are rejected, and whether a reader has a high level of statistical proficiency. I argue that rank ordering predictors is a sound method for evaluating the feasibility of hypotheses and one that is resistant to some of the criticisms that are levied against complicated methodologies in Murphy’s (Reference Murphy2021) focal article.
In what follows, I highlight the advantages of using a correlation matrix to determine relative importance via dominance analysis. Importantly, however, I do so within the context of the specific issues that were raised by Murphy (Reference Murphy2021) and with the overall goal of reducing the methodological divide between research and practice.
Using Table 1 to determine the relative importance of correlated predictors: A dominance analysis approach
In most quantitative studies, the goal is to determine which predictor, or subset of predictors, accounts for the most variance in a criterion of interest. Determining the relative importance of a predictor (i.e., relative to the total number of variables in a study) allows researchers and practitioners to ascertain which predictor accounts for the most unique variance in a criterion (Budescu, Reference Budescu1993; LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). Dominance analysis is one of the most common methods for determining relative importance because it produces the incremental predictive validity of each predictor through directly comparing each predictor with all other predictors (Braun et al., Reference Braun, Converse and Oswald2019). The correlation matrix can be the data source to conduct a dominance analysis, without the need for the original data. For instance, scholars can use the “dominanceanalysis” package in R (Navarrete & Soares, Reference Navarrete and Soares2020) to carry out this procedure.
For the purposes of this commentary, I focus on the use of a general dominance weight (Braun et al., Reference Braun, Converse and Oswald2019), which is essentially a predictor’s average R
2 across all submodels—which allows for a simple and intuitive rank ordering of predictors. Moreover, I respond to Murphy’s (Reference Murphy2021) rationale for suggesting the use of correlations and semipartial correlations to evaluate hypotheses. In the next sections, I discuss why dominance analysis may be a better option for accomplishing this goal.
Dominance analysis as a tool for evaluating hypotheses
A primary argument for increasing the use of bivariate correlations is the need to evaluate the feasibility of hypotheses and to determine whether a contradiction in complex analyses is likely to occur (Murphy, Reference Murphy2021). For instance, when evaluating the relationships between independent variables (X), mediators (M), moderators (Z), and outcomes (Y), Murphy notes that there should be several considerations. Among these considerations, I focus on two: (a) in moderation hypotheses, the relationship between X and Y should be stronger than the relationship between Z and Y and (b) in mediation hypotheses, the X and Y relationship should also be stronger than that of M and Y.
It can be argued that if the goal is to evaluate whether X is a better (or more important) predictor of Y than Z or M, looking at correlations or semipartial correlations may not be the best method. First, unless predictors (X, M, and Z) are uncorrelated, the R 2 values that are derived from their bivariate correlations with Y do not address the unique variance explained (Budescu, Reference Budescu1993); thus, it is inconclusive whether X truly is a better predictor of Y than Z or M. Second, even if semipartial correlations are used as a feasibility test (e.g., the effect of X on Y, controlling for Z or M), the difference in effects that is observed between a significant and nonsignificant p value may be meaningless (Gelman & Stern, Reference Gelman and Stern2006). Finally, when covariates are included in a model, the R 2 of predictors can vary depending on the order of entry of other variables (Azen & Budescu, Reference Azen and Budescu2006; Budescu, Reference Budescu1993). Stated differently, X may be perceived to be a “better” predictor of Y than Z or M depending on the covariates that are included.
Considering these limitations, it would mean that describing bivariate or semipartial correlations may not have the intended effect that Murphy (Reference Murphy2021) had in mind. That is, it may not provide scholars with the tools that are necessary for critically evaluating the feasibility of moderation and mediational hypotheses. Instead of reporting bivariate or semipartial correlations, it would be much more useful to report the general dominance weight and the percentage of average unique R 2 accounted for by each predictor (LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). An option for reporting supplemental dominance analysis results would be to run a multiple regression model with all independent variables, moderators, mediators, and covariates entered simultaneously.
This essentially allows for a rank ordering of predictors, and it gives a cleaner interpretation of whether X is, truly, a better predictor than M or Z, in the presence of any covariates. This would allow scholars to evaluate the feasibility of hypotheses (e.g., the idea that X should be a better predictor of Y than M or Z) in a robust and intuitive manner. The dominance analysis procedure aligns with Murphy’s (Reference Murphy2021) arguments about leveraging Table 1 to evaluate hypotheses; though, notably, this sentiment is taken a step further. Finally, dominance analysis does not rely on the p value to determine importance (Azen & Budescu, Reference Azen and Budescu2006), which circumvents the problematic influence of sample size on significance testing.
The replicability and verifiability of dominance analysis results
Murphy (Reference Murphy2021) argued that a major problem with complex analyses was the fact that there are substantial errors in reporting, citing work by Cortina et al. (Reference Cortina, Green, Keeler and Vandenberg2017) who found that a large portion of SEM studies reported an incorrect df. It is proposed that reasons for mistakes largely pertain to the specification of model parameters (Cortina et al., Reference Cortina, Green, Keeler and Vandenberg2017).
Dominance analysis does not have this problem, particularly when the correlation matrix is used as the data source. Due to the correlation matrix having utility in the absence of original data, consumers of research can more readily replicate or verify dominance analysis results. This is because the sum of general dominance weights across all predictors in a model equals the total R 2 (Azen & Budescu, Reference Azen and Budescu2003; Braun et al., Reference Braun, Converse and Oswald2019). Therefore, the general dominance weight can be divided by the total R 2 to provide users with a percentage of unique variance explained for an outcome (LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). Verifying dominance analysis results, therefore, can be accomplished without a high level of statistical proficiency in comparison with SEM or other methods such as HLM. This, in turn, should provide readers with more confidence in the results. For an illustrative example of a dominance analysis procedure using a correlation matrix, see Kleine et al. (Reference Kleine, Rudolph and Zacher2019).
Conclusion
In this commentary, I brought to light some issues about using bivariate and semipartial correlations to infer whether moderation and mediational hypotheses are warranted. These issues are grounded in the fact that R 2 values in correlated predictors may not address unique variance and the notion that significant versus nonsignificant effects that are observed among predictors may not be meaningfully different from one another. To counteract this, it was proposed that the correlation matrix be used to consistently report the relative importance of predictors through dominance analysis. With relative importance information, scholars should be armed with a useful tool for evaluating the feasibility of moderation and mediational hypotheses—the notion that X should be a better predictor of Y than M or Z. Furthermore, dominance analysis results are readily verifiable if correlation matrices are used, and when covariates are included in models, the substantive conclusions about predictor importance holds. In reality, regardless of whether consumers of research are academics or practitioners or whether they understand the complex methods employed, the substantive information provided by dominance analysis will add value to studies and increase scientific rigor without confusing readers.
You have
Access
In his focal article, Murphy (Reference Murphy2021) raised legitimate concerns regarding the need to comprehensively report the contents of Table 1. This sentiment is grounded in the idea that complex analyses isolate people in the organizational sciences and the increased usage of correlations should help bridge the gap between overly complicated procedures and sound science. However, a critical topic that was overlooked by the author pertains to the limitations that are inherent to an overreliance on correlations and semipartial correlations for evaluating the feasibility of hypotheses. For this reason, I propose an alternative method for evaluating hypotheses: dominance analysis (Azen & Budescu, Reference Azen and Budescu2003; Budescu, Reference Budescu1993). This method goes beyond bivariate correlations yet is far less complicated than traditional structural equation modeling (SEM) and hierarchical linear modeling (HLM) techniques.
In this commentary, I recommend that researchers report the relative importance of predictors by using dominance analysis because it leverages the correlation matrix that is available in Table 1 and provides more reliable information for evaluating hypotheses. Dominance analysis information is intuitive to researchers and practitioners because it simply rank orders predictors (Azen & Budescu, Reference Azen and Budescu2003). Ostensibly, providing dominance analysis results adds value to studies regardless of model complexity, whether null hypotheses are rejected, and whether a reader has a high level of statistical proficiency. I argue that rank ordering predictors is a sound method for evaluating the feasibility of hypotheses and one that is resistant to some of the criticisms that are levied against complicated methodologies in Murphy’s (Reference Murphy2021) focal article.
In what follows, I highlight the advantages of using a correlation matrix to determine relative importance via dominance analysis. Importantly, however, I do so within the context of the specific issues that were raised by Murphy (Reference Murphy2021) and with the overall goal of reducing the methodological divide between research and practice.
Using Table 1 to determine the relative importance of correlated predictors: A dominance analysis approach
In most quantitative studies, the goal is to determine which predictor, or subset of predictors, accounts for the most variance in a criterion of interest. Determining the relative importance of a predictor (i.e., relative to the total number of variables in a study) allows researchers and practitioners to ascertain which predictor accounts for the most unique variance in a criterion (Budescu, Reference Budescu1993; LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). Dominance analysis is one of the most common methods for determining relative importance because it produces the incremental predictive validity of each predictor through directly comparing each predictor with all other predictors (Braun et al., Reference Braun, Converse and Oswald2019). The correlation matrix can be the data source to conduct a dominance analysis, without the need for the original data. For instance, scholars can use the “dominanceanalysis” package in R (Navarrete & Soares, Reference Navarrete and Soares2020) to carry out this procedure.
For the purposes of this commentary, I focus on the use of a general dominance weight (Braun et al., Reference Braun, Converse and Oswald2019), which is essentially a predictor’s average R 2 across all submodels—which allows for a simple and intuitive rank ordering of predictors. Moreover, I respond to Murphy’s (Reference Murphy2021) rationale for suggesting the use of correlations and semipartial correlations to evaluate hypotheses. In the next sections, I discuss why dominance analysis may be a better option for accomplishing this goal.
Dominance analysis as a tool for evaluating hypotheses
A primary argument for increasing the use of bivariate correlations is the need to evaluate the feasibility of hypotheses and to determine whether a contradiction in complex analyses is likely to occur (Murphy, Reference Murphy2021). For instance, when evaluating the relationships between independent variables (X), mediators (M), moderators (Z), and outcomes (Y), Murphy notes that there should be several considerations. Among these considerations, I focus on two: (a) in moderation hypotheses, the relationship between X and Y should be stronger than the relationship between Z and Y and (b) in mediation hypotheses, the X and Y relationship should also be stronger than that of M and Y.
It can be argued that if the goal is to evaluate whether X is a better (or more important) predictor of Y than Z or M, looking at correlations or semipartial correlations may not be the best method. First, unless predictors (X, M, and Z) are uncorrelated, the R 2 values that are derived from their bivariate correlations with Y do not address the unique variance explained (Budescu, Reference Budescu1993); thus, it is inconclusive whether X truly is a better predictor of Y than Z or M. Second, even if semipartial correlations are used as a feasibility test (e.g., the effect of X on Y, controlling for Z or M), the difference in effects that is observed between a significant and nonsignificant p value may be meaningless (Gelman & Stern, Reference Gelman and Stern2006). Finally, when covariates are included in a model, the R 2 of predictors can vary depending on the order of entry of other variables (Azen & Budescu, Reference Azen and Budescu2006; Budescu, Reference Budescu1993). Stated differently, X may be perceived to be a “better” predictor of Y than Z or M depending on the covariates that are included.
Considering these limitations, it would mean that describing bivariate or semipartial correlations may not have the intended effect that Murphy (Reference Murphy2021) had in mind. That is, it may not provide scholars with the tools that are necessary for critically evaluating the feasibility of moderation and mediational hypotheses. Instead of reporting bivariate or semipartial correlations, it would be much more useful to report the general dominance weight and the percentage of average unique R 2 accounted for by each predictor (LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). An option for reporting supplemental dominance analysis results would be to run a multiple regression model with all independent variables, moderators, mediators, and covariates entered simultaneously.
This essentially allows for a rank ordering of predictors, and it gives a cleaner interpretation of whether X is, truly, a better predictor than M or Z, in the presence of any covariates. This would allow scholars to evaluate the feasibility of hypotheses (e.g., the idea that X should be a better predictor of Y than M or Z) in a robust and intuitive manner. The dominance analysis procedure aligns with Murphy’s (Reference Murphy2021) arguments about leveraging Table 1 to evaluate hypotheses; though, notably, this sentiment is taken a step further. Finally, dominance analysis does not rely on the p value to determine importance (Azen & Budescu, Reference Azen and Budescu2006), which circumvents the problematic influence of sample size on significance testing.
The replicability and verifiability of dominance analysis results
Murphy (Reference Murphy2021) argued that a major problem with complex analyses was the fact that there are substantial errors in reporting, citing work by Cortina et al. (Reference Cortina, Green, Keeler and Vandenberg2017) who found that a large portion of SEM studies reported an incorrect df. It is proposed that reasons for mistakes largely pertain to the specification of model parameters (Cortina et al., Reference Cortina, Green, Keeler and Vandenberg2017).
Dominance analysis does not have this problem, particularly when the correlation matrix is used as the data source. Due to the correlation matrix having utility in the absence of original data, consumers of research can more readily replicate or verify dominance analysis results. This is because the sum of general dominance weights across all predictors in a model equals the total R 2 (Azen & Budescu, Reference Azen and Budescu2003; Braun et al., Reference Braun, Converse and Oswald2019). Therefore, the general dominance weight can be divided by the total R 2 to provide users with a percentage of unique variance explained for an outcome (LeBreton et al., Reference LeBreton, Ployhart and Ladd2004). Verifying dominance analysis results, therefore, can be accomplished without a high level of statistical proficiency in comparison with SEM or other methods such as HLM. This, in turn, should provide readers with more confidence in the results. For an illustrative example of a dominance analysis procedure using a correlation matrix, see Kleine et al. (Reference Kleine, Rudolph and Zacher2019).
Conclusion
In this commentary, I brought to light some issues about using bivariate and semipartial correlations to infer whether moderation and mediational hypotheses are warranted. These issues are grounded in the fact that R 2 values in correlated predictors may not address unique variance and the notion that significant versus nonsignificant effects that are observed among predictors may not be meaningfully different from one another. To counteract this, it was proposed that the correlation matrix be used to consistently report the relative importance of predictors through dominance analysis. With relative importance information, scholars should be armed with a useful tool for evaluating the feasibility of moderation and mediational hypotheses—the notion that X should be a better predictor of Y than M or Z. Furthermore, dominance analysis results are readily verifiable if correlation matrices are used, and when covariates are included in models, the substantive conclusions about predictor importance holds. In reality, regardless of whether consumers of research are academics or practitioners or whether they understand the complex methods employed, the substantive information provided by dominance analysis will add value to studies and increase scientific rigor without confusing readers.