Extending the idea from the focal article regarding the role of descriptive statistics in a primary study (Table 1 in a study; Murphy, Reference Murphy2021), Ihope to encourage researchers to develop a more expansive understanding of how Table 1s (across studies) may prove to be valuable for meta-analytic inquiries in this commentary. Specifically, Idiscuss two ways that descriptive statistics may contribute novel insights to meta-analyses.
Although anyone who has embarked on a meta-analysis project will be intimately familiar with Table 1 (and find the lack thereof frustrating), researchers usually have a focal construct in mind when they are attempting to code the data from Table 1 in each study. During this process, variables that are not considered either an antecedent or a consequence of the focal construct are usually not coded. Furthermore, information that is not required for meta-analytic estimation is typically discarded in subsequent analyses. These practices may be understandable if researchers are attempting to summarize the average effect sizes in a given research domain.
However, a point that routinely gets overlooked is that documenting, discussing, and explaining the heterogeneity of effect sizes should be an equally—if not more—important goal of meta-analyses. Nevertheless, compared with the average population effect size, this heterogeneity receives far less attention in the discussion section of published meta-analyses (DeSimone et al., Reference DeSimone, Köhler and Schoen2019). Relatedly, as described in the focal article, the heterogeneity of effect sizes across situations (and the existence of potential moderators) is rarely referenced or used by researchers who read and cite published meta-analyses (Carlson & Ji, Reference Carlson and Ji2011; DeSimone et al., Reference DeSimone, Köhler and Schoen2019). This is unfortunate, as many questions of great theoretical and practical importance in industrial-organizational psychology focus on “it depends.” Furthermore, even the published meta-analyses that do incorporate moderators in an attempt to explain such heterogeneity are largely focused on the methodological aspects of the research (e.g., cross-sectional versus time lagged). In sharp contrast, theory-driven moderators remain relatively rare (Yuan et al., Reference Yuan, Morgeson and LeBreton2020).
Toward that end, information from Table 1 (and going beyond such as sample characteristics) can be thought of as a concise summary of important contextual aspects of the study sample. In other words, descriptive statistics may prove valuable in meta-analyses by providing information about important sources of contextual influences that may help explain the heterogeneity of effect sizes across situations. For example, the sample mean of trust in upper management may provide useful contextual information about how the sampled employees feel about their employer in general and, therefore, be used as a theoretically meaningful moderator (provided that it functions equivalently across situations, has been measured with similar instruments, and has been converted to the same scale). Further, the standard deviation (SD) of climate may be used as a proxy of climate strength as another important contextual aspect (Schneider et al., Reference Schneider, Salvaggio and Subirats2002). Given that these moderators may be interrelated due to different reasons (e.g., sample mean and SD are especially likely to be interrelated close to the maximum and minimum values; Lindell & Brandt, Reference Lindell and Brandt2000), using them together as moderators in advanced analyses such as meta-analytic regression helps account for their interdependence (Steel & Kammeyer-Mueller, Reference Steel and Kammeyer-Mueller2002).
These examples are intended only to illustrate the methodological issues and choices that arise regarding how best to operationalize theory-driven moderators from information that is reported in Table 1. The larger—and primary—message is that every piece of information from Table 1 may be theoretically important to consider, as it may reveal contextual aspects that can influence the direction and/or magnitude of relationships. The team conflict meta-analysis that was undertaken by de Wit et al. (Reference de Wit, Greer and Jehn2012) provides an exemplar of this approach. In their meta-analysis, these authors made painstaking efforts to connect theory with information reported in Table 1 that could act as theoretically meaningful moderators. For example, they incorporated the sample mean of task and relationship conflict as moderators to shed light on the heterogeneous relationship between team conflict and team performance that is routinely observed in that research domain.
Although the first way that descriptive statistics may enhance meta-analyses is concerned with connecting theory with the information available in Table 1, the second way that Table 1 may lead to novel insights requires researchers to use Table 1 to challenge existing knowledge and assumptions. Given its very nature, theory may constrain, but not facilitate, this approach. In turn, researchers need to think outside the box to glean a broader picture that might otherwise be lost when issues are viewed through the smaller frame of a specific study.
In a thought-provoking methodological article, Köhler et al. (Reference Köhler, Cortina, Kurtessis and Gölz2015) correlated 628 pairs of reliability coefficients from studies that were published in the Journal of Applied Psychology and the Academy of Management Journal (2004–2011) and found substantial overlap between them. Their results challenged the independence assumptions made by psychometric meta-analysis—namely, that study artifacts are independent from each other (Schmidt & Hunter, Reference Schmidt and Hunter2015). In a similar vein, Yuan et al. (Reference Yuan, Morgeson and LeBreton2020) linked a situational variable (i.e., safety climate) to criterion reliability (i.e., safety performance) and further challenged the assumption in psychometric meta-analysis that situational moderators are independent from study artifacts. Taken together, these findings (based on information readily available in Table 1) suggest the independence assumptions underlying psychometric meta-analysis are untenable (James et al., Reference James, Demaree, Mulaik and Ladd1992).
Notably, in these two papers the authors tested the relationship between variables that normally would not attract researchers’ attention in a meta-analysis (i.e., predictor and criterion reliability in Köhler et al., Reference Köhler, Cortina, Kurtessis and Gölz2015; sample mean of safety climate and criterion reliability in Yuan et al., Reference Yuan, Morgeson and LeBreton2020). Yet it is exactly through these unconventional analyses that were based on information from Table 1 that the authors found evidence that would challenge the long-held and rarely contested assumptions underlying the prominent meta-analytic method in industrial-organizational psychology. Indeed, the way that psychometric meta-analysis illustrates the role of reliability is by examining it through the smaller “picture frame” of a single study (i.e., how measurement unreliability attenuates the relationship between variables; see Chapter 2 in Schmidt & Hunter, Reference Schmidt and Hunter2015). By adopting this smaller lens, researchers risk overlooking the possible interrelatedness among study artifacts and moderators (all of which are reported in Table 1) across studies. In other words, better use of information across Table 1s can lead to novel discoveries that would greatly challenge existing knowledge and assumptions.
In conclusion, Table 1s can and should play an even more important role in meta-analytic inquiries. Exploiting them more fully will not only facilitate theory-driven tests of situational moderators that may help explain heterogeneous effect sizes across situations but also lead to novel insights that can challenge existing knowledge and assumptions. It is true that as part of the Journal Article Reporting Standards (American Psychological Association, 2020), researchers are expected to opt in—that is, to include Table 1 in their paper. However, as highlighted in the focal article, many of us have become accustomed to Table 1’s existence and opted out of fully realizing its potential in both primary and secondary research. It is my hope that the two ways of capitalizing on its value that are illustrated in this commentary can help researchers think more critically and creatively about how they may incorporate information from Table 1s when conducting a meta-analysis. Let’s make better use of Table 1s.
You have
Access
Extending the idea from the focal article regarding the role of descriptive statistics in a primary study (Table 1 in a study; Murphy, Reference Murphy2021), Ihope to encourage researchers to develop a more expansive understanding of how Table 1s (across studies) may prove to be valuable for meta-analytic inquiries in this commentary. Specifically, Idiscuss two ways that descriptive statistics may contribute novel insights to meta-analyses.
Although anyone who has embarked on a meta-analysis project will be intimately familiar with Table 1 (and find the lack thereof frustrating), researchers usually have a focal construct in mind when they are attempting to code the data from Table 1 in each study. During this process, variables that are not considered either an antecedent or a consequence of the focal construct are usually not coded. Furthermore, information that is not required for meta-analytic estimation is typically discarded in subsequent analyses. These practices may be understandable if researchers are attempting to summarize the average effect sizes in a given research domain.
However, a point that routinely gets overlooked is that documenting, discussing, and explaining the heterogeneity of effect sizes should be an equally—if not more—important goal of meta-analyses. Nevertheless, compared with the average population effect size, this heterogeneity receives far less attention in the discussion section of published meta-analyses (DeSimone et al., Reference DeSimone, Köhler and Schoen2019). Relatedly, as described in the focal article, the heterogeneity of effect sizes across situations (and the existence of potential moderators) is rarely referenced or used by researchers who read and cite published meta-analyses (Carlson & Ji, Reference Carlson and Ji2011; DeSimone et al., Reference DeSimone, Köhler and Schoen2019). This is unfortunate, as many questions of great theoretical and practical importance in industrial-organizational psychology focus on “it depends.” Furthermore, even the published meta-analyses that do incorporate moderators in an attempt to explain such heterogeneity are largely focused on the methodological aspects of the research (e.g., cross-sectional versus time lagged). In sharp contrast, theory-driven moderators remain relatively rare (Yuan et al., Reference Yuan, Morgeson and LeBreton2020).
Toward that end, information from Table 1 (and going beyond such as sample characteristics) can be thought of as a concise summary of important contextual aspects of the study sample. In other words, descriptive statistics may prove valuable in meta-analyses by providing information about important sources of contextual influences that may help explain the heterogeneity of effect sizes across situations. For example, the sample mean of trust in upper management may provide useful contextual information about how the sampled employees feel about their employer in general and, therefore, be used as a theoretically meaningful moderator (provided that it functions equivalently across situations, has been measured with similar instruments, and has been converted to the same scale). Further, the standard deviation (SD) of climate may be used as a proxy of climate strength as another important contextual aspect (Schneider et al., Reference Schneider, Salvaggio and Subirats2002). Given that these moderators may be interrelated due to different reasons (e.g., sample mean and SD are especially likely to be interrelated close to the maximum and minimum values; Lindell & Brandt, Reference Lindell and Brandt2000), using them together as moderators in advanced analyses such as meta-analytic regression helps account for their interdependence (Steel & Kammeyer-Mueller, Reference Steel and Kammeyer-Mueller2002).
These examples are intended only to illustrate the methodological issues and choices that arise regarding how best to operationalize theory-driven moderators from information that is reported in Table 1. The larger—and primary—message is that every piece of information from Table 1 may be theoretically important to consider, as it may reveal contextual aspects that can influence the direction and/or magnitude of relationships. The team conflict meta-analysis that was undertaken by de Wit et al. (Reference de Wit, Greer and Jehn2012) provides an exemplar of this approach. In their meta-analysis, these authors made painstaking efforts to connect theory with information reported in Table 1 that could act as theoretically meaningful moderators. For example, they incorporated the sample mean of task and relationship conflict as moderators to shed light on the heterogeneous relationship between team conflict and team performance that is routinely observed in that research domain.
Although the first way that descriptive statistics may enhance meta-analyses is concerned with connecting theory with the information available in Table 1, the second way that Table 1 may lead to novel insights requires researchers to use Table 1 to challenge existing knowledge and assumptions. Given its very nature, theory may constrain, but not facilitate, this approach. In turn, researchers need to think outside the box to glean a broader picture that might otherwise be lost when issues are viewed through the smaller frame of a specific study.
In a thought-provoking methodological article, Köhler et al. (Reference Köhler, Cortina, Kurtessis and Gölz2015) correlated 628 pairs of reliability coefficients from studies that were published in the Journal of Applied Psychology and the Academy of Management Journal (2004–2011) and found substantial overlap between them. Their results challenged the independence assumptions made by psychometric meta-analysis—namely, that study artifacts are independent from each other (Schmidt & Hunter, Reference Schmidt and Hunter2015). In a similar vein, Yuan et al. (Reference Yuan, Morgeson and LeBreton2020) linked a situational variable (i.e., safety climate) to criterion reliability (i.e., safety performance) and further challenged the assumption in psychometric meta-analysis that situational moderators are independent from study artifacts. Taken together, these findings (based on information readily available in Table 1) suggest the independence assumptions underlying psychometric meta-analysis are untenable (James et al., Reference James, Demaree, Mulaik and Ladd1992).
Notably, in these two papers the authors tested the relationship between variables that normally would not attract researchers’ attention in a meta-analysis (i.e., predictor and criterion reliability in Köhler et al., Reference Köhler, Cortina, Kurtessis and Gölz2015; sample mean of safety climate and criterion reliability in Yuan et al., Reference Yuan, Morgeson and LeBreton2020). Yet it is exactly through these unconventional analyses that were based on information from Table 1 that the authors found evidence that would challenge the long-held and rarely contested assumptions underlying the prominent meta-analytic method in industrial-organizational psychology. Indeed, the way that psychometric meta-analysis illustrates the role of reliability is by examining it through the smaller “picture frame” of a single study (i.e., how measurement unreliability attenuates the relationship between variables; see Chapter 2 in Schmidt & Hunter, Reference Schmidt and Hunter2015). By adopting this smaller lens, researchers risk overlooking the possible interrelatedness among study artifacts and moderators (all of which are reported in Table 1) across studies. In other words, better use of information across Table 1s can lead to novel discoveries that would greatly challenge existing knowledge and assumptions.
In conclusion, Table 1s can and should play an even more important role in meta-analytic inquiries. Exploiting them more fully will not only facilitate theory-driven tests of situational moderators that may help explain heterogeneous effect sizes across situations but also lead to novel insights that can challenge existing knowledge and assumptions. It is true that as part of the Journal Article Reporting Standards (American Psychological Association, 2020), researchers are expected to opt in—that is, to include Table 1 in their paper. However, as highlighted in the focal article, many of us have become accustomed to Table 1’s existence and opted out of fully realizing its potential in both primary and secondary research. It is my hope that the two ways of capitalizing on its value that are illustrated in this commentary can help researchers think more critically and creatively about how they may incorporate information from Table 1s when conducting a meta-analysis. Let’s make better use of Table 1s.