Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017) describe two main sources of uncertainty that are usually reported in a meta-analysis: uncertainty about the value of the underlying mean correlation (which they describe as SE(rxy
)) and uncertainty about the individual values of rho that arise from the random-effects variance (the square root of which they describe as SD(rho)). They proceed to recommend descriptions of small, medium, and large values of each uncertainty that meta-analysts should report and consider for interpretation. However, there exists a simpler solution to expressing and interpreting such uncertainty.
The prediction interval (Borenstein, Hedges, Higgins, & Rothstein, Reference Borenstein, Hedges, Higgins and Rothstein2009) is analogous to the credibility interval but incorporates uncertainty about both the mean and the variance of the underlying distribution of rho. Borenstein et al.’s (Reference Borenstein, Hedges, Higgins and Rothstein2009, p. 133) formula is:
where M* is the summary estimate (overall mean), t refers to the t distribution (with df = number of effect sizes less two) rather than the unit normal because the underlying variance is estimated, the term under the radical on the left refers to the random-effects variance (estimated Var(rho)), and the final term refers to the variance of the mean. Reporting the prediction interval communicates a plausible range in which population correlations are expected to be found based on the current evidence. Such a presentation is easier to grasp than are cutoffs for standard deviations or standard errors. This is not to say that information about the confidence interval for the mean and the size of the random effects variance is unimportant. The prediction interval just expresses such quantities in a single summary that is more easily interpreted.
However, in my opinion, this is not the main issue with the typical presentation of meta-analysis results. Tett et al. (Reference Tett, Hundley and Christiansen2017) nicely described typical meta-analyses in I-O psychology. Meta-analysts commonly compute and report credibility intervals, and then ignore them. It has been my experience that people find the credibility interval difficult to understand; they interpret it the same way as they do a confidence interval (Whitener, Reference Whitener1990). This is analogous to a tailor measuring the waist and inseam for 1,000 people, getting a precise average size, stitching one pair of pants based on the average, and declaring that their new pants are properly one-size-fits-all through the science of Trouser Generalization. If the population random-effects variance is greater than zero, then the data are consistent with the notion that the population correlations really are different in different places, just as some people are taller than others (and therefore need longer trousers).
Paterson, Harms, Steel, and Crede (Reference Paterson, Harms, Steel and Crede2016) reviewed over 250 meta-analyses in the organizational sciences. They report an approximate average rho of .28 and an approximate average SD(rho) of .14 (cf. Oswald & McCloy's [Reference Oswald, McCloy and Murphy2003] “large” SD(rho) value of .125). If a distribution is normal, then going up and down about 2 standard deviations will cover about 95% of the cases. Based on the Paterson et al.’s (Reference Paterson, Harms, Steel and Crede2016) data, on average, after a meta-analysis, we can be confident that the true correlation in any local setting is between 0 and .56. Did we learn much from the meta-analysis? That interval is over one-quarter of the possible range of the correlation coefficient; I suggest that there is a great deal of uncertainty remaining after the typical meta-analysis is completed.
The main difficulty is that the random-effects variance is problematic, and it is largely ignored in our reports; at a minimum, we should interpret it. The prediction interval is a start. A large SD(rho) warns that our understanding of the phenomenon is shallow. A large SD(rho) should serve as a trigger for developing models that explain the impact of different measures, demographics of participants, work content, organization type, cultural context, and so forth. By explaining the variation in outcomes, SD(rho) can be reduced to the point where it is trivial. Such modeling of effect sizes is difficult because of the large number of potential moderators compared to the small numbers of studies (Schmidt & Hunter, Reference Schmidt and Hunter2015). However, such modeling is possible, and it is essential if we are to claim an understanding rather than a simple description of organizational phenomena.
You have
Access
Tett, Hundley, and Christiansen (Reference Tett, Hundley and Christiansen2017) describe two main sources of uncertainty that are usually reported in a meta-analysis: uncertainty about the value of the underlying mean correlation (which they describe as SE(rxy )) and uncertainty about the individual values of rho that arise from the random-effects variance (the square root of which they describe as SD(rho)). They proceed to recommend descriptions of small, medium, and large values of each uncertainty that meta-analysts should report and consider for interpretation. However, there exists a simpler solution to expressing and interpreting such uncertainty.
The prediction interval (Borenstein, Hedges, Higgins, & Rothstein, Reference Borenstein, Hedges, Higgins and Rothstein2009) is analogous to the credibility interval but incorporates uncertainty about both the mean and the variance of the underlying distribution of rho. Borenstein et al.’s (Reference Borenstein, Hedges, Higgins and Rothstein2009, p. 133) formula is:
where M* is the summary estimate (overall mean), t refers to the t distribution (with df = number of effect sizes less two) rather than the unit normal because the underlying variance is estimated, the term under the radical on the left refers to the random-effects variance (estimated Var(rho)), and the final term refers to the variance of the mean. Reporting the prediction interval communicates a plausible range in which population correlations are expected to be found based on the current evidence. Such a presentation is easier to grasp than are cutoffs for standard deviations or standard errors. This is not to say that information about the confidence interval for the mean and the size of the random effects variance is unimportant. The prediction interval just expresses such quantities in a single summary that is more easily interpreted.
However, in my opinion, this is not the main issue with the typical presentation of meta-analysis results. Tett et al. (Reference Tett, Hundley and Christiansen2017) nicely described typical meta-analyses in I-O psychology. Meta-analysts commonly compute and report credibility intervals, and then ignore them. It has been my experience that people find the credibility interval difficult to understand; they interpret it the same way as they do a confidence interval (Whitener, Reference Whitener1990). This is analogous to a tailor measuring the waist and inseam for 1,000 people, getting a precise average size, stitching one pair of pants based on the average, and declaring that their new pants are properly one-size-fits-all through the science of Trouser Generalization. If the population random-effects variance is greater than zero, then the data are consistent with the notion that the population correlations really are different in different places, just as some people are taller than others (and therefore need longer trousers).
Paterson, Harms, Steel, and Crede (Reference Paterson, Harms, Steel and Crede2016) reviewed over 250 meta-analyses in the organizational sciences. They report an approximate average rho of .28 and an approximate average SD(rho) of .14 (cf. Oswald & McCloy's [Reference Oswald, McCloy and Murphy2003] “large” SD(rho) value of .125). If a distribution is normal, then going up and down about 2 standard deviations will cover about 95% of the cases. Based on the Paterson et al.’s (Reference Paterson, Harms, Steel and Crede2016) data, on average, after a meta-analysis, we can be confident that the true correlation in any local setting is between 0 and .56. Did we learn much from the meta-analysis? That interval is over one-quarter of the possible range of the correlation coefficient; I suggest that there is a great deal of uncertainty remaining after the typical meta-analysis is completed.
The main difficulty is that the random-effects variance is problematic, and it is largely ignored in our reports; at a minimum, we should interpret it. The prediction interval is a start. A large SD(rho) warns that our understanding of the phenomenon is shallow. A large SD(rho) should serve as a trigger for developing models that explain the impact of different measures, demographics of participants, work content, organization type, cultural context, and so forth. By explaining the variation in outcomes, SD(rho) can be reduced to the point where it is trivial. Such modeling of effect sizes is difficult because of the large number of potential moderators compared to the small numbers of studies (Schmidt & Hunter, Reference Schmidt and Hunter2015). However, such modeling is possible, and it is essential if we are to claim an understanding rather than a simple description of organizational phenomena.