Crossref Citations
This article has been cited by the following publications. This list is generated based on data provided by Crossref.
Oswald, Frederick L.
Ercan, Seydahmet
McAbee, Samuel T.
Ock, Jisoo
and
Shaw, Amy
2015.
Imperfect Corrections or Correct Imperfections?: Psychometric Corrections in Meta-Analysis – ADDENDUM.
Industrial and Organizational Psychology,
Vol. 8,
Issue. 2,
p.
305.
Samba, Codou
Van Knippenberg, Daan
and
Miller, C. Chet
2018.
The impact of strategic dissent on organizational outcomes: A meta‐analytic integration.
Strategic Management Journal,
Vol. 39,
Issue. 2,
p.
379.
Oswald, Frederick L.
2019.
Measuring and modeling cognitive ability: Some comments on process overlap theory..
Journal of Applied Research in Memory and Cognition,
Vol. 8,
Issue. 3,
p.
296.
Oswald, Frederick L.
and
Tackett, Jennifer L.
2020.
Multilevel Modeling and Meta-Analysis: Epilogue to the Invited Forums.
Advances in Methods and Practices in Psychological Science,
Vol. 3,
Issue. 1,
p.
138.
Hambrick, David Z.
Macnamara, Brooke N.
and
Oswald, Frederick L.
2020.
Is the Deliberate Practice View Defensible? A Review of Evidence and Discussion of Issues.
Frontiers in Psychology,
Vol. 11,
Issue. ,
Wiernik, Brenton M.
and
Dahlke, Jeffrey A.
2020.
Obtaining Unbiased Results in Meta-Analysis: The Importance of Correcting for Statistical Artifacts.
Advances in Methods and Practices in Psychological Science,
Vol. 3,
Issue. 1,
p.
94.
Morris, Scott B.
2023.
Meta-Analysis in Organizational Research: A Guide to Methodological Options.
Annual Review of Organizational Psychology and Organizational Behavior,
Vol. 10,
Issue. 1,
p.
225.
Plonsky, Luke
Hu, Yuhang
Sudina, Ekaterina
and
Oswald, Frederick L.
2023.
Current Approaches in Second Language Acquisition Research.
p.
304.
Becker, Betsy Jane
and
Zhang, Qian
2024.
Advances in meta‐analysis: A unifying modelling framework with measurement error correction.
British Journal of Mathematical and Statistical Psychology,
Vol. 77,
Issue. 3,
p.
395.
Wang, Pengda
Myeong, Hwayeon
and
Oswald, Frederick L.
2024.
On putting the horse (raters and criteria) before the cart (variance components in ratings).
Industrial and Organizational Psychology,
Vol. 17,
Issue. 3,
p.
309.
Ock, Jisoo
Oswald, Frederick L
and
Tippins, Nancy T
2026.
If At First You Don’t Succeed: Effect of Job Applicant Retesting on the Practical Outcomes of Selection.
Journal of Business and Psychology,

There is understandable concern by LeBreton, Scherer, and James (Reference LeBreton, Scherer and James2014) that psychometric corrections in organizational research are nothing more than a form of statistical hydraulics. Statistical corrections for measurement error variance and range restriction might inappropriately ratchet observed effects upward into regions of practical significance and publication glory—at the expense of highly questionable results.
We share this concern. Of course, effect sizes based on high-quality measurement and representative sampling from populations of interest are preferred over psychometric corrections applied to effects subject to measurement error variance and range restriction. But there are at least two major reasons to consider the latter. First and foremost, we must contend with the reality of our research: Measures are never perfect reflections of their intended constructs, and a variety of range restriction effects exist with respect to the population to which inferences are made (e.g., effects of recruiting, self-selection, and other nonrandom sampling). Statistically modeling measurement error variance and range restriction attempts to take these realities into account when estimating latent relationships.
Second, assuming that the psychometric model and corrections are realistic (a strong assumption that we revisit shortly), the resulting magnitudes and patterns of latent relationships might be more accurate and interpretable than the uncorrected effects observed in the data. But you don't get something for nothing in making these psychometric corrections: Statistically, effects with lower reliability and greater range restriction increase the standard error of corrected effects (they increase by a factor of the multiplicative corrections themselves; Schmidt & Hunter, Reference Schmidt and Hunter2015, p. 142). As an analogy, psychometric corrections move the telescope toward the correct constellations (there is less systematic distortion); however, the cost of movement is a blurrier picture (larger standard errors). Thus, corrected effects will be higher (disattenuated) in the simplest case of classical-test-theory corrections for reliability and correction for direct range restriction, but the associated inaccuracy will be higher as well. Meta-analysis treats this phenomenon appropriately, namely that effects with lower reliability and greater range restriction will not contribute to meta-analysis results as much as effects with higher reliability and less range restriction will.
Thus, we appreciate the concern about statistical hydraulics artificially inflating the results of meta-analysis, although we equally appreciate (a) the unavoidable reality of measurement error variance and range restriction in our findings, (b) the potential benefits of modeling these artifacts in terms of a psychometric correction, and (c) the principled approach that meta-analysis takes in weighing stronger corrections proportionately less. Psychometric corrections have obviously been a major focus of the Schmidt and Hunter (Reference Schmidt and Hunter2015) approach to meta-analysis and validity generalization, and yet these psychometric artifacts are equally important for the effect sizes within any individual study. Perhaps this point is most obvious in studies involving structural equation modeling (SEM). The previous issues for meta-analysis generally apply here as well: SEM corrects structural relationships for measurement error variance, but when there are low factor loadings (i.e., low communality estimates of the indicators), this comes at the cost of larger standard errors associated with the structural estimates of causality between latent variables. Considering meta-analysis and SEM together as methods in which psychometric corrections are routinely implemented, the problem does not lie in the correction formulas themselves, as LeBreton et al. (Reference LeBreton, Scherer and James2014) correctly imply (being formulas, they have no ill intent); instead, the problem for meta-analysis—and more generally, any attempt to correct observed effects for statistical artifacts—lies in what psychometric corrections might be appropriate.
Psychometric corrections for measurement error variance, for instance, quite literally substitute the observed variance of measures by a proportion of the variance, expressed as their reliability coefficients. For instance, a reliability of .65 for X and .70 for Y means that correlations should be reestimated on the basis of 65% of the observed variance in X and 70% of the observed variance in Y (the covariance between X and Y does not change). This in turn increases (disattenuates) estimates of effect size. LeBreton et al. (Reference LeBreton, Scherer and James2014) considered such a correction to yield an estimated correlation that would be obtained under conditions of perfect measurement. From this perspective, they are right to consider this estimate as always hypothetical and, as such, not very useful. But, in alternative and equally legitimate perspective, the corrected effect is the estimate that results when the focus is on only the reliable variance of the constituent measures. From this latter perspective, psychometric corrections are useful and interpretable but tempered by the previously mentioned increase in error associated with corrected estimates.
Generalizability theory has long demonstrated that reliability can be modeled as a multifaceted phenomenon. Multiple sources of internal consistency, alternate forms, and test–retest reliability can be modeled simultaneously in what is called the generalized coefficient of equivalence and stability (Le, Schmidt, & Putka, Reference Le, Schmidt and Putka2009), and other sources such as interrater agreement and reliability can be modeled as well. Moreover, there are many forms of range restriction and enhancement, such as when a predictor of interest is part of a composite that was used for selection (Sackett, Lievens, Berry, & Landers, Reference Sackett, Lievens, Berry and Landers2007), when there are multiple hurdles in a selection process, or when self-selection effects, recruiting efforts, and selection decisions complexly affect validity (Newman & Lyon, Reference Newman and Lyon2009). In addition, reliability and range restriction can be statistically related when correlations between the two can be modeled in a meta-analysis (Raju, Anselmi, Goodman, & Thomas, Reference Raju, Anselmi, Goodman and Thomas1998). These artifacts also can be substantively related, such as by organizational climate (James, Demaree, Mulaik, & Ladd, Reference James, Demaree, Mulaik and Ladd1992) or by other moderator effects that may tend to go unmodeled in meta-analysis (Russell & Gilliland, Reference Russell and Gilliland1995).
The true challenge, then, comes in identifying the appropriate sources of variance viewed as error (Oswald & McCloy, Reference Oswald, McCloy and Murphy2003) and, thus, appropriate sources of measurement error variance and range restriction for applying psychometric corrections in light of the numerous practical constraints in conducting research (e.g., missing data, finite numbers of raters, access to primary data for meta-analyses). It is problematic that meta-analyses and primary studies often lack the data we would like to have in order to make appropriate corrections, but if we increase our sensibility and sensitivity to different sources of variance, perhaps our research and publishing practices will encourage such data, corrections, and associated reporting requirements.
In conclusion, we agree with LeBreton et al. (Reference LeBreton, Scherer and James2014) that is not the psychometric equations in meta-analysis that are problematic; it is the inappropriate modeling of reliability and range restriction. Modeling reliability and range restriction appropriately requires that meta-analysts revisit fundamental questions when considering and coding each study: How big is the inferential leap from measure to construct, between sample to population, and what are the measure–sample confounds that might exist (e.g., measurement nonequivalence caused by third variables)? How are questions about reliability and range restriction being asked and addressed (if not statistically corrected for)? How sensitive are meta-analysis results to changes in assumptions (for an argument to use sensitivity analysis more often, see Aytug, Rothstein, Zhou, & Kern, Reference Aytug, Rothstein, Zhou and Kern2011)?
Not making use of psychometric corrections may not be an entirely appropriate answer to these questions, because we know that statistical artifacts distort our understanding of latent relationships. At the same time, assumption-laden psychometric corrections might not be entirely appropriate either, because usually we know there are both substantive and statistical influences that cannot be modeled. Therefore, any result in a meta-analysis concerning the nature of observed or corrected effects should not be viewed as definitive. When the research interest is on estimating latent relationships in meta-analysis, or in individual studies, we, like most, prefer continued improvements in measurement, research design, and sampling, and—barring that—we prefer providing both uncorrected and corrected results, with an emphasis on the imperfect corrections rather than correct imperfections.