Hostname: page-component-7b7567c8c-fvjsw Total loading time: 0 Render date: 2026-11-15T05:18:25.178Z Has data issue: false hasContentIssue false
  • English
  • Français

Why Are Experts Correlated? Decomposing Correlations Between Judges

Published online by Cambridge University Press:  01 January 2025

Stephen B. Broomell and
David V. Budescu
Stephen B. Broomell*
Affiliation:
Department of Psychology, University of Illinois
David V. Budescu
Affiliation:
Department of Psychology, University of Illinois and Department of Psychology, Fordham University
*
Requests for reprints should be sent to Stephen B. Broomell, Department of Psychology, University of Illinois, 603 E. Daniel Street, Champaign, IL 61820, USA. E-mail: sbroome2@uiuc.edu
Get accessRights & Permissions [Opens in a new window]

Abstract

We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.

Keywords

information aggregationcorrelationdependenceexpert advice

Information

Type
Theory and Methods
Information
Psychometrika , Volume 74 , Issue 3 , September 2009, pp. 531 - 553
Copyright
Copyright © 2009 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

This work was supported by the National Science Foundation under Awards SES 02-41434 and 03-45925. The first author was supported by a grant from the National Institutes of Health under Ruth L. Kirschstein National Research Service Award PHS 2 T32 MH014257 (“Quantitative Methods for Behavioral Research”) to the University of Illinois at Urbana-Champaign.

References

Ariely, D., Au, W.T., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S., Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147Google ScholarPubMed
Ashton, R.H. (1986). Combining the judgments of experts: How many and which ones?. Organizational Behavior and Human Decision Processes, 38, 405–414CrossRefGoogle Scholar
Ashton, A.H., Ashton, R.H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508CrossRefGoogle Scholar
Azen, R., Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148CrossRefGoogle ScholarPubMed
Budescu, D.V. (2006). Confidence in aggregation of opinions from multiple sources. In Fiedler, K., Juslin, P. (Eds.), Information sampling and adaptive cognition (pp. 327–354). Cambridge: Cambridge University PressGoogle Scholar
Budescu, D.V., Yu, H.T. (2007). Aggregation of opinions based on correlated cues and advisors. Journal of Behavioral Decision Making, 20, 153–177CrossRefGoogle Scholar
Clemen, R.T., Winkler, R.L. (1985). Limits for precision and value of information from dependent sources. Operations Research, 33, 427–442CrossRefGoogle Scholar
Clemen, R.T., Winkler, R.L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46CrossRefGoogle Scholar
Dawes, R.M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582CrossRefGoogle Scholar
Einhorn, H.J. (1974). Expert Judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571CrossRefGoogle Scholar
Hammond, K.R., Stewart, T.R. (2001). The essential Brunswik: beginnings, explications, application, London: Oxford University PressCrossRefGoogle Scholar
Hammond, K.R., Wilkins, M.M., Todd, F.J. (1966). A research paradigm for the study of interpersonal learning. Psychological Bulletin, 65, 221–232CrossRefGoogle Scholar
Hogarth, R.M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46CrossRefGoogle Scholar
Hursch, C.J., Hammond, K.R., Hursch, J.L. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60CrossRefGoogle ScholarPubMed
Johnson, T.R., Budescu, D.V., Wallsten, T.S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140CrossRefGoogle Scholar
Miller, S. (2008). Supporting joint human-computer judgment under uncertainty. Unpublished Dissertation at the University of Illinois at Urbana-Champaign.CrossRefGoogle Scholar
Morris, P.A. (1986). Comment on Genest and Zideck’s “Combining probability distributions: A critique and annotated bibliography”. Statistical Science, 1, 141–144CrossRefGoogle Scholar
Shanteau, J. (2001). What does it mean when experts disagree?. In Salas, E., Klein, G. (Eds.), Linking expertise and naturalistic decision making, Earlbaum: MahwaGoogle Scholar
Schmidt, F.L., Johnson, R.H., Gugel, J.F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2, 345–357CrossRefGoogle Scholar
Wallsten, T.S., Budescu, D.V., Erev, I., Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–2683.0.CO;2-M>CrossRefGoogle Scholar
Wallsten, T.S., Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18CrossRefGoogle Scholar
Weiss, D.J., Shanteau, J. (2003). The vice of consensus and the virtue of consistency. In Shanteau, J., Johnson, P., Smith, C. (Eds.), Psychological explorations of competent decision making, Cambridge: Cambridge University PressGoogle Scholar
Weiss, D.J., Shanteau, J. (2003). Empirical assessment of expertise. Human Factors, 45, 104–116CrossRefGoogle Scholar
Winkler, R.L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685CrossRefGoogle Scholar
Winkler, R.L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488CrossRefGoogle Scholar
Winkler, R.L., Poses, R.M. (1993). Evaluating and combining physician’s probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543CrossRefGoogle Scholar
Yaniv, I., Choshen-Hillel, S., Milyavsky, M. (2009). Spurious consensus and opinion revision: Why might people be more confident in their less accurate judgments. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 558–563Google ScholarPubMed