Hostname: page-component-74656f9884-j7cmm Total loading time: 0 Render date: 2026-11-09T12:46:05.081Z Has data issue: false hasContentIssue false
  • English
  • Français

Cognitive heterogeneity and complex belief elicitation

Published online by Cambridge University Press:  14 March 2025

Ingrid Burfurd  and
Tom Wilkening Open the ORCID record for Tom Wilkening [Opens in a new window]
Ingrid Burfurd*
Affiliation:
Centre for Market Design, Department of Economics, The University of Melbourne, Melbourne, Australia
Tom Wilkening*
Affiliation:
Deparment of Economics, The University of Melbourne, Melbourne, Australia
*
ingrid.burfurd@gmail.com
tom.wilkening@unimelb.edu.au
Get access Rights & Permissions [Opens in a new window]

Abstract

The Stochastic Becker-DeGroot-Marschak (SBDM) mechanism is a theoretically elegant way of eliciting incentive-compatible beliefs under a variety of risk preferences. However, the mechanism is complex and there is concern that some participants may misunderstand its incentive properties. We use a two-part design to evaluate the relationship between participants’ probabilistic reasoning skills, task complexity, and belief elicitation. We first identify participants whose decision-making is consistent and inconsistent with probabilistic reasoning using a task in which non-Bayesian modes of decision-making lead to violations of stochastic dominance. We then elicit participants’ beliefs in both easy and hard decision problems. Relative to Introspection, there is less variation in belief errors between easy and hard problems in the SBDM mechanism. However, there is a greater difference in belief errors between consistent and inconsistent participants. These results suggest that while the SBDM mechanism encourages individuals to think more carefully about beliefs, it is more sensitive to heterogeneity in probabilistic reasoning. In a follow-up experiment, we also identify participants with high and low fluid intelligence with a Raven task, and high and low proclivities for cognitive effort using an extended Cognitive Reflection Test. Although performance on these tasks strongly predict errors in both the SBDM mechanism and Introspection, there is no significant interaction effect between the elicitation mechanism and either ability or effort. Our results suggest that mechanism complexity is an important consideration when using elicitation mechanisms, and that participants’ probabilistic reasoning is an important consideration when interpreting elicited beliefs.

Keywords

Belief elicitationProbabilistic reasoningCognitionComplexityObserver effectC81C91

Information

Type
Original Paper
Information
Experimental Economics , Volume 25 , Issue 2 , April 2022 , pp. 557 - 592
Copyright
Copyright © 2021 Economic Science Association

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s10683-021-09722-x) contains supplementary material, which is available to authorized users.

We gratefully acknowledge the financial support of the Australian Research Council (DE140101014) as well as the Faculty of Business and Economics at the University of Melbourne. Raw data, the programs used to analyze the data, and the programs used to run the experiment are available at https://osf.io/ex26f/.

References

Allen, F (1987). Notes-discovering personal probabilities when utility functions are unknown. Management Science, 33(4), 542–544. 10.1287/mnsc.33.4.542CrossRefGoogle Scholar
Andersen, S, Fountain, J, Harrison, GW, & Rutström, EE (2014). Estimating subjective probabilities. Journal of Risk and Uncertainty, 48(3), 207–229. 10.1007/s11166-014-9194-zCrossRefGoogle Scholar
Basso, D, Pesarin, F, & Salmaso, L (2009). Permutation tests for stochastic ordering and anova: theory and applicaitons with R, New York: LLC:Springer Science+Business Media.Google Scholar
Becker, GM, DeGroot, MH, & Marschak, J (1964). Measuring utility by a single-response sequential method. Behavioral Science, 9(3), 226–232. 10.1002/bs.3830090304CrossRefGoogle ScholarPubMed
Bors, DA, & Stokes, TL (1998). Raven’s advanced progressive matrices: Norms for first-year university students and the development of a short form. Educational and Psychological Measurement, 58(3), 382–395. 10.1177/0013164498058003002CrossRefGoogle Scholar
Brañas-Garza, P, Garcia-Munoz, T, & González, RH (2012). Cognitive effort in the beauty contest game. Journal of Economic Behavior & Organization, 83(2), 254–260. 10.1016/j.jebo.2012.05.018CrossRefGoogle Scholar
Brañas-Garza, P, Kujal, P, & Lenkei, B (2019). Cognitive reflection test: Whom, how, when. Journal of Behavioral and Experimental Economics, 82, 306–307. 10.1016/j.socec.2019.101455CrossRefGoogle Scholar
Burfurd, I, & Wilkening, T (2018). Experimental guidance for eliciting beliefs with the Stochastic Becker-DeGroot-Marschak mechanism. Journal of the Economic Science Association, 4(1), 15–28. 10.1007/s40881-018-0046-5CrossRefGoogle Scholar
Burks, SV, Carpenter, JP, Goette, L, & Rustichini, A (2009). Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proceedings of the National Academy of Sciences, 106(19), 7745–7750. 10.1073/pnas.0812360106CrossRefGoogle ScholarPubMed
Cacioppo, JT, Petty, RE, & Feng Kao, C (1984). The efficient assessment of need for cognition. Journal of Personality Assessment, 48(3), 306–307. 10.1207/s15327752jpa4803_13CrossRefGoogle ScholarPubMed
Carpenter, J, Graham, M, & Wolf, J (2013). Cognitive ability and strategic sophistication. Games and Economic Behavior, 80, 115–130. 10.1016/j.geb.2013.02.012CrossRefGoogle Scholar
Cattell, RB (1963). Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology, 54(1), 1–22. 10.1037/h0046743CrossRefGoogle Scholar
Charness, G, & Levin, D (2005). When optimal choices feel wrong: A laboratory study of Bayesian updating, complexity, and affect. The American Economic Review, 95(4), 1300–1309. 10.1257/0002828054825583CrossRefGoogle Scholar
Charness, G, Karni, E, & Levin, D (2007). Individual and group decision making under risk: An experimental study of Bayesian updating and violations of first-order stochastic dominance. Journal of Risk and Uncertainty, 35(2), 129–148. 10.1007/s11166-007-9020-yCrossRefGoogle Scholar
Charness, G, Rustichini, A, & van de Ven, J (2018). Self-confidence and strategic behavior. Experimental Economics, 21, 72–98. 10.1007/s10683-017-9526-3CrossRefGoogle Scholar
Chen, DL, Schonger, M, & Wickens, C (2016). oTree-an open-source platform for laboratory, online, and field experiments. Journal of Behavioral and Experimental Finance, 9, 88–97. 10.1016/j.jbef.2015.12.001CrossRefGoogle Scholar
Cokely, ET, Galesic, M, Schulz, E, Ghazal, S, & Garcia-Retamero, R (2012). Measuring risk literacy: The Berlin Numeracy Test. Judgment and Decision Making, 7(1), 25–47. 10.1017/S1930297500001819CrossRefGoogle Scholar
Dal Bó, E, Dal Bó, P, & Eyster, E (2017). The demand for bad policy when voters underappreciate equilibrium effects. The Review of Economic Studies, 85(2), 964–998.CrossRefGoogle Scholar
Danz, D., Vesterlund, L, & Wilson, A. (2020). Belief elicitation: Limiting truth telling with information on incentives. Tech. rep., CESifo Working Paper No. 8048.CrossRefGoogle Scholar
Ducharme, WM, & Donnell, ML (1973). Intrasubject comparison of four response modes for subjective probability assessment. Organizational Behavior and Human Performance, 10(1), 108–117. 10.1016/0030-5073(73)90007-XCrossRefGoogle Scholar
Fagerland, MW, & Sandvik, L (2009). The Wilcoxon-Mann-Whitney test under scrutiny. Statistics in Medicine, 28(10), 1487–1497. 10.1002/sim.3561CrossRefGoogle ScholarPubMed
Fischbacher, U (2007). z-tree: Zurich toolbox for ready-made economic experiments. Experimental Economics, 10(2), 171–178. 10.1007/s10683-006-9159-4CrossRefGoogle Scholar
Frederick, S (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42. 10.1257/089533005775196732CrossRefGoogle Scholar
Freeman, DJ, & Mayraz, G (2019). Why choice lists increase risk taking. Experimental Economics, 22, 131–154. 10.1007/s10683-018-9586-zCrossRefGoogle Scholar
Gigerenzer, G (1984). External validity of laboratory experiments: The frequency-validity relationship. The American Journal of Psychology, 97, 185–195. 10.2307/1422594CrossRefGoogle Scholar
Gigerenzer, G, & Hoffrage, U (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102, 684–704. 10.1037/0033-295X.102.4.684CrossRefGoogle Scholar
Good, P (2000). Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses, 2New York: Springer Science+Business Media 10.1007/978-1-4757-3235-1CrossRefGoogle Scholar
Greiner, B (2015). Subject pool recruitment procedures: Organizing experiments with ORSEE. Journal of the Economic Science Association, 1(1), 114–125. 10.1007/s40881-015-0004-4CrossRefGoogle Scholar
Grether, D. M. (1981). Financial incentive effects and individual decision-making. Tech. rep., California Institute of Technology, Working Papers 401.Google Scholar
Hao, L, & Houser, D (2012). Belief elicitation in the presence of naïve respondents: An experimental study. Journal of Risk and Uncertainty, 44(2), 161–180. 10.1007/s11166-011-9133-1CrossRefGoogle Scholar
Harrison, GW, Phillips, RD, & Andersen, T (2014). Subjective beliefs and statistical forecasts of financial risks: The chief risk officer project. Contemporary challenges in risk management, London: Palgrave Macmillan 163–202.Google Scholar
Harrison, GW, Martínez-Correa, J, & Swarthout, JT (2013). Inducing risk neutral preferences with binary lotteries: A reconsideration. Journal of Economic Behavior & Organization, 94, 145–159. 10.1016/j.jebo.2012.09.008CrossRefGoogle Scholar
Harrison, GW, Martínez-Correa, J, & Swarthout, JT (2014). Eliciting subjective probabilities with binary lotteries. Journal of Economic Behavior & Organization, 101, 430–448. 10.1016/j.jebo.2014.02.011CrossRefGoogle Scholar
Harrison, GW, Martínez-Correa, J, & Swarthout, JT (2015). Eliciting subjective probabily distributions with binary lotteries. Economics Letters, 101, 68–71. 10.1016/j.econlet.2014.12.027CrossRefGoogle Scholar
Harrison, GW, Martínez-Correa, J, & Swarthout, JT (2017). Scoring rules for subjective probability distributions. Journal of Economic Behavior & Organization, 134, 430–448. 10.1016/j.jebo.2016.12.001CrossRefGoogle Scholar
Heß, S (2017). Randomization inference with Stata: A guide and software. Stata Journal, 17(3), 630–651. 10.1177/1536867X1701700306CrossRefGoogle Scholar
Hollard, G, Massoni, S, & Vergnaud, JC (2016). In search of good probability assessors: An experimental comparison of elicitation rules for confidence judgments. Theory and Decision, 80(3), 363–387. 10.1007/s11238-015-9509-9CrossRefGoogle Scholar
Holt, CA (2006). Markets, games, and strategic behavior: Recipes for interactive learning, Boston: Pearson Addison Wesley.Google Scholar
Holt, CA, & Smith, AM (2009). An update on Bayesian updating. Journal of Economic Behavior & Organization, 69(2), 125–134. 10.1016/j.jebo.2007.08.013CrossRefGoogle Scholar
Holt, CA, & Smith, AM (2016). Belief elicitation with a synchronized lottery choice menu that is invariant to risk attitudes. American Economic Journal: Microeconomics, 8(1), 110–39.Google Scholar
Horn, JL, & Cattell, RB (1966). Refinement and test of the theory of fluid and crystallized general intelligences. Journal of Educational Psychology, 57(5), 253 10.1037/h0023816CrossRefGoogle ScholarPubMed
Hossain, T, & Okui, R (2013). The binarized scoring rule. The Review of Economic Studies, 80(3), 984–1001. 10.1093/restud/rdt006CrossRefGoogle Scholar
Huepe, D, Roca, M, Salas, N, Canales-Johnson, A, Rivera-Rei, ÁA, Zamorano, L et al., (2011). Fluid intelligence and psychosocial outcome: From logical problem solving to social adaptation. PLoS One, 6(9), e24858 10.1371/journal.pone.0024858CrossRefGoogle ScholarPubMed
Juanchich, M, Dewberry, C, Sirota, M, & Narendran, S (2016). Cognitive reflection predicts real-life decision outcomes, but not over and above personality and decision-making styles. Journal of Behavioral Decision Making, 29(1), 52–59. 10.1002/bdm.1875CrossRefGoogle Scholar
Karni, E (2009). A mechanism for eliciting probabilities. Econometrica, 77(2), 603–606. 10.3982/ECTA7833Google Scholar
Koh, B. H. (2017). Belief elicitation: QSR vs. BSR. In:. Essays in leadership and contests. University of Melbourne PhD Thesis.Google Scholar
Li, S (2017). Obviously strategy-proof mechanisms. American Economic Review, 107(11), 3257–3287. 10.1257/aer.20160425CrossRefGoogle Scholar
Li, Y, Baldassi, M, Johnson, EJ, & Weber, EU (2013). Complementary cognitive capabilities, economic decision making, and aging. Psychology and Aging, 28(3), 595 10.1037/a0034172CrossRefGoogle ScholarPubMed
Lilleholt, L. (2019). Cognitive ability and risk aversion: A systematic review and meta analysis. Judgment & Decision Making, 14(3), 234–279.CrossRefGoogle Scholar
Machina, MJ, & Schmeidler, D (1992). A more robust definition of subjective probability. Econometrica, 60(4), 745–780. 10.2307/2951565CrossRefGoogle Scholar
Montgomery, DC (2017). Design and Analysis of Experiments, 9Hoboken, NJ: John Wiley & Sons Inc..Google Scholar
Obrecht, NA, Chapman, GB, & Gelman, R (2009). An encounter frequency account of how experience affects likelihood estimation. Memory & Cognition, 37(5), 632–643. 10.3758/MC.37.5.632CrossRefGoogle ScholarPubMed
Oechssler, J, Roider, A, & Schmitz, PW (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72(1), 147–152. 10.1016/j.jebo.2009.04.018CrossRefGoogle Scholar
Offerman, T, Sonnemans, J, van de Kuilen, G, & Wakker, PP (2009). A truth serum for non-Bayesians: Correcting proper scoring rules for risk attitudes. The Review of Economic Studies, 76(4), 1461–1489. 10.1111/j.1467-937X.2009.00557.xCrossRefGoogle Scholar
Pauly, M, Brunner, E, & Konietschke, F (2015). Asymptotic permutation tests in general factorial designs. Journal of the Royal Statistical Society Series B (Statistical Methodology), 77(2), 461–473. 10.1111/rssb.12073CrossRefGoogle Scholar
Perasin, F (2001). Multivariate Permutation Tests with Applications in Biostatistics, Chichester: Wiley & Sons.Google Scholar
Primi, C, Morsanyi, K, Chiesi, F, Donati, MA, & Hamilton, J (2016). The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making, 29(5), 453–469. 10.1002/bdm.1883CrossRefGoogle Scholar
Salmaso, L. (2003). Synchronized permutation tests in 2k factorial designs. Communications in Statistics - Theory and Methods, 32(7), 1419–1437. https://doi.org/10.1081/STA-120021566.CrossRefGoogle Scholar
Savage, LJ (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336), 783–801. 10.1080/01621459.1971.10482346CrossRefGoogle Scholar
Schipolowski, S, Wilhelm, O, & Schroeders, U (2014). On the nature of crystallized intelligence: The relationship between verbal ability and factual knowledge. Intelligence, 46, 156–168. 10.1016/j.intell.2014.05.014CrossRefGoogle Scholar
Schlag, K. H., & Tremewan, J. (2020). Simple belief elicitation: An experimental evaluation. Tech. rep., MPRA Paper 98187. Germany: University Library of Munich.Google Scholar
Schlag, KH, Tremewan, J, & van der Weele, JJ (2013). A penny for your thoughts: A survey of methods for eliciting beliefs. Experimental Economics, 18(3), 1–34.Google Scholar
Schotter, A, & Trevino, I (2014). Belief elicitation in the laboratory. Annual Review of Economics, 6(1), 103–128. 10.1146/annurev-economics-080213-040927CrossRefGoogle Scholar
Smith, CA (1961). Consistency in statistical inference and decision. Journal of the Royal Statistical Society Series B (Methodological), 23(1), 1–37. 10.1111/j.2517-6161.1961.tb00388.xCrossRefGoogle Scholar
Stanovich, KE, & West, RF (2000). Individual differences in reasoning: Implications for the rationality debate?. Behavioral and Brain Sciences, 23(5), 645–665. 10.1017/S0140525X00003435CrossRefGoogle ScholarPubMed
Thomson, KS, & Oppenheimer, DM (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99–113. 10.1017/S1930297500007622CrossRefGoogle Scholar
Toplak, ME, West, RF, & Stanovich, KE (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275 10.3758/s13421-011-0104-1CrossRefGoogle ScholarPubMed
Trautmann, ST, & van de Kuilen, G (2015). Belief elicitation: A horse race among truth serums. The Economic Journal, 125(589), 2116–2135. 10.1111/ecoj.12160CrossRefGoogle Scholar
Winkler, RL, & Murphy, AH (1968). Good probability assessors. Journal of Applied Meteorology, 7(5), 751–758. 10.1175/1520-0450(1968)007<0751:PA>2.0.CO;22.0.CO;2>CrossRefGoogle Scholar
Supplementary material: File

Burfurd and Wilkening supplementary material

Supplemental Material
Download Burfurd and Wilkening supplementary material(File)
File 295.8 KB