Hostname: page-component-6898cfcb55-th4rj Total loading time: 0 Render date: 2026-11-11T13:14:28.963Z Has data issue: false hasContentIssue false
  • English
  • Français

Hierarchical Bayesian Modeling for Test Theory Without an Answer Key

Published online by Cambridge University Press:  01 January 2025

Zita Oravecz,
Royce Anders and
William H. Batchelder
Zita Oravecz*
Affiliation:
University of California, Irvin
Royce Anders
Affiliation:
University of California, Irvin
William H. Batchelder
Affiliation:
University of California, Irvin
*
Requests for reprints should be sent to Zita Oravecz, UCI, Department of Cognitive Sciences, 3213 Social & Behavioral Sciences Gateway Building, Irvine, CA 92697-5100, USA. E-mail: zoravecz@uci.edu
Get accessRights & Permissions [Opens in a new window]

Abstract

Cultural Consensus Theory (CCT) models have been applied extensively across research domains in the social and behavioral sciences in order to explore shared knowledge and beliefs. CCT models operate on response data, in which the answer key is latent. The current paper develops methods to enhance the application of these models by developing the appropriate specifications for hierarchical Bayesian inference. A primary contribution is the methodology for integrating the use of covariates into CCT models. More specifically, both person- and item-related parameters are introduced as random effects that can respectively account for patterns of inter-individual and inter-item variability.

Keywords

Cultural Consensus TheoryBayesian statisticshierarchical modelcovariate modeling

Information

Type
Original Paper
Information
Psychometrika , Volume 80 , Issue 2 , June 2015, pp. 341 - 364
Copyright
Copyright © 2013 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Baer, R.D., Weller, S.C., Alba Garcia, J.G., de Glazer, M., Trotter, R., & Pachter, L. et al. (2003). A cross-cultural approach to the study of the folk illness nervios. Culture, Medicine and Psychiatry, 27, 315–337.CrossRefGoogle Scholar
Batchelder, W.H., & Anders, R. (2012). Cultural consensus theory: comparing different concepts of cultural truth. Journal of Mathematical Psychology, 56, 316–332.CrossRefGoogle Scholar
Batchelder, W.H., Kumbasar, E., & Boyd, J. (1997). Consensus analysis of three-way social network data. The Journal of Mathematical Sociology, 22, 29–58.CrossRefGoogle Scholar
Batchelder, W.H., & Riefer, D.M. (1999). Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review, 6(1), 57–86.CrossRefGoogle ScholarPubMed
Batchelder, W.H., & Romney, A.K. (1988). Test theory without an answer key. Psychometrika, 53, 71–92.CrossRefGoogle Scholar
Batchelder, W.H., Strashny, A., & Romney, A. (2010). Cultural Consensus Theory: aggregating continuous responses in a finite interval.Google Scholar
Bimler, D. (2013). Two applications of the points-of-view model to subject variations in sorting data. Quality and Quantity, 47(2), 775–790.CrossRefGoogle Scholar
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord, F.M., & Novick, M.R. (Eds.), Statistical theories of mental test scores. Reading: Addison-Wesley.Google Scholar
Buhrmester, M., Kwang, T., & Gosling, S.D. (2011). Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data?. Perspectives on psychological science.Google Scholar
Comrey, A.L. (1962). The minimum residual method of factor analysis. Psychological Reports, 11, 15–18.CrossRefGoogle Scholar
Congdon, P. (2003). Applied Bayesian modelling. New York: Wiley.CrossRefGoogle Scholar
De Boeck, P. (2008). Random item IRT models. Psychometrika, 73, 533–559.CrossRefGoogle Scholar
De Boeck, P., & Wilson, M. (2004). Explanatory item response models: a generalized linear and nonlinear approach. New York: Springer.CrossRefGoogle Scholar
Dey, D.K., Gelfand, A.E., Swartz, T.B., & Vlachos, P.K. (1998). A simulation-intensive approach for checking hierarchical models. Test, 7(2), 325–346.CrossRefGoogle Scholar
Fischer, G., & Molenaar, I. (1995). Rasch models: foundations, recent developments, and applications. New York: Springer.CrossRefGoogle Scholar
Gelman, A., Carlin, J., Stern, H., & Rubin, D. (2004). Bayesian data analysis. New York: Chapman & Hall.Google Scholar
Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press.Google Scholar
Hopkins, A. (2011). Use of network centrality measures to explain individual levels of herbal remedy cultural competence among the Yucatec Maya in Tabi, Mexico. Field Methods, 23(3), 307–328.CrossRefGoogle ScholarPubMed
Hruschka, D.J., Kalim, N., Edmonds, J., & Sibley, L. (2008). When there is more than one answer key: cultural theories of postpartum hemorrhage in Matlab, Bangladesh. Field Methods, 20, 315–337.CrossRefGoogle Scholar
Iannucci, A., & Romney, A. (1990). Consensus in the judgment of personality traits among friends and acquaintances. Journal of Quantitative Anthropology, 4, 279–295.Google Scholar
Jackman, S. (2009). Bayesian analysis for the social sciences. New York: Wiley.CrossRefGoogle Scholar
Karabatsos, G., & Batchelder, W.H. (2003). Markov chain estimation methods for test theory without an answer key. Psychometrika, 68, 373–389.CrossRefGoogle Scholar
Klauer, K. (2010). Hierarchical multinomial processing tree models: a latent—trait approach. Psychometrika, 75, 70–98.CrossRefGoogle Scholar
Kruschke, J.K. (2011). Doing Bayesian data analysis: a tutorial with R and BUGS. New York: Academic Press.Google Scholar
Lee, M.D. (2011). How cognitive modeling can benefit from hierarchical Bayesian models. Journal of Mathematical Psychology, 55, 1–7.CrossRefGoogle Scholar
Lunn, D.J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS—a Bayesian modeling framework: concepts, structure, and extensibility. Statistics and Computing, 10, 325–337.CrossRefGoogle Scholar
Macmillan, N.A., & Creelman, C.D. (2005). Detection theory: a users guide (2nd ed.). Mahwah: Erlbaum.Google Scholar
Merkle, E.C., Smithson, M., & Verkuilen, J. (2011). Hierarchical models of simple mechanisms underlying confidence in decision makings. Journal of Mathematical Psychology, 55, 57–67.CrossRefGoogle Scholar
Miller, E. (2011). Maternal health and knowledge and infant health outcomes in the Ariaal people of northern Kenya. Social Science & Medicine, 73(8), 1266–1274.CrossRefGoogle ScholarPubMed
Mood, A.M., Graybill, F.A., & Boes, D.C. (1974). Introduction to the theory of statistics. New York: McGraw-Hill.Google Scholar
Morey, R.D. (2011). A Bayesian hierarchical model for the measurement of working memory capacity. Journal of Mathematical Psychology, 55(1), 8–24.CrossRefGoogle Scholar
Oravecz, Z., Vandekerckhove, J., & Batchelder, W. H. (in press). Bayesian cultural consensus theory. Field Methods.Google Scholar
Plummer, M. (2011). Rjags: Bayesian graphic models using MCMC (R package version 2.2.0-3). http://CRAN.R-project.org/package=rjags.Google Scholar
Rasch, G. (1960). Probabilistic models for some intelligent and attainment tests. Copenhagen: Danish Institute for Educational Research.Google Scholar
Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: applications and data analysis methods. Newbury Park: Sage.Google Scholar
Riefer, D.M., Knapp, B.R., Batchelder, W.H., Bamber, D., & Manifold, V. (2002). Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychological Assessment, 14(2), 184.CrossRefGoogle ScholarPubMed
Robert, C.P., & Casella, G. (2004). Monte Carlo statistical methods. New York: Springer.CrossRefGoogle Scholar
Romney, A.K., & Batchelder, W.H. (1999). Cultural consensus theory. In Wilson, R., & Keil, F. (Eds.), The MIT encyclopedia of the cognitive sciences (pp. 208–209). Cambridge: MIT Press.Google Scholar
Romney, A.K., Weller, S.C., & Batchelder, W.H. (1986). Culture as consensus: a theory of culture and informant accuracy. American Anthropologist, 88(2).CrossRefGoogle Scholar
Rouder, J., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604.CrossRefGoogle ScholarPubMed
Scheibehenne, B., Rieskamp, J., & Wagenmakers, E. J. (2013). Testing adaptive toolbox models: a Bayesian hierarchical approach. Psychological Review, 120, 39–64.CrossRefGoogle ScholarPubMed
Smith, J.B., & Batchelder, W.H. (2010). Beta-MPT: multinomial processing tree models for addressing individual differences. Journal of Mathematical Psychology, 54(1), 167–183.CrossRefGoogle Scholar
Snijders, T., & Bosker, R. (1999). Multilevel analysis: an introduction to basic and advanced multilevel modeling. Thousand Oaks: Sage.Google Scholar
Spearman, C.E. (1904). ‘General intelligence’ objectively determined and measured. The American Journal of Psychology, 15, 72–101.CrossRefGoogle Scholar
Spiegelhalter, D.J., Best, N.G., Carlin, B.P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society, Series B, 6, 583–640.CrossRefGoogle Scholar
Sprouse, J., Fukuda, S., Ono, H., & Kluender, R. (2011). Reverse island effects and the backward search for a licensor in multiple wh-questions. Syntax, 14(2), 179–203.CrossRefGoogle Scholar
Weller, S.C. (2007). Cultural consensus theory: applications and frequently asked questions. Field Methods, 19, 339–368.CrossRefGoogle Scholar
Weller, S.C., Baerm, R.D., Pachter, L.M., Trotter, R., Glazer, M., & de Alba Garcia, J.G. et al. (1999). Latino beliefs about diabetes. Diabetes Care, 22, 722–728.CrossRefGoogle ScholarPubMed
Weller, S.C., Pachter, L.M., Trotter, R.T., & Baer, R.D. (1993). Empacho in four Latino groups: a study of intra- and inter-cultural variation in beliefs. Medical Anthropology, 15(2), 109–136.CrossRefGoogle ScholarPubMed
Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.J. (2010). Bayesian parameter estimation in the expectancy valence model of the Iowa gambling task. Journal of Mathematical Psychology, 54, 14–27.CrossRefGoogle Scholar
Yoshino, R. (1989). An extension of the “test theory without answer key” by Batchelder and Romney and its application to an analysis of data on national consciousness. Proceedings of the Institute of Statistical Mathematics, 37, 171–188. (in Japanese).Google Scholar