Hostname: page-component-7b7567c8c-8tzhz Total loading time: 0 Render date: 2026-11-19T14:50:47.134Z Has data issue: false hasContentIssue false

Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters

Published online by Cambridge University Press:  01 January 2025

Kilem Li Gwet*
Affiliation:
Stataxis Consulting
*
Requests for reprints should be sent to Kilem Li Gwet, Sr. Statistical Consultant, STATAXIS Consulting, 20315 Marketree Place, Montgomery Village, MD 20886, USA. E-mail: gwet62@hotmail.com

Abstract

Most inter-rater reliability studies using nominal scales suggest the existence of two populations of inference: the population of subjects (collection of objects or persons to be rated) and that of raters. Consequently, the sampling variance of the inter-rater reliability coefficient can be seen as a result of the combined effect of the sampling of subjects and raters. However, all inter-rater reliability variance estimators proposed in the literature only account for the subject sampling variability, ignoring the extra sampling variance due to the sampling of raters, even though the latter may be the biggest of the variance components. Such variance estimators make statistical inference possible only to the subject universe. This paper proposes variance estimators that will make it possible to infer to both universes of subjects and raters. The consistency of these variance estimators is proved as well as their validity for confidence interval construction. These results are applicable only to fully crossed designs where each rater must rate each subject. A small Monte Carlo simulation study is presented to demonstrate the accuracy of large-sample approximations on reasonably small samples.

Information

Type
Theory and Methods
Copyright
Copyright © 2008 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable