Hostname: page-component-7b7567c8c-mnklg Total loading time: 0 Render date: 2026-11-12T18:31:32.067Z Has data issue: false hasContentIssue false
  • English
  • Français

Nominal Scale Agreement Among Observers

Published online by Cambridge University Press:  01 January 2025

Hubert J. A. Schouten
Hubert J. A. Schouten*
Affiliation:
Institute of Biostatistics, Erasmus University Rotterdam
*
Requests for reprints should be sent to Hubert J. A. Schouten, Department of Medical Informaties and Statistics, University of Limburg, PO Box 616, 6200 MD Maastricht, THE NETHERLANDS.
Get accessRights & Permissions [Opens in a new window]

Abstract

Experiments are considered where each of a sample of subjects is assigned to one of C categories separately by each of a fixed or varying group of observers. Building on earlier publications, general procedures are proposed to analyze agreements and disagreements among observers. In the case of a varying group of observers, it is shown that it is not necessary to demand a constant number of observers per subject. In the case of a fixed group of observers, the problem of missing data is considered.

The procedures are illustrated within the context of two clinical diagnosis examples. In the first example it is investigated which categories are relatively hard to distinguish from one another; a new theorem is applied that shows a useful property of the statistic kappa. In the second example it is investigated if a subgroup of observers can be found with a significantly higher degree of interobserver agreement.

Keywords

agreementkappamissing data

Information

Type
Original Paper
Information
Psychometrika , Volume 51 , Issue 3 , September 1986, pp. 453 - 466
Copyright
Copyright © 1986 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The author gratefully acknowledges the valuable suggestions by W. Molenaar, R. van Strik, R. Popping and the referees.

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.CrossRefGoogle ScholarPubMed
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322–328.CrossRefGoogle Scholar
Efron, B. (1982). The jackknife, the bootstrp and other resampling plans, Philadelphia: S.I.A.M..CrossRefGoogle Scholar
Efron, B., Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37, 36–48.CrossRefGoogle Scholar
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378–382.CrossRefGoogle Scholar
Fleiss, J. L., Davies, M. (1982). Jackknifing functions of multinomial frequencies, with an application to a measure of concordance. American Journal of Epidemiology, 115, 841–845.CrossRefGoogle ScholarPubMed
James, I. R. (1983). Analysis of nonagreements among multiple raters. Biometrics, 39, 651–657.CrossRefGoogle Scholar
Kraemer, H. C. (1980). Extension of the kappa coefficient. Biometrics, 36, 207–216.CrossRefGoogle ScholarPubMed
Parr, W. C., Tolley, H. D. (1982). Jackknifing in categorical data analysis. The Australian Journal of Statistics, 24, 67–79.CrossRefGoogle Scholar
Schouten, H. J. A. (1980). Measuring pairwise agreement among many observers. Biometrical Journal, 22, 497–504.CrossRefGoogle Scholar
Schouten, H. J. A. (1982). Measuring pairwise agreement among many observers, II: Some improvements and additions. Biometrical Journal, 24, 431–435.CrossRefGoogle Scholar
Schouten, H. J. A. (1982). Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Statistica Neerlandica, 36, 45–61.CrossRefGoogle Scholar
Schouten, H. J. A. (1985). Statistical Measurement of Interobserver Agreement. Unpublished doctoral dissertation, Erasmus University Rotterdam.Google Scholar
Van den Berge, J. H., Schouten, H. J. A., Boomstra, S., van Drunen Littel, S., Braakman, R. (1979). Interobserver agreement in assessment of ocular signs in coma. Journal of Neurology, Neurosurgery and Psychiatry, 42, 1163–1168.CrossRefGoogle ScholarPubMed