Hostname: page-component-7b7567c8c-wxvwp Total loading time: 0 Render date: 2026-11-16T15:24:47.266Z Has data issue: false hasContentIssue false

A Cautionary Note on using Internal Cross Validation to Select the Number of Clusters

Published online by Cambridge University Press:  01 January 2025

Abba M. Krieger
Affiliation:
Department of Statistics, University of Pennsylvania
Paul E. Green*
Affiliation:
Department of Marketing, University of Pennsylvania
*
Requests for reprints should be sent to Paul Green, Marketing Department, The Wharton School, University of Pennsylvania, 1400 Steinberg Hall-Dietrich Hall, Philadelphia PA 19104-6371.

Abstract

A highly popular method for examining the stability of a data clustering is to split the data into two parts, cluster the observations in Part A, assign the objects in Part B to their nearest centroid in Part A, and then independently cluster the Part B objects. One then examines how close the two partitions are (say, by the Rand measure). Another proposal is to split the data into k parts, and see how their centroids cluster. By means of synthetic data analyses, we demonstrate that these approaches fail to identify the appropriate number of clusters, particularly as sample size becomes large and the variables exhibit higher correlations.

Information

Type
Original Paper
Copyright
Copyright © 1999 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable