To save content items to your account,
please confirm that you agree to abide by our usage policies.
If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account.
Find out more about saving content to .
To save content items to your Kindle, first ensure no-reply@cambridge.org
is added to your Approved Personal Document E-mail List under your Personal Document Settings
on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part
of your Kindle email address below.
Find out more about saving to your Kindle.
Note you can select to save to either the @free.kindle.com or @kindle.com variations.
‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi.
‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Over 45 years ago, William Revelle proposed a reliability measure based on the worst split-half of a test or scale, commonly known as Revelle’s beta, to assess the general factor saturation. However, to this day, there is no reliable method for computing this measure, as existing approaches are either computationally infeasible or insufficiently accurate in identifying the worst split-half. This difficulty arises because the number of candidate splits increases exponentially with the number of items. In this article, we show that computing Revelle’s beta is conceptually equivalent to divisive (“top-down”) hierarchical clustering. This insight allows us to reduce the number of candidate splits to a quadratic problem, making the computation feasible. We specify theoretical conditions under which this approach is guaranteed to recover the worst split-half. To validate the efficiency of our approach, we conduct simulation studies and analyze real-world data. Code implementations accompanying this work are available online, together with Supplementary Material.
Early prediction of depressive and anxiety disorders is challenging due to substantial heterogeneity in risk pathways. Conventional machine-learning models trained on aggregated populations may obscure subgroup-specific mechanisms and limit interpretability for prevention. We evaluated whether a hybrid unsupervised–supervised framework can identify meaningful subgroups and yield more interpretable risk prediction.
Methods
We analyzed cohort data of 15,897 Japanese adults who completed baseline (August–September 2020) and 6-month follow-up (February–March 2021) surveys and did not screen positive for depressive and anxiety disorders at baseline (K6 score < 13). Using 169 baseline demographic, psychosocial, lifestyle, and behavioral variables, we performed hierarchical clustering to derive data-driven subgroups. Within each cluster, we trained Random Forest models to predict incident screened depressive and anxiety disorders at follow-up (K6 ≥ 13) and interpreted predictors using SHapley Additive exPlanations (SHAP).
Results
The overall 6-month incidence was 6.23%. A five-cluster solution revealed two high-risk subgroups: an older-adult profile with poor quality of life (12.9%) and a working-parent profile characterized by work–family overload (29.8%). Compared with a global model trained on the full sample, the cluster-then-predict framework showed broadly similar overall performance but performed better in the highest-risk subgroup and revealed more differentiated predictor profiles. Loneliness, health-related quality of life, happiness, and personality traits predominated in clusters with moderate adversity, whereas lifestyle disruption (sleep, diet, and irregular routines) characterized the high-risk late-life subgroup and alcohol dependence and work–family burden characterized the high-risk working-parent subgroup.
Conclusions
Addressing risk-factor heterogeneity before prediction may enable more interpretable, context-tailored prevention strategies.
In this chapter we present some of the most popular approaches to clustering, and discuss general techniques for evaluating and validating the quality of a data partition.
This paper presents a nonspatial operationalization of the Krumhansl (1978, 1982) distance-density model of similarity. This model assumes that the similarity between two objects i and j is a function of both the interpoint distance between i and j and the density of other stimulus points in the regions surrounding i and j. We review this conceptual model and associated empirical evidence for such a specification. A nonspatial, tree-fitting methodology is described which is sufficiently flexible to fit a number of competing hypotheses of similarity formation. A sequential, unconstrained minimization algorithm is technically presented together with various program options. Three applications are provided which demonstrate the flexibility of the methodology. Finally, extensions to spatial models, three-way analyses, and hybrid models are discussed.
Statisticians typically estimate the parameters of latent class and latent profile models using the Expectation-Maximization algorithm. This paper proposes an alternative two-stage approach to model fitting. The first stage uses the modified k-means and hierarchical clustering algorithms to identify the latent classes that best satisfy the conditional independence assumption underlying the latent variable model. The second stage then uses mixture modeling treating the class membership as known. The proposed approach is theoretically justifiable, directly checks the conditional independence assumption, and converges much faster than the full likelihood approach when analyzing high-dimensional data. This paper also develops a new classification rule based on latent variable models. The proposed classification procedure reduces the dimensionality of measured data and explicitly recognizes the heterogeneous nature of the complex disease, which makes it perfect for analyzing high-throughput genomic data. Simulation studies and real data analysis demonstrate the advantages of the proposed method.
A theorem is proved stating that the set of all “minimax links”, defined as links minimizing, over paths, the maximum length of links in any path connecting a pair of objects comprising nodes in an undirected weighted graph, comprise the union of all minimum spanning trees of that graph. This theorem is related to methods of fitting network models to dissimilarity data, particularly a method called “Pathfinder” due to Schvaneveldt and his colleagues, as well as to single linkage clustering, and results concerning the relationship between minimum spanning trees and single linkage hierarchical trees.
The psychometric and classification literatures have illustrated the fact that a wide class of discrete or network models (e.g., hierarchical or ultrametric trees) for the analysis of ordinal proximity data are plagued by potential degenerate solutions if estimated using traditional nonmetric procedures (i.e., procedures which optimize a STRESS-based criteria of fit and whose solutions are invariant under a monotone transformation of the input data). This paper proposes a new parametric, maximum likelihood based procedure for estimating ultrametric trees for the analysis of conditional rank order proximity data. We present the technical aspects of the model and the estimation algorithm. Some preliminary Monte Carlo results are discussed. A consumer psychology application is provided examining the similarity of fifteen types of snack/breakfast items. Finally, some directions for future research are provided.
This paper presents a new procedure called TREEFAM for estimating ultrametric tree structures from proximity data confounded by differential stimulus familiarity. The objective of the proposed TREEFAM procedure is to quantitatively “filter out” the effects of stimulus unfamiliarity in the estimation of an ultrametric tree. A conditional, alternating maximum likelihood procedure is formulated to simultaneously estimate an ultrametric tree, under the unobserved condition of complete stimulus familiarity, and subject-specific parameters capturing the adjustments due to differential unfamiliarity. We demonstrate the performance of the TREEFAM procedure under a variety of alternative conditions via a modest Monte Carlo experimental study. An empirical application provides evidence that the TREEFAM outperforms traditional models that ignore the effects of unfamiliarity in terms of superior tree recovery and overall goodness-of-fit.
For the clustering problem with general (not necessarily symmetric) relational constraints, different sets of feasible clusterings, also called clustering types, determined by the same relation, can be defined. In this paper some clustering types are discussed and adaptations of the hierarchical clustering method compatible with these clustering types are proposed.
With reference to a questionnaire aimed at assessing the performance of Italian nursing homes on the basis of the health conditions of their patients, we investigate two relevant issues: dimensionality of the latent structure and discriminating power of the items composing the questionnaire. The approach is based on a multidimensional item response theory model, which assumes a two-parameter logistic parameterization for the response probabilities. This model represents the health status of a patient by latent variables having a discrete distribution and, therefore, it may be seen as a constrained version of the latent class model. On the basis of the adopted model, we implement a hierarchical clustering algorithm aimed at assessing the actual number of dimensions measured by the questionnaire. These dimensions correspond to disjoint groups of items. Once the number of dimensions is selected, we also study the discriminating power of every item, so that it is possible to select the subset of these items which is able to provide an amount of information close to that of the full set. We illustrate the proposed approach on the basis of the data collected on 1,051 elderly people hosted in a sample of Italian nursing homes.
Ordinal network representations are graph-theoretic representations of proximity data. They seek to provide parsimonious representations of the ordinal (nonmetric) information in observed proximity data by means of the minimum-path-length distance of a connected and weighted graph. In contrast to traditional tree-based graph-theoretic approaches, ordinal network representation is not limited to but includes the representation by trees. Asymmetry in the proximity data and violations of zero-minimality are allowed for. The paper explores fundamental representation and uniqueness results and discusses a method of constructing ordinal network representations. A simple strategy for handling the problem of errors in the data is described and illustrated.
This paper presents a new methodology concerned with the estimation of ultrametric trees calibrated on subjects' pairwise proximity judgments of stimuli, capturing subject heterogeneity using a finite mixture formulation. We assume that a number of unobserved classes of subjects exist, each having a different ultrametric tree structure underlying the pairwise proximity judgments. A new likelihood based estimation methodology is presented for those finite mixtures of ultrametric trees, that accommodates ultrametric as well as other external constraints. Various assumptions on the correlation of the error of the dissimilarities are accommodated. The performance of the method to recover known ultrametric tree structures is investigated on synthetic data. An empirical application to published data from Schiffman, Reynolds, and Young (1981) is provided. The ability to deal with external constraints on the tree-topology is demonstrated, and a comparison with an alternative clustering based method is made.
Kerala, a humid tropical southern state, is the third largest producer of Tamarind (Tamarindus indica L.) in India. This tree spice is an important livelihood option for the rural society, especially in Palakkad district which is the lead producer of tamarind in the state with a few accepted primitive cultivars, viz. ‘valanpuli’, ‘madhurapuli’ and ‘thenpuli’. A survey was conducted to identify the fruit variability and document the sweet tamarind types of Palakkad. This study revealed the variability and weight of 30 fruits was seen to be the most indicative variable of tamarind in Kerala. The pod length varied from 5.28 to 23.41 cm and fruit weight from 4.83 to 43.40 g. Hierarchical clustering on principal component analysis resulted in six clusters. The clusters I, IV and V represented collections with fruit weight more than 20 g. Box plot diagrams revealed the high real pulp value in sample KTJ 162 (9.57) and high fruit length in KTJ 44 (21.68 cm). Among 113 collections, 18 samples were collected as perceived sweet types based on local enquiry. In the scatter plot between total soluble solids and ratio between total soluble solids and total titrable acidity, two samples were spotted as sweet types with acidity below 8%. Further molecular characterization and systematic crop improvement programmes are required for conserving the eroding gene pool of tamarind in Kerala and to develop sweet types for commercial production.
Chapter 1 makes use of two empirical approaches. Its first part uses property law from 136 jurisdictions in an unsupervised machine-learning method (hierarchical clustering) that divide these jurisdictions into 10 legal families. Unlike the traditional wisdom that highlights the difference between common law and civil law, this chapter finds that, in terms of property doctrines, a trichotomy better describes the legal systems: one big group is jurisdictions affected by French property law; another big group is composed of jurisdictions that follow or resemble German property law; and the final group contains common-law jurisdictions, Nordic countries, and a number of socialist jurisdictions. The second part of Chapter 1 re-combines 156 jurisdictions into 149 countries, and computes the correlation coefficients among each country pair, to show dyadic similarities in property law.
Under unsupervised learning, clustering or cluster analysis is first studied. Clustering methods are grouped into non-hierarchical (including K-means clustering) and hierarchical clustering. Self-organizing maps can be used as a clustering method or as a discrete non-linear principal component analysis method. Autoencoders are neural network models that can be used for non-linear principal component analysis. Non-linear canonical correlation analysis can also be performed using neural network models.
This chapter begins with a discussion of classes of problem complexity. It then focuses on hierarchical pyramids that have been accepted for the last three decades as computational models of the visual system of primates, including humans. Anatomical and behavioral results that support this claim are reviewed. First, the speed-accuracy tradeoff results in visual perception are reviewed. Second, mental size transformation is described and it is shown that they can be explained by a pyramid model. With these results related to visual perception in hand, it is shown how the pyramid model can be applied to the TSP. Pyramid models are characterized by self-similarity across space and scale. This self-similarity is a form of invariance, also known as symmetry. A pyramid model produces near-optimal TSP tours by using a global-to-local (coarse-to-fine) sequence of computational steps. Good performance of pyramid models on TSP comes with a price when the geometry of a TSP problem is perturbed by inserting obstacles. Finally, subjects are tested in a real-life application of the TSP, where they collect tennis balls on a tennis court. The fact that humans are capable of producing near-optimal TSP tours in almost “no time” suggests that they solve the task of minimizing a tour's length without ever measuring the length of the tour. Instead, they infer the concept of direction in the search space, and use the direction, rather than the distance, when they navigate within the problem spaces.
High-dimensional datasets on cultural characters contribute to uncovering insights about factors that influence cultural evolution. Because cultural variation in part reflects descent processes with a hierarchical structure – including the descent of populations and vertical transmission of cultural traits – methods designed for hierarchically structured data have potential to find applications in the analysis of cultural variation. We adapt a network-based hierarchical clustering method for use in analysing cultural variation. Given a set of entities, the method constructs a similarity network, hierarchically depicting community structure among them. We illustrate the approach using four datasets: pronunciation variation in the US mid-Atlantic region, folklore variation in worldwide cultures, phonemic variation across worldwide languages and temporal variation in first names in the US. In these examples, the method provides insights into processes that affect cultural variation, uncovering geographic and other influences on observed patterns and cultural characters that make important contributions to them.
Recent improvements in robotic arms have increased their interest in many areas such as the industry and biomedical sectors. Path planning is an essential part of the robotic arm, since most automated factories seek to move things from one place to another with obstacles providing the shortest route. This paper presents a novel optimal path planning algorithm based on the 3D cubic Bézier curve with three shape parameters and its geometric properties and hierarchical clustering. The proposed method utilizes a feature vector which is obtained from curvature, torsion, and path length of candidate curves. A hierarchical clustering is applied to determine curve pairs. Then, a multi-objective function is used to determine the best curve pair, which gives the best curve for the robotic arm. Besides forming the optimal 3D cubic Bézier path, the optimal ruled and developable path surfaces are obtained. In addition to presenting theoretical results, this work also demonstrates the proposed method on several Kinova Gen3 robotic arm cases.
Astronomers in the past clearly recognized the irregular nature of the galaxy distribution in the nearby Universe. Both Herschel (ca. 1800) and Shapley (ca. 1932) detected and described these effects. They both named specific regions in the sky that are crowded with galaxies and other regions that are significantly deficient in galaxies. However, the scientific views of Hubble published in 1936 overshadowed these early results, and based on his beliefs (with no significant substantiating evidence) Hubble asserted that the Universe (both locally and at great distances) is isotropic and homogeneous. Hubble’s 1936 analysis used counts of faint galaxies to show that the Universe – with galaxies as designated “markers” in space – extends in depth to the greatest limits he was able to obtain at Mt. Wilson Observatory. In the 1930s, Holmberg and others set the foundation for hierarchical structure formation to explain the origin of groups and clusters of galaxies.
Clustering is the process of examining a collection of “points,” and grouping the points into “clusters” according to some distance measure. The goal is that points in the same cluster have a small distance from one another, while points in different clusters are at a large distance from one another. A suggestion of what clusters might look like was seen in Fig. 1.1. However, there the intent was that there were three clusters around three different road intersections, but two of the clusters blended into one another because they were not sufficiently separated. Our goal in this chapter is to offer methods for discovering clusters in data. We are particularly interested in situations where the data is very large, and/or where the space either is high-dimensional, or the space is not Euclidean at all. We shall therefore discuss several algorithms that assume the data does not fit in main memory. However, we begin with the basics: the two general approaches to clustering and the methods for dealing with clusters in a non-Euclidean space.