1. Introduction
Recent work in historical psychology and behavioral science has begun to argue that computational analysis of historical corpora can support stronger claims about past populations than earlier proxy-based approaches permitted. One prominent version of this ambition appears in recent discussions of “historical large language models” (HLLMs), proposed as generative models trained on historical corpora that might supply populations of simulated historical participants (Varnum et al. Reference Varnum, Baumard, Atari and Gray2024). The proposal is explicitly framed as a move beyond earlier corpus-based approaches, which inferred past values, stereotypes, norms, and related psychological tendencies from textual patterns such as word frequencies and co-occurrences, and therefore relied on comparatively indirect proxies. The newer promise is stronger: HLLMs would allow researchers to query simulated participants with modern psychological instruments, behavioral measures, surveys, scales, or experimental tasks, thereby generating “somewhat more direct” quantitative data about traits, attitudes, and behavioral tendencies in past populations. The discussion of HLLMs frames the broader ambition at issue here. The central worked case, however, is Contextualized Construct Representation (CCR), a recently developed pipeline for scoring classical Chinese texts in relation to contemporary psychological constructs. CCR is not itself an HLLM: it does not simulate historical respondents answering questionnaires. It is nevertheless useful because it implements a concrete and inspectable version of the commensurating move on which the HLLM ambition depends: contemporary psychological constructs are rendered into historically available textual proxies, mapped onto surviving corpora, and converted into scalar scores. Both belong to a wider historical-psychology program that treats surviving cultural artifacts as informative traces of past minds and seeks to reconstruct psychological variation across time from texts, institutions, material remains, and related historical evidence (Muthukrishna et al. Reference Muthukrishna, Henrich and Slingerland2021; Atari and Henrich Reference Atari and Henrich2023; Martins and Baumard Reference Martins and Baumard2022; Baumard et al. Reference Baumard, Safra, Martins and Chevallier2024). Related work in psychology has likewise argued that language can function as an informative record of thought, value, and social meaning, encouraging the use of large textual corpora as evidence about minds and cultures beyond the laboratory (Jackson et al. Reference Jackson, Watts, List, Puryear, Drabble and Lindquist2022; Jackson and Atari Reference Jackson and Atari2025). None of this is inherently misguided. Historical inquiry often proceeds through mediated traces, and efforts to recover patterns of cognition, value, and sociality from such materials can be both intelligible and illuminating. The issue is what evidential burden must be met before a text-analytic pipeline counts as measurement rather than construct-guided comparison.
The CCR pipeline was developed for classical Chinese corpora by Chen et al. (Reference Chen, Li, Li and Atari2024). It begins with contemporary psychometric constructs such as traditionalism, norm strength, and collectivism. It then uses a multilingual quote-recommendation model, supplemented by manual filtering, to convert questionnaire items into classical Chinese; embeds both the transformed construct items and historical texts in a shared semantic space; and assigns texts scalar scores on the basis of their semantic similarity to those items (Chen et al. Reference Chen, Li, Li and Atari2024).Footnote 1 The pipeline is further refined through indirect supervised contrastive learning based on pseudo-ground truth derived from title similarity, and it is presented as validated through benchmarking against external historical data, including coded attitudes of Song officials toward reform (Chen et al. Reference Chen, Li, Li and Atari2024). The result is a methodologically elaborate workflow in which contemporary psychological constructs are rendered historically legible and used to score passages within a learned representational space. In concrete terms, CCR estimates how strongly particular historical texts express selected contemporary constructs. For example, a questionnaire item intended to capture traditionalism is translated into, or retrieved as, classical Chinese expressions judged to approximate the construct; historical passages are then scored according to their proximity to those construct expressions in the learned semantic space. In this setting, success can take several forms: improved internal coherence, stronger benchmark performance on annotated materials, and downstream results in which the resulting scores recover annotated distinctions, support questionnaire-item classification, or align with historically coded variables in theoretically expected ways (Chen et al. Reference Chen, Li, Li and Atari2024, Reference Chen, Atari, Slingerland, Hong, Fu, Wang, Schulz, Bol and Henrich2025). I refer to these forms of performance collectively as predictive success. By this I do not mean prediction in the narrow sense of forecasting future events. I mean successful performance on tasks where the scoring regime is asked to recover, classify, or align with information not identical to the score itself, but made legible within the same constructed evidential space. Such successes are genuine, and often nontrivial. The question is whether they validate the underlying construct-to-trace relation or instead show that the regime performs well within the representational space it has helped to construct.
This paper argues that, in historically mediated measurement regimes, predictive success may justify comparative use within an induced representational space without thereby validating the construct-to-trace relation required for measurement. More generally, it contends that validation is not exhausted by reliability, calibration, benchmark success, or downstream utility. In historically and socially mediated inquiry, validation must also require that the representational mapping itself be exposed to evidence not generated solely by successful performance within the regime that the mapping helps to organize. The broader issue is what I call endogenous validation: a structure in which a construct is operationalized through a representational mapping; that mapping organizes the field of admissible signals; success is assessed within that field; and that success is then taken to support the adequacy of the mapping itself. The problem is not iteration as such, but the transfer of warrant from performance within a regime to the representational assumptions that determine what success can count as evidence for. The central methodological danger, on this view, is the conflation of successful construct commensuration with validated measurement. Commensuration, in the relevant sense, is not a neutral technical preprocessing step. It is the production of comparability through a shared metric imposed across heterogeneous materials, one that transforms qualities into quantities, reduces disparate information to a common scale, and thereby alters what becomes salient, comparable, or even visible (Espeland and Stevens Reference Espeland and Stevens1998). Here, the point is not merely that historical texts are scored, but that they are first made scoreable under a contemporary construct vocabulary. This achievement may be real and indispensable: it can stabilize comparison, construct workable proxies, and make disciplined inference possible under conditions of heterogeneity. But these are not yet measurement achievements in the stronger sense. A regime may successfully organize historical materials under a common construct vocabulary while leaving unresolved whether the resulting score tracks the target property in the target population strongly enough to support the substantive claim. Success in benchmark or downstream analyses may show that a model discriminates texts in ways that align with imported construct categories, recovers labels produced by present-day annotators, or outperforms alternatives on internally defined tasks (cf. Chen et al. Reference Chen, Li, Li and Atari2024; Varnum et al. Reference Varnum, Baumard, Atari and Gray2024). It does not, by itself, show that the resulting scores measure the psychology of past populations rather than project a contemporary construct vocabulary onto historically surviving textual forms. In historically mediated psychological modeling, downstream success may justify comparative reliance within a regime without yet establishing the adequacy of the proxy relation that structures it.
The distinction matters beyond historical psychology. In economics and the quantitative social sciences, tractable proxies often provide valuable information about harder-to-observe targets without becoming identical with what they stand in for. Satellite luminosity, for example, has been used fruitfully as a proxy for economic activity where conventional data are poor or missing, but its usefulness does not erase the distinction between proxy success and validated measurement of the target phenomenon (Chen and Nordhaus Reference Chen and Nordhaus2011; Henderson et al. Reference Henderson, Storeygard and Weil2012). Historical-psychological text analysis sharpens this general problem because the target is remote, benchmarks are sparse, and substantial representational work is required to render historical materials comparable under present constructs. CCR is therefore best understood as a commensurating technology: it produces a space in which heterogeneous texts can be compared under a common psychological metric, while leaving the construction of that space methodologically load-bearing. Historically mediated psychological modeling brings familiar concerns about modeling, classification, proxy use, and evidential uptake into a setting where they are integral rather than peripheral. The regime’s apparent directness depends on representational decisions whose methodological role may become less visible as downstream performance improves. This is why the argument is adjacent to several familiar lines of criticism without being reducible to any one of them. Work on proxy measurement shows that the central issue is not indirectness alone, but the inferential route by which a proxy is taken to bear on its target (Wilson and Boudinot Reference Wilson and Garrett Boudinot2022). Work on construct validation and measurement validity likewise shows that operational success can leave the relation between measure and target underdescribed or overburdened (Alexandrova and Haybron Reference Alexandrova and Haybron2016; Zhao Reference Zhao2023; Higgins et al. Reference Higgins, Gillett, Kaplan and Ross2024). The present claim is more specific: in historically mediated inquiry, success within a constructed evidential regime can be taken to support the adequacy of the proxy relation through which that regime became available.
The contribution, then, is a diagnosis of endogenous validation in regimes where commensuration, scoring, and validation are tightly coupled. The paper’s claim is narrower than blanket skepticism about modeling and stronger than the modest point that proxies are imperfect. It does not argue that historical corpora cannot support psychological inference, that imported constructs are always illegitimate, or that benchmark success is evidentially worthless. Its claim is instead that the risk of overextending measurement language is especially acute in socially and historically structured domains, where surviving corpora are selective, psychological salience is filtered through genre and archival survival, and independent benchmarks are often sparse. Under those conditions, methodological success can bear more epistemic weight than it can secure. Even proponents of HLLMs acknowledge that surviving historical texts are not representative samples of past populations and that benchmarks for ancient settings may be unavailable or scarce (Varnum et al. Reference Varnum, Baumard, Atari and Gray2024). These are not merely practical inconveniences. They are conditions under which the burden of showing that a score tracks a target property becomes heavier, not lighter. The deeper risk, then, is not simply that the framework remains undertested. It is that success within the regime can begin to stand in for unresolved validity claims. Once a pipeline yields interpretable scalar outputs, correlates with theoretically congenial variables, and outperforms comparators on benchmark tasks, what has been established may be the stability and usefulness of a commensurating metric, not yet the stronger claim that past psychology has been measured in the sense required by the surrounding substantive argument. The relevant question is therefore whether any identifiable pattern of divergence could rationally place that measurement relation under evidential pressure. A regime may be bounded, theory-laden, and operationally useful without thereby supplying such a condition. Where no such condition is available, downstream success may justify comparative reliance while still failing to warrant the stronger claim that the target attribute has been measured. The distinctive problem is thus not background dependence as such, but the misallocation of evidential force across levels of the regime.
The argument proceeds in four steps. First, it distinguishes construct commensuration from historical measurement and clarifies why the two are liable to be conflated in current historical-LLM discourse. Second, it reconstructs the CCR pipeline as a worked case of construct-guided historical-text scoring. Third, it shows how benchmark success and external correlations can shift epistemic burden in ways that allow unresolved validity questions to recede from view. Fourth, it argues that this burden-shifting dynamic is especially consequential in socially structured domains, where corpus composition, annotation practices, and the admissibility conditions of interpretation are historically filtered in advance. The result is not an argument against computational historical psychology as such, but a limit claim about evidential scope in historically mediated measurement.
2. A minimal analytic framework
The framework that follows is diagnostic. It separates three things that can otherwise be conflated in discussions of computational historical psychology: successful scoring within a constructed space, validation of the proxy relation, and validation of the historical claim. The account of validity assumed here is realist in a limited but important sense. I take measurement validity to require more than reliability, benchmark performance, or coherence among scores: a valid measure must stand in an appropriate relation to the attribute it purports to measure. This position is close to the instrument-based account defended by Borsboom et al. (Reference Borsboom, Mellenbergh and van Heerden2004), for whom a test is valid for measuring an attribute only if the attribute exists and variations in that attribute causally produce variation in measurement outcomes. It is also compatible with Hood’s (Reference Hood2009, Reference Hood2012) argument that this conception of validity entails realist commitments in psychological testing, since validity cannot be reduced to successful use or interpretive convenience. However, this position does not require rejecting broader construct-validational or argument-based accounts, which treat validity as an evidential judgment about the interpretations and uses of scores rather than as a property of the instrument alone (Cronbach and Meehl Reference Cronbach and Meehl1955; Messick Reference Messick1995; Kane Reference Kane2001). The point is narrower: in historically mediated text-scoring regimes, evidence for the interpretation and use of scores may itself be generated within the commensurating structure that makes those scores available. The burden, then, is not only to show that the scores are useful, coherent, or predictively successful. It is to show that the measure–target relation on which such interpretations depend has been adequately tested rather than merely presupposed. That concern is continuous with recent work on construct validation, measurement models, and proxy failure, which has emphasized that operationally useful measures may still fail to secure the relation between measure and target required for stronger validity claims (Alexandrova and Haybron Reference Alexandrova and Haybron2016; Jacobs and Wallach Reference Jacobs and Wallach2021; Wilson and Boudinot Reference Wilson and Garrett Boudinot2022; Higgins et al. Reference Higgins, Gillett, Kaplan and Ross2024). The additional claim made here is that historically mediated text-scoring regimes sharpen this problem because the relevant attribute, textual trace, and admissible evidential field are jointly organized through commensurating operations (Espeland and Stevens Reference Espeland and Stevens1998). Validation must therefore show not only that a scoring regime performs well, but that the construct-to-trace relation on which its outputs depend has been exposed to credible conditions of failure.
This point does not depend on settling the realism–empiricism debate in favor of realism. A constructive empiricist might regard empirical adequacy as sufficient for some scientific purposes while remaining agnostic about the deeper metaphysical status of the attributes being modeled. The present concern arises, however, when such success is used to license a stronger representational role for the output: when scores are treated not merely as empirically useful summaries but as measurements of psychological attributes, evidence for population-level traits, or explanatory variables in accounts of historical change. In such cases, the issue is not only whether the regime performs well on observable tasks, but also whether the interpretation attached to that performance is warranted. The problem therefore remains even for a more empiricist reader: if a scoring regime’s empirical success is generated within an induced construct space, that success may justify constrained predictive or comparative use without justifying the further claim that the scores measure the target attribute. The dispute is thus not over whether performance matters, but over what kind of epistemic warrant it supplies.
To keep that question from being absorbed into general claims about performance, the framework below makes explicit the elements across which warrant is often transferred too quickly. The notation is purely schematic. Its purpose is to distinguish elements that ordinary discussion often collapses: the space of admissible possibilities, the rules by which departures are classified, and the procedure by which commitments are revised. In historically mediated measurement regimes, a further element must also be made explicit: the commensurating operations by which heterogeneous historical materials are rendered comparable under a shared metric. The framework preserves the broader sense of predictive success introduced above, while distinguishing successful performance within a regime from the adequacy of the construct-to-trace mapping on which its evidential significance depends. Without that distinction, downstream success can too easily be treated as though it directly bore on the mapping’s validity.
To see where that transfer of warrant occurs, it is useful first to describe the ordinary structure of a performance-oriented regime before asking what happens when the space in which success occurs has itself been produced through commensuration. Let such a regime be characterized by (i) a field of admissible counterfactual positions, F—for example, a hypothesis space, model class, intervention space, or outcome partition; (ii) an evaluative mapping, E, that assigns epistemic standing—confirmation, anomaly, tolerable deviation, noise—relative to F; and (iii) an updating rule, U, that revises commitments under exposure. F is thus the regime’s partition of admissible counterfactual divergence: each member specifies a way the system could depart from expectation while remaining intelligible within the regime’s evaluative architecture. A regime exhibits internal openness when, holding F, it updates responsively under U and registers error relative to E. Standard accounts of prediction, model checking, and confirmation primarily theorize this sense of openness. Bayesian conditionalization redistributes credence over hypotheses already admitted (Jeffreys Reference Jeffreys1961; Howson and Urbach Reference Howson and Urbach2006). Falsificationist testing evaluates outcomes against antecedently specified refuters (Popper Reference Popper1959). Posterior predictive checking assesses fit relative to selected likelihoods and model classes (Gelman and Shalizi Reference Gelman and Shalizi2013). Iterative optimization, more generally, adjusts parameters or weights within an established representational frame (Earman Reference Earman1992). In each case, exposure can alter commitments within the space without thereby placing the articulation of the space itself at issue.
In historically mediated text-measurement pipelines, however, F is not simply given. It is induced by a commensuration scheme that renders historical materials legible under a contemporary construct vocabulary. To capture that structure, let such a regime also include (iv) a construct specification, C (for example, an imported psychometric instrument or item set); (v) a commensuration operator, T, that renders C historically comparable by mapping it into linguistically and materially available proxy forms; and (vi) a metricization rule, M, that maps historical texts into scalar outputs within the induced construct space. In these regimes, the operative field is best understood as F(C,T,M): the space of differences that can register as evidence, anomaly, noise, or revisional pressure given the construct anchor, the transformation that makes it historically legible, and the metric that maps texts into comparable scores. This is precisely why commensuration can be indispensable without being identical with measurement. It can stabilize comparison by converting heterogeneous materials into a common metric that constructs proxies, simplifies difference, and reorganizes what counts as salient or comparable, while still leaving unresolved whether the resulting scores track the target property in the target population rather than merely a disciplined scheme of commensuration (Espeland and Stevens Reference Espeland and Stevens1998; Messick Reference Messick1995; Kane Reference Kane2001). A regime exhibits external openness, by contrast, only when exposure can trigger revision of F itself—when the space of admissible alternatives, and thus the partition that determines what counts as a live challenge, is revisable. In historically mediated measurement, that requires that exposure be able to bear not only on parameter values or model fit, but on C, T, or M themselves: the imported construct family, the admissibility conditions under which historical expressions count as realizations of that construct, or the scoring rule by which texts are mapped into the induced space must themselves be revisable in response to recalcitrant patterns. Call such revision structural rearticulation. Structural rearticulation is not mere recalibration, parameter adjustment, or model replacement within an accepted class. It is a change in what the regime can treat as a meaningful alternative. Positions previously excluded from the space of intelligible deviation become candidates for uptake, or positions once treated as live cease to count as such. At that point, revision no longer concerns commitments within a fixed space; it concerns the construct-to-trace mapping by which the space was constituted.
This yields a sharper criterion. The relevant notion of external openness here requires some endogenous condition under which patterns of divergence cannot be assimilated as internal variation and instead place the structure of F(C,T,M) under evidential pressure. Absent such a condition, recursive updating may remain fully responsive while leaving the articulation of the operative space insulated from revision. The relevant question is therefore not whether a regime can learn, recalibrate, or improve, but whether its conditions of improvement include any principled route by which exposure can bear on the boundary conditions of admissible alternatives. Standard predictive frameworks often direct evidential assessment toward movement and comparison within an articulated field of possibilities rather than toward revision of the field as such (Earman Reference Earman1992; Gelman and Shalizi Reference Gelman and Shalizi2013). That limitation becomes sharper under recursive updating, where improved performance can rationally stabilize reliance on the operative space without independently testing its boundaries. This is the point at which recursive entrenchment becomes possible: once commensuration is operationally successful, the ordering it imposes on the material can begin to function less as a provisional scaffold than as a settled evidential background. For that reason, the epistemic risk extends beyond underdetermination at the level of measurement to the naturalization of a representational scheme whose very success can obscure the assumptions and exclusions on which it depends. Such a pipeline can be internally open—highly responsive by its own evaluative standards—while lacking a principled route by which exposure could bear on the construct-to-trace mapping that made its evidential categories available.
2.1 Recursive entrenchment
The distinction introduced above allows the central problem of historical-text measurement to be stated more precisely. In construct-guided historical-text scoring regimes, the central problem is not simply that updating proceeds within an articulated space of admissible alternatives. It is that later success can feed authority back into the earlier commensurating decisions that made the target measurable. Where construct specification, transformation, metricization, and validation are tightly coupled, recursive improvement may entrench the operative construct frame without independently testing whether the score tracks the target property in the historical population. Let C be an imported construct specification, T a commensuration operator that renders C historically legible, M a metricization rule that maps transformed construct items and historical texts into scalar outputs, and V a validation regime that evaluates those outputs through benchmark tasks, annotation recovery, external correlations, or downstream prediction. Let the operative admissibility field be F(C,T,M): the structured field within which textual differences become legible as construct-relevant evidence, anomaly, noise, or revision pressure. Further, let performance under V improve through repeated refinement of M or adjacent stages of the pipeline—for example, through stronger benchmark performance, more stable discrimination, or better recovery of selected labels. Suppose further that agents accept the following norms:
(N1) improved performance under V is evidence of methodological adequacy;
(N2) adequacy judgments license projection, reliance, and substantive inference relative to the task at hand.
The crucial point is that nothing in N1 or N2 requires the adequacy being licensed to be measurement adequacy rather than task-relative usefulness, yet in practice the latter can be rhetorically upgraded into the former. Under these conditions, sustained improvement can rationally increase confidence not only in the scalar outputs produced by M, but also in continued reliance on C and T as the operative construct-to-trace mapping. The advantage of separating C, T, M, and V is that it becomes possible to distinguish questions that are otherwise easily conflated: whether the scoring regime performs well under the chosen validation conditions; whether the outputs are stable and discriminating within the induced metric space; and whether anything in those operations places the construct anchor or commensurating transformation itself under evidential pressure.
If improved benchmark or predictive performance is taken as evidence of adequacy, and adequacy licenses projection and reliance, then success under V will support continued use of the pipeline for the task at hand. Yet neither M nor V necessarily evaluates whether C is historically apt or whether T preserves the target relation across linguistic, generic, and temporal distance. The metricization rule generates scores through criteria that substantially depend on the induced space, and the validation regime may assess those scores through benchmark or downstream criteria without thereby placing the construct anchor or commensurating transformation under independent pressure. Under those conditions, recursive improvement can stabilize reliance on the regime without supplying independent warrant that the construct-to-trace mapping is itself adequate. Recursive entrenchment arises when better performance rationally reinforces reliance on that mapping without independently establishing that the score tracks the target property in the historical population.
This structure also clarifies why CCR is the appropriate worked case for a paper framed by HLLM proposals. Although CCR does not simulate historical respondents, it already contains the relevant inferential architecture: contemporary constructs are rendered into historically available proxy expressions, embedded with historical texts, and transformed into scalar outputs within an induced representational space. If those scores recover annotated contrasts, correlate with independently coded variables, or outperform alternative configurations, that success is evidentially meaningful: it shows that the scoring procedure is discriminating and useful within the induced construct space. But unless failure on such tasks could rationally motivate revision of the construct anchor or of the transformation by which it was made historically legible, the validation regime has not yet tested the relation on which the stronger measurement claim depends.
2.2 Endogenous validation and the limits of benchmark success
The abstract structure just identified can now be specified for the Chen et al. (Reference Chen, Li, Li and Atari2024, Reference Chen, Atari, Slingerland, Hong, Fu, Wang, Schulz, Bol and Henrich2025) pipeline by asking a narrower question: What, in this case, would have to be true for benchmark success to count as evidence for measurement rather than merely for task performance? The pipeline can be mapped onto the analytic structure introduced above. Contemporary questionnaires supply the construct anchor, C. Quote retrieval plus manual filtering supply the commensuration operator, T, by defining which historical expressions count as construct-relevant realizations. Embedding and cosine-similarity aggregation supply the metricization rule, M, by turning transformed items and historical texts into scalar loading scores. Benchmark tasks, annotated datasets, downstream regressions, and historically interpreted patterns then function as the validation regime, V. Benchmarking can be evidentially significant, but it is not automatically external in the sense required to validate a measurement claim. A benchmark is external only if success or failure on it can place the commensurating transformation itself under evidential pressure—if failure would rationally motivate revision of C or T, rather than merely tuning M or selecting a better-performing configuration within the same induced space.
That condition is difficult to satisfy here for two related reasons. First, the score is generated through construct-guided transformations before benchmarking begins. Contemporary instruments are selected as anchors, rendered into classical Chinese through retrieval and manual filtering, embedded alongside historical texts, and converted into construct loadings through cosine similarity. Chen et al. (Reference Chen, Atari, Slingerland, Hong, Fu, Wang, Schulz, Bol and Henrich2025) describe the method in just these terms: contemporary questionnaire items are transformed into face-valid classical Chinese sentences and used to quantify the salience of psychological constructs in historical texts; the method then computes semantic similarity between the transformed items and texts to deliver a score for each Chinese text. When benchmark tasks evaluate performance downstream of this induced construct space, success is evidence of stability and discriminative power within that space. It is not automatically evidence that the induced space measures the intended historical attribute rather than operationalizing a contemporary construct vocabulary over historically surviving forms (Espeland and Stevens Reference Espeland and Stevens1998). Nor does such downstream success by itself establish the stronger measure-to-attribute relation required for a measurement claim (Borsboom et al. Reference Borsboom, Mellenbergh and van Heerden2004; Jacobs and Wallach Reference Jacobs and Wallach2021). Second, benchmark sparsity and representativeness constraints make downstream success liable to bear more epistemic weight than it can justify: the public proposal motivating such pipelines explicitly emphasizes simulated participants and more direct measurement while acknowledging that surviving texts are often unrepresentative and that suitable benchmarks may be sparse or unavailable (Varnum et al. Reference Varnum, Baumard, Atari and Gray2024). Under such conditions, validation may show that the pipeline detects stable and discriminating textual patterns, without yet establishing that the induced metric is historically adequate as a measure of the target attribute.
Genre adjustment does not resolve this problem, because it occurs only after the psychological scores have already been generated. In the DC3-Dynasty mixed-effects model—a book-level model that estimates the relationship while treating dynasty as a fixed effect and genre as a random effect—the reported kinship–psychology relationship is estimated after accounting for variation across genres and systematic differences between dynasties. But that is a downstream adjustment to associations among already-produced scores, not a demonstration that the score-generating procedure is interpretively equivalent across corpora organized by different genre systems, quotation practices, and lexical conventions. This limitation matters because genre is not external to the scoring process. Chen et al. rely, for the spatial analysis, on local gazetteers, which they describe as a relatively standardized literary genre, while the temporal analysis uses the multi-genre DC3 corpus. Their genre-decomposed analyses also suggest that the signal is not uniform across textual kinds: the strongest effects appear in history, philosophy, and literature, while weaker effects are concentrated in Buddhist and Daoist texts. Genre therefore helps shape the textual regularities through which quotation retrieval, filtering, and embedding-based similarity become possible; register and genre are major organizers of linguistic variation rather than incidental textual background (Biber and Conrad Reference Biber and Conrad2019). A future application to another historical language could therefore test whether the pipeline is portable to another archive, but not by itself whether the resulting scores are commensurate in the stronger sense required for measurement—namely, whether the same scalar output tracks the same underlying construct relation across historically different traditions of textual production, preservation, and expression.
The significance of this point is not confined to historical psychology. The same structural problem arises in other research programs whenever downstream success within a constructed evidential regime is allowed to stand in for independent pressure on the proxy relation that organizes that regime. In psychometrics and psychiatric measurement, reliability or practical usefulness may fall short of construct validity (Cronbach and Meehl Reference Cronbach and Meehl1955; Borsboom et al. Reference Borsboom, Mellenbergh and van Heerden2004). In algorithmic fairness, strong benchmark performance may coexist with a mis-specified or normatively inadequate measurement model (Jacobs and Wallach Reference Jacobs and Wallach2021). In bibliometrics and computational social science, operationally useful indicators may remain imperfect stand-ins for the phenomena they are taken to measure (Hicks et al. Reference Hicks, Wouters, Waltman, de Rijcke and Rafols2015; Olteanu et al. Reference Olteanu, Castillo, Diaz and Kiciman2019). The point, then, is not that historical cases are uniquely resistant to validation, but that endogeneity becomes a general risk wherever success inside an induced evidential space is treated as sufficient warrant for the proxy relation that made that space intelligible at the outset.
2.3 Internal openness and frame contestability
The case just examined clarifies the paper’s positive epistemological claim. The issue is not merely that predictive inquiry proceeds within a bounded or theory-saturated frame. It is that recursively structured validation can produce a specific warrant-transfer error: success that properly bears on technical adequacy within an induced regime is allowed to bear as well on the adequacy of the construct-to-trace relation that makes the regime intelligible as a measurement procedure. The consequence is not skepticism about validation, but a stricter account of what validation must establish in endogenously structured inquiry. In such settings, a satisfactory account of validation must distinguish at least three questions. First, there is technical adequacy: whether a system is reliable, well calibrated, or successful across benchmark and task-performance settings relative to a given task environment. Second, there is representational adequacy: whether the operationalization or proxy relation tracks the target property it is taken to measure. Third, there is boundary adequacy: whether the field of admissible signals, contrasts, and alternatives through which the target has been rendered legible has itself been exposed to meaningful critical pressure. Standard validation practices are often strongest at the first level while remaining substantially weaker at the second and third. The relevant distinction is therefore between internal openness and frame contestability.
Internal openness concerns whether a regime can register error, improve fit, recover labels, or adjust under exposure while holding fixed the proxy relation that structures what counts as relevant evidence. Frame contestability concerns whether that relation is itself vulnerable to evidence: whether some pattern of divergence could rationally motivate revision of the construct anchor, the commensurating transformation, or the admissibility conditions under which a historical text counts as realizing the target construct at all. This point bears directly on the worry that contemporary psychological categories may not project cleanly onto past cultures, but the difficulty is not exhausted by asking whether a modern category applies. In these techniques, category and interpretation are mutually entangled. A contemporary psychological construct does not enter the pipeline only after historical meaning has been independently recovered; it helps determine which historical expressions are recognized as relevant in the first place. The commensurating transformation therefore does interpretive work on historical language while assigning that language a place within a present construct space. If the historical semantic field was organized by different contrasts, saliences, or inferential roles, that difference may not appear as a failure of the method. It may instead be absorbed as lexical variation, weak signal, genre effect, or residual noise. The risk is that modern categories may structure historical visibility itself, making evidence most readily available when it can be routed through the categories that organize the inquiry. This is why the problem is not merely one of translation or lexical approximation. Historical semantics and conceptual history have long emphasized that words acquire significance within historically specific fields of use, contrast, and social practice, rather than carrying stable meanings across contexts as detachable semantic units (Skinner Reference Skinner1969; Koselleck Reference Koselleck2004; Hacking Reference Hacking and Pickering1992). The relevant pressure would therefore have to come from comparisons that could show the construct-to-trace mapping itself to be unstable or historically inappropriate: materially different construct anchors, corpus-specific scoring procedures, rival historically grounded operationalizations, or triangulation against independently structured sources might yield divergent score profiles that could not be absorbed as mere tuning variation within the existing regime (cf. Hammerton and Munafò Reference Hammerton and Munafò2021). Without such pressure, a regime may be internally open, and indeed highly responsive, while remaining insulated from contestation at the level of the frame.
This is the sense in which the problem is not solved by ordinary predictive responsiveness. Standard accounts of prediction, model checking, and confirmation primarily theorize internal openness: responsiveness to evidence within an already constituted field of hypotheses, contrasts, model classes, or scoring possibilities. Bayesian conditionalization redistributes credence across hypotheses already admitted to a field of live possibilities (Savage Reference Savage1954; Jeffreys Reference Jeffreys1961; Howson and Urbach Reference Howson and Urbach2006). Posterior predictive checking, model comparison, and sensitivity analysis enhance responsiveness to data by identifying misfit, comparing members of a model class, or testing robustness under specified variations in assumptions (Kass and Raftery Reference Kass and Raftery1995; Gelman and Shalizi Reference Gelman and Shalizi2013). More generally, formal accounts of confirmation and updating show how evidence can alter commitments within an established representational frame, without by themselves supplying criteria for when that frame’s constitutive proxy relation should itself be revised (Earman Reference Earman1992). Iterative optimization procedures make the same limitation visible in computational form: they may revise parameters, weights, or decision rules while leaving the operative representation of the target intact. These procedures matter because they make inquiry corrigible within a regime. But they do not by themselves determine when evidential pressure should be taken to challenge the regime’s organizing representational commitments rather than merely recalibrate performance within them. The resulting ambiguity is precisely what makes warrant-transfer errors possible: the limitation is easy to miss because predictive success is often read as evidence of increasingly secure access to the target attribute, rather than as performance within a structured evidential regime.
This problem is adjacent to, but not reducible to, several familiar concerns in philosophy of science and statistics. It is not simply the problem of unconceived alternatives, where scientists fail to entertain possible competitors to an accepted theory (Stanford Reference Stanford2006). Nor is it merely the problem of confirmation under restricted model classes, where evidential support may be overstated because comparison occurs only within a bounded set of candidates (Berk Reference Berk1966; Dawid et al. Reference Dawid, Hartmann and Sprenger2015). It is also distinct from convergence under misspecification, where estimators or posteriors stabilize around the best available approximation despite model error (Kleijn and van der Vaart Reference Kleijn and van der Vaart2006). Those literatures identify genuine limits on what success can establish under bounded alternatives. The present claim is narrower and more structural: success achieved within a recursively constructed evidential regime can be allowed to bear on the adequacy of the proxy relation through which the regime was constituted as a site of measurement. The positive upshot is therefore a revised diagnostic standard: predictive success, benchmark robustness, and downstream usefulness can support a measurement claim only where the representational bridge on which those achievements depend is itself exposed to identifiable conditions of failure or revision. Absent such conditions, success may warrant comparative use, exploratory inference, and constrained practical reliance, but it does not by itself warrant the stronger conclusion that the target attribute has been measured. What is required, in other words, is not more internal openness alone, but credible routes by which evidence could contest the frame that organizes admissible signals, operational contrasts, and proxy relations from the outset.
3. Commensuration, evidential authority, and the politics of validation
Historically mediated predictive measurement regimes are increasingly important for scientific inference about cultural, psychological, and institutional change, especially in work that treats historical texts, artifacts, and databases as evidence for reconstructing past minds and explaining present-day psychological variation (Muthukrishna et al. Reference Muthukrishna, Henrich and Slingerland2021; Varnum et al. Reference Varnum, Baumard, Atari and Gray2024). In such regimes, targets that are not directly observable—cultural values, psychological dispositions, institutional orientations, or long-run patterns of social change—are rendered scoreable by translating heterogeneous traces into a portable metric space. Because such scores are portable, they can travel beyond the immediate research setting into broader comparative and policy-adjacent claims. The claim here is not that the specific CCR or HLLM systems discussed above have already been adopted as policy instruments. It is that once historically mediated interpretations are converted into scores for constructs such as collectivism, norm strength, traditionalism, or trust, they become available for comparative ranking, causal explanation, civilizational narrative, and policy-adjacent claims about cultural populations. This concern is not merely hypothetical: Chen et al.’s (Reference Chen, Li, Li and Atari2024) CCR pipeline explicitly includes norm tightness and norm looseness among the questionnaires converted into classical Chinese and used for classification and psychological-measure tasks; cultural tightness has been mapped across 31 Chinese provinces and related to innovation, urbanization, and happiness (Chua et al. Reference Chua, Huang and Jin2019); and tightness–looseness has been extended across both diachronic and comparative settings, including American cultural change from 1800 to 2000 and global variation among non-industrial societies (Jackson et al. Reference Jackson, Gelfand, De and Fox2019, Reference Jackson, Gelfand and Ember2020). Related historical measures, especially ancestral kinship tightness, have been used in economics to explain contemporary variation in cooperation, trust, punishment, moral values, and institutions (Enke Reference Enke2019). The broader-impact concern is therefore one of epistemic portability: measurement rhetoric allows construct scores to travel more easily than the interpretive qualifications on which they depend.
Classic work on commensuration emphasizes that turning qualities into quantities is not a neutral preprocessing step. It produces comparability by imposing a common metric across heterogeneous materials, thereby simplifying local difference and reorganizing what becomes salient, comparable, and actionable (Espeland and Stevens Reference Espeland and Stevens1998). This reorganization is epistemically indispensable, but it is also a mode of power: commensuration can stabilize hierarchies by privileging what is legible to the metric and treating residual heterogeneity as noise, nuisance, or idiosyncrasy. This is a shift in epistemic authority, not merely in method. Portability and standardization move judgment away from historically situated interpretation and toward the actors, institutions, and infrastructures that control the metric and its validation. In global governance and public administration, indicators have therefore been analyzed not merely as descriptive summaries but as technologies that alter standard-setting, decision-making, and contestation by making particular forms of comparison authoritative (Davis et al. Reference Davis, Kingsbury and Engle Merry2012). Recent work extends this point through the sociology of quantification and evidence cultures. Quantification scholarship stresses that commensurating devices not only enable comparison; they help construct the evidence cultures in which numbers acquire legitimacy and practical force (Mennicken and Espeland Reference Mennicken and Nelson Espeland2019). Evidence-culture analysis likewise argues that evidence carries political as well as epistemic authority. On this view, evidence-based regimes may persist despite critique because the routines that certify validity, together with the institutions organized around them, continually renew the authority of the very standards under challenge (Straßheim Reference Straßheim2024). The upshot is a structural risk: when success within a commensurated regime is repeatedly taken as warrant for the regime’s own framing assumptions, predictive performance can function as an engine of naturalization, making the conditions of measurability harder to see and harder to challenge.
Historically mediated regimes intensify these risks because they typically involve severe constraints on independent ground truth, selective and politically structured data sources, and extensive representational work to make traces scoreable. Digitization is not neutral preservation but a process of politically consequential selection, cataloguing, classification, metadata assignment, and access, all of which shape what pasts become computationally available (Zaagsma Reference Zaagsma2023). Related scholarship on archival absence and silence shows that digitized collections can further entrench what is already overrepresented while rendering omissions harder to detect, so that what becomes legible to computation is conditioned not only by survival but by prior and ongoing regimes of description and exclusion (Ortolja-Baird and Nyhan Reference Ortolja-Baird and Nyhan2022). In historically oriented projects, the archive is not merely the input. It is part of the regime’s boundary conditions: an upstream structuring of what can count as evidence. Where independent validation is scarce and the archive is selective, a commensurating scheme can acquire evidential authority it has not independently earned. This concern is related to the familiar Duhem–Quine point that evidence bears on a conjunction of commitments rather than on an isolated hypothesis: apparent confirmation or disconfirmation may attach to the target claim, the measurement procedure, background theory, or auxiliary assumptions (Quine Reference Quine1951; Duhem Reference Duhem1954). The problem here is more specific. In historically mediated scoring regimes, the relevant auxiliaries include the commensurating operations that define what can count as evidence at all: construct selection, archive composition, proxy formation, embedding architecture, scoring rules, and benchmark design. In this setting, evidential misallocation runs in a distinctive direction. Evidence for the usefulness of an induced scoring scheme may be treated as evidence that the scheme measures the historical attribute itself, rather than as evidence about performance within a particular hypothesis-and-auxiliary package. Evidence-culture analysis underscores why this matters: evidential standards carry institutional and political authority, and regimes can persist partly because validation routines help renew the authority of the very standards under dispute. If validation practices never place the proxy relation under pressure, success will tend to naturalize the frame and marginalize alternative specifications of the target.
Three practical implications follow. First, claims should be scaled to evidence: operationally useful scoring should not be redescribed as validated measurement of historically remote attributes. Second, validation should be stratified: predictive performance and benchmark alignment can support technical adequacy, but representational adequacy requires evidence about the construct-to-trace mapping (Cronbach and Meehl Reference Cronbach and Meehl1955; Borsboom et al. Reference Borsboom, Mellenbergh and van Heerden2004; Jacobs and Wallach Reference Jacobs and Wallach2021). Third, validation designs should exert pressure on the frame itself. Benchmark comparison, downstream genre controls, and subgroup decomposition may improve confidence in performance within a regime, but they do not by themselves establish that the operative construct anchor and commensurating transformation preserve the same construct interpretation across sources, genres, or historical settings. Validation designs must place those commitments under evidential strain: alternative construct anchors, source- or genre-stratified tests of construct stability, and triangulation against evidence streams not already induced by the same transformation. Consider a historical-text pipeline that assigns high scores for construct X to texts that are semantically close to a historically retrieved proxy set for X. If those scores also correlate with an expected historical variable, the result is evidentially meaningful: it supports the usefulness of the induced scoring scheme and may provide convergent evidence that the pipeline is capturing a historically relevant signal. What it does not yet establish is that the proxy set adequately represents construct X, that the construct anchor is historically appropriate, or that semantic proximity to the proxy set licenses interpreting the resulting score as a measurement of X. A credible condition of failure need not require direct access to the historical psychological attribute. In a CCR-style case, it might arise if an equally defensible historically grounded proxy set—for example, one built from indigenous legal, ritual, or moral vocabulary rather than questionnaire-derived construct items—yielded a systematically different ranking or temporal profile of the same corpus. It might also arise if source- or genre-stratified analyses showed that the score tracked genre conventions, quotation practices, or corpus composition more strongly than the intended construct. In such cases, the problem would not simply be weaker model performance; it would be instability in the relation between construct, proxy, and historical trace. Without such exposure, evidence for the stability or usefulness of a scoring scheme can be misallocated as evidence that the scheme measures the target attribute.
4. When commensuration becomes historical authority
The stakes of the foregoing argument are not exhausted by the claim that predictive success may be overread as validated measurement. In historically mediated inquiry, the deeper consequence is that a commensurating regime can convert selective, heterogeneous, and interpretively contested materials into portable psychological facts. The problem is therefore not only epistemic inflation. It is the production of historical authority through a metric whose conditions of legibility may too easily recede from view. That shift matters because historical inquiry has traditionally depended on explicit sensitivity to mediation: archive, transmission, social position, rhetorical convention, and the uneven survival of evidence. Humanities scholarship has long treated archival absence not as a neutral gap in the record but as an effect of power (Trouillot Reference Trouillot1995). Recent work in digital history and digital humanities has similarly argued that digitization often reproduces rather than erases those asymmetries (Ortolja-Baird and Nyhan Reference Ortolja-Baird and Nyhan2022; Zaagsma Reference Zaagsma2023). Selection for digitization, cataloguing, metadata, and access shape what becomes discoverable and usable as historical evidence, while digital availability can give already dominant corpora and canonical materials renewed reach and authority (Guldi Reference Guldi2023; Jensen Reference Jensen2021; Zaagsma Reference Zaagsma2023). A scoring regime that renders texts commensurable under a shared construct vocabulary therefore does not eliminate mediation. It reorganizes it. Features that would otherwise demand interpretive caution—genre, transmission, archival survival, social position, and rhetorical convention—can be redescribed as manageable noise within a tractable inferential system. The effect is not simply to simplify a difficult evidential field, but to change what counts, within that field, as disciplined historical knowledge.
This is where the epistemological diagnosis presented here connects directly to power. Once a commensurated historical score is treated as measurement, what is legible to the metric can acquire priority over what is legible only to thicker forms of historical interpretation. The issue is not that quantitative results are inherently coercive or that interpretive judgment is beyond criticism. It is that, under measurement rhetoric, the burden of justification can silently reverse. Instead of the metric having to show that it tracks the target in a historically adequate way, rival readings must increasingly justify themselves against the authority of the score. Once constructs such as tightness–looseness, individualism–collectivism, or social trust are treated as scoreable across periods, regions, or societies, the resulting differences can cease to function merely as exploratory outputs. They become available as evidence that historically remote populations instantiated more or less of a modern psychological trait, and from there can be carried forward into rankings, civilizational narratives, explanations of institutional divergence, or claims about the deep cultural roots of present-day behavior. At that point, the issue is no longer only overinterpretation, but downstream stabilization: the score can circulate as evidence while the representational choices that produced it become harder to recover and contest.
Historically mediated scores are rarely inert. Once they circulate, they can guide corpus selection, motivate explanatory variables, privilege certain annotations, and encourage the search for further downstream regularities. Under those conditions, commensuration becomes constitutive: the metric helps make the historical world it purports to measure by reorganizing attention, inference, and explanation around what it can register (Espeland and Sauder Reference Espeland and Sauder2007). What cannot be translated into the construct space is increasingly liable to appear irrelevant, residual, or methodologically intractable. Contemporary constructs may not simply measure what is present in the archive; they may determine which archival features become legible as evidence. Where that possibility remains untested, downstream success may register increasing competence within that partition without showing that it has been vindicated as measurement.
The consequences are not confined to academic method. Historical claims formed in this way can travel outward into public and institutional discourse. Once expressed as scores, they are easier to mobilize in debates about culture, governance, development, education, or collective identity. That mobility is part of their appeal, but also what makes them dangerous when the evidential scope of the measure exceeds what has been secured. A commensurated representation of the past can quickly become an input into contemporary narratives about which societies are more trusting, more disciplined, more individualistic, or more psychologically “advanced.” The risk is especially acute where the surviving corpus is shaped by elite male authorship, canonical transmission, and institutions of preservation that disproportionately retain the perspectives of socially dominant groups. In such cases, the archive is not merely selective in the thin sense of being incomplete. It may also be normatively saturated, preserving texts that articulate and justify hierarchy, exclusion, patriarchy, and other forms of domination while leaving alternative experiences, idioms, and forms of life radically underrepresented. When such materials are scored as if they transparently indexed the psychology or culture of a population, elite self-description can become evidence about collective traits, and historically situated justifications for inequality can become neutral-seeming comparative facts. The point is not that history should never be rendered comparable, nor that quantification must always distort. Comparison is often indispensable, and commensuration can be methodologically productive. The point is that when comparison hardens into measurement rhetoric under conditions of weak independent validation, the resulting authority can exceed its warrant in consequential ways. It can flatten local difference, naturalize the present-day categories through which the past has been made legible, and reinforce the authority of what is already best preserved, best digitized, and easiest to score. In that respect, the danger is not just that one gets the past wrong. It is that one acquires a disciplined, scalable, and institutionally usable way of being wrong that is harder to contest precisely because it looks like measurement.
The constructive implication is therefore not to retreat from historical-text modeling, but to raise the threshold for what such models may be taken to establish. Where scores are used comparatively, their conditions of construction should remain visible. Where they are used explanatorily, the proxy relation should remain contestable. And where they support broad claims about historical populations, institutions, or trajectories, the burden should fall not only on demonstrating predictive success but on specifying credible conditions under which the representational mapping itself would count as having failed. Without that discipline, historically mediated text analysis risks converting one way of organizing historical evidence into the authoritative form of the past itself.
5. Conclusion
The argument developed here demands greater precision about what historically mediated predictive procedures establish. In endogenously validated regimes, predictive fit, benchmark alignment, and downstream usefulness may all be real achievements. But they do not by themselves show that the construct-to-trace relation on which those achievements depend has been vindicated as measurement. Where that distinction is blurred, success within a commensurating regime can be mistaken for independent warrant for the relation that made the regime possible. The danger is not simply overconfidence in one method, but the quiet conversion of one disciplined way of organizing evidence into the authoritative shape of the phenomenon itself. The constructive response is not retreat but redesign. If historically mediated predictive inquiry is to mature, it should aim not only at stronger pipelines, larger corpora, or better benchmarks, but at validation designs that make representational failure more visible: alternative construct anchors, rival operationalizations, source-stratified tests of construct stability, and triangulation against evidence streams not already induced by the same commensurating transformation. What is needed is not less comparison, but comparison under conditions where its own terms remain contestable. The central question for historically mediated measurement is therefore no longer simply whether a model works, but whether the conditions under which it works are also conditions under which it could be shown to be wrong. That is the standard by which predictive success ceases merely to entrench a frame and begins, instead, to justify belief.
Acknowledgements
I gratefully acknowledge the support of The Center for Hellenic Studies, which provided the time and resources to develop the broader research program of which this article forms a part. I also thank the two anonymous reviewers for their careful reading and constructive suggestions, which strengthened the manuscript.
Funding statement
None to declare.
Declarations
The author was formerly a postdoctoral researcher in the Culture, Cognition, and Coevolution Laboratory at Harvard University. She has not participated in the laboratory’s text-analysis research since 2024 and was not involved in the China corpus research discussed in this article.
