Hostname: page-component-74656f9884-8r8nv Total loading time: 0 Render date: 2026-11-07T01:05:58.857Z Has data issue: false hasContentIssue false
  • English
  • Français

The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models

Published online by Cambridge University Press:  28 September 2026

Timo Freiesleben Open the ORCID record for Timo Freiesleben [Opens in a new window]  and
Sebastian Zezulka Open the ORCID record for Sebastian Zezulka [Opens in a new window]
Timo Freiesleben
Affiliation:
Munich Center for Mathematical Philosophy, LMU Munich, Germany
Sebastian Zezulka*
Affiliation:
Cluster of Excellence: “Machine Learning: New Perspectives for Science”, University of Tübingen, Tübingen, Germany
*
Corresponding author: Sebastian Zezulka; Email: sebastian.zezulka@uni-tuebingen.de
Rights & Permissions [Opens in a new window]

Abstract

Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning research and scientific inquiry. However, benchmark scores at best measure performance relative to a specific dataset and learning problem. Drawing substantial scientific inferences requires additional assumptions. Adapting ideas from psychological validity theory, we propose validity conditions that make these assumptions explicit. In two case studies—ImageNet and the Fragile Families Challenge—we show how benchmark results can support inferences about research progress and limits of predictability, situating predictive benchmarking as a distinct epistemic practice in machine learning.

Information

Type
Article
Information
Philosophy of Science , First View , pp. 1 - 30
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2026. Published by Cambridge University Press on behalf of Philosophy of Science Association

1. Introduction

Predictive benchmarking is the central evaluation strategy in machine learning. By now, scientists use benchmarks to address diverse scientific questions: Do we make progress on image classification? Are life outcomes unpredictable? Do large language models (LLMs) exhibit general capabilities? This raises epistemological questions about what predictive benchmarks measure and how we can appraise the validity of inferences drawn from such measurement. To answer these questions, we begin by tracing the historical development of predictive benchmarking, from a methodology for evaluating machine learning systems to an epistemic tool used across the sciences to justify claims that benchmarks were not originally designed to support.

In the early days of artificial intelligence research, the Defense Advanced Research Projects Agency (DARPA) was one of its main funders. Much like today, researchers promised to solve complex and abstract engineering tasks like language translation or object recognition within just a few months or years, ultimately never coming close to delivering on their promises. A revealing episode played out in 1966, when Minsky and Papert, whose research was strongly supported by DARPA at the time, assigned a group of students a project aimed at artificially constructing “a significant portion of a visual system” over the course of just one summer (Papert Reference Papert1966). Following substantial cuts in government funding and disappointed expectations, DARPA employees were faced with the difficult question of assessing whether progress had been made on any of the problems.

DARPA answered this question in the late 1980s by adopting the Common Task Framework (Liberman Reference Liberman2010; Donoho Reference Donoho2017). Instead of the grand claims about solving artificial general intelligence that dominated the early days of artificial intelligence (AI), the Common Task Framework shifted the focus to concrete (usually supervised) learning problems. Data for building models to solve these problems was made publicly available, and crucially, performance was evaluated on unseen holdout data. DARPA had both the authority and the resources to establish the Common Task Framework as the new norm in AI research, thereby laying the groundwork for modern predictive benchmarking and fostering the shift from symbolic AI toward machine learning.Footnote 1

A predictive benchmark has four components: (1) a learning problem specifying the prediction target; (2) a standardized dataset split into a training set and an evaluation set; (3) an evaluation metric; and (4) a public leaderboard, ranking the models by the scores they obtained.Footnote 2 Take the famous Modified National Institute of Standards and Technology (MNIST) benchmark as an example (Lecun et al. Reference Lecun, Bottou, Bengio and Haffner1998). It consists of a seemingly simple learning problem: Given a small grayscale image of a handwritten digit, identify which digit it represents, from 0 to 9 (Yadav and Bottou Reference Yadav, Bottou, Wallach, Larochelle, Beygelzimer, d’Alché-Buc, Fox and Garnett2019). Performance of models is evaluated by their accuracy on the evaluation set. For some years, MNIST was a proving ground for new machine learning techniques, and a public leaderboard still tracks model results. But times have changed, and with current methods reaching about 99.87% accuracy, the benchmark is considered too easy and essentially “solved.”

Predictive benchmarks have since evolved from a tool for making funding decisions into the dominant methodology for evaluating machine learning models and coordinating research programs around shared problems.Footnote 3 They have become the “iron rule” of machine learning (Hardt Reference Hardt2026): Justification for algorithmic innovation largely depends on outperforming the “state of the art,” as measured by a relevant benchmark. With the growing influence of machine learning over the last decade, predictive benchmarks have transcended their engineering roots and are emerging as a relevant methodology in the empirical sciences. This became very clear in 2024, when benchmark-driven research earned its first Nobel Prize: John Jumper and Demis Hassabis were honored for their protein structure prediction models, which substantially outperformed all previous methods on the Critical Assessment of Protein Structure Prediction (CASP)Footnote 4 benchmark challenge (Jumper et al. Reference Jumper, Evans, Pritzel, Green, Figurnov, Ronneberger, Tunyasuvunakool, Bates, Žídek, Potapenko, Bridgland, Meyer, Kohl, Ballard, Cowie, Bernardino Romera-Paredes, Jain, Adler, Back, Petersen, Reiman, Clancy, Zielinski, Steinegger, Pacholska, Berghammer, Bodenstein, Silver, Vinyals, Senior, Kavukcuoglu, Kohli and Hassabis2021). And it is not only structural biology—prominent predictive benchmarks now exist across many scientific fields, including meteorology (Rasp et al. Reference Rasp, Dueben, Scher, Weyn, Mouatadid and Thuerey2020), medicine (Irvin et al. Reference Irvin, Rajpurkar, Ko, Yu, Silviana Ciurea-Ilcus, Marklund, Haghgoo, Ball, Shpanskaya, Seekins, Mong, Halabi, Sandberg, Jones, Larson, Langlotz, Patel, Lungren and Ng2019), neuroscience (Schrimpf et al. Reference Schrimpf, Kubilius, Hong, Majaj, Rajalingham, Issa, Kar, Bashivan, Jonathan Prescott-Roy, Schmidt, Yamins and DiCarlo2018), materials science (Dunn et al. Reference Dunn, Wang, Ganose, Dopp and Jain2020), legal studies (Guha et al. Reference Guha, Nyarko, Ho, Ré, Chilton, Aditya, Alex Chohlas-Wood, Waldon, Rockmore, Zambrano, Talisman, Hoque, Surani, Fagan, Sarfaty, Dickinson, Porat, Hegland, Wu, Nudell, Niklaus, Nay, Choi, Tobia, Hagan, Ma, Livermore, Nikon Rasumov-Rahe, Kolt, Henderson, Rehaag, Goel, Gao, Williams, Gandhi, Zur, Iyer, Li, Oh, Naumann, Globerson, Saenko, Hardt and Levine2023), psychology (Fried et al. Reference Fried, Proppert and Rieble2023), and sociology (Sivak et al. Reference Sivak, Pankowska, Mendrik, Emery, Javier Garcia-Bernardo, Karpinska, Maineri, Mulder, Nissim and Stulp2024).

Despite the central role that benchmarks play in machine learning and in the sciences, they have received little attention from philosophers. Philosophers of machine learning have primarily concentrated on issues of epistemic opacity (Creel Reference Creel2020; Boge Reference Boge2021; Sullivan Reference Sullivan2022; Duede Reference Duede2023) and explainable AI (Zednik and Boelsen Reference Zednik and Boelsen2022; Watson Reference Watson2022; Buchholz and Grote Reference Buchholz and Grote2023; Freiesleben et al. Reference Freiesleben, König, Molnar and Tejero-Cantero2024). A rich philosophical literature examines (big) data along its representational and relational dimensions (Bogen and Woodward Reference Bogen and Woodward1988; Leonelli Reference Leonelli2015; Leonelli Reference Leonelli and Edward2020), but not in the context of benchmarking. Formal epistemologists, providing a priori guarantees for learning algorithms, have typically adopted a theoretical perspective (Herrmann Reference Herrmann2020; Sterkenburg and Grünwald Reference Sterkenburg and Grünwald2021; Lin Reference Lin2022; Grote et al. Reference Grote, Genin and Sullivan2024b; Sterkenburg Reference Sterkenburg2024), largely overlooking the empirically driven methodology of predictive benchmarking that underpins much of contemporary machine learning research. Yet it is precisely because benchmarking structures entire research programs and informs scientific inquiry and policy decisions that it demands philosophical scrutiny. Without an epistemic basis, researchers risk drawing misleading or irreproducible inferences from benchmark scores. This article contributes such a basis, showing how predictive benchmarks can serve as epistemic tools capable of supporting a wide range of scientific inferences, if researchers explicitly consider the relevant validity conditions. Specifically, we address the following question: How can benchmark scores support substantive scientific inferences within machine learning and across the sciences?

We argue that, analogous to psychometric and educational tests, predictive benchmarks are best understood as measurement tools (sec. 2). The resulting measurements M(F) warrant a generalization inference about the expected predictive performance of the set of models F on a given learning problem, given appropriate sampling conditions. Drawing more substantial inferences I from the measurements requires additional assumptions A to render the inferences valid, that is,

upper M left parenthesis upper F right parenthesis and upper A right arrow upper I period $M(F)\; \wedge \;A\; \to \;I.$

The study of these assumptions is often referred to as validity theory (Kane Reference Kane2013; American Educational Research Association et al. 2014). We adapt the argument-based validity framework to the context of predictive benchmarks, comprising four steps: (1) defining the intended inference, (2) specifying necessary validity conditions for this inference, (3) providing evidence for and against the conditions, and (4) constraining the inference accordingly. We specify validity conditions under which substantial scientific inferences can be drawn from benchmark scores, establishing predictive benchmarks as key sites of conceptual and theoretical reasoning in machine learning and the sciences more broadly (sec. 3).

We illustrate the framework with two case studies that reflect different stakeholders, the distinct inferences they seek to draw from benchmark scores, and the corresponding validity concerns that arise. The first examines ImageNet for computer vision, where the learning problem is to map images to predefined classes (sec. 4). Within the field, ImageNet serves as a proxy for measuring methodological progress on the broader task of image classification. For this role to be justified, in addition to a generalization inference, one must also draw an extrapolation inference from performance on ImageNet to performance on other such image-classification problems, and a conceptualization inference from performance on ImageNet to image classification more generally. We analyze the conditions under which these inferences are valid and the types of evidence required to support them.

The second case study considers the Fragile Families Challenge (FFC) from sociology, which focuses on predicting life outcomes of young people (sec. 5). The uniformly low predictive performance of models in this setting has led researchers to question whether such outcomes are fundamentally unpredictable. In addition to the preceding sub-inference, justifying this inference requires establishing additional predictability conditions that rule out alternative explanations, including limited sample size or unobserved features. In section 6, we conclude by outlining the broader relevance of validity theory for evaluating machine learning systems, in particular as a tool for assessing the capabilities of LLMs and the limits of benchmarking.

2. Predictive benchmarks are measurement tools

Predictive benchmarks are best understood as measurement instruments for the predictive performance of prediction models on specific learning problems. As such, they are analogous to psychological and educational tests that measure latent psychological traits of test takers (Blili-Hamelin and Hancox-Li Reference Blili-Hamelin and Hancox-Li2023; Cardoso et al. Reference Cardoso, Santos, Kawasaki Francês, Prudêncio, Alves, Cerri and Prati2020).Footnote 5 Consider the example of intelligence tests: Human test takers solve a battery of test items, such as sequence completion problems, to obtain a test score. This score is then interpreted as indicative of a trait of the test takers, conceptualized as constructs like their general cognitive ability. Subsequently, it may be used to draw further inferences or make decisions, such as selecting job candidates (Glymour Reference Glymour1998; Cave Reference Cave2020). In the case of predictive benchmarking, prediction models (test takers) are evaluated on data instances in the benchmark (test items), yielding a performance score (test score). From this score, researchers draw various inferences, for example, about the predictive performance of models (constructs) to select the most suitable prediction models.

In psychological and educational measurement, well-developed methodological frameworks are used to justify inferences from test scores, commonly subsumed under the term validity theory (American Educational Research Association et al. 2014). They require the specification of validity conditions that warrant the inferences (Messick Reference Messick1989; Kane Reference Kane2013). Depending on the construct of interest, they are often complemented by a theoretical characterization of that construct (Cronbach and Meehl Reference Cronbach and Meehl1955; Borsboom et al. Reference Borsboom, Mellenbergh and van Heerden2003; Millsap Reference Millsap2012). One expects similar efforts when it comes to validating the inferences drawn from benchmark scores.

The validity conditions for a generalization inference from benchmark scores are well established; this is arguably the standard form of inference in machine learning. Benchmark scores are used to infer a model’s performance on randomly sampled and previously unseen data. In this setting, little explicit measurement theory is required, because the problem can be formulated as a statistical estimation task: The empirical error is an estimator for the expected error. This procedure is commonly referred to as the holdout method (Grote et al. Reference Grote, Genin and Sullivan2024b; Hardt Reference Hardt2026).

Let’s unpack this inference. Evaluating a set of models on a benchmark yields numerical scores, the empirical error as defined by the evaluation metric.Footnote 6 For a prediction model f, an evaluation dataset De with features xi and labels yi, and an evaluation metric script l $\ell$ , the empirical error is defined as

ModifyingAbove upper R With caret left parenthesis f right parenthesis colon equals StartFraction 1 Over StartAbsoluteValue upper D Subscript e Baseline EndAbsoluteValue EndFraction sigma summation Underscript left parenthesis x Subscript i Baseline comma y Subscript i Baseline right parenthesis element of upper D Subscript e Baseline Endscripts script l left parenthesis f left parenthesis x Subscript i Baseline right parenthesis comma y Subscript i Baseline right parenthesis period $\hat R(f): = {1 \over {\mid {D_e}\mid }}\sum\limits_{({x_i},{y_i}) \in {D_e}} \ell \left( {f({x_i}),{y_i}} \right).$

That is, the evaluation metric script l $\ell$ is used to compute an error for each instance left parenthesis x Subscript i Baseline comma y Subscript i Baseline right parenthesis $(x_i, y_i)$ by measuring the discrepancy between the model prediction f left parenthesis x Subscript i Baseline right parenthesis $f(x_i)$ and the corresponding label y Subscript i $y_i$ . These instance-level errors are then averaged over all examples in the evaluation dataset to obtain the empirical error.

To interpret the scores, we must connect them to the theoretical quantity or construct that the scores are intended to measure. In the case of a generalization inference, this is the expected error. For a model f and an evaluation metric script l $\ell$ , the expected error is defined as

upper R left parenthesis f right parenthesis colon equals double struck upper E Subscript upper X comma upper Y Baseline left bracket script l left parenthesis f left parenthesis upper X right parenthesis comma upper Y right parenthesis right bracket comma $R(f): = {\mathbb{E}_{X,Y}}\left[ {\ell \left( {f(X),Y} \right)} \right],$

where (X,Y) are random variables drawn from a fixed but unknown data distribution, from which the evaluation data upper D Subscript e $D_{e}$ are assumed to be sampled.Footnote 7 The expected error captures how the models would, on average, perform on new instances from this population, effectively representing the model’s performance were we able to observe an infinite number of data points.

Inferring the expected errors from benchmark scores requires supporting assumptions—we will call them generalization conditions—that connect them. The validity conditions must provide a sound inferential bridge between the benchmark scores and the intended interpretation. In this case, sufficient conditions are as follows:

  1. (i) The evaluation data are independent of the models, implying, for example, that one cannot use the evaluation data for model training.

  2. (ii) The evaluation data are statistically independent samples left parenthesis x Subscript i Baseline comma y Subscript i Baseline right parenthesis $(x_i, y_i)$ drawn from one identical distribution over (X, Y), the independent and identically distributed (i.i.d.) assumption.

  3. (iii) The evaluation data are sufficiently large so that the empirical error provides a reasonable approximation of the expected error.

Together, these conditions justify the generalization inference via the law of large numbers and provide theoretical error bounds for finite samples (Hardt Reference Hardt2026). However, the three conditions are not strictly necessary: Weaker variants may suffice, depending on the specific requirements of the inference. Condition (i) can be weakened, because some dependencies between test data and models, such as adaptivity, do not always invalidate the inference from test error to expected error (Hardt Reference Hardt2026, chap. 5). However, the corresponding error bounds for finite samples, which capture the expected discrepancy between test and true error, may become substantially wider, increasing the required sample size for reliable inference (Dwork et al. Reference Dwork, Feldman, Hardt, Pitassi, Reingold, Roth, Cortes, Lawrence, Lee, Sugiyama and Garnett2015). Likewise, condition (ii) can be relaxed so that the test data are a random sample from an underlying population (Höltgen and Williamson Reference Höltgen and Charles Williamson2026) or, particularly in the case of time series, to assumptions like stationarity or ergodicity (Andrews Reference Andrews1988; Sandrić Reference Sandrić2017). Similarly, the requirements of condition (iii) depend on the specific inference considered: Whereas some require tight error bounds and thus large sample sizes, others can be adequately supported by smaller samples.

In any case, when the generalization conditions are warranted, they allow us to interpret benchmark scores as an implementation of the holdout method and the resulting empirical errors as estimates of the expected errors. This is a strong result, and it already enables us to draw a range of important inferences from benchmark scores: about a model’s expected error on the learning problem; about how different models compare in their expected error on this problem; and even about whether particular learning algorithms, architectural choices, or hyperparameter settings yield models with lower expected errors. Generalization inferences are the basis and constitute an important step for many scientific inferences. In the words of Kane (Reference Kane2013), “generalizability is a necessary condition for the validity of almost all test score interpretations because almost all of these interpretations involve generalization over some universe of generalization that goes beyond the observations actually made” (29).

Although generalization inferences are relevant, in the empirical sciences and in machine learning, we often seek to draw more involved inferences from benchmark scores, which the three conditions noted previously, by themselves, cannot support. Recall our motivating examples: Researchers may wish not merely to infer that models perform well on a benchmark such as ImageNet, but that they are making progress on the broader task of image classification, or that consistently poor benchmark performance across many prediction models indicates principled limitations on the predictability of life outcomes. These inferences extend beyond statistical standards of estimation and call for a richer structure of validity considerations than is currently in place for benchmarking, potentially changing the target of estimation itself. In psychological testing, such conceptual, theoretical, and normative considerations are systematically organized under the umbrella of validity theory (American Educational Research Association et al. 2014). Analyzing the inferences drawn from predictive benchmarks as measurements brings into focus the central role that supporting conditions of validity play. Validity theory provides a powerful framework for conceptualizing the diverse and substantive inferences drawn from predictive benchmarks.

3. Validity theory for score interpretation

Two major philosophical positions can be distinguished on what validity is all about (Alexandrova and Haybron Reference Alexandrova and Haybron2016; Zhao Reference Zhao2023). The correspondence account treats validity as a feature of the measurement instrument or its outcomes, for example, by establishing a causal relationship between the measurement and the construct (Borsboom et al. Reference Borsboom, Mellenbergh and van Heerden2004). In contrast, the argument-based account understands validity as a feature of the score interpretation or its use, rather than of the measurement instrument, focusing on the logical relationship between the measurement and the intended inference (Kane Reference Kane2013). Like Hood (Reference Hood2009), we take these accounts as complementary rather than mutually exclusive: The correspondence account provides an ontology, and the argument-based account provides an epistemology. Because we adopt an epistemological rather than an ontological perspective on benchmarks and examine them as epistemic tools for drawing valid inferences, we follow the argument-based account of validity in this article.

The argument-based account of validity has two key components: (1) the interpretation or use argument and (2) the validity argument (Kane Reference Kane2013, 9). The first specifies the intended inference from the benchmark scores and the conditions under which the inference is valid; together with the measurement, these conditions should imply the intended inference. The second examines the conditions and provides empirical and conceptual warrants for them, potentially requiring us to adapt the intended inference. The uncertainties of these warrants carry over to the validity of the inference. In practice, the validity of an intended inference is established in a four-step procedure that is widely recognized as a standard in psychological and educational measurement (American Educational Research Association et al. 2014). It is defined by the following iterative sequence of instructions, where steps 1 and 2 correspond to specifying the interpretation or use argument, and steps 3 and 4 correspond to establishing the validity argument:

  1. 1. Define the intended inference. The inference intended to be drawn from the benchmark scores must be made specific. This includes a specification of the relevant constructs, the scope of the inference, and the substeps of the inference.

  2. 2. Specify validity conditions. Validity conditions must be specified that make all substeps of the intended inference and, consequently, the intended inference valid. These secure, among others, the relevant statistical assumptions and the relevance of the operationalization of the construct.

  3. 3. Provide evidence for and against the conditions. Researchers must provide evidence of whether and to what degree the validity conditions hold. Here, evidence is understood broadly and may include benchmark scores for relevant subgroups, results from other studies, and conceptual or domain-specific theoretical insights.

  4. 4. Constrain the inference. In light of the evidence, the validity conditions and, consequently, the intended inference will only be supported to some degree. There will be evidence strengthening some and evidence undermining other assumptions. Scientists must constrain the scope and content of the intended inference accordingly and then validate the inference again using the steps outlined previously.

Applying this framework in practice imposes substantial demands on researchers and, in most cases, is not straightforward. Even specifying the intended inference can be challenging. Although researchers often start with broad, high-level inferences, there is a trade-off: The more general the intended inference is, the stronger become the required validity conditions and the more extensive must the supporting evidence be (Kane Reference Kane2013). In practice, researchers often arrive at a well-specified intended inference only by an iterative process, starting from a broad formulation and refining it over multiple passes through the framework.

A key prerequisite for applying the framework is to decompose the intended inference into a set of well-motivated substeps, which we refer to as sub-inferences. This decomposition is necessary because intended inferences are often complex and highly context specific. Although it may be possible to formulate validity conditions directly for a given intended inference, such conditions would typically reflect this context and would be difficult to reuse, limiting the usefulness and standardization of the framework. At the same time, many inferences share common inferential steps. For example, consider the generalization inference that underlies most scientific inferences from benchmark scores. It is possible to formulate general validity conditions that can be applied across contexts for sub-inferences of this kind. A key contribution we make is to identify a set of relevant sub-inferences and to specify corresponding validity conditions. We provide an overview in table 1. Some of the inferences and conditions, especially those about generalization, extrapolation, and decision, are inspired by Kane (Reference Kane2013).

Table 1.

The table summarizes the five sub-inferences discussed in this article, specifying their associated validity conditions. Here, f denotes the model, upper D Subscript e $D_e$ the evaluation data, script l $\ell$ the evaluation metric, and d(s) a decision rule based on scores s. script upper B $\mathcal{B}$ is a set of benchmarks, and T represents a theoretical task. The arrow on the right indicates that as more theoretical inferences are drawn, an increasing number of validity conditions must be satisfied

A table summarizing sub-inferences and their validity conditions.
Table 1. Long description

A table with two columns: Inference and Conditions. The Inference column lists five sub-inferences: Generalization, Extrapolation, Conceptualization, Predictability, and Decision. Each sub-inference has associated conditions specified in the Conditions column. The table includes rows detailing the conditions for each sub-inference. For Generalization, conditions include independence and size of evaluation data. For Extrapolation, conditions involve robust performance and informative metrics. For Conceptualization, conditions relate to the learning problem and task performance. For Predictability, conditions include model diversity and feature measurement. For Decision, conditions involve utility, fairness, and systemic effects. The table also notes that as more theoretical inferences are drawn, more validity conditions must be satisfied.

Specifying the appropriate validity conditions that render any specific inference valid is an important task. In the following, we identify sufficient conditions, but as discussed in the previous section for the generalization inference, these need not be necessary. Weaker validity conditions impose less demanding requirements on the supporting evidence and may thus be preferable.

Any inference is only as strong as the evidential support for its validity conditions and can be undermined by evidence that challenges the conditions. The framework thus also provides a structured method for invalidation: Inferences can be weakened or even rejected by undermining the relevant validity conditions with new evidence. As suggested by Kane (Reference Kane2013), the inferential steps can be seen as spans of a bridge connecting benchmark scores, validity conditions, and conclusions. A single failing span may invalidate the inference, even if others remain intact. Accordingly, researchers should focus their efforts on identifying the weakest points in the validity argument.

In general, the validity of an inference is gradual, and its assessment evolves over time (Kane Reference Kane2013). More substantial or wide-ranging inferences from benchmark scores rest on more sub-inferences and, accordingly, more validity conditions. As the theoretical implications of an inference increase, we move from generalization toward extrapolation, conceptualization, decision conditions, and beyond.

The argument-based account of validity may at first appear abstract, offering a broad perspective but limited concrete guidance for specific applications. The validity conditions we propose are general in that they apply across a wide range of settings while still allowing for the flexibility needed to accommodate contextual variation. Ultimately, the conceptual subtleties of the account in benchmarking become fully apparent only in application. In the next two sections, we therefore examine two case studies. First, we analyze an inference about research progress in the context of ImageNet, a widely used benchmark in computer vision. Second, we turn to an inference about the predictability of life outcomes based on the FFC.

4. ImageNet: Do we make progress on image classification?

The ImageNet ChallengeFootnote 8 is one of the most influential predictive benchmarks in the history of computer vision and machine learning in general (Deng et al. Reference Deng, Dong, Socher, Li, Kai and Li2009). It was at the center of the deep learning revolution, when the convolutional neural network AlexNet (Krizhevsky et al. Reference Krizhevsky, Sutskever, Hinton, Fernando Pereira, Burges and Kilian2012) outperformed the competing models by about 11 percentage points in 2012. Moreover, key algorithmic innovations in deep learning, like dropout (Srivastava et al. Reference Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov2014) and batch normalization (Ioffe and Szegedy Reference Ioffe, Szegedy, Bach and Blei2015), were adopted because of their effectiveness in improving performance on ImageNet.

The ImageNet Challenge, particularly its classification component, became so paradigmatic in computer vision that it effectively served as a measure of progress. Within 5 years, researchers were able to reduce the reported error rates from 28.2 percent sign $28.2\%$ in 2010 to 3.57 percent sign $3.57\%$ in 2015 (Nguyen et al. Reference Nguyen, Fookes, Ross and Sridharan2018). Why can performance gains on ImageNet be taken as evidence that models have improved on image classification generally? The four-step approach introduced earlier allows us to evaluate when this inference is valid by analyzing the underlying validity conditions.

4.1. The intended inference

Computer vision researchers may aim to infer that the error rates of models on ImageNet reflect their error rates on image-classification tasks generally. If this inference holds, improvements on ImageNet can legitimately be interpreted as advances in image classification and, by extension, in computer vision. This would also justify the community’s intense concentration on ImageNet.

The scope of this inference is fairly wide and ultimately concerns the theoretical notion of an image-classification task. The construct of interest is the error rate on such image-classification tasks. Connecting the scores on ImageNet to this construct requires three sub-inferences: First, the error rates on the ImageNet evaluation dataset must reflect the error rates on ImageNet-like data (generalization inference); second, they must reflect the error rates on alternative image-classification benchmarks (extrapolation inference); and finally, they must reflect error rates in image-classification tasks generally (conceptualization inference).

4.2. Specifying validity conditions and providing relevant evidence

To avoid jumps between the conditions supporting the inferences and their respective justifications, we present each validity condition together with a critical discussion of the supportive and contradictory evidence.

Generalization inference. We discussed the three standard generalization conditions in section 2, requiring that (i) the model is independent of the evaluation data, (ii) the samples are statistically independent and drawn from an identical distribution (i.i.d.), and (iii) the evaluation dataset is sufficiently large. We now consider the relevant evidence for each in turn.

The ImageNet data were used over many years, during which researchers could adapt to prior results and successful model architectures, violating the independence of the models and the evaluation data (Blum and Hardt Reference Blum and Hardt2015). Recht et al. (Reference Recht, Roelofs, Schmidt, Shankar, Chaudhuri and Salakhutdinov2019) recreated ImageNet, mirroring the original data-collection strategy, to empirically study this violation by reevaluating successful models like AlexNet and ResNet on the new data. By construction, these new data are independent of the models.Footnote 9 They indeed found that models performed about 11% to 14% worse on the recreated data, but model rankings remained intact. Interestingly, the performance gap narrowed for more recent, higher-accuracy models. For this reason, the authors attribute the observed performance drops not to adaptivity, which should at the very least not decrease over time, but instead to subtle distribution shifts in data collection. We return to this discussion in the context of the extrapolation inference.

The i.i.d. assumption is not directly testable. It serves as a guideline for data collection and puts strong conceptual limitations on the population for which we get valid estimates. ImageNet images were sourced from web search engines and labeled by at least 10 Mechanical Turk workers per image (Deng et al. Reference Deng, Dong, Socher, Li, Kai and Li2009; Denton et al. Reference Denton, Hanna, Amironesei, Smart and Nicole2021). Duplicates and labels on which workers did not agree were filtered out to ensure accurate labels, but this introduces potential sampling biases (Torralba and Efros Reference Torralba and Efros2011; Shirali and Hardt Reference Shirali and Hardt2025), which we discuss further in the conceptualization inference.

The evaluation set is large enough to yield reliable error estimates with roughly 100 images per class. Russakovsky et al. (Reference Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg and Fei-Fei2015) empirically confirm this by estimating bootstrap-based confidence intervals, showing low variation in the resulting scores.

Extrapolation inference Extrapolation inferences connect expected errors on a specific benchmark with expected errors on alternative benchmarks. In practice, errors often vary substantially across closely related benchmarks (Torralba and Efros Reference Torralba and Efros2011; Herrmann et al. Reference Herrmann, Probst, Hornung, Jurinovic and Boulesteix2021, Reference Herrmann, Lange, Eggensperger, Casalicchio, Wever, Feurer, Rügamer, Hüllermeier, Boulesteix, Bischl, Salakhutdinov, Kolter, Heller, Weller, Oliver, Scarlett and Berkenkamp2024), grounding the need for validity conditions that link ImageNet scores to alternative data distributions, learning tasks, and evaluation metrics. Two sufficient conditions are as follows:

  1. (i) The performance scores from ImageNet are similar across relevant alternative learning problems and data distributions.

  2. (ii) The performance scores for a given evaluation metric are informative about scores from other relevant evaluation metrics.

The two conditions ensure that model performance is robust over relevant alternative operationalizations of the task. Robustness is a causal term: The benchmark scores must remain sufficiently stable under changes to a modifier, here the operationalizations (Freiesleben and Grote Reference Freiesleben and Grote2023). What counts as relevant alternatives is determined by the scope of the inference. In our case, this includes essentially all alternative operationalizations of the intensional task of image classification. Extrapolation and conceptualization inferences are closely related and can often be treated as a single inferential step. Kane (Reference Kane2013, 28) groups them together, only distinguishing between the respective validity conditions: Extrapolation primarily relies on empirical evidence, whereas conceptualization relies mostly on conceptual or theoretical arguments. Nevertheless, we deem it useful to distinguish between extrapolation and conceptualization inferences, because they serve different interpretative goals and have different targets: The first concerns performance on alternative benchmarks, whereas the second concerns performance on the underlying intensional task.

Let us clarify the difference between the two extrapolation conditions before turning to their evidential support. One might be tempted to unify them into a single requirement of robustness of benchmark scores across alternative learning problems, data distributions, and evaluation metrics. However, robustness across metrics differs from robustness across learning problems and data distributions. Although the latter can be captured using the causal notion of robustness of Freiesleben and Grote (Reference Freiesleben and Grote2023), changes to the evaluation metrics will typically change the scores beyond any reasonable tolerance threshold. As such, robustness across metrics either reduces to a weaker requirement, such as the stability of rankings, or must be understood in an information-theoretic sense, where benchmark scores under one metric allow us to infer scores under others.

There is substantial evidence that ImageNet scores robustly rank the performance of models on a broad class of image-classification problems, whereas absolute performance scores drop drastically even under small changes. Kornblith et al. (Reference Kornblith, Shlens and Le2019) evaluated 16 models trained on ImageNet on alternative image-classification tasks like CIFAR-10/100. The ImageNet scores showed near-perfect rank preservation and were predictive of the performance on the tasks. These findings indicate stronger improvements in cross-task generalization compared with early results by Torralba and Efros (Reference Torralba and Efros2011). New image datasets like LAIONet (Shirali and Hardt Reference Shirali and Hardt2025) and ImageNot (Salaudeen and Hardt Reference Salaudeen and Hardt2026) provide distributional alternatives to ImageNet. Like Recht et al. (Reference Recht, Roelofs, Schmidt, Shankar, Chaudhuri and Salakhutdinov2019), they observe large absolute performance drops but stable rankings compared with ImageNet scores. In particular, Salaudeen and Hardt (Reference Salaudeen and Hardt2026) compare six eminent learning algorithms that have improved performances on ImageNet and show that they realize similar performance gains when trained on ImageNot. Nevertheless, small corruptions or perturbations in the data cause sharp performance drops, raising concerns about the domain on which they are robust (Goodfellow et al. Reference Goodfellow, Shlens and Szegedy2015; Geirhos et al. Reference Geirhos, Temme, Rauber, Schütt, Bethge, Wichmann, Bengio, Wallach, Larochelle, Grauman, Cesa-Bianchi and Garnett2018; Hendrycks and Dietterich Reference Hendrycks and Dietterich2019; Tsipras et al. Reference Tsipras, Santurkar, Engstrom, Ilyas, Madry, Hal Daumé and Singh2020).

Turning to the evaluation metric, we again find evidence that model rankings are largely consistent, especially across top-1, top-5, and multilabel accuracy (Russakovsky et al. Reference Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg and Fei-Fei2015; Shankar et al. Reference Shankar, Roelofs, Mania, Fang, Recht, Schmidt, Daumé and Singh2020). Naturally, the different metrics shift the absolute performance scores. However, when models were evaluated on expert-provided multilabels, allowing for several labels per image, the relative gains between the models diminished (Beyer et al. Reference Beyer, Hénaff, Kolesnikov, Zhai and van den Oord2020). This indicates metric sensitivity at finer levels of comparison than ordinal rankings.

Conceptualization inference. Intended inferences often target intensional tasks rather than specific benchmarks. A conceptualization inference links expected errors on a benchmark to expected errors on a task. Establishing this link requires conditions that connect the extensional description of the task (a benchmark) with its intensional description (Schlangen Reference Schlangen2021; Liao et al. Reference Liao, Taori, Raji and Schmidt2021). Three sufficient conditions are as follows:

  1. (i) The benchmark captures essential aspects of the theoretical task, for example, image classification.

  2. (ii) The dataset adequately represents the relevant classes.

  3. (iii) The evaluation metric meaningfully reflects predictive performance.

Unlike the former two sub-inferences, the conceptualization inference calls for a more theoretical assessment. The evaluation must address the following questions: Does the learning problem, specified by the input features and target classes of the ImageNet Challenge, adequately capture the relevant aspects of the intensional task of image classification? Does the ImageNet evaluation dataset cover the full breadth of meaning associated with each class? Does the evaluation metric used in the ImageNet Challenge capture what it means to be successful at image classification? We address each of these questions in turn.

Even though the class selection has a clear bias, ImageNet makes a strong effort to capture a wide range of classes relevant to human image classification. The learning problem in ImageNet is to map images, each assigned a single label, to 5 out of 1,000 classes, which aligns with established practices in image classification (Torralba and Efros Reference Torralba and Efros2011). The 1,000 classes used in the challenge were first randomly chosenFootnote 10 from more than 5,000 classes in the full ImageNet database, which is structured according to the WordNet hierarchy and thus provides a widely accepted ontological basis (Miller Reference Miller1995; Deng et al. Reference Deng, Dong, Socher, Li, Kai and Li2009). At the same time, the class selection is very task specific and was later adopted accordingly. For example, 90 classes of different dog breeds were added to increase the between-image difficulty, requiring models to pick up on fine-grained structures (Russakovsky et al. Reference Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg and Fei-Fei2015).

The curators explicitly aimed to capture visual diversity along dimensions like size, pose, and lighting (Deng et al. Reference Deng, Dong, Socher, Li, Kai and Li2009). Nevertheless, the use of an English word database, web-data limitations, image exclusion criteria, annotator priming, and high-agreement filtering during labeling, which can limit diversity, remain as sources of specific task biases (Torralba and Efros Reference Torralba and Efros2011; Tsipras et al. Reference Tsipras, Santurkar, Engstrom, Ilyas, Madry, Hal Daumé and Singh2020; Shirali and Hardt Reference Shirali and Hardt2025). Probably the most fundamental limitation is that each image only gets a single label, even when multiple may apply. Shankar et al. (Reference Shankar, Roelofs, Mania, Fang, Recht, Schmidt, Daumé and Singh2020) estimate that approximately 20% of images of ImageNet have multiple plausible labels. This reflects practical annotation constraints but reduces the benchmark’s fidelity to general classification.

The evaluation metric used in the ImageNet Challenge has limited conceptual plausibility. ImageNet Challenge uses the top-5 metric: A prediction is correct if the assigned label is among the model’s five guesses. This compensates for the challenge that often multiple plausible labels exist, but it can also lead to distortions. For instance, the prediction {“miniature schnauzer,” “tiger,” “pirate,” “shovel,” “fan”} scores perfectly for an image labeled as “miniature schnauzer,” despite four very implausible guesses. In contrast, the prediction {“standard schnauzer,” “giant schnauzer,” “mailbox,” “street sign,” “Irish terrier”} counts as an error, even though all labels are plausible for an image showing a dog sitting on a street. Multilabel accuracy would better reflect this classification performance and is standard in other benchmarks (Everingham et al. Reference Everingham, Gool, Williams, Winn and Zisserman2009; Everingham et al. Reference Everingham, Ali Eslami, Van Gool, Williams, Winn and Zisserman2014).

4.3. Constraining the inference

In light of this discussion, one must be cautious when linking benchmark scores on ImageNet to absolute expected error rates on image-classification problems more generally. Several of the validity conditions are only partially supported by the available evidence. Although the generalization inference is well supported, adaptivity remains a concern. The extrapolation inference is substantially weakened by the high sensitivity to the specific sampling domain, implying that scores do not reliably transfer across benchmarks. The conceptualization inference is plausible within a certain scope, but the labeling approach and the resulting choice of the top-5 metric introduce important limitations.

Restricting the scope of the interpretation to model rankings allows for a more modest and better supported inference. In particular, the extrapolation inference, which is problematic for absolute scores, can be recovered for rankings because models with higher ImageNet performance also tend to perform better on other image-classification benchmarks. The inference based on rankings is therefore more robust and, as Salaudeen and Hardt (Reference Salaudeen and Hardt2026) argue, sufficient to justify the use of a benchmark like ImageNet for tracking engineering progress in computer vision.

5. Fragile Families Challenge: Are life outcomes predictable?

The FFC, held in 2017, established a predictive benchmark for sociology using data from the Fragile Families and Child Wellbeing Study (Salganik et al. Reference Salganik, Lundberg, Kindel and McLanahan2019). Predictive benchmarks have the potential to serve as powerful evaluation tools in the quantitative social sciences, complementing explanatory in-sample methods (Verhagen Reference Verhagen2022; Shmueli Reference Shmueli2010; Hofman et al. Reference Hofman, Sharma and Watts2017). They allow for direct comparisons between different statistical models (Rocca and Yarkoni Reference Rocca and Yarkoni2021), testing of model assumptions (Buchholz and Grote Reference Buchholz and Grote2023), and investigation of theoretical questions like the predictability of life outcomes. Results from benchmark challenges like the FFC, therefore, have implications for public policy and are of significance for social theory.

The benchmark challenge was to predict at least one out of six individual life outcomes of 15-year-old teenagers, such as their grade point average (GPA), a psychological construct of perseverance called grit, or whether their household experienced material hardship in the past year (Salganik et al. Reference Salganik, Lundberg, Kindel and McLanahan2019). Built on a 15-year longitudinal survey, more than 4,200 families were interviewed six times. In the end, 160 models were submitted, ranging from conventional statistical methods to various machine learning approaches.

The results, however, were sobering: All submitted models performed poorly in predicting the different outcomes. On the reported evaluation metric, where 1 indicates perfect accuracy and 0 corresponds to predictions no better than the respective average in the training data, even the best models only achieved scores between 0.23 $0.23$ for material hardship, 0.19 $0.19$ for GPA, and 0.03 $0.03$ for lay-off.

5.1. The intended inference

Social scientists may aim to infer from the benchmarking challenge that the best scores achieved in the FFC represent an upper bound on the expected error when predicting individual life outcomes. If this inference is valid, the FFC results would suggest that life outcomes are inherently difficult to predict—a fundamental limitation rather than an artifact of the survey data or the available modeling techniques.

This inference is very general, giving it an extremely broad scope that spans all life-outcome prediction tasks. The construct of interest is an upper bound on the generalization error. Linking the performance score from the FFC to this bound requires five intermediate steps. First, error rates on the FFC evaluation dataset must generalize to error rates on FFC-like data (generalization inference). Second, they must extend to error rates on alternative life-outcome benchmarks (extrapolation inference). Third, they should capture error rates across life-outcome prediction tasks more broadly (conceptualization inference). Fourth, they must reflect the upper bound of the generalization error for all such tasks (predictability inference). Lastly, we discuss possible consequences from this inference and potential uses (decision inference).

5.2. Specifying validity conditions and providing relevant evidence

Generalization inference. The generalization inference from the FFC scores is well supported by the careful study design. The final scores are estimated using data from the latest survey wave, which was not publicly available at that time. This ensures independence between the models and the evaluation data.Footnote 11 The challenge data were collected as part of the Future Families and Child Wellbeing Study, a representative survey of nonmarital births in large US cities (Reichman et al. Reference Reichman, Teitler, Garfinkel and McLanahan2001). The main threat to generalization lies in the uncertainty in identifying a “winning” model. When bootstrapping several test sets from the evaluation data, the top-ranked model changes in more than half the cases for GPA and nearly a quarter for children’s grit (Salganik et al. Reference Salganik, Lundberg, Kindel, Ahearn, Khaled Al-Ghoneim, Altschul, Brand, Carnegie, Compton, Datta, Davidson, Filippova, Gilroy, Goode, Jahani, Kashyap, Kirchner, McKay, Morgan, Pentland, Polimis, Raes, Rigobon, Roberts, Stanescu, Suhara, Usmani, Wang, Adem, Alhajri, AlShebli, Amin, Amos, Argyle, Livia Baer-Bositis, Chung, Eggert, Faletto, Fan, Freese, Gadgil, Gagné, Gao, Halpern-Manners, Hashim, Hausen, He, Higuera, Hogan, Horwitz, Hummel, Jain, Jin, Jurgens, Kaminski, Karapetyan, Kim, Leizman, Liu, Möser, Mack, Mahajan, Mandell, Marahrens, Mercado-Garcia, Mocz, Mueller-Gastell, Musse, Niu, Nowak, Omidvar, Or, Ouyang, Pinto, Porter, Porter, Qian, Rauf, Sargsyan, Schaffner, Schnabel, Schonfeld, Sender, Tang, Tsurkov, van Loon, Varol, Wang, Wang, Wang, Wang, Weissman, Whitaker, Wolters, Woon, Wu, Wu, Yang, Yin, Zhao, Zhu, Brooks-Gunn, Engelhardt, Hardt, Knox, Levy, Narayanan, Stewart, Watts and McLanahan2020). Thus, the score rankings partly reflect sampling artifacts rather than genuine performance differences between the models. Nevertheless, changes in the rankings are not a major obstacle to the intended inference because all models performed poorly.

Extrapolation inference. The FFC results align with a broad body of research on the predictability of life outcomes, showing that although these outcomes can often be predicted better than chance, overall predictive performance remains low. This empirically supports the extrapolation inference. The FFC alone covers six different prediction targets, already indicating a certain robustness of the results. Despite slightly higher scores for material hardship and GPA, predictability is low across all targets. Other studies generally find a similar pattern when predicting recidivism, individual-level mortalities, or the risk of long-term unemployment, respectively (Dressel and Farid Reference Dressel and Farid2018; Breen and Seltzer Reference Breen and Seltzer2023; Kuikka Reference Kuikka2024; Kern et al. Reference Kern, Bach, Mautner and Kreuter2021; Desiere and Struyven Reference Desiere and Struyven2020; Desiere et al. Reference Desiere, Langenbucher and Struyven2019). Zheng and Cheng (Reference Zheng and Cheng2025) report more mixed results specifically for individuals in their 40s: Although log total income and employment status were poorly predictable, log hourly wages reached a substantially higher score of 0.53 $0.53$ on the same metric as used in the FFC. Perdomo et al. (Reference Perdomo, Britton, Hardt and Abebe2025) also report high accuracy rates for student dropout prediction. Building on extensive administrative labor and health records from Denmark, Savcisens et al. (Reference Savcisens, Tina Eliassi-Rad, Mortensen, Lilleholt, Rogers, Zettler and Lehmann2023) recently trained a transformer model to predict individual mortality. Although the model outperforms their simple baseline models, it also only achieves an accuracy of about 78 percent sign $78 \%$ .

Conceptualization inference. The conceptualization inference is generally well supported by the close connection of the FFC to social science theory. The data are drawn from the Fragile Families and Child Wellbeing Study, a well-documented survey with extensive theoretical grounding (Reichman et al. Reference Reichman, Teitler, Garfinkel and McLanahan2001). The prediction targets cover different types of life outcomes, and their operationalizations are established in the field. Nevertheless, life outcomes are a highly heterogeneous type, and the predictability of any specific one may well vary with its scope and specific cultural and societal conditions. Studying the predictability of life outcomes therefore requires making explicit the conditions under which sufficient stability can be expected (Liou et al. Reference Liou, Bailey, Baldwin, Duckworth and Tay2023; Northcott Reference Northcott2022).

A central motivation of the survey underlying the FFC was to examine the role and impact of unwed fathers on child well-being, a group that is often underrepresented in surveys because such fathers are harder to identify (Reichman et al. Reference Reichman, Teitler, Garfinkel and McLanahan2001). This focus on so-called “fragile families” in urban contexts introduces a systematic bias in its data coverage: On average, participating families are poorer and more marginalized than the general US population and face more uncertainties. It is also to be expected that predicting outcomes at the age of 15 is particularly challenging because adolescence is a rather volatile period of life. In contrast, outcomes in later adulthood are considered more predictable because many life trajectories are more stable at this stage (Zheng and Cheng Reference Zheng and Cheng2025). The evaluation metric underlying the reported normalized out-of-sample upper R squared $R^2$ scores is the mean squared error (MSE), which disproportionately penalizes large deviations. The choice reflects the research interest of identifying families with “particularly unexpected outcomes” (Salganik et al. Reference Salganik, Lundberg, Kindel, Ahearn, Khaled Al-Ghoneim, Altschul, Brand, Carnegie, Compton, Datta, Davidson, Filippova, Gilroy, Goode, Jahani, Kashyap, Kirchner, McKay, Morgan, Pentland, Polimis, Raes, Rigobon, Roberts, Stanescu, Suhara, Usmani, Wang, Adem, Alhajri, AlShebli, Amin, Amos, Argyle, Livia Baer-Bositis, Chung, Eggert, Faletto, Fan, Freese, Gadgil, Gagné, Gao, Halpern-Manners, Hashim, Hausen, He, Higuera, Hogan, Horwitz, Hummel, Jain, Jin, Jurgens, Kaminski, Karapetyan, Kim, Leizman, Liu, Möser, Mack, Mahajan, Mandell, Marahrens, Mercado-Garcia, Mocz, Mueller-Gastell, Musse, Niu, Nowak, Omidvar, Or, Ouyang, Pinto, Porter, Porter, Qian, Rauf, Sargsyan, Schaffner, Schnabel, Schonfeld, Sender, Tang, Tsurkov, van Loon, Varol, Wang, Wang, Wang, Wang, Weissman, Whitaker, Wolters, Woon, Wu, Wu, Yang, Yin, Zhao, Zhu, Brooks-Gunn, Engelhardt, Hardt, Knox, Levy, Narayanan, Stewart, Watts and McLanahan2020).

Predictability inference. The intended inference does not only pertain to the prediction errors of a specific set of models but rather to a general limit on the predictability of life outcomes, and thus to a property of the target phenomenon itself. Establishing such a predictability inference requires two validity conditions:

  1. (i) The set of evaluated models must be sufficiently diverse and the data sufficiently large for the best-performing models to approximate the Bayes-optimal model.

  2. (ii) The observed low predictability must not be driven by unmeasured features or the operationalization of the target variable.

The conditions rule out alternative explanations for the low predictive performance observed in the FFC. The first ensures that limited performance cannot be attributed to a restricted set of modeling approaches unable to capture the relationship between features and outcomes or to insufficient data for model training. Under this condition, the best-performing models should approximate the theoretically achievable optimum given the available features and targets, namely, the Bayes-optimal model (Hastie et al. Reference Hastie, Tibshirani and Friedman2009). If life outcomes are fundamentally unpredictable, the error of the Bayes-optimal model is high, which means that all predictors perform poorly. However, the Bayes-optimal model may perform poorly for different reasons. The second condition ensures that the poor performance is indeed due to a high irreducible error or high stochasticity in the prediction target, not to uninformative or unmeasured features or a misspecified target variable. With 160 teams independently submitting models based on diverse modeling approaches, the FFC takes a substantial step in this direction. The submissions cover the relevant modeling approaches available at the time of the challenge.Footnote 12 Nevertheless, the weak performance of machine learning methods could also be explained by the relatively small size of the training data relative to the large number of features. A total of 4,242 families, split into training, test, and evaluation sets, is a large cohort for a longitudinal survey, considering the costs of a 15-year study. Yet the dataset remains small relative to its 12,942 features. This puts limitations on the intended interpretation as providing bounds on predictability.

The poor predictive performance may also result from inaccurate or incomplete data, a possibility investigated by Lundberg et al. (Reference Lundberg, Rachel Brown-Weinstock, Pachman, Nelson, Yang, Edin and Salganik2024). To identify limitations that contribute to the prediction errors beyond the sample size, they interviewed 40 children and their families from the FFC, focusing on teenagers whose GPA predictions were inaccurate. From the interviews, they identified three important sources of errors in the FFC: first, imperfectly measured features, such as coarse ones that hide important variation, or recording errors; second, unrecorded features (e.g., one child’s higher-than-predicted GPA was explained by a strong social network outside her immediate family, a factor not accounted for in the survey); third, the presence of unmeasurable features, such as events that occurred after the survey was conducted.

Decision inference. Validity considerations extend beyond the interpretation of benchmark scores and may also include decision inferences, which combine the scores with a practical decision rule. This rule specifies conditions under which a particular course of action is chosen. To illustrate this, model selection based on weather forecasting benchmarks like WeatherBench (Rasp et al. Reference Rasp, Dueben, Scher, Weyn, Mouatadid and Thuerey2020, Reference Rasp, Hoyer, Merose, Langmore, Battaglia, Russell, Alvaro Sanchez-Gonzalez, Carver, Agrawal, Chantry, Bouallegue, Dueben, Bromberg, Sisk, Barrington, Bell and Sha2024) is embedded in high-stakes decision-making: Forecasts guide critical decisions, from issuing early warnings for extreme weather to planning energy market operations. Energy providers rely on weather models to forecast solar radiation and wind speeds in order to coordinate renewable energy supply with backup fossil fuel systems. The three conditions for validating such decision inferences, outlined in Kane (Reference Kane2013, 48), are as follows:

  1. (i) The decision rule reflects the utilities of a course of action for the intended goal.

  2. (ii) The decision rule is fair across relevant population subgroups.

  3. (iii) The application of the decision rule does not produce undesirable systemic effects compared with alternative rules.Footnote 13

The conditions jointly support the validity of the decision rule and, provided the score interpretation is sound, the resulting decision inference. Importantly, interpreting a score and using it for decision-making are not the same—neither does a valid score interpretation alone justify its use for decision-making, nor does rejecting the application in a given context necessarily undermine the interpretation. Consistent with this distinction, we agree with Kane (Reference Kane2013) that the evaluation of the consequences of both interpreting and using benchmark scores is an essential component of a comprehensive and responsible validity analysis. The consequences of interpreting and using benchmark scores must inform the evidential standards applied.

Predictions of life outcomes similar to those in the FFC regularly inform public policy decisions and the allocation of resources (Perdomo et al. Reference Perdomo, Britton, Hardt and Abebe2025; Zezulka and Genin Reference Zezulka and Genin2024). These algorithmic policies often result in differential treatments across social groups (Corbett-Davies et al. Reference Corbett-Davies, Gaebler, Nilforoshan, Shroff and Goel2023; Berk et al. Reference Berk, Kuchibhotla and Tchetgen2023). The three conditions just listed provide general guidelines for validation of such decision inferences.

The intended inference is about a knowledge claim rather than a decision. Nevertheless, such an interpretation can have indirect consequences by influencing future research and policy. Intended and unintended consequences thus arise primarily from the inductive risk associated with the scientific inference drawn (Rudner Reference Rudner1953; Douglas Reference Douglas2000). If we were to mistakenly conclude that life outcomes are only weakly predictable (as the scores suggest), this could impose social costs in the form of less effective and consistent policy interventions (Kleinberg et al. Reference Kleinberg, Lakkaraju, Leskovec, Ludwig and Mullainathan2017). Conversely, if we erroneously conclude that life outcomes are highly predictable, this could lead to inflated expectations of algorithmic policies, harming potential beneficiaries and misallocating (public) resources. The FFC results thus bear directly on the feasibility and legitimacy of algorithmic governance (Wang et al. Reference Wang, Kapoor, Barocas and Narayanan2024). Weighing these competing risks inevitably involves value judgments. Given the careful design of the challenge and the balance of potential costs and benefits, it seems reasonable to treat the results as an important caveat to the development of algorithmic policies and, importantly, to expect low average accuracy and large subgroups for which the models consistently perform poorly.

5.3. Constraining the inference

Despite uniformly low scores of all models across the six FFC targets, the results do not support the strong intended inference that life outcomes are inherently unpredictable. Although the generalization inference is well supported, the extrapolation inference is less stable because some life outcomes appear to be substantially more predictable. Similarly, the conceptualization inference is fragile, given the heterogeneity of a construct such as “life outcomes.” The predictability conditions, particularly data coverage and sample size, are especially critical: With larger datasets and more informative features, some life outcomes may, after all, prove to be more predictable.

In light of the validity warrants discussed earlier, a more constrained inference is better supported. The benchmark scores and the validity conditions support the inference that although many individual life outcomes can be predicted better than chance, the overall predictive performance on the data typically available is very likely to be low. This holds particularly for young and marginalized social groups. The challenge also provides evidence that simple models using only a few variables often are competitive with more complex ones.

6. Towards a benchmarking epistemology

Predictive benchmarks are powerful methods for the empirical sciences, as the two case studies show. Benchmark scores, together with relevant conditions of validity, can support a wide range of substantial inferences, from tracking research progress in machine learning to assessing the predictability of life events. Drawing on the analogy with psychological testing, we have argued that any such inference requires the explicit specification and evaluation of its validity conditions. The four-step procedure provides a practical guide for interpreting and using benchmark scores.

Analyzing benchmarking as a measurement procedure from a validity-theoretic perspective is valuable beyond these specific inferences. Following, we argue that this framework provides (1) a unifying language for conceptually dispersed discussions in the benchmarking literature, (2) a theoretical grounding for the evaluation of the capabilities of LLMs, (3) insights into the social epistemology of benchmarks, and (4) a clear framework to address the limits of benchmarking.

6.1. Unifying language for benchmarking problems

Researchers in machine learning have raised several concerns about the practice of benchmarking. However, the community still lacks a unifying language for articulating problems with benchmarks and the inferences drawn from benchmarking scores. Validity theory provides exactly such a language.

Consider, for example, the influential discussion in computer vision on the problem of adaptivity, the sequential reuse of evaluation data over time, which tends to yield inconsistent scores (Blum and Hardt Reference Blum and Hardt2015; Yadav and Bottou Reference Yadav, Bottou, Wallach, Larochelle, Beygelzimer, d’Alché-Buc, Fox and Garnett2019; Recht et al. Reference Recht, Roelofs, Schmidt, Shankar, Chaudhuri and Salakhutdinov2019). Within validity theory, this is a problem of generalization: Adaptivity violates the second generalization condition—namely, that the evaluation data and the tested model must be independent. The significance of adaptivity becomes clearer from an argument-based perspective because generalization underpins most scientific inferences drawn from benchmark scores. Responses to adaptivity likewise reflect the structure of the argument-based account: Either they weaken the intended inference, shifting from generalization claims about scores to claims about ordinal rankings (Recht et al. Reference Recht, Roelofs, Schmidt, Shankar, Chaudhuri and Salakhutdinov2019; Salaudeen and Hardt Reference Salaudeen and Hardt2026), or they propose weaker validity conditions that nonetheless suffice to support the original generalization inference (Dwork et al. Reference Dwork, Feldman, Hardt, Pitassi, Reingold, Roth, Cortes, Lawrence, Lee, Sugiyama and Garnett2015; Hardt Reference Hardt2026).

Others have also adopted a validity-theoretic perspective on benchmarks. For instance, Yee (Reference Yee2024) examines construct legitimacy, criterion validity, and construct validity in evaluating machine learning models in the context of counterterrorism. Recently, Salaudeen et al. (Reference Salaudeen, Reuel, Ahmed, Bedi, Robertson, Sundar, Domingue, Wang and Koyejo2025) propose a “claim-centred framework” for evaluating general-purpose AI systems that is closely related to our approach. The attribution of capabilities to AI systems on the basis of benchmark scores has spurred growing interest in psychometric testing and validity (Wang et al. Reference Wang, Jiang, Jose Hernandez-Orallo, Chen, Sun, Luo and Xie2026; Sühr et al. Reference Sühr, Dorner, Salaudeen, Kelava and Samadi2025; Ye et al. Reference Ye, Jin, Xie, Zhang and Song2025).

6.2. Guiding LLM capability testing

With the development of generative AI and LLMs, the role of benchmarks in machine learning has changed. Although these models are typically trained as next-token predictors, some researchers aim to attribute capabilities to them and have begun to develop benchmarks to draw inferences about such latent constructs. For example, benchmarks have been designed to test whether models have capabilities like a stable sense of morality (Jiang et al. Reference Jiang, Hwang, Bhagavatula, Bras, Liang, Dodge, Sakaguchi, Forbes, Borchardt, Gabriel, Tsvetkov, Etzioni, Sap, Rini and Choi2021), a theory of mind (Strachan et al. Reference Strachan, Albergo, Borghini, Pansardi, Scaliti, Gupta, Saxena, Rufo, Panzeri, Manzi, Graziano and Becchio2024), or legal reasoning (Masry et al. Reference Masry, Long, Tan, Joty and Hoque2022). Attributing such capabilities to LLMs has been widely criticized both for anthropomorphizing machine learning models (Shanahan Reference Shanahan2024; Sypherd et al. Reference Sypherd, Tang, Belle, Jacquet, Brochier and Yama2025) and particularly for lacking a sound methodology (Shapira et al. Reference Shapira, Levy, Alavi, Zhou, Choi, Goldberg, Sap and Shwartz2024; Zhuang et al. Reference Zhuang, Liu, Pardos, Kyllonen, Zu, Huang, Wang, Chen, Singh, Fazel, Hsu, Lacoste-Julien, Berkenkamp, Maharaj, Wagstaff and Zhu2025).

Although we agree that the anthropomorphization of machine learning tools carries risks, we also believe that LLM capability tests can serve an important inferential role. LLMs are assessed as general-purpose models; however, users are typically concerned with their usefulness for specific downstream tasks. Thus, capability scores must provide evaluation at a level of abstraction that is informative for a wide range of tasks while remaining feasible to implement. This makes the lack of methodological standards highly problematic. Most of the targeted capabilities cannot be straightforwardly operationalized in a benchmark; they are what Kane (Reference Kane2013) calls theoretical properties, instead of measurable attributes.Footnote 14 Validity theory in turn was explicitly developed to infer latent constructs from measurable test scores. For this reason, it can guide LLM capability testing in establishing evaluation and inferential standards.

Cronbach and Meehl (Reference Cronbach and Meehl1955), Whitely (Reference Whitely1983), and Borsboom et al. (Reference Borsboom, Mellenbergh and van Heerden2003) have argued for different approaches to developing theories about latent constructs, using nomological networks, process theories, or structural causal models, respectively. The argument-based account of validity also allows for the integration of such theories and supports inferences about latent constructs. The validity conditions of such theorization inferences from Kane (Reference Kane2013) can be fruitfully adopted when reasoning about LLM capabilities. In addition to the generalization, extrapolation, and conceptualization conditions, they require (i) a theory that explicates the latent construct, explains its relationship to other such constructs, and provides a justification of the plausibility of the theory, and (ii) evaluation data that includes samples representing the measurable attributes that the theory links to the construct of interest.

Decades of theorizing human cognition demonstrate that such conditions are more easily stated than met (McGrew Reference McGrew, Dawn and Patti2005). First, it remains elusive how LLM “cognition” operates and whether it can be meaningfully compared to human cognition, making such transfers particularly challenging (Shanahan Reference Shanahan2024). Second, even if suitable theoretical models were available, they would need to be rigorously tested before supporting any inference. In particular, they should undergo convergent and discriminant analyses to ensure that the theory coheres with empirical results (Cronbach and Meehl Reference Cronbach and Meehl1955; Messick Reference Messick1989; Strauss and Smith Reference Strauss and Smith2009). Research on human testing highlights the difficulties in designing adequate assessment tests that are broad enough to capture the behavioral implications of a latent construct without becoming too encompassing. Tests are constantly at risk of either failing to represent certain aspects of the construct, called construct underrepresentation, or being influenced by factors irrelevant to the construct, referred to as construct irrelevance (Strauss and Smith Reference Strauss and Smith2009). Construct irrelevance, in particular, is difficult to detect, because depending on the external factors included, it may either facilitate or hinder test performance (Messick Reference Messick1989).

In agreement with Sühr et al. (Reference Sühr, Dorner, Salaudeen, Kelava and Samadi2025), we believe that LLM researchers should not simply apply tests designed for humans to LLMs. At the same time, we are also convinced that methodological transfers like the one we propose here from psychological testing to assessing LLM capabilities can be very fruitful.

6.3. Social dimensions of benchmarking

So far, we used validity theory to analyze the epistemic role of predictive benchmarks in drawing inferences, largely leaving aside their important social role in structuring research communities. How do benchmarks shape research practices and communities? And who holds the power to establish benchmarks within a community?

These questions have begun to receive some attention. Raising concerns about biased data and task selection, Koch et al. (Reference Koch, Denton, Hanna, Foster, Vanschoren and Yeung2021) show that research is increasingly concentrated on a small set of benchmark datasets, often repurposed from other contexts and provided by a few powerful institutions. Dehghani et al. (Reference Dehghani, Tay, Gritsenko, Zhao, Houlsby, Diaz, Metzler and Vinyals2021) argue that when a single benchmark dominates a field of research, this favors methods that are already well aligned with the benchmark’s idiosyncrasies. Conversely, where multiple benchmarks coexist, researchers may cherry-pick those on which their models happen to perform best. These results point to a fragility of benchmark-based evaluation, which is especially serious when model assessment is reduced to benchmarking. Hooker (Reference Hooker1995) further cautions that framing scientific research as a competition promotes a race for marginal numerical improvements at the expense of more substantive progress.

Although validity theory cannot replace the development of a substantial social epistemology of benchmarking, it draws attention to the relations between the curators of a benchmark and those intending to draw inferences from it. Typically, machine learning practitioners and scientists employ benchmarks they did not curate themselves. As a consequence, many of the validity conditions discussed earlier, including generalization and extrapolation conditions, are outside their control. This underscores the need for benchmark curators to maintain close communication with benchmark users, ensuring that benchmarks are designed to support relevant inferences and that underlying design choices are clearly communicated in the corresponding publications. The social dimension of benchmarking highlights challenges for drawing valid inference from benchmark scores. Any comprehensive validity theory of benchmarking will have to address these challenges in future work.

6.4. Limits of the benchmarking epistemology

Predictive benchmarks were originally developed to shift the focus of evaluation away from broad claims about artificial general intelligence and toward measuring performance on concrete and narrowly defined learning problems. Ironically, with the advent of modern LLMs, researchers have returned to making such claims about general capabilities—this time on the basis of benchmark scores. This raises the question of the limits of benchmarks: What can be meaningfully measured by and inferred from them, and what cannot?

Validity theory, in particular the argument-based account, teaches an important lesson here: The more abstract and complex the intended inferences become, the stronger must the corresponding validity conditions in the interpretation or use argument be (Kane Reference Kane2013). Stronger validity conditions impose higher demands on supporting evidence while also creating more opportunities for potential undermining evidence. As a result, such inferences will often fail because some of the validity conditions are only partially supported or even undermined. For strong theoretical inferences—which are, of course, of particular interest in the empirical sciences—these failure modes can often be traced to benchmarks “thinning the world” by focusing on easily quantifiable aspects of a phenomenon, underrepresenting the construct of interest, or capturing irrelevant features of it (Raji et al. Reference Raji, Denton, Bender, Hanna, Paullada, Vanschoren and Yeung2021; Denton et al. Reference Denton, Hanna, Amironesei, Smart and Nicole2021; Orr and Kang Reference Orr and Kang2024). This danger becomes particularly acute when benchmarks come to function as authoritative proxies for entire tasks, as ImageNet once did for image classification, gaining legitimacy through widespread adoption within the research community (Orr and Crawford Reference Orr and Crawford2024).

On first sight, benchmarking may appear to be the tool to establish the mystical ideal of a theory-free science—a science devoid of human theoretical preconceptions and shaped only by empirical data and predictions (Andrews Reference Andrews2025). However, the critical evaluation of benchmarking as a measurement procedure demonstrates the importance of a validity theory and makes clear that theory, for example, in the form of validity conditions, remains indispensable for drawing inferences from benchmark scores. In other words, predictive benchmarking is a key site of conceptual and theoretical reasoning in machine learning, and the limits of the benchmarking epistemology are not determined by the benchmarks themselves but by the strength and justification of the inferences we draw from them.

Acknowledgments

We thank Konstantin Genin, Jan-Willem Romeijn, Wolfgang Spohn, Bob Williamson, Tom Sterkenburg, Thomas Grote, Markus Ahlers, Katharina Holzhey, Kino Zhao, Moritz Hardt, Luis Winckelmann, Tim Schreier, and Alex Braun for helpful discussions and valuable feedback. We are also grateful to the two anonymous reviewers for their careful reading and insightful comments, which substantially improved the article.

Funding Statement

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG; German Research Foundation) through the Emmy Noether Research Group “From Bias to Knowledge: The Epistemology of Machine Learning” (project number 511917847) and under Germany’s Excellence Strategy, EXC 2064/1 (project number 390727645). The authors acknowledge support for Timo Freiesleben from the Carl Zeiss Stiftung (Project: “Certification and Foundations of Safe Machine Learning Systems in Healthcare”) and the Munich Center for Machine Learning. They also thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Sebastian Zezulka.

Declarations

None to declare.

Footnotes

1 The Common Task Framework echoed ideas introduced as early as 1959, when Bill Highleyman outlined many of the principles in his PhD thesis on handwritten character recognition (Orr and Crawford Reference Orr and Crawford2024). Highleyman also shared his dataset with other researchers—by mail.

2 Recent benchmarks like Super General Language Understanding Evaluation (SuperGLUE) (Wang et al. Reference Wang, Pruksachatkun, Nangia, Singh, Michael, Hill, Levy, Bowman, Wallach, Larochelle, Beygelzimer, d’Alché-Buc, Fox and Garnett2019) or Holistic Evaluation of Language Models (HELM) (Bommasani et al. Reference Bommasani, Liang and Lee2023) often encompass several learning problems, datasets, and metrics. This raises additional questions about the aggregation of scores (Zhang and Hardt Reference Zhang, Hardt, Salakhutdinov, Kolter, Heller, Weller, Oliver, Scarlett and Berkenkamp2024).

3 Hardt and Recht (Reference Hardt and Recht2022), chap. 8, Paullada et al. (Reference Paullada, Raji, Bender, Denton and Hanna2021), and Koch and Peterson (Reference Koch and Peterson2024) discuss the historical development of predictive benchmarks.

4 The CASP challenge, dating back to 1994, is among the oldest scientific benchmark challenges (Donoho Reference Donoho2024).

5 Mussgnug (Reference Mussgnug2022) analyzes the output of machine learning models as measurements, rather than predictive benchmarks.

6 The evaluation metric does not need to coincide with the loss function used to train the model, nor must the model be trained for the learning problem on which it is evaluated.

7 For a discussion on the interpretation of statistical assumptions, see Zhao (Reference Zhao2025).

8 Known as ILSVRC, the ImageNet Large Scale Visual Recognition Challenge.

9 Several studies have since examined the empirical impact of this information leakage for other benchmarks (Yadav and Bottou Reference Yadav, Bottou, Wallach, Larochelle, Beygelzimer, d’Alché-Buc, Fox and Garnett2019; Roelofs et al. Reference Roelofs, Shankar, Recht, Sara Fridovich-Keil, Miller, Schmidt, Wallach, Larochelle, Beygelzimer, d’Alché-Buc, Fox and Garnett2019; Miller et al. Reference Miller, Krauth, Recht, Schmidt, Daumé and Singh2020).

10 With minor manual postprocessing to exclude obscure classes (Russakovsky et al. Reference Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg and Fei-Fei2015).

11 The reported scores may still be overestimated as a result of the finite sample effects: Estimates that correct for this indeed are slightly smaller, but also less precise. Overall, they show patterns similar to those of the original results (Salganik et al. Reference Salganik, Lundberg, Kindel, Ahearn, Khaled Al-Ghoneim, Altschul, Brand, Carnegie, Compton, Datta, Davidson, Filippova, Gilroy, Goode, Jahani, Kashyap, Kirchner, McKay, Morgan, Pentland, Polimis, Raes, Rigobon, Roberts, Stanescu, Suhara, Usmani, Wang, Adem, Alhajri, AlShebli, Amin, Amos, Argyle, Livia Baer-Bositis, Chung, Eggert, Faletto, Fan, Freese, Gadgil, Gagné, Gao, Halpern-Manners, Hashim, Hausen, He, Higuera, Hogan, Horwitz, Hummel, Jain, Jin, Jurgens, Kaminski, Karapetyan, Kim, Leizman, Liu, Möser, Mack, Mahajan, Mandell, Marahrens, Mercado-Garcia, Mocz, Mueller-Gastell, Musse, Niu, Nowak, Omidvar, Or, Ouyang, Pinto, Porter, Porter, Qian, Rauf, Sargsyan, Schaffner, Schnabel, Schonfeld, Sender, Tang, Tsurkov, van Loon, Varol, Wang, Wang, Wang, Wang, Weissman, Whitaker, Wolters, Woon, Wu, Wu, Yang, Yin, Zhao, Zhu, Brooks-Gunn, Engelhardt, Hardt, Knox, Levy, Narayanan, Stewart, Watts and McLanahan2020).

12 Details on all submissions are provided in the appendix of Salganik et al. (Reference Salganik, Lundberg, Kindel, Ahearn, Khaled Al-Ghoneim, Altschul, Brand, Carnegie, Compton, Datta, Davidson, Filippova, Gilroy, Goode, Jahani, Kashyap, Kirchner, McKay, Morgan, Pentland, Polimis, Raes, Rigobon, Roberts, Stanescu, Suhara, Usmani, Wang, Adem, Alhajri, AlShebli, Amin, Amos, Argyle, Livia Baer-Bositis, Chung, Eggert, Faletto, Fan, Freese, Gadgil, Gagné, Gao, Halpern-Manners, Hashim, Hausen, He, Higuera, Hogan, Horwitz, Hummel, Jain, Jin, Jurgens, Kaminski, Karapetyan, Kim, Leizman, Liu, Möser, Mack, Mahajan, Mandell, Marahrens, Mercado-Garcia, Mocz, Mueller-Gastell, Musse, Niu, Nowak, Omidvar, Or, Ouyang, Pinto, Porter, Porter, Qian, Rauf, Sargsyan, Schaffner, Schnabel, Schonfeld, Sender, Tang, Tsurkov, van Loon, Varol, Wang, Wang, Wang, Wang, Weissman, Whitaker, Wolters, Woon, Wu, Wu, Yang, Yin, Zhao, Zhu, Brooks-Gunn, Engelhardt, Hardt, Knox, Levy, Narayanan, Stewart, Watts and McLanahan2020).

13 For example, in educational settings, standardized testing may influence what is taught in ways that extend beyond its intended role in performance assessment.

14 Raji et al. (Reference Raji, Denton, Bender, Hanna, Paullada, Vanschoren and Yeung2021) argue that only well-specified intensional tasks can be faithfully operationalized by benchmarks. Many general-purpose language benchmarks, such as SuperGLUE, may be seen to violate this principle (Dorner et al. Reference Dorner, Sühr, Samadi and Kelava2023; Millière and Buckner Reference Millière and Buckner2024; Zhang and Hardt Reference Zhang, Hardt, Salakhutdinov, Kolter, Heller, Weller, Oliver, Scarlett and Berkenkamp2024; Grote et al. Reference Grote, Freiesleben and Berens2024a).

References

Alexandrova, Anna, and Haybron, Daniel M.. 2016. “Is Construct Validation Valid?” Philosophy of Science 83 (5):1098–109. https://doi.org/10.1086/687941.CrossRefGoogle Scholar
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. 2014. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.Google Scholar
Andrews, Donald W. K. 1988. “Laws of Large Numbers for Dependent Non-Identically Distributed Random Variables.” Econometric Theory 4 (3):458–67. https://doi.org/10.1017/s0266466600013396.CrossRefGoogle Scholar
Andrews, Mel. 2025. “The Immortal Science of ML: Machine Learning & the Theory-Free Ideal.” Erkenntnis 91 (6):2685–707. https://doi.org/10.1007/s10670-025-01010-x.CrossRefGoogle Scholar
Berk, Richard A., Kuchibhotla, Arun Kumar, and Tchetgen, Eric. 2023. “Fair Risk Algorithms.” Annual Review of Statistics and Its Application 10 (1):165–87. https://doi.org/10.1146/annurev-statistics-033021-120649.CrossRefGoogle Scholar
Beyer, Lucas, Hénaff, Olivier J., Kolesnikov, Alexander, Zhai, Xiaohua, and van den Oord, Aäron. 2020. “Are We Done with ImageNet?” arXiv. https://doi.org/10.48550/arXiv.2006.07159.CrossRefGoogle Scholar
Blili-Hamelin, Borhane, and Hancox-Li, Leif. 2023. “Making Intelligence: Ethical Values in IQ and ML Benchmarks.” In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 271–84. New York: Association for Computing Machinery. https://doi.org/10.1145/3593013.3593996.CrossRefGoogle Scholar
Blum, Avrim, and Hardt, Moritz. 2015. “The Ladder: A Reliable Leaderboard for Machine Learning Competitions.” In Proceedings of the 32nd International Conference on Machine Learning, vol. 37, 1006–14. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Boge, Florian J. 2021. “Two Dimensions of Opacity and the Deep Learning Predicament.” Minds and Machines 32 (1):43–75. https://doi.org/10.1007/s11023-021-09569-4.CrossRefGoogle Scholar
Bogen, James, and Woodward, James. 1988. “Saving the Phenomena.” The Philosophical Review 97 (3):303–52. https://doi.org/10.2307/2185445.CrossRefGoogle Scholar
Bommasani, Rishi, Liang, Percy, and Lee, Tony. 2023. “Holistic Evaluation of Language Models.” Annals of the New York Academy of Sciences 1525 (1):140–46. https://doi.org/10.1111/nyas.15007.CrossRefGoogle ScholarPubMed
Borsboom, Denny, Mellenbergh, Gideon J., and van Heerden, Jaap. 2003. “The Theoretical Status of Latent Variables.” Psychological Review 110 (2):203–19. https://doi.org/10.1037/0033-295x.110.2.203.CrossRefGoogle ScholarPubMed
Borsboom, Denny, Mellenbergh, Gideon J., and van Heerden, Jaap. 2004. “The Concept of Validity.” Psychological Review 111 (4):1061–71. https://doi.org/10.1037/0033-295x.111.4.1061.CrossRefGoogle ScholarPubMed
Breen, Casey, and Seltzer, Nathan. 2023. “Demographic Perspectives on Predicting Individual-Level Mortality.” SocArXiv. https://doi.org/10.31235/osf.io/znsqg.CrossRefGoogle Scholar
Buchholz, Oliver, and Grote, Thomas. 2023. “Predicting and Explaining with Machine Learning Models: Social Science as a Touchstone.” Studies in History and Philosophy of Science 102:60–69. https://doi.org/10.1016/j.shpsa.2023.10.004.CrossRefGoogle ScholarPubMed
Cardoso, Lucas F. F., Santos, Vitor C. A., Kawasaki Francês, Regiane S., Prudêncio, Ricardo B. C., and Alves, Ronnie C. O.. 2020. “Decoding Machine Learning Benchmarks.” In Intelligent Systems, edited by Cerri, Ricardo and Prati, Ronaldo C., 412–25. Springer International. https://doi.org/10.1007/978-3-030-61380-8_28.CrossRefGoogle Scholar
Cave, Stephen. 2020. “The Problem with Intelligence: Its Value-Laden History and the Future of AI.” In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 29–35. New York: Association for Computing Machinery. https://doi.org/10.1145/3375627.3375813.CrossRefGoogle Scholar
Corbett-Davies, Sam, Gaebler, Johann D., Nilforoshan, Hamed, Shroff, Ravi, and Goel, Sharad. 2023. “The Measure and Mismeasure of Fairness.” Journal of Machine Learning Research 24 (312):1–117.Google Scholar
Creel, Kathleen A. 2020. “Transparency in Complex Computational Systems.” Philosophy of Science 87 (4):568–89. https://doi.org/10.1086/709729.CrossRefGoogle Scholar
Cronbach, Lee J., and Meehl, Paul E.. 1955. “Construct Validity in Psychological Tests.” Psychological Bulletin 52 (4):281–302. https://doi.org/10.1037/h0040957.CrossRefGoogle ScholarPubMed
Dehghani, Mostafa, Tay, Yi, Gritsenko, Alexey A., Zhao, Zhe, Houlsby, Neil, Diaz, Fernando, Metzler, Donald, and Vinyals, Oriol. 2021. “The Benchmark Lottery.” arXiv. https://doi.org/10.48550/arXiv.2107.07002.CrossRefGoogle Scholar
Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Kai, Li, and Li, Fei-Fei. 2009. “ImageNet: A Large-Scale Hierarchical Image Database.” In IEEE Conference on Computer Vision and Pattern Recognition, 248–55. New York: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/cvpr.2009.5206848.Google Scholar
Denton, Emily, Hanna, Alex, Amironesei, Razvan, Smart, Andrew, and Nicole, Hilary. 2021. “On the Genealogy of Machine Learning Datasets: A Critical History of ImageNet.” Big Data & Society 8 (2):1–14. https://doi.org/10.1177/20539517211035955.CrossRefGoogle Scholar
Desiere, Sam, Langenbucher, Kristine, and Struyven, Ludo. 2019. “Statistical Profiling in Public Employment Services.” OECD Social, Employment and Migration Working Papers. https://doi.org/10.1787/b5e5f16e-en.CrossRefGoogle Scholar
Desiere, Sam, and Struyven, Ludo. 2020. “Using Artificial Intelligence to Classify Jobseekers: The Accuracy-Equity Trade-Off.” Journal of Social Policy 50 (2):367–85. https://doi.org/10.1017/s0047279420000203.CrossRefGoogle Scholar
Donoho, David. 2017. “50 Years of Data Science.” Journal of Computational and Graphical Statistics 26 (4):745–66. https://doi.org/10.1080/10618600.2017.1384734.CrossRefGoogle Scholar
Donoho, David. 2024. “Data Science at the Singularity.” Harvard Data Science Review 6 (1). https://doi.org/10.1162/99608f92.b91339ef.Google Scholar
Dorner, Florian, Sühr, Tom, Samadi, Samira, and Kelava, Augustin. 2023. “Do Personality Tests Generalize to Large Language Models?” In Socially Responsible Language Modelling Research (SoLaR) Workshop at NeurIPS 2023. Amherst, MA: OpenReview. https://openreview.net/forum?id=zKDSfGhCoK.Google Scholar
Douglas, Heather. 2000. “Inductive Risk and Values in Science.” Philosophy of Science 67 (4):559–79. https://doi.org/10.1086/392855.CrossRefGoogle Scholar
Dressel, Julia, and Farid, Hany. 2018. “The Accuracy, Fairness, and Limits of Predicting Recidivism.” Science Advances 4 (1):eaao5580. https://doi.org/10.1126/sciadv.aao5580.CrossRefGoogle ScholarPubMed
Duede, Eamon. 2023. “Deep Learning Opacity in Scientific Discovery.” Philosophy of Science 90 (5):1089–99. https://doi.org/10.1017/psa.2023.8.CrossRefGoogle Scholar
Dunn, Alexander, Wang, Qi, Ganose, Alex, Dopp, Daniel, and Jain, Anubhav. 2020. “Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm.” npj Computational Materials 6 (1): article 138. https://doi.org/10.1038/s41524-020-00406-3.CrossRefGoogle Scholar
Dwork, Cynthia, Feldman, Vitaly, Hardt, Moritz, Pitassi, Toni, Reingold, Omer, and Roth, Aaron. 2015. “Generalization in Adaptive Data Analysis and Holdout Reuse.” In Advances in Neural Information Processing Systems, edited by Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R., vol. 28, 2350–58. Red Hook, NY: Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2015/file/bad5f33780c42f2588878a9d07405083-Paper.pdf.Google Scholar
Everingham, Mark, Ali Eslami, S. M., Van Gool, Luc, Williams, Christopher K. I., Winn, John, and Zisserman, Andrew. 2014. “The Pascal Visual Object Classes Challenge: A Retrospective.” International Journal of Computer Vision 111 (1):98–136. https://doi.org/10.1007/s11263-014-0733-5.CrossRefGoogle Scholar
Everingham, Mark, Gool, Luc Van, Williams, Christopher K. I., Winn, John, and Zisserman, Andrew. 2009. “The Pascal Visual Object Classes (VOC) Challenge.” International Journal of Computer Vision 88 (2):303–38. https://doi.org/10.1007/s11263-009-0275-4.CrossRefGoogle Scholar
Freiesleben, Timo, and Grote, Thomas. 2023. “Beyond Generalization: A Theory of Robustness in Machine Learning.” Synthese 202 (4):109. https://doi.org/10.1007/s11229-023-04334-9.CrossRefGoogle Scholar
Freiesleben, Timo, König, Gunnar, Molnar, Christoph, and Tejero-Cantero, Álvaro. 2024. “Scientific Inference with Interpretable Machine Learning: Analyzing Models to Learn about Real-World Phenomena.” Minds and Machines 34 (3). https://doi.org/10.1007/s11023-024-09691-z.CrossRefGoogle Scholar
Fried, Eiko I., Proppert, Ricarda K. K., and Rieble, Carlotta L.. 2023. “Building an Early Warning System for Depression: Rationale, Objectives, and Methods of the WARN-D Study.” Clinical Psychology in Europe 5 (3):e10075. https://doi.org/10.32872/cpe.10075.CrossRefGoogle Scholar
Geirhos, Robert, Temme, Carlos R. M., Rauber, Jonas, Schütt, Heiko H., Bethge, Matthias, and Wichmann, Felix A.. 2018. “Generalisation in Humans and Deep Neural Networks.” In Advances in Neural Information Processing Systems, vol. 31, edited by Bengio, Sammy, Wallach, Hanna, Larochelle, Hugo, Grauman, Kristen, Cesa-Bianchi, Nicolò, and Garnett, Roman, 7549–61. Red Hook, NY: Curran Associates.Google Scholar
Glymour, Clark. 1998. “What Went Wrong? Reflections on Science by Observation and the Bell Curve.” Philosophy of Science 65 (1):1–32. https://doi.org/10.1086/392624.CrossRefGoogle Scholar
Goodfellow, Ian J., Shlens, Jonathon, and Szegedy, Christian. 2015. “Explaining and Harnessing Adversarial Examples.” In International Conference on Learning Representations. arXiv. https://arxiv.org/abs/1412.6572.Google Scholar
Grote, Thomas, Freiesleben, Timo, and Berens, Philipp. 2024a. “Foundation Models in Healthcare Require Rethinking Reliability.” Nature Machine Intelligence 6 (12):1421–23. https://doi.org/10.1038/s42256-024-00924-5.CrossRefGoogle Scholar
Grote, Thomas, Genin, Konstantin, and Sullivan, Emily. 2024b. “Reliability in Machine Learning.” Philosophy Compass 19 (5):e12974. https://doi.org/10.1111/phc3.12974.CrossRefGoogle Scholar
Guha, Neel, Nyarko, Julian, Ho, Daniel, Ré, Christopher, Chilton, Adam, Aditya, K, Alex Chohlas-Wood, Austin Peters, Waldon, Brandon, Rockmore, Daniel, Zambrano, Diego, Talisman, Dmitry, Hoque, Enam, Surani, Faiz, Fagan, Frank, Sarfaty, Galit, Dickinson, Gregory, Porat, Haggai, Hegland, Jason, Wu, Jessica, Nudell, Joe, Niklaus, Joel, Nay, John, Choi, Jonathan, Tobia, Kevin, Hagan, Margaret, Ma, Megan, Livermore, Michael, Nikon Rasumov-Rahe, Nils Holzenberger, Kolt, Noam, Henderson, Peter, Rehaag, Sean, Goel, Sharad, Gao, Shang, Williams, Spencer, Gandhi, Sunny, Zur, Tom, Iyer, Varun, and Li, Zehua. 2023. “LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models.” In Advances in Neural Information Processing Systems, vol. 36, edited by Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., 44123–279. Red Hook, NY: Curran Associates.Google Scholar
Hardt, Moritz. 2026. The Emerging Science of Machine Learning Benchmarks. Princeton, NJ: Princeton University Press. https://mlbenchmarks.org/.Google Scholar
Hardt, Moritz, and Recht, Benjamin. 2022. Patterns, Predictions, and Actions: Foundations of Machine Learning. Princeton, NJ: Princeton University Press.Google Scholar
Hastie, Trevor, Tibshirani, Robert, and Friedman, Jerome. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer.10.1007/978-0-387-84858-7CrossRefGoogle Scholar
Hendrycks, Dan, and Dietterich, Thomas. 2019. “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.” In International Conference on Learning Representations. OpenReview. https://openreview.net/forum?id=HJz6tiCqYm.Google Scholar
Herrmann, Daniel A. 2020. “PAC Learning and Occam’s Razor: Probably Approximately Incorrect.” Philosophy of Science 87 (4):685–703. https://doi.org/10.1086/709786.CrossRefGoogle Scholar
Herrmann, Moritz, Lange, F. Julian D., Eggensperger, Katharina, Casalicchio, Giuseppe, Wever, Marcel, Feurer, Matthias, Rügamer, David, Hüllermeier, Eyke, Boulesteix, Anne-Laure, and Bischl, Bernd. 2024. “Position: Why We Must Rethink Empirical Research in Machine Learning.” In Proceedings of the 41st International Conference on Machine Learning, vol. 235, edited by Salakhutdinov, Ruslan, Kolter, Zico, Heller, Katherine, Weller, Adrian, Oliver, Nuria, Scarlett, Jonathan, and Berkenkamp, Felix, 18228–47. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Herrmann, Moritz, Probst, Philipp, Hornung, Roman, Jurinovic, Vindi, and Boulesteix, Anne-Laure. 2021. “Large-Scale Benchmark Study of Survival Prediction Methods Using Multi-Omics Data.” Briefings in Bioinformatics 22 (3):bbaa167. https://doi.org/10.1093/bib/bbaa167.CrossRefGoogle ScholarPubMed
Höltgen, Benedikt, and Charles Williamson, Robert. 2026. “The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World.” In Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency, 5219–36. New York: Association for Computing Machinery. https://doi.org/10.1145/3805689.3806478.CrossRefGoogle Scholar
Hofman, Jake M., Sharma, Amit, and Watts, Duncan J.. 2017. “Prediction and Explanation in Social Systems.” Science 355 (6324):486–88. https://doi.org/10.1126/science.aal3856.CrossRefGoogle ScholarPubMed
Hood, S. Brian. 2009. “Validity in Psychological Testing and Scientific Realism.” Theory & Psychology 19 (4):451–73. https://doi.org/10.1177/0959354309336320.CrossRefGoogle Scholar
Hooker, J. N. 1995. “Testing Heuristics: We Have It All Wrong.” Journal of Heuristics 1 (1):33–42. https://doi.org/10.1007/bf02430364.CrossRefGoogle Scholar
Ioffe, Sergey, and Szegedy, Christian. 2015. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” In Proceedings of the 32nd International Conference on Machine Learning, vol. 37, edited by Bach, Francis and Blei, David, 448–56. Cambridge, MA: Proceedings of Machine Learning Research. https://proceedings.mlr.press/v37/ioffe15.html.Google Scholar
Irvin, Jeremy, Rajpurkar, Pranav, Ko, Michael, Yu, Yifan, Silviana Ciurea-Ilcus, Chris Chute, Marklund, Henrik, Haghgoo, Behzad, Ball, Robyn, Shpanskaya, Katie, Seekins, Jayne, Mong, David A., Halabi, Safwan S., Sandberg, Jesse K., Jones, Ricky, Larson, David B., Langlotz, Curtis P., Patel, Bhavik N., Lungren, Matthew P., and Ng, Andrew Y.. 2019. “CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.” Proceedings of the AAAI Conference on Artificial Intelligence 33 (1):590–97. https://doi.org/10.1609/aaai.v33i01.3301590.CrossRefGoogle Scholar
Jiang, Liwei, Hwang, Jena D., Bhagavatula, Chandra, Bras, Ronan Le, Liang, Jenny, Dodge, Jesse, Sakaguchi, Keisuke, Forbes, Maxwell, Borchardt, Jon, Gabriel, Saadia, Tsvetkov, Yulia, Etzioni, Oren, Sap, Maarten, Rini, Regina, and Choi, Yejin. 2021. “Can Machines Learn Morality? The Delphi Experiment.” arXiv. https://doi.org/10.48550/arXiv.2110.07574.CrossRefGoogle Scholar
Jumper, John, Evans, Richard, Pritzel, Alexander, Green, Tim, Figurnov, Michael, Ronneberger, Olaf, Tunyasuvunakool, Kathryn, Bates, Russ, Žídek, Augustin, Potapenko, Anna, Bridgland, Alex, Meyer, Clemens, Kohl, Simon A. A., Ballard, Andrew J., Cowie, Andrew, Bernardino Romera-Paredes, Stanislav Nikolov, Jain, Rishub, Adler, Jonas, Back, Trevor, Petersen, Stig, Reiman, David, Clancy, Ellen, Zielinski, Michal, Steinegger, Martin, Pacholska, Michalina, Berghammer, Tamas, Bodenstein, Sebastian, Silver, David, Vinyals, Oriol, Senior, Andrew W., Kavukcuoglu, Koray, Kohli, Pushmeet, and Hassabis, Demis. 2021. “Highly Accurate Protein Structure Prediction with AlphaFold.” Nature 596 (7873):583–89. https://doi.org/10.1038/s41586-021-03819-2.CrossRefGoogle ScholarPubMed
Kane, Michael T. 2013. “Validating the Interpretations and Uses of Test Scores.” Journal of Educational Measurement 50 (1):1–73. https://doi.org/10.1111/jedm.12000.CrossRefGoogle Scholar
Kern, Christoph, Bach, Ruben L., Mautner, Hannah, and Kreuter, Frauke. 2021. “Fairness in Algorithmic Profiling: A German Case Study.” arXiv. https://doi.org/10.48550/arXiv.2108.04134.CrossRefGoogle Scholar
Kleinberg, Jon, Lakkaraju, Himabindu, Leskovec, Jure, Ludwig, Jens, and Mullainathan, Sendhil. 2017. “Human Decisions and Machine Predictions.” The Quarterly Journal of Economics 133 (1):237–93. https://doi.org/10.1093/qje/qjx032.CrossRefGoogle ScholarPubMed
Koch, Bernard, Denton, Emily, Hanna, Alex, and Foster, Jacob G.. 2021. “Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research.” In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, edited by Vanschoren, J. and Yeung, S.. Red Hook, NY: Curran Associates.Google Scholar
Koch, Bernard J., and Peterson, David. 2024. “From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution.” arXiv. https://doi.org/10.48550/arXiv.2404.06647.CrossRefGoogle Scholar
Kornblith, Simon, Shlens, Jonathon, and Le, Quoc V.. 2019. “Do Better ImageNet Models Transfer Better?” In IEEE Conference on Computer Vision and Pattern Recognition, 2661–71. New York: Institute of Electrical and Electronics Engineers.Google Scholar
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” In Advances in Neural Information Processing Systems, vol. 25, edited by Fernando Pereira, Christopher J. Burges, Leon Bottou, and Kilian, Q. Weinberger, 1097–105. Red Hook, NY: Curran Associates.Google Scholar
Kuikka, Sanni. 2024. “The (Un)Predictability of Early (Un)Employment: A Machine Learning Approach.” Socius: Sociological Research for a Dynamic World 10. https://doi.org/10.1177/23780231241286655.CrossRefGoogle Scholar
Lecun, Yan, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86 (11):2278–324. https://doi.org/10.1109/5.726791.CrossRefGoogle Scholar
Leonelli, Sabina. 2015. “What Counts as Scientific Data? A Relational Framework.” Philosophy of Science 82 (5):810–821. https://doi.org/10.1086/684083.CrossRefGoogle ScholarPubMed
Leonelli, Sabina. 2020. “Scientific Research and Big Data.” In The Stanford Encyclopedia of Philosophy, edited by Edward, N. Zalta. Stanford: Stanford University Press.Google Scholar
Liao, Thomas, Taori, Rohan, Raji, Inioluwa Deborah, and Schmidt, Ludwig. 2021. “Are We Learning Yet? A Meta Review of Evaluation Failures across Machine Learning.” In Advances in Neural Information Processing Systems. OpenReview. https://openreview.net/pdf?id=mPducS1MsEK.Google Scholar
Liberman, Mark. 2010. “Obituary: Fred Jelinek.” Computational Linguistics 36 (4):595–99. https://doi.org/10.1162/coli_a_00032.CrossRefGoogle Scholar
Lin, Hanti. 2022. “Modes of Convergence to the Truth: Steps toward a Better Epistemology of Induction.” The Review of Symbolic Logic 15 (2):277–310. https://doi.org/10.1017/s1755020321000605.CrossRefGoogle Scholar
Liou, Gloria, Bailey, Drew H., Baldwin, Chayce R., Duckworth, Angela Lee, and Tay, Louis. 2023. “Why Life Outcomes Are Hard to Predict.” PsyArXiv. https://doi.org/10.31234/osf.io/7q2rx.CrossRefGoogle Scholar
Lundberg, Ian, Rachel Brown-Weinstock, Susan Clampet-Lundquist, Pachman, Sarah, Nelson, Timothy J., Yang, Vicki, Edin, Kathryn, and Salganik, Matthew J.. 2024. “The Origins of Unpredictability in Life Outcome Prediction Tasks.” Proceedings of the National Academy of Sciences 121 (24):e2322973121. https://doi.org/10.1073/pnas.2322973121.CrossRefGoogle ScholarPubMed
Masry, Ahmed, Long, Do, Tan, Jia Qing, Joty, Shafiq, and Hoque, Enamul. 2022. “ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.” In Findings of the Association for Computational Linguistics: ACL 2022, 2263–79. Stroudsburg, PA: Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-acl.177.CrossRefGoogle Scholar
McGrew, Kevin S. 2005. “The Cattell-Horn-Carroll Theory of Cognitive Abilities: Past, Present, and Future.” In Contemporary Intellectual Assessment: Theories, Tests, and Issues, edited by Dawn, P. Flanagan and Patti, L. Harrison, 136–81. New York: Guilford Press.Google Scholar
Messick, Samuel. 1989. “Meaning and Values in Test Validation: The Science and Ethics of Assessment.” Educational Researcher 18 (2):5–11. https://doi.org/10.3102/0013189x018002005.CrossRefGoogle Scholar
Miller, George A. 1995. “WordNet: A Lexical Database for English.” Communications of the ACM 38 (11):39–41. https://doi.org/10.1145/219717.219748.CrossRefGoogle Scholar
Miller, John, Krauth, Karl, Recht, Benjamin, and Schmidt, Ludwig. 2020. “The Effect of Natural Distribution Shift on Question Answering Models.” In Proceedings of the 37th International Conference on Machine Learning, vol. 119, edited by Daumé, Hal III and Singh, Aarti, 6905–16. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Millière, Raphaël, and Buckner, Cameron. 2024. “A Philosophical Introduction to Language Models—Part II: The Way Forward.” arXiv. https://doi.org/10.48550/arXiv.2405.03207.CrossRefGoogle Scholar
Millsap, Roger E. 2012. Statistical Approaches to Measurement Invariance. New York: Routledge. https://doi.org/10.4324/9780203821961.CrossRefGoogle Scholar
Mussgnug, Alexander Martin. 2022. “The Predictive Reframing of Machine Learning Applications: Good Predictions and Bad Measurements.” European Journal for Philosophy of Science 12 (3): article 55. https://doi.org/10.1007/s13194-022-00484-8.CrossRefGoogle Scholar
Nguyen, Kien, Fookes, Clinton, Ross, Arun, and Sridharan, Sridha. 2018. “Iris Recognition with Off-the-Shelf CNN Features: A Deep Learning Perspective.” IEEE Access 6:18848–55. https://doi.org/10.1109/access.2017.2784352.CrossRefGoogle Scholar
Northcott, Robert. 2022. “Reflexivity and Fragility.” European Journal for Philosophy of Science 12 (3): article 43. https://doi.org/10.1007/s13194-022-00474-w.CrossRefGoogle ScholarPubMed
Orr, Will, and Crawford, Kate. 2024. “The Social Construction of Datasets: On the Practices, Processes, and Challenges of Dataset Creation for Machine Learning.” New Media & Society 26 (9):4955–72. https://doi.org/10.1177/14614448241251797.CrossRefGoogle Scholar
Orr, Will, and Kang, Edward B.. 2024. “AI as a Sport: On the Competitive Epistemologies of Benchmarking.” In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. New York: Association for Computing Machinery. https://doi.org/10.1145/3630106.3659012.Google Scholar
Papert, Seymour A. 1966. “The Summer Vision Project.” AI Memo AIM-100. Massachusetts Institute of Technology, Artificial Intelligence Laboratory. https://hdl.handle.net/1721.1/6125.Google Scholar
Paullada, Amandalynne, Raji, Inioluwa Deborah, Bender, Emily M., Denton, Emily, and Hanna, Alex. 2021. “Data and Its (Dis)Contents: A Survey of Dataset Development and Use in Machine Learning Research.” Patterns 2 (11):100336. https://doi.org/10.1016/j.patter.2021.100336.CrossRefGoogle ScholarPubMed
Perdomo, Juan Carlos, Britton, Tolani, Hardt, Moritz, and Abebe, Rediet. 2025. “Difficult Lessons on Social Prediction from Wisconsin Public Schools.” In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 2682–704. New York: Association for Computing Machinery. https://doi.org/10.1145/3715275.3732175.CrossRefGoogle Scholar
Raji, Deborah, Denton, Emily, Bender, Emily M., Hanna, Alex, and Paullada, Amandalynne. 2021. “AI and the Everything in the Whole Wide World Benchmark.” In Advances in Neural Information Processing Systems, vol. 1, edited by Vanschoren, J. and Yeung, S.. Red Hook, NY: Curran Associates.Google Scholar
Rasp, Stephan, Dueben, Peter D., Scher, Sebastian, Weyn, Jonathan A., Mouatadid, Soukayna, and Thuerey, Nils. 2020. “WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting.” Journal of Advances in Modeling Earth Systems 12 (11):e2020MS002203M. https://doi.org/10.1029/2020ms002203.CrossRefGoogle Scholar
Rasp, Stephan, Hoyer, Stephan, Merose, Alexander, Langmore, Ian, Battaglia, Peter, Russell, Tyler, Alvaro Sanchez-Gonzalez, Vivian Yang, Carver, Rob, Agrawal, Shreya, Chantry, Matthew, Bouallegue, Zied Ben, Dueben, Peter, Bromberg, Carla, Sisk, Jared, Barrington, Luke, Bell, Aaron, and Sha, Fei. 2024. “WeatherBench 2: A Benchmark for the Next Generation of Data-Driven Global Weather Models.” Journal of Advances in Modeling Earth Systems 16 (6):e2023MS004019. https://doi.org/10.1029/2023ms004019.CrossRefGoogle Scholar
Recht, Benjamin, Roelofs, Rebecca, Schmidt, Ludwig, and Shankar, Vaishaal. 2019. “Do ImageNet Classifiers Generalize to ImageNet?” In Proceedings of the 36th International Conference on Machine Learning, vol. 97, edited by Chaudhuri, Kamalika and Salakhutdinov, Ruslan, 5389–400. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Reichman, Nancy E., Teitler, Julien O., Garfinkel, Irwin, and McLanahan, Sara S.. 2001. “Fragile Families: Sample and Design.” Children and Youth Services Review 23 (4–5):303–26. https://doi.org/10.1016/s0190-7409(01)00141-4.CrossRefGoogle Scholar
Rocca, Roberta, and Yarkoni, Tal. 2021. “Putting Psychology to the Test: Rethinking Model Evaluation Through Benchmarking and Prediction.” Advances in Methods and Practices in Psychological Science 4 (3):251524592110268. https://doi.org/10.1177/25152459211026864.CrossRefGoogle Scholar
Roelofs, Rebecca, Shankar, Vaishaal, Recht, Benjamin, Sara Fridovich-Keil, Moritz Hardt, Miller, John, and Schmidt, Ludwig. 2019. “A Meta-Analysis of Overfitting in Machine Learning.” In Advances in Neural Information Processing Systems, edited by Wallach, Hanna, Larochelle, Hugo, Beygelzimer, Alina, d’Alché-Buc, Florence, Fox, Emily, and Garnett, Roman, vol. 32, 9179–89. Red Hook, NY: Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2019/file/ee39e503b6bedf0c98c388b7e8589aca-Paper.pdf.Google Scholar
Rudner, Richard. 1953. “The Scientist Qua Scientist Makes Value Judgments.” Philosophy of Science 20 (1):1–6. https://doi.org/10.1086/287231.CrossRefGoogle Scholar
Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei, Li. 2015. “ImageNet Large Scale Visual Recognition Challenge.” International Journal of Computer Vision 115 (3):211–52. https://doi.org/10.1007/s11263-015-0816-y.CrossRefGoogle Scholar
Salaudeen, Olawale, and Hardt, Moritz. 2026. “ImageNot: A Contrast with ImageNet Preserves Model Rankings.” Transactions on Machine Learning Research 2835-8856. https://openreview.net/forum?id=YVbhMerXv9.Google Scholar
Salaudeen, Olawale, Reuel, Anka, Ahmed, Ahmed, Bedi, Suhana, Robertson, Zachary, Sundar, Sudharsan, Domingue, Ben, Wang, Angelina, and Koyejo, Sanmi. 2025. “Measurement to Meaning: A Validity-Centered Framework for AI Evaluation.” ArXiv. https://doi.org/10.48550/arxiv.2505.10573.CrossRefGoogle Scholar
Salganik, Matthew J., Lundberg, Ian, Kindel, Alexander T., Ahearn, Caitlin E., Khaled Al-Ghoneim, Abdullah Almaatouq, Altschul, Drew M., Brand, Jennie E., Carnegie, Nicole Bohme, Compton, Ryan James, Datta, Debanjan, Davidson, Thomas, Filippova, Anna, Gilroy, Connor, Goode, Brian J., Jahani, Eaman, Kashyap, Ridhi, Kirchner, Antje, McKay, Stephen, Morgan, Allison C., Pentland, Alex, Polimis, Kivan, Raes, Louis, Rigobon, Daniel E., Roberts, Claudia V., Stanescu, Diana M., Suhara, Yoshihiko, Usmani, Adaner, Wang, Erik H., Adem, Muna, Alhajri, Abdulla, AlShebli, Bedoor, Amin, Redwane, Amos, Ryan B., Argyle, Lisa P., Livia Baer-Bositis, Moritz Büchi, Chung, Bo-Ryehn, Eggert, William, Faletto, Gregory, Fan, Zhilin, Freese, Jeremy, Gadgil, Tejomay, Gagné, Josh, Gao, Yue, Halpern-Manners, Andrew, Hashim, Sonia P., Hausen, Sonia, He, Guanhua, Higuera, Kimberly, Hogan, Bernie, Horwitz, Ilana M., Hummel, Lisa M., Jain, Naman, Jin, Kun, Jurgens, David, Kaminski, Patrick, Karapetyan, Areg, Kim, E. H., Leizman, Ben, Liu, Naijia, Möser, Malte, Mack, Andrew E., Mahajan, Mayank, Mandell, Noah, Marahrens, Helge, Mercado-Garcia, Diana, Mocz, Viola, Mueller-Gastell, Katariina, Musse, Ahmed, Niu, Qiankun, Nowak, William, Omidvar, Hamidreza, Or, Andrew, Ouyang, Karen, Pinto, Katy M., Porter, Ethan, Porter, Kristin E., Qian, Crystal, Rauf, Tamkinat, Sargsyan, Anahit, Schaffner, Thomas, Schnabel, Landon, Schonfeld, Bryan, Sender, Ben, Tang, Jonathan D., Tsurkov, Emma, van Loon, Austin, Varol, Onur, Wang, Xiafei, Wang, Zhi, Wang, Julia, Wang, Flora, Weissman, Samantha, Whitaker, Kirstie, Wolters, Maria K., Woon, Wei Lee, Wu, James, Wu, Catherine, Yang, Kengran, Yin, Jingwen, Zhao, Bingyu, Zhu, Chenyun, Brooks-Gunn, Jeanne, Engelhardt, Barbara E., Hardt, Moritz, Knox, Dean, Levy, Karen, Narayanan, Arvind, Stewart, Brandon M., Watts, Duncan J., and McLanahan, Sara. 2020. “Measuring the Predictability of Life Outcomes with a Scientific Mass Collaboration.” Proceedings of the National Academy of Sciences 117 (15):8398–403. https://doi.org/10.1073/pnas.1915006117.CrossRefGoogle ScholarPubMed
Salganik, Matthew J., Lundberg, Ian, Kindel, Alexander T., and McLanahan, Sara. 2019. “Introduction to the Special Collection on the Fragile Families Challenge.” Socius: Sociological Research for a Dynamic World 5:237802311987158. https://doi.org/10.1177/2378023119871580.CrossRefGoogle Scholar
Sandrić, Nikola. 2017. “A Note on the Birkhoff Ergodic Theorem.” Results in Mathematics 72 (1):715–30. https://doi.org/10.1007/s00025-017-0681-9.CrossRefGoogle Scholar
Savcisens, Germans, Tina Eliassi-Rad, Lars Kai Hansen, Mortensen, Laust Hvas, Lilleholt, Lau, Rogers, Anna, Zettler, Ingo, and Lehmann, Sune. 2023. “Using Sequences of Life-Events to Predict Human Lives.” Nature Computational Science 4 (1):43–56. https://doi.org/10.1038/s43588-023-00573-5.CrossRefGoogle ScholarPubMed
Schlangen, David. 2021. “Targeting the Benchmark: On Methodology in Current Natural Language Processing Research.” In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Stroudsburg, PA: Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-short.85.Google Scholar
Schrimpf, Martin, Kubilius, Jonas, Hong, Ha, Majaj, Najib J., Rajalingham, Rishi, Issa, Elias B., Kar, Kohitij, Bashivan, Pouya, Jonathan Prescott-Roy, Franziska Geiger, Schmidt, Kailyn, Yamins, Daniel L. K., and DiCarlo, James J.. 2018. “Brain-Score: Which Artificial Neural Network for Object Recognition Is Most Brain-Like?” BioRxiv. https://doi.org/10.1101/407007.CrossRefGoogle Scholar
Shanahan, Murray. 2024. “Talking about Large Language Models.” Communications of the ACM 67 (2):68–79. https://doi.org/10.1145/3624724.CrossRefGoogle Scholar
Shankar, Vaishaal, Roelofs, Rebecca, Mania, Horia, Fang, Alex, Recht, Benjamin, and Schmidt, Ludwig. 2020. “Evaluating Machine Accuracy on ImageNet.” In Proceedings of the 37th International Conference on Machine Learning, vol. 119, edited by Daumé, Hal III and Singh, Aarti, 8634–44. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Shapira, Natalie, Levy, Mosh, Alavi, Seyed Hossein, Zhou, Xuhui, Choi, Yejin, Goldberg, Yoav, Sap, Maarten, and Shwartz, Vered. 2024. “Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models.” In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 2257–73. Stroudsburg, PA: Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-long.138.Google Scholar
Shirali, Ali, and Hardt, Moritz. 2025. “What Makes ImageNet Look Unlike LAION.” Transactions on Machine Learning Research 2835-8856. https://openreview.net/forum?id=IrBYuh9W3T.Google Scholar
Shmueli, Galit. 2010. “To Explain or to Predict?” Statistical Science 25 (3):289–310. https://www.jstor.org/stable/41058949.10.1214/10-STS330CrossRefGoogle Scholar
Sivak, Elizaveta, Pankowska, Paulina, Mendrik, Adriënne, Emery, Tom, Javier Garcia-Bernardo, Seyit Höcük, Karpinska, Kasia, Maineri, Angelica, Mulder, Joris, Nissim, Malvina, and Stulp, Gert. 2024. “Combining the Strengths of Dutch Survey and Register Data in a Data Challenge to Predict Fertility (PreFer).” Journal of Computational Social Science 7:1403–31. https://doi.org/10.1007/s42001-024-00275-6.CrossRefGoogle Scholar
Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” The Journal of Machine Learning Research 15 (1):1929–58.Google Scholar
Sterkenburg, Tom F. 2024. “Statistical Learning Theory and Occam’s Razor: The Core Argument.” Minds and Machines 35 (1):3. https://doi.org/10.1007/s11023-024-09703-y.CrossRefGoogle Scholar
Sterkenburg, Tom F., and Grünwald, Peter D.. 2021. “The No-Free-Lunch Theorems of Supervised Learning.” Synthese 199 (3–4):9979–10015. https://doi.org/10.1007/s11229-021-03233-1.CrossRefGoogle Scholar
Strachan, James W. A., Albergo, Dalila, Borghini, Giulia, Pansardi, Oriana, Scaliti, Eugenio, Gupta, Saurabh, Saxena, Krati, Rufo, Alessandro, Panzeri, Stefano, Manzi, Guido, Graziano, Michael S. A., and Becchio, Cristina. 2024. “Testing Theory of Mind in Large Language Models and Humans.” Nature Human Behaviour 8 (7):1285–95. https://doi.org/10.1038/s41562-024-01882-z.CrossRefGoogle ScholarPubMed
Strauss, Milton E., and Smith, Gregory T.. 2009. “Construct Validity: Advances in Theory and Methodology.” Annual Review of Clinical Psychology 5 (1):1–25. https://doi.org/10.1146/annurev.clinpsy.032408.153639.CrossRefGoogle ScholarPubMed
Sühr, Tom, Dorner, Florian E., Salaudeen, Olawale, Kelava, Augustin, and Samadi, Samira. 2025. “Stop Evaluating AI with Human Tests, Develop Principled, AI-Specific Tests Instead.” ArXiv. https://doi.org/10.48550/arxiv.2507.23009.CrossRefGoogle Scholar
Sullivan, Emily. 2022. “Understanding from Machine Learning Models.” The British Journal for the Philosophy of Science 73 (1):109–33. https://doi.org/10.1093/bjps/axz035.CrossRefGoogle Scholar
Sypherd, Chris, Tang, Weizhi, and Belle, Vaishak. 2025. “Breaking the Illusion: Revisiting LLM Anthropomorphism.” In The 4th International Conference on Human and Artificial Rationalities, edited by Jacquet, Baptiste, Brochier, Emmanuel, and Yama, Hiroshi, 1–19. London: Springer Nature.Google Scholar
Torralba, Antonio, and Efros, Alexei A.. 2011. “Unbiased Look at Dataset Bias.” In IEEE Conference on Computer Vision and Pattern Recognition, 1521–28. New York: Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/cvpr.2011.5995347.Google Scholar
Tsipras, Dimitris, Santurkar, Shibani, Engstrom, Logan, Ilyas, Andrew, and Madry, Aleksander. 2020. “From ImageNet to Image Classification: Contextualizing Progress on Benchmarks.” In Proceedings of the 37th International Conference on Machine Learning, vol. 199, edited by Hal Daumé, III and Singh, Aarti, 9625–35. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Verhagen, Mark D. 2022. “A Pragmatist’s Guide to Using Prediction in the Social Sciences.” Socius: Sociological Research for a Dynamic World 8:1–17. https://doi.org/10.1177/23780231221081702.CrossRefGoogle Scholar
Wang, Alex, Pruksachatkun, Yada, Nangia, Nikita, Singh, Amanpreet, Michael, Julian, Hill, Felix, Levy, Omer, and Bowman, Samuel. 2019. “SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.” In Advances in Neural Information Processing Systems, vol. 32, edited by Wallach, Hanna, Larochelle, Hugo, Beygelzimer, Alina, d’Alché-Buc, Florence, Fox, Emily, and Garnett, Roman, vol. 32, 3266–80. Red Hook, NY: Curran Associates.Google Scholar
Wang, Angelina, Kapoor, Sayash, Barocas, Solon, and Narayanan, Arvind. 2024. “Against Predictive Optimization: On the Legitimacy of Decision-Making Algorithms That Optimize Predictive Accuracy.” ACM Journal on Responsible Computing 1 (1):1–45. https://doi.org/10.1145/3636509.CrossRefGoogle Scholar
Wang, Xiting, Jiang, Liming, Jose Hernandez-Orallo, David Stillwell, Chen, Shiqiang, Sun, Luning, Luo, Fang, and Xie, Xing. 2026. “Evaluating General-Purpose AI with Psychometrics.” Communications of the ACM 69 (5):92–102.10.1145/3769688CrossRefGoogle Scholar
Watson, David S. 2022. “Conceptual Challenges for Interpretable Machine Learning.” Synthese 200 (2): article 65. https://doi.org/10.1007/s11229-022-03485-5.CrossRefGoogle Scholar
Whitely, Susan E. 1983. “Construct Validity: Construct Representation versus Nomothetic Span.” Psychological Bulletin 93 (1):179–97. https://doi.org/10.1037/0033-2909.93.1.179.CrossRefGoogle Scholar
Yadav, Chhavi, and Bottou, Leon. 2019. “Cold Case: The Lost MNIST Digits.” In Advances in Neural Information Processing Systems, vol. 32, edited by Wallach, Hanna, Larochelle, Hugo, Beygelzimer, Alina, d’Alché-Buc, Florence, Fox, Emily, and Garnett, Roman, 13465–74. Red Hook, NY: Curran Associates.Google Scholar
Ye, Haoran, Jin, Jing, Xie, Yuhang, Zhang, Xin, and Song, Guojie. 2025. “Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement.” ArXiv. https://doi.org/10.48550/arxiv.2505.08245.CrossRefGoogle Scholar
Yee, Adrian K. 2024. “Construct Validity in Automated Counterterrorism Analysis.” Philosophy of Science 92 (3):566–83. https://doi.org/10.1017/psa.2024.65.CrossRefGoogle Scholar
Zednik, Carlos, and Boelsen, Hannes. 2022. “Scientific Exploration and Explainable Artificial Intelligence.” Minds and Machines 32 (1):219–39. https://doi.org/10.1007/s11023-021-09583-6.CrossRefGoogle Scholar
Zezulka, Sebastian, and Genin, Konstantin. 2024. “From the Fair Distribution of Predictions to the Fair Distribution of Social Goods: Evaluating the Impact of Fair Machine Learning on Long-Term Unemployment.” In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 1984–2006. New York: Association for Computing Machinery. https://doi.org/10.1145/3630106.3659020.CrossRefGoogle Scholar
Zhang, Guanhua, and Hardt, Moritz. 2024. “Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks.” In Proceedings of the 41st International Conference on Machine Learning, vol. 235, edited by Salakhutdinov, Ruslan, Kolter, Zico, Heller, Katherine, Weller, Adrian, Oliver, Nuria, Scarlett, Jonathan, and Berkenkamp, Felix, 58984–9002. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Zhao, Kino. 2023. “Measuring the Nonexistent: Validity before Measurement.” Philosophy of Science 90 (2):227–44. https://doi.org/10.1017/psa.2023.3.CrossRefGoogle Scholar
Zhao, Kino. 2025. “What Are Statistical Assumptions About? An Answer from Perspectivism.” Harvard Data Science Review 7 (1). https://doi.org/10.1162/99608f92.7fcd521d.CrossRefGoogle Scholar
Zheng, Haowen, and Cheng, Siwei. 2025. “Social Rigidity across and within Generations: A Predictive Approach.” Sociological Methods & Research 54 (4):1683–725. https://doi.org/10.1177/00491241251347984.CrossRefGoogle Scholar
Zhuang, Yan, Liu, Qi, Pardos, Zachary, Kyllonen, Patrick C, Zu, Jiyun, Huang, Zhenya, Wang, Shijin, and Chen, Enhong. 2025. “Position: AI Evaluation Should Learn from How We Test Humans.” In Proceedings of the 42nd International Conference on Machine Learning, edited by Singh, Aarti, Fazel, Maryam, Hsu, Daniel, Lacoste-Julien, Simon, Berkenkamp, Felix, Maharaj, Tegan, Wagstaff, Kiri, and Zhu, Jerry, 82483–508. Cambridge, MA: Proceedings of Machine Learning Research.Google Scholar
Figure 0

Table 1. Table 1 long description.The table summarizes the five sub-inferences discussed in this article, specifying their associated validity conditions. Here, f denotes the model, De$D_e$ the evaluation data, ℓ$\ell$ the evaluation metric, and d(s) a decision rule based on scores s. B$\mathcal{B}$ is a set of benchmarks, and T represents a theoretical task. The arrow on the right indicates that as more theoretical inferences are drawn, an increasing number of validity conditions must be satisfied