2.1. Scientific Questions and Underdetermination

Here I outline the formal framework for epistemic reliability (Kelly Reference Kelly1996). I define a scientific problem in terms of a set W of possible worlds and a countable set I of information states. For a world $w\in W$, I use I(w) to denote the set of information states in w. Informally, an information state represents a body of evidence, and I(w) denotes all evidence that we will eventually observe in w. Formally, we identify $E\in I$ with the set of possible worlds compatible with that evidence. A hypothesis h is a subset of possible worlds, and a question 𝒬 is a partition of W into countably many answers. A hypothesis h entails an answer if it is contained entirely within that answer.

To illustrate this concretely, consider the following toy example: At the end of each day, we record whether it rained in Pittsburgh, marking 1 for rain or 0 for no rain. Each possible world w corresponds to an infinite binary sequence, and I(w) is the set of finite initial subsequences of w. The hypothesis “it will rain twice in the first week” corresponds to the set of all worlds with exactly two 1’s in the first seven entries. Similarly, the question “will it ever stop raining” is the partition of W into two cells: worlds with finitely many 1’s and worlds with infinitely many.

This provides a formal characterization of underdetermination in scientific inquiry. For our present purposes, we distinguish between local and global underdetermination. Intuitively, a hypothesis h is locally underdetermined if any finite amount of evidence is insufficient to deductively verify or refute it. Formally, h is locally undetermined in w iff, for any finite evidence E ₁, …, $E_n\in I\left(w\right)$, there exists a world $w^{\prime }\notin h$ such that $w^{\prime }\in {\cap }_{i=1}^n{E}_i$ (i.e., iff any finite amount of evidence in w is consistent with some world not in h). If h is locally undetermined in every world, we drop the “in w.” A locally underdetermined hypothesis is nonverifiable, although as we will see in the next section, it may be verifiable in the limit. A globally underdetermined hypothesis, however, could not be deductively verified even if we observed the entire data stream at once. Formally, h is globally underdetermined in w iff there exists $w^{\prime }\notin h$ such that $w^{\prime }\in {\cap }_{E\in I\left(w\right)}E$ (i.e., iff some world not in h is compatible with all evidence in w). This notion will be especially important in the scientific contexts we consider for section 3.

2.2. Methods and Convergence

In the formal learning theory framework, verification and refutation are defined as success criteria on methods. A method is a map $M:I\to 𝒫\left(W\right)$ from information states to hypotheses. For a hypothesis h, we say that M deductively verifies h iff the following conditions hold:

1. 1. (Infallibility) For all $E\in I\left(w\right)$, $w\in M\left(E\right)$.

2. 2. (Convergence) $w\in h$ iff there exists $E\in I\left(w\right)$ such that $M\left(F\right)\subseteq M\left(E\right)\subseteq h$ for all $F\in I\left(w\right)$ such that $F\subseteq E$.

Intuitively, Infallibility stipulates that the method’s outputs (conclusions) are deductively entailed by its inputs (evidence), and Convergence stipulates that M will converge to h (or some hypothesis entailing h) iff h is true. Similarly, M refutes h iff M verifies h^c, M decides h iff it both verifies and refutes h, and a hypothesis is verifiable/refutable/decidable iff there exists a method that verifies/refutes/decides it. Returning to our example, the hypothesis h = “it will rain twice in the first week” is clearly decidable by the following method: output $h\cup h^c$ (the trivial hypothesis) for the first seven observations, then count the number of 1’s and output h or h_c accordingly.

Of course, Infallibility is too strict a requirement for nearly all scientific inquiry (hence the problem of induction). By dropping this condition, however, we obtain a weaker success criteria for M. Specifically, a method verifies h in the limit iff it satisfies condition 2, and a hypothesis is limit verifiable iff such a method exists for it. Thus, we can still deductively prove that a method will converge to the truth in the limit, even if we cannot deductively verify a hypothesis in the short run. To illustrate this, consider h = “it will rain only finitely many times.” This hypothesis is locally underdetermined and cannot be verified or refuted by any method: any finite evidence is consistent with a world with finitely many days of rain and a world with infinitely many. However, h is limit verifiable by the following method: if it did not rain today, output h, otherwise output h^c. In any world where h is true, there must exist a last day of rain, which means that on this last day, M will correctly stabilize to h. Conversely, in a world where h is false, M will either stabilize to h^c (if it never stops raining) or never stabilize at all. Therefore h is unverifiable but verifiable in the limit.

While we will not use them here, it is worth noting that all of these concepts have precise topological analogues, and there exists a natural hierarchy of topological properties that corresponds to the hierarchy of solvability criteria. This allows us to better analyze the solvability of a problem in terms of its topological properties. Furthermore, Genin and Kelly (Reference Genin and Kelly2017) extend this reasoning to the statistical setting, deriving precise notions of statistical verifiability and statistical verifiability in the limit in the same topological framework.

3.1. Technical Challenges: Severe Global Underdetermination

Recall the motivating principle of “feasibility contextualism”: identify our epistemic constraints (underdetermination) and succeed as well as possible given those constraints. If we take “success in the limit” as our baseline success criterion, we can apply this to problems that are, at worst, locally underdetermined. However, certain features characteristic of human-focused research often induce a very strong form of global underdetermination.

In order to make sense of these claims, however, we must first provide some minimal characterization of the research contexts we want to address. For this purpose, economics is an informative case to consider, as economists deal both with human-generated data and mathematical models of the data-generating process. For an economic realist, a model is an attempt to (approximately) represent some real-world data-generating process, and we can at least make conceptual sense of epistemic reliability. We can very generally represent an economic research problem as follows: ${\left\{\underset{\_}{Z}\right\}}_{t\in T}$ denotes a data stream where T indexes the time steps of observation, and each ${\underset{\_}{Z}}_t$ corresponds to some panel of measurements. These measurements may be at the microlevel, denoting the behavior of individual agents (e.g., firm or consumer), or the macrolevel, denoting aggregate measures or indices (e.g., gross domestic product or unemployment). The chosen set of measurements determines our information basis I. The hypothesis space ℋ is typically a model structure or class of models, posited by the economist. This, in conjunction with any theoretical assumptions, determines a parameter space, and we can equate each allowable parameterization with a possible world. The economist’s challenge is to construct a method $M:I\to H$ that converges in the limit to the true model/model parameters, for some notion of convergence.

There are several sources of underdetermination common to economic research. The most common is simply the low quality of available data: economic data, particularly at the microlevel, are often biased and woefully incomplete (Friedman Reference Friedman1953; Windrum, Fagiolo, and Moneta Reference Windrum, Fagiolo and Moneta2007). Furthermore, for reasons both practical and ethical, it is often extremely difficult to generate useful experimental data in economics. Economists therefore have to rely on incomplete, often biased, observational data alone, with only marginal control over the data-collection process. This challenge is certainly not unique to economics: psychologists and cognitive scientists face similar difficulties, particularly when the variables of interest are unobservable cognitive states on which we cannot directly intervene. The resulting difficulty of performing controlled experiments severely limits the experimenters’ ability to infer causal connections between real-world variables; any two causal models that induce the same joint probability distribution over observed variables are empirically equivalent and cannot be distinguished without controlled experimentation. For some applications this may be sufficient, but many applications (e.g., designing effective policy or medical interventions) require causal knowledge beyond the scope of observational data.

A second source of underdetermination is the abundance of available models and modeling paradigms. Statisticians and econometricians are endlessly innovative in designing flexible model classes for time-series data, and the economist must often choose between two or more empirically equivalent modeling paradigms (Haavelmo Reference Haavelmo1944). In some cases, a single model class will contain multiple (sometimes infinitely many) empirically equivalent models; this occurs in econometrics, for example, when using overcomplete dictionaries to estimate complex functions (e.g., Clyde and Wolpert Reference Clyde and Wolpert2007; Belloni and Chernozhukov Reference Belloni and Chernozhukov2011). Such issues also arise in models that represent the unobservable inner workings of human decision processes, as is sometimes the case in behavioral micromodels and often the case in cognitive science and psychology. As it is impossible for statistical inference to decide between observationally equivalent hypotheses, global underdetermination is built into the hypothesis space itself.

A third source of underdetermination is the nature and prevalence of ceteris paribus (CP) clauses in economic hypotheses, which weaken a hypothesis of the form “all A are B” to “all else being equal, all A are B.” There is some debate as to whether, and to what degree, CP clauses occur in natural sciences (e.g., Cartwright Reference Cartwright1995; Earman and Roberts Reference Earman and Roberts1999), but it is widely acknowledged that most economic hypotheses contain a CP clause, either implicitly or explicitly (Hausman Reference Hausman1988; Cartwright Reference Cartwright, Earman, Glymour and Mitchell2002). In this context, we can understand a CP clause as an auxiliary hypothesis of unknown dimension; the clause specifies that our model is only true so long as certain conditions hold, but the exact conditions are left unspecified. This results in a particularly bad instance of the Duhem-Quine thesis: not only is the main hypothesis inextricably tied to a set of auxiliary hypotheses, but we cannot even specify what those auxiliary hypotheses are. A thorough study of underdetermination in economics (Sawyer, Beed, and Sankey Reference Sawyer, Beed and Sankey1997) demonstrates some technical implications of CP clauses for hypothesis testing. In time-series econometrics, for example, the modeler attempts to distinguish between “signals” that capture the causal connections of interest and “noise” that accounts for random, less systematic causes that disturb the variables of interest. In such cases, it is possible for noises to be stronger than signals, compounded by the possibility that the causal mechanisms of interest may undergo structural changes themselves (Valente Reference Valente2005). Thus, the underspecified nature of auxiliary hypotheses in economics can make precise refutations impossible, and econometricians will sometimes respond to diagnostic test failure with ad hoc adjustments like changing the estimation procedure, rather than respecifying the theory (Hendry Reference Hendry1980).

3.2. Conceptual Challenges: The Nature and Purpose of Scientific Hypotheses

The previous analysis assumes a certain “realist” perspective to begin with: the view that a model is an attempt to accurately represent some (often causal) aspect of the real world. Yet many economists, cognitive scientists, psychologists, and associated philosophers take a decidedly nonrealist stance on the nature of hypotheses. Even within economics, there is a wide range of “nonrealist” perspectives on the status and purpose of hypotheses, and I identify two recurring concerns among these views that complicate the interpretation of epistemic reliability criteria on a conceptual level.

The first concern relates to the notion of hypothesis specifically. In epistemic reliability, we assume a prespecified set of relevant possibilities and identify hypotheses with subsets of possibilities. This notion, however, is difficult to reconcile with how hypotheses and models are often discussed in economics.Footnote ² Sugden (Reference Sugden2000), for example, argues that an economic model is not a set of “real” possibilities but a counterfactual world constructed by the economist. Such an object says nothing about the world; a model supports conceptual exploration, enabling deductive inferences about what the world would look like if the assumptions in the model were true. Bridging the gap between model world and real world requires a separate inductive reasoning process, establishing the “robustness” of the model against violations or modifications of its assumptions. A less formalized, more historicist viewpoint asserts that economic hypotheses are, essentially, carefully crafted stories, which we use to explain historical events and draw lessons for the future (e.g., McCloskey Reference McCloskey1998, 23–28). Neither conception, however, is particularly well suited to representation as subsets of a fixed set of relevant possibilities. Even if we equate individual models with relevant possibilities, the prevalence of CP clauses, noise terms of unknown dimension, and other underspecified simplifying assumptions make it difficult to explicitly identify a model with a set of possible worlds.

The second concern relates to how we evaluate hypotheses, independent of what kinds of objects we understand them to be. Underlying epistemic reliability is the assumption that, when faced with multiple hypotheses compatible with the same evidence, our method for selecting one should be evaluated in terms of its truth conduciveness. In the formal setup, the truth conditions for a hypothesis are built right into the framework—a hypothesis h is a set of possible worlds, and h is true iff it contains the true world. But many scientists (especially economists) take a decidedly instrumentalist stance toward their research, asserting that we cannot (or at least, have no reason to) establish firm truth conditions for a hypothesis. Rather, a hypothesis is a tool used to achieve some end—typically prediction of future data—and the only metric for evaluating a hypothesis is how well its predictions match with our observations (e.g., Friedman Reference Friedman1953). There are (at least) two versions of instrumentalism; one version maintains a sort of quasi realism, granting that the assumptions of a hypothesis at least refer to some part of the real world but asserting that it does not matter whether those assumptions are true. The alternative version denies that theoretical assumptions have any connection to things in the real world; in this view, the assumptions underlying a model are simply a compact way of representing the model’s predictions. This latter form of instrumentalism is also common in certain areas of cognitive modeling. For example, many cognitive scientists take an “ideal Bayesian observer” approach to modeling perception, in which the goal is not to accurately model the actual causal processes underlying human cognition but to provide an “as if” explanation at the computational level (Jones and Love Reference Jones and Love2011). In either case, the instrumentalist is unconcerned with assessing the “truth” of the hypotheses, and it therefore makes little sense to evaluate the instrumentalist’s methods in terms of truth conduciveness.

4.1. Goal-Directed Learning and Reliability

As in the epistemic setting, we define a problem in terms of a set W of possible worlds and a set I of information states, with I(w) denoting all evidence that will eventually be observed in w. Unlike the epistemic setting, the hypothesis space may be any abstract representation space and need not correspond to explicitly defined subsets of W (although we continue to use the w subscript to index the “true” world from which evidence is generated). This allows the framework to accommodate nonrealists who are uncomfortable with the metaphysical implications of a known, predefined space of possibilities. As before, we assume that evidence is observed in trials, using $M:I\to \mathscr{H}$ to denote the total evidence at the nth trial, and define a method ℋ to be a map from information states to hypotheses.

By allowing ℋ to be any representation space, we lose the well-defined truth conditions built into the epistemic framework. In order to evaluate a hypothesis, we must understand (a) the aims of our inquiry and (b) how we may act on our inferences so as to reach those aims. That is, in order to model goal-directed learning, we must model not only what knowledge can be acquired but how that knowledge will be put to use. To this end, we draw on the Adaptively Rational Learning framework presented in Wellen and Danks (Reference Wellen and Danks2016). In this framework, a learner consists of an inference method M and a decision mechanism D, which deterministically assigns each hypothesis to an action. Actions can include decisions such as setting a price or choosing a medical treatment, interventions such as imposing an incentive or penalty, or inferences such as predictions about the future.

We assume that learning is motivated by some research goal G, represented as a value function V_G(a, w), which assesses how well our goal is satisfied by a given action in a given world. We are intentionally vague about the nature of the goal, beyond its operationalization as a value function. Research may be motivated by data-driven goals (e.g., prediction), policy goals (e.g., optimizing interest rates, identifying effective tax interventions), or more value-laden social goals, in the vein of Dewey’s (Reference Dewey and McDermott1973) pragmatist view of science. Goals may also be quantitatively evaluated (e.g., an explicit measure of forecasting accuracy) or qualitatively evaluated (e.g., replicating some qualitative, “stylized” facts about the data). We can also represent purely epistemic goals, so long as we have some way to make sense of “truth conditions” for our hypotheses.Footnote ³ Importantly, goals may be strict, requiring an exact solution to some optimization problem, or approximate, only requiring a solution within some wide margin of error. While we do not consider specific examples here, sufficiently weak goals may permit short-run performance guarantees (just as sufficiently tractable epistemic problems may permit short-run convergence guarantees). Thus, our notion of goal here is very general, and the only necessary criterion is that the goal can be evaluated through some value function.

Given this notation, we define an environmental learning problem as $𝒫=\{W,I,\mathscr{H},{\rm A},D,V_G\}$, where Α is the set of possible actions, D is the decision mechanism, and V_G is a goal, operationalized as a value function. For the purpose of this discussion, we assume that the decision mechanism is fixed throughout learning and deterministically outputs a best action in pursuit of the goal. For simplicity of presentation, we also assume that V_G is binary valued. With this established, we can define the following pragmatic reliability criteria:

Definition 1. For a hypothesis $h\in \mathscr{H}$, a method M pragmatically verifies h in the limit under D,Footnote ⁴ iff for every world $w\in W$, M converges to h in w iff h induces a goal-satisfying action in w under D. Formally, this is written as

$\begin{array}{}\text{(1)}& \forall w\in W\left((\exists E\in I(w\left)\right)\left((\forall F\subseteq E)\left(M\right(F)=M(E)=h)\right)\right)\leftrightarrow V_G\left(D\right(h),w)=1.\end{array}$

Similarly, M pragmatically solves 𝒫 in the limit (under D) iff for every world $w\in W$, there exists an information state $E\in I\left(w\right)$ such that

1. 1. $M\left(E\right)=h$ for some h that satisfies $V_G\left(D\right(h),w)=1$, and

2. 2. For all information states $F\subseteq E$, $D\left(M\right(F\left)\right)=D\left(h\right)$.

Intuitively, M pragmatically verifies h so long as it only converges to h in worlds where h induces the best possible action, and M pragmatically solves 𝒫 so long as it converges to a hypothesis that induces the best possible action in every possible world. Thus, we can extend the notion of reliability in the limit to a case in which hypotheses may not be representable as subsets of possible worlds or we cannot or have no need to make sense of “truth conduciveness.” Even in such cases, we can still sensibly talk about convergence in the limit to a goal-satisfying hypothesis, given our goals and constraints.

4.2. Pragmatic Underdetermination and the Coarsening Operation

While the above definitions extend the notion of reliability beyond a strictly epistemic setting, they do not address the problem of underdetermination. We reviewed the epistemic notion of underdetermination in sections 2 and 3, but when we add pragmatic constraints to the framework, we must also consider a pragmatic notion of underdetermination. In particular, pragmatic underdetermination occurs when the decision mechanism assigns two different hypotheses to the same action. That is, if two different hypotheses result in the same plan of action for achieving a particular goal, then there is no pragmatic reason to prefer one hypothesis over the other. Just as no method can limit verify a hypothesis that is globally underdetermined, no method can pragmatically limit verify a hypothesis that is pragmatically underdetermined, per definition 1. However, there is a principled way to extract a coarser but tractable problem from a pragmatically underdetermined one.

To this end, let $𝒫=\{W,I,\mathscr{H},{\rm A},D,V_G\}$ be an environmental learning problem with deterministic D, and let ∼_D be a relation over ℋ defined as $h_1{\sim }_D{h}_2\leftrightarrow D\left(h_1\right)=D\left(h_2\right)$ (following Wellen and Danks Reference Wellen and Danks2016). This relation induces a partition of the original hypothesis space into pragmatic indistinguishability classes, as any two hypotheses within the same cell result in the same action under D. We can then transform 𝒫 into a new problem $\widehat{𝒫}=\{W,I,\hat{\mathscr{H}},{\rm A},D,V_G\}$, where $\hat{\mathscr{H}}$ is the set of equivalence classes induced under ∼_D. Since these equivalence classes group together hypotheses that map to the same action, any two hypotheses from two distinct classes will map to distinct actions. Thus, the coarsened problem $\widehat{𝒫}$ eliminates the pragmatic underdetermination faced by the original problem 𝒫.

The coarsening operation is neither new nor unique to the pragmatic learning paradigm. This general principle of extracting coarser but more tractable problems from finer ones is used, whether implicitly or explicitly, to justify scientific practices across a range of domains. A recent example in causation is Causal Feature Learning (Chalupka, Eberhardt, and Perona Reference Chalupka, Eberhardt and Perona2017), which addresses the problem of discovering high-level macrocauses from nonexperimental microvariable data. The authors endorse the view that “macrovariables should be thought of as task-specific” and propose a method for partitioning a space of microvariables into a hypothesis space of macrocauses, with respect to some set of target outputs (141). The operation underlying this framework groups together microvariables that have indistinguishable effects on the output. This is an explicit application of the coarsening operation to construct an optimal hypothesis space for identifying macrocauses of a set of effects.

A less explicit, more historical example of coarsening is seen in the first sections of Skinner (Reference Skinner1953, 23–39), wherein he outlines and justifies a framework for behaviorism. The purpose of behaviorism, Skinner writes, is to provide a “functional analysis” of behavior, that is, an understanding of the “external variables of which behavior is a function” (35) and how we can manipulate those external variables to control that behavior. The relevant hypothesis space is therefore a set of possible functions relating external circumstances to behavioral outputs. While Skinner does not formally specify all such functions, he characterizes a common “causal chain consisting of three links: (1) an operation performed upon the organism from without—for example, water deprivation; (2) an inner condition—for example physiological or psychic thirst; and (3) a kind of behavior—for example, drinking” (34). Under this view, it might seem that a natural characteristic of any plausible hypothesis would be separability into two functions $f=g\circ h$: one mapping external conditions to hidden internal states and one mapping internal states to actions. But, as Skinner points out, “the second link is useless in the control of behavior unless we can manipulate it” (34). That is, if g, h and g′, h′ are two distinct pairs of functions that, when composed, induce the same map from external inputs to behavioral outputs, then the hypotheses $g\circ h$ and $g^{\prime }\circ h^{\prime }$ are indistinguishable with respect to the goal of behavioral analysis. The solution, therefore, is to coarsen our hypothesis space: we group together all pairs of functions that result in the same input-output map and simply “examine the third link as a function of the first” (35). Thus, we can interpret Skinner’s justification for the “behaviorist” hypothesis space as the pragmatic indistinguishability classes of some more general space of cognitive behavior functions, taken with respect to the goal of predicting and controlling behavior.

4.3. Epistemic versus Pragmatic Underdetermination

In our framework, a problem may face both pragmatic and epistemic underdetermination, and it is important to understand how these notions are distinct and how they relate to each other. Intuitively, epistemic underdetermination occurs when our epistemic constraints (i.e., the available data and possible answers under consideration) do not allow us to distinguish between two or more hypotheses. Pragmatic underdetermination occurs when our pragmatic constraints (goals, available actions, decision mechanism, etc.) do not require us to distinguish between two or more hypotheses. Whereas the traditional reliability framework deals with what we can learn from a given data source, the pragmatic extension deals with what we ought to learn. The latter, normative question is, of course, influenced by both our epistemic and pragmatic constraints: we ought not try to learn anything we cannot learn, and we ought not try to learn anything we need not learn. Thus, determining what we ought to learn requires a framework that can accommodate both kinds of constraints, which we achieve by relaxing the notions of hypothesis and truth condition and applying a coarsening operation to the hypothesis space.

To better illustrate these principles, and how they help accommodate the issues raised in section 3, we will consider some variations of the toy problem used as an example in the introduction. Recall the original problem: an urn contains some unknown number of red or blue marbles, and we observe a sequence of draws. In each step of the sequence, we observe either a null draw N (with probability 1 if the urn is empty or probability 1/2 if the urn is nonempty) or a single marble drawn uniformly at random from those remaining in the urn. The original problem is to infer the initial contents of the urn from this sequence, and we can represent the hypothesis space ℋ as the set of all ordered pairs of nonnegative integers. As we previously explained, this problem is locally but not globally underdetermined. In particular, this problem is solved in the limit by the method $M\left(E\right)=(\#r,\#b)$, where #r and #b are the number of red and blue marbles in the data sequence E, respectively.

Now consider a pragmatic extension of this problem: we are in a room with four levers, corresponding to the following four possibilities: (a) the urn is (initially) empty, (b) the urn contains some red and no blue marbles, (c) the urn contains some blue and no red marbles, and (d) the urn contains some red and some blue marbles. The hypothesis space remains unchanged, but now we must map our hypothesis to an action and choose one of the four levers. If we choose the correct lever we get a reward, otherwise we get no reward. This problem faces pragmatic underdetermination: there are multiple distinct hypotheses that map to the same action (e.g., (3, 0) and (4, 0) both map to lever b), and we therefore have no reason to distinguish between these hypotheses. We can therefore eliminate this pragmatic underdetermination by partitioning the original hypothesis space into four cells, one corresponding to each action. This allows us to define a method $\hat{M}$ that pragmatically solves the coarsened problem $\hat{\mathscr{H}}$ in the limit.

Now consider the original problem, and suppose that the data stream has been corrupted, as is often the case in sciences with human-generated data. In particular, suppose that the first draw of the sequence is obscured, so that we do not know whether the first draw was R, B, or N. In this case, the problem faces epistemic underdetermination: because we cannot observe the first draw, we cannot deduce the original contents of the urn, even if we could observe the complete data stream at once (minus the corrupted draw). This is a case in which succeeding as well as necessary surpasses what is possible. However, we can still apply the same principles to identify the strongest weakening of the original problem that is solvable in the limit. In particular, suppose that the full (corrupted) data stream contains #r red marbles and #b blue marbles. As we cannot observe the first draw, this evidence is compatible with three possibilities: (#r, #b), ( $\#r+1$, #b), and (#r, $\#b+1$). Thus, if we define a new hypothesis space by grouping hypotheses into the appropriate triples, we obtain a weaker version of the original problem that is still solvable in the limit by the method $M\left(E\right)=\left\{\right(\#r,\#b),(\#r+1,\#b),(\#r,\#b+1\left)\right\}$ (i.e., the method that outputs a disjunction of the three hypotheses). While this method is not guaranteed to converge to the correct singular hypothesis, it is guaranteed to converge to a set of three possibilities that contains the correct hypothesis, which is the best possible outcome, given our epistemic constraints.

This helps illustrate the core principles of our framework and how they apply to the issues discussed in section 3. The epistemic part of the framework determines what can and cannot be reliably learned and motivates the feasibility contextualism principle: given an inductive learning problem, identify your epistemic constraints, then succeed as well as possible (Kelly Reference Kelly, Baltag and Smets2014). The pragmatic setting, however, provides additional criteria for justifying inferences. In particular, the decision function and goal determine which distinctions are necessary to learn in order to satisfy the goal. Depending on the goal, then, succeeding “as well as possible” may be unnecessary, and given limited resources and finite time, learning more than necessary is often a luxury we cannot afford. We can therefore motivate our pragmatic reliability framework with a modified feasibility contextualism principle: identify your epistemic constraints, then succeed as well as necessary.Footnote ⁵

5.1. Prediction as a Research Goal

When our goal is prediction, the decision mechanism D outputs a predicted information state $D\left(h\right)\in I$.Footnote ⁶ Even if a hypothesis h does not explicitly correspond to a subset of W, we can identify the output of D with the set of possibilities compatible with that prediction (even if we do not explicitly know what those possibilities are).

We refer to D as completely predictive if, for every h, D(h) outputs a complete information state for some world $w\in W$. Formally, we define the total information in w to be $\mathrm{Tot}\left(w\right)={\cap }_{E\in I\left(w\right)}E$, and the complete information in w to be $\mathrm{Com}\left(w\right)=\{w^{\prime }\in W|\mathrm{Tot}\left(w^{\prime }\right)=\mathrm{Tot}\left(w\right)\}$. Intuitively, if we have the total data in w, this means that we have all of the available evidence in w. If we have the complete data in w, this means that we have all of the available evidence in w, and we know that we have all of the available evidence in w. This distinction is subtle, but important, so it is worth briefly considering an example in which the distinction is relevant.

To this end, suppose that W is the set of all partial recursive functions from ℕ to ℕ, and the data in each world f consist of input/output pairs (n, f(n)). Let f be any such function defined on every input except 1, and let f′ be a function defined on every input in N that is identical to f on all inputs not equal to 1 (i.e., $f^{\prime }\left(n\right)=f\left(n\right)$ for all $n\ne 1$). The total data in world f are the set of all input/output pairs $\mathrm{Tot}\left(f\right)=\left\{\right(n,f\left(n\right)\left)\right|n\ne 1\}$, and the total data in world f′ are equal to $\mathrm{Tot}\left(f\right)\cup \left\{\right(1,f^{\prime }\left(1\right)\}$. Thus, the total data in world f are compatible with world f′: even if we observed every input/output pair in world f, this would not be sufficient to rule out the possibility that we are in world f′. If, however, we observe all the evidence in f and we know that this is all the available evidence, then we could rule out f′, as we would know that the evidence is missing the extra input/output pair that distinguishes f′ from f. Thus, this illustrates a problem in which the total data are insufficient to distinguish f from f′, but the complete data are sufficient to make this distinction.

A practical example of a completely predictive mechanism is a linear regression model. In function estimation, the data consist of input/output pairs (X, Y), possible worlds are functions relating X to Y, and the complete data in any world F are the full list of input-output pairs (X, F(X)). For a hypothesis with parameters β, we can predict the value of Y from any value of X by computing $Y=\beta X$. Therefore, we can generate all of the input-output pairs, which constitute the complete data in some world.

With these formalizations, we can define empirical adequacy as follows: if D is completely predictive, we define the oracle goal G _oracle by the value function V_G(D(h), w), which outputs 1 iff $D\left(h\right)=\mathrm{Com}\left(w\right)$. Intuitively, this specifies that under D, h is an oracle for w; that is, D(h) predicts everything we will eventually observe in w and nothing we will not. This corresponds to van Fraassen’s (Reference van Fraassen1980) notion of empirical adequacy, which we can now operationalize as a research goal.

5.2. Pragmatic and Epistemic Underdetermination

When D is completely predictive, the resulting indistinguishability classes coincide in an important way with the epistemic notion of global underdetermination. In particular, for two hypotheses h ₁, $h_2\subset W$ (in the epistemic sense) we define h ₁ and h ₂ to be epistemically indistinguishable iff the problem of distinguishing between them is globally underdetermined by available data. Formally, h ₁ and h ₂ are epistemically indistinguishable iff for any $w\in W$,

$\begin{array}{}\text{(2)}& h_1\cap \mathrm{Tot}\left(w\right)\ne \varnothing \leftrightarrow h_2\cap \mathrm{Tot}\left(w\right)\ne \varnothing \mathrm{.}\end{array}$

Recalling that h ₁ and h ₂ are pragmatically indistinguishable iff $D\left(h_1\right)=D\left(h_2\right)$, we obtain the following result:

Note that in the pragmatic framework, h ₁ and h ₂ are simply points in some representation space and may have no explicit relation to subsets of W. If D is a prediction generator, however, then D(h) is an information state, which we can equate with the set of possibilities in which that information would be observed (even if we do not explicitly know what those possibilities are). Under D, the representation space ℋ induces a corresponding hypothesis space of predictions D(ℋ). Theorem 1 shows that if D is completely predictive, the epistemic indistinguishability classes of D(ℋ) are exactly the pragmatic indistinguishability classes of ℋ, mapped under D.

A corollary of this theorem better illustrates the relation between empirical adequacy (as a pragmatic goal) and “truth seeking” (as an epistemic goal):

Intuitively, this corollary gives us a dual characterization of empirical adequacy. Under the epistemic view, empirical adequacy is the strongest problem that is limit solvable in every possible world. Under the pragmatic view, empirical adequacy is the hardest problem to pragmatically solve in the limit. To put this another way, recall the principle of feasibility contextualism (identify your epistemic constraints, then succeed as well as possible) and our pragmatic modification (identify your epistemic constraints, then succeed as well as necessary). In this language, corollary 1 shows that, when our goal is empirical adequacy, succeeding as well as necessary is the same as succeeding as well as possible. We can therefore think of empirical adequacy as a sort of boundary point between epistemically (i.e., truth-seeking) and pragmatically (i.e., goal-satisfying) solvable problems.