2. Carnapian inductive logic

The rules of inference studied by Carnap go back to Laplace. They are known as “rules of succession.” A rule of succession takes as input a finite sequence of past observations. A sequence of observations is given by a sequence of random variables ${\bf X}_n = (X_1, \ldots, X_n)$ . Each $X_i$ takes a value in a finite set of observational outcomes $\{1, \ldots, t\}$ . A rule of succession then outputs predictive probabilities for the next observation, $X_{n+1}$ , given an assignment of values for $X_1, \ldots, X_n$ :

(1) $\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_n}) = {{{n_i} + {\alpha _i}} \over {n + \sum\nolimits_j {{\alpha _j}} }}.$

The counts $n_i$ represent the number of times outcome i appears among $X_1, \ldots, X_n$ . The weight $\alpha_j$ determines the initial probability assigned to outcome j. In the Carnapian setting, the weights are positive if the sequence of random variables is infinite; otherwise, they can be negative. A Carnapian rule thus assigns probabilities to events about the immediate future given past observations.

The logical foundations of the predictive probabilities in (1) are fully understood. An especially lucid treatment is given in Zabell (Reference Zabell1982), who derives (1) from three assumptions that are based on the ideas of Johnson (Reference Johnson1932): exchangeability, Johnson’s sufficientness postulate, and a regularity assumption. Regularity only requires that each finite sequence of outcomes has a positive probability. Exchangeability and Johnson’s sufficientness postulate are more substantial: they express order invariance for unconditional and conditional probabilities.

Exchangeability says that

$\mathbb{P}(X_1 = x_1, \ldots, X_n = x_n) = \mathbb{P}(X_1 = x_{\tau(1)}, \ldots, X_n = x_{\tau(n)})$

for each n, all possible sequences of outcomes $x_1, \ldots, x_n$ , and for any permutation $\tau$ of $\{1, \ldots, n\}$ . In words, a probability assignment is exchangeable if it is invariant under any reordering of the outcomes. A consequence of exchangeability is that the counts $n_1, \ldots, n_t$ are a sufficient statistic: knowing how often each outcome is observed among $X_1, \ldots, X_n$ completely determines the probability distribution $\mathbb{P}$ since each sequence with the same counts $n_1, \ldots, n_t$ must have the same probability.

Johnson’s sufficientness postulate applies this idea to conditional probabilities. It says that observing an outcome i only depends on i, $n_i$ , and the total sample size n:

$\mathbb{P}(X_{n+1} = i \mid {\bf X}_n) = f_i(n_i, n).$

When combined with exchangeability and regularity, Johnson’s sufficientness postulate implies that there exist constants $\alpha_j$ such that (1) holds.

The connections between Carnapian inductive systems and Bayesian inference are also well understood. The basic insight is the following. According to de Finetti’s theorem, any exchangeable infinite sequence of random variables $X_1, X_2, \ldots$ Footnote ¹ is a mixture of multinomial distributions:

(2) $\mathbb{P}(X_1 = x_1, \ldots, X_n = x_n) = \int_{\Delta_{t-1}} p_1^{n_1} p_2^{n_2} \cdots p_t^{n_t}\,d\mu(p_1, \ldots, p_{t-1}).$

Here, the numbers $n_i$ are again the counts of observational outcomes. The integral ranges over the $(t-1)$ -dimensional simplex $\Delta_{t-1} = \{(p_1, p_2, \ldots, p_t)\ \colon \; p_j \geq 0,\, {\bf\sum}_j \;p_j = 1\}$ . (The simplex has dimension $t-1$ since $p_t = 1 - {\bf\sum}_{j = 1}^{t-1} \; p_j$ ). The simplex $\Delta_{t-1}$ can be thought of as the set of all possible chances of outcomes: $p_j$ is the chance that outcome j occurs at any stage of the observational process. The integrand in (2) articulates this by aggregating chances into an independent and identically distributed setup in which each of the $n_j$ occurrences of j happens with chance $p_j$ . However, the chances are unknown. That’s why (2) integrates over all chances relative to a prior $\mu$ that represents the initial uncertainty about chances.

The prior $\mu$ can be used to forge a connection between de Finetti’s theorem and Carnapian inductive logic. If $\mu$ takes on a specific functional form (when it is a so-called Dirichlet prior), then the predictive probabilities that result from the representation (2) coincide with the inductive system in (1) (with the parameters $\alpha_j$ determining the Dirichlet distribution).

There is no need here to go into the mathematical specifics, which are explained elsewhere (e.g., Skyrms Reference Skyrms, Ferguson, Shapley and MacQueen1996; Romeijn Reference Romeijn, Gabbay, Hartmann and Woods2011). Two issues are more important for us. The first concerns the philosophical lessons to be drawn from the connection between Carnapian inductive logic and Bayesian statistics. I will say a bit more about this in the last section of this paper. For now, let me just make a brief remark about the bridge that has been built by putting inductive logic in contact with de Finetti’s theorem. One way to read de Finetti’s theorem is as a pragmatic reduction of chances to degrees of belief (Skyrms Reference Skyrms1984). According to this point of view, chances are nothing but stable components of an agent’s exchangeable credences. What is missing in this picture is an explanation of the prior $\mu$ . The prior $\mu$ ranges over chances. But how can an agent elicit such a prior if chances are only something buried deep in the structure of her credences? The Carnapian approach, being purely predictive, circumvents this question. Exchangeability and Johnson’s sufficientness postulate (as well as regularity) only speak about sequences of observations, and they do so in a way that fully determines a specific prior $\mu$ . These postulates can thus be viewed as an “observational” characterization of a specific class of priors (the Dirichlet priors) that specifies an unobservable theoretical structure.

The second issue has to do with the limitations of exchangeability. Exchangeability, which expresses order invariance, is clearly not always a good assumption. Carnapian inductive systems and de Finetti’s theorem have been extended in many different ways.Footnote ² What most of these generalizations have in common is a commitment to order invariance at some level. For instance, Markov chains involve chances of transitioning from one observational state to another (instead of chances of outcomes simpliciter). Carnapian inductive logic for Markov chains requires order invariance within certain subsequences of observations, namely those that follow a particular outcome. The relevant concept, Markov exchangeability, is weaker than exchangeability, but it does assume a certain kind of order invariance.

However, nonstationary environments don’t exhibit the kind of stability that underlies exchangeability and its generalizations. Instead of stable patterns that persist throughout the observational process, such environments are characterized by transient short-term trends. If we find ourselves in an epistemic situation of this kind, order invariance may not be a plausible assumption.

3. Exponential smoothing

In the forecasting literature, exponential smoothing is one of the most prominent rules for nonstationary epistemic settings. Exponential smoothing goes back to the work of Brown (Reference Brown1959) and Holt (Reference Holt1957) in operations research. Using the notation of the last section, the rule is given by the following update procedure:

(3) $\mathbb{P}(X_{n+1} = i \mid X_1 = x_1, \ldots, X_n = x_n) = \beta^n\mathbb{P}({\boldsymbol X}_1 = i) + (1-\beta) \sum\limits_{m=1}^{n} \beta^{n-m} \chi_{i}(x_m).$

Here, the parameter $\beta$ is in [0,1], and $\chi_{i}$ is the indicator of outcome type i (i.e., $\chi_{i}(x_m) = 1$ if $x_m = i$ and $\chi_{i}(x_m) = 0$ otherwise). According to (3), the initial probability of i, $\mathbb{P}(X_1 = i)$ , influences its predictive probability but is discounted, unless $\beta = 1$ , in which case there is no learning. The remaining predictive weight, $1-\beta$ , comes from observations of i. More recent observations of i are weighed more heavily, the weights being regulated exponentially by the parameter $\beta$ .

In contrast to Carnapian inductive logic and most of its variants, exponential smoothing ages out older observations. As a result, it clearly fails to be order invariant. The time step at which an observation of type i was made matters a lot in determining the predictive probability of observing another i. Thus, exponential smoothing is a qualitatively plausible predictive system for settings in which trends and patterns do not need to persist for long periods of time.

The question I wish to answer goes beyond the broad features of exponential smoothing and concerns its specific mathematical form. Aging out of past observation can be done in many different ways. Is there something special or principled about the exponential discounting of the past as expressed in (3)? In the next section, we will see that inductive logic can shed some light on this question. It does so by identifying the types of invariance that underlie exponential smoothing. My approach turns out to be different from the usual approach, which is based on ideas about order invariance. While my analysis does not rely on order invariance, it is inspired by an alternative axiomatic derivation of Carnapian inductive logic that was put forward by Costantini (Reference Costantini1979). Costantini’s paper addresses a technical wrinkle in the standard derivation of Carnapian inductive logic, which relies on Johnson’s sufficientness postulate. It is well known that the postulate is empty when there are only two types of observations, for then the number of times one type was observed and the total number of observations determine the number of times the other type was observed. In his paper, Costantini replaced Johnson’s sufficientness postulate with an invariance requirement for relevance quotients. In the context of Carnapian inductive logic, relevance quotients are ratios of the form

${{\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = j)} \over {\mathbb{P}({X_n} = i\mid {{\bf{X}}_{n - 1}})}}.$

This ratio quantifies the relevance an observation j has on the predictive probability of i, given the history of observations ${\bf X}_{n-1}$ . Costantini’s invariance condition on relevance quotients states that for any $i \ne j$ and $k \ne l$ ,

(4) ${{\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = j)} \over {\mathbb{P}({X_n} = i\mid {{\bf{X}}_{n - 1}})}} = {{\mathbb{P}({X_{n + 1}} = k\mid {{\bf{X}}_{n - 1}},{X_n} = l)} \over {\mathbb{P}({X_n} = k\mid {{\bf{X}}_{n - 1}})}}.$

The relevance quotient on the left side of this equation has the same number of times i has been observed, $n_i$ , in the numerator and the denominator. The right side likewise fixes $n_k$ . The invariance condition says that the effect an observation of type $j \ne i$ has on the predictive probability of i is the same as the effect an observation of type $l \ne k$ has on the predictive probability for k, regardless of i, j and k, l (as long as each pair has distinct outcomes). Costantini’s invariance of relevance quotients entails (1) in the presence of exchangeability and regularity, and it does so whenever there are at least two types of observational outcomes.

In the next section, we will see that invariance conditions for relevance quotients are also key to understanding the principles that underlie exponential smoothing. The other key is a generalization of exchangeability that applies to settings in which order invariance fails.

Before spelling this out in more detail, let me comment briefly on the significance of exponential smoothing. In data science and machine learning, rules of succession (i.e., Carnapian inductive logic) fall under the rubric of additive smoothing. Smoothing methods, such as additive or exponential smoothing, are often used to filter out noise from data in order to discover underlying trends or patterns. This is related to curve fitting, but is mathematically the same as predicting unobserved data points given a set of observed data, which is the principal domain of inductive logic. Exponential smoothing has applications in a wide array of fields.Footnote ³ Especially noteworthy are recent applications of exponential smoothing in machine learning, such as hybrids of exponential smoothing and recurrent neural networks (Smyl Reference Smyl2020; Smyl et al. Reference Smyl, Dudek and Pełłka2023). In these settings, exponential smoothing is used to identify the main components of highly complex data sets. Thus, aside from the intrinsic value of understanding the inductive logic of exponential smoothing, the principles studied in the next section may also enhance our understanding of certain machine learning algorithms that utilize exponential smoothing.

4. Inductive logic for exponential smoothing

It is well known that exponential smoothing (3) can be equivalently expressed as a convex combination of previous predictive probabilities and current observations. We prove this result for the sake of completeness.

Lemma 1. Exponential smoothing (3) is equivalent to

(5) $\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}, X_n = x_n) =\beta \mathbb{P}({\it X}_n = i \mid {\bf X}_{n-1}) + (1-\beta) \chi_{i}(x_n).$

Proof. Suppose (3) holds for all n. Then

$\beta \mathbb{P}(X_n = i \mid X_1 = x_1, \ldots, X_{n-1} = x_{n-1}) + (1-\beta) \chi_{i}(x_n)$

is equal to

$\beta \big[\beta^{n-1}\mathbb{P}(X_1 = i) + (1-\beta) \sum\limits_{m=1}^{n-1} \beta^{n-1-m} \chi_{i}(x_m)\big] + (1-\beta) \chi_{i}(x_n),$

which simplifies to

$\beta^n\mathbb{P}(X_1 = i) + (1-\beta) \sum\limits_{m=1}^{n} \beta^{n-m} \chi_{i}(x_m).$

This yields one direction. Now suppose that (5) holds. Then (3) follows by simply applying the recursion to each n. QED

The convex combination (5) says that the old predictive probability of i receives weight $\beta$ , with the remaining weight being attached to a new observation of i if it occurs; otherwise, the new weight goes to a different category.

I am now going to introduce a set of postulates that together suffice to derive exponential smoothing as a predictive system. In what follows, we consider an infinite sequence of random variables $X_1, X_2, \ldots$ taking on values in a finite set that has at least two elements. The finite case is a bit more complex and will be discussed in the next section.

As mentioned above, regularity is a basic assumption in Carnapian inductive logic. We also assume it for exponential smoothing, since it guarantees that all conditional probabilities are elementary. Formally, regularity says that

(6) $\mathbb{P}(X_1 = x_1, \ldots, X_n = x_n) \gt 0$

for all n and all possible combinations $x_1, \ldots, x_n$ of observational outcomes.

The second assumption is new and has more substance. It was introduced in Berti et al. (Reference Berti, Pratelli and Rigo2004), who refer to it as distributions being conditionally independent. The sequence $X_1, X_2, \ldots$ is conditionally independently distributed if

(7) $\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n}) = \mathbb{P}(X_{n+k} = i \mid {\bf X}_{n})$

for all n and i, and for all $k\geq 2$ . Conditional independence in distribution is an invariance condition for predictive probabilities. Conditional on the evidence at time n, any future observation of an outcome has the same probability regardless of when it occurs.

Exchangeable sequences of random variables are conditionally independently distributed, but a conditionally independently distributed sequence of random variables can fail to be exchangeable (Berti et al. Reference Berti, Pratelli and Rigo2004, Reference Berti, Dreassi, Pratelli and Rigo2021). As it turns out, conditional independence in distribution is the probabilistic invariance required for an inductive logic of exponential smoothing, where it plays the same role as exchangeability does in the standard setting. However, its significance goes beyond this particular context. It is a cornerstone of predictive approaches in statistics (Fong et al. Reference Fong, Holmes and Walker2023; Falck et al. Reference Falck, Wang and Holmes2024).

The remaining postulates make use of relevance quotients. They are slightly different from Costantini’s relevance quotients (4) in that they refer to $X_{n+1}$ in the numerator and the denominator. In the presence of conditional independence, there is no difference between the two ways of expressing relevance quotients, because (7) implies that $\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}) = \mathbb{P}(X_n = i \mid {\bf X}_{n-1})$ and $\mathbb{P}(X_{n+1} = i) = \mathbb{P}(X_n = i)$ . Since exchangeability implies conditional independence (7), $X_{n+1}$ could be used for Costantini’s relevance quotients (4) in the same way. I prefer using just $X_{n+1}$ because it fixes a single hypothesis about future observations that is evaluated in the context of inductive relevance.

I will refer to the first postulate about relevance quotients as invariance of inductive relevance. It comes in two parts. The first part states that

(8) ${{\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = j)} \over {\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}})}} = {{\mathbb{P}({X_{n + 1}} = i\mid {X_1} = j)} \over {\mathbb{P}({X_{n + 1}} = i)}}$

for all n, i, and $j \ne i$ : the relevance j has for an outcome i at time $n+1$ is the same as the initial relevance j has on that outcome regardless of the intermediate history. The same is assumed for the relevance an outcome has on all other categories taken together. That is, for all n and for all outcomes i,

(9) ${{\mathbb{P}({X_{n + 1}}\ne i\mid {{\bf{X}}_{n - 1}},{X_n} = i)} \over {\mathbb{P}({X_{n + 1}}\ne i\mid {{\bf{X}}_{n - 1}})}} = {{\mathbb{P}({X_{n + 1}}\ne i\mid {X_1} = i)} \over {\mathbb{P}({X_{n + 1}}\ne i)}}.$

Clearly, (9) does not just follow from (8), or vice versa. The denominators and numerators in (9) are sums, but the ratio of these sums is generally not equal to the sum of ratios like (8). Equation (9) is a somewhat roundabout way of expressing the inductive relevance that a category has on itself.

I call the final postulate invariance of initial relevance. It also has two parts. First,

(10) ${{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_2} = i)}} = {{\mathbb{P}({X_2} = k\mid {X_1} = l)} \over {\mathbb{P}({X_2} = k)}}$

for all i, j, k, l such that $i \ne j$ and $k \ne l$ . Second,

(11) ${{\mathbb{P}({X_2}\ne i\mid {X_1} = i)} \over {\mathbb{P}({X_2}\ne i)}} = {{\mathbb{P}({X_2} = j\mid {X_1} = i)} \over {\mathbb{P}({X_2} = j)}}$

for all i and all $j \ne i$ . According to (10), the initial relevance an outcome has on another is the same across all pairs of distinct outcomes, and (11) says that the relevance an outcome has on all the others, taken together, is the same as the relevance it has on each distinct outcome, taken individually. Thus, there is no privileged outcome that is affected by i more or less positively than the other categories.

It is now not difficult to prove that exponential smoothing can be derived from the aforementioned axioms.

Theorem 1. Suppose $X_1, X_2, \ldots$ is an infinite sequence of random variables taking values in a finite set $\{1, \ldots, t\}$ , $t \geq 2$ , that is regular (6), conditionally independent (7), satisfies invariance of inductive relevance, (8) and (9), as well as invariance of initial relevance, (10) and (11). Then there exists a $\beta \in (0,1]$ such that

$\mathbb{P}(X_{n+1} = i \mid X_1 = x_1, \ldots, X_n = x_n) = \beta^n\mathbb{P}(X_1 = i) + (1-\beta) \sum\limits_{m=1}^{n} \beta^{n-m} \chi_{i}(x_m)$

for all n. The parameter $\beta$ is given by $\beta = {\mathbb{P}(X_2 = i \mid X_1 = j)}/{\mathbb{P}(X_1 = i)}$ , where i and j can be any two distinct outcomes.

Proof. It suffices to show that there is a $\beta \in (0,1]$ such that the convex recursion (5) holds for all n. Regularity is used throughout the proof without being mentioned explicitly.

We consider two cases. First, suppose $x_n = j \ne i$ . Then (5) is given by

(12) $\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}, X_n = x_n) = \beta \,\mathbb{P}(X_{n} = i \mid {\bf X}_{n-1}).$

Let

${\beta _{ij}} = {{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_1} = i)}}.$

Conditional independence in distribution (7) implies

${{\mathbb{P}({X_{n + 1}} = i\mid {X_1} = j)} \over {\mathbb{P}({X_{n + 1}} = i)}} = {{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_1} = i)}} = {\beta _{ij}}.$

Invariance of inductive relevance (8) and (7) imply that

${\beta _{ij}} = {{\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = j)} \over {\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}})}} = {{\mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = j)} \over {\mathbb{P}({X_n} = i\mid {{\bf{X}}_{n - 1}})}}.$

Hence,

$\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}, X_n = j) = \beta_{ij} \mathbb{P}(X_{n} = i \mid {\bf X}_{n-1}).$

Now, invariance of initial relevance (10) implies that $\beta_{ij} = \beta_{kl}$ for all $i \ne j$ and $k \ne l$ . So we can set $\beta = \beta_{ij}$ for any two distinct outcomes i,j. As a consequence, (12) holds for this $\beta$ .

For the second case, consider again (5) and let $x_n = i$ . We need to show that for the $\beta$ determined in the previous case the recursion (5) is satisfied:

(13) $\eqalign{\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}, X_n = i) & = \beta \mathbb{P}(X_{n} = i \mid {\bf X}_{n-1}) + 1-\beta \nonumber \\ & = 1 - \beta(1 - \mathbb{P}(X_{n} = i \mid {\bf X}_{n-1})).}$

To this end, note first that (7) and invariance of inductive relevance (9) imply that

${{\mathbb{P}({X_{n + 1}}\ne i\mid {{\bf{X}}_{n - 1}},{X_n} = i)} \over {\mathbb{P}({X_{n + 1}}\ne i\mid {{\bf{X}}_{n - 1}})}} = {{\mathbb{P}({X_2}\ne i\mid {X_1} = i)} \over {\mathbb{P}({X_2}\ne i)}},$

while invariance of initial relevance, (10) and (11), together with (7) yield

$\eqalign{{{\mathbb{P}({X_2}\ne i\mid {X_1} = i)} \over {\mathbb{P}({X_2}\ne i)}} & = {{\mathbb{P}({X_2} = j\mid {X_1} = i)} \over {\mathbb{P}({X_2} = j)}} = {{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_2} = i)}} \cr & = {{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_1} = i)}} = \beta, }$

where $\beta$ is the same number as in the previous paragraph. Putting these equations together, we get

${{1 - \mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}},{X_n} = i)} \over {1 - \mathbb{P}({X_{n + 1}} = i\mid {{\bf{X}}_{n - 1}})}} = \beta .$

This is equivalent to

$1 - \mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1}, X_n = i) = \beta (1 - \mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1})).$

Equation (13) follows from this by applying (7) to $\mathbb{P}(X_{n+1} = i \mid {\bf X}_{n-1})$ . Altogether, this shows that there exists a $\beta$ such that (3) holds for all n and i.

It remains to show that $0 \lt \beta \leq 1$ . If $\beta \lt 0$ , then

${{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_1} = i)}} \lt 0,$

which is impossible. If $\beta = 0$ , then

${{\mathbb{P}({X_2} = i\mid {X_1} = j)} \over {\mathbb{P}({X_1} = i)}} = 0.$

Hence $\mathbb{P}(X_2 = i \mid X_1 = j) = 0$ , and so

$\mathbb{P}(X_1 = i, X_2 = j) = \mathbb{P}(X_2 = i \mid X_1 = j) \mathbb{P}(X_1 = j) = 0,$

which contradicts regularity.

Now suppose $\beta \gt 1$ . We know that, for all n,

$\mathbb{P}(X_{n+1} = i \mid X_1 = x_1, \ldots X_n = x_n) = \beta^n \mathbb{P}(X_1 = i) + (1-\beta) \sum_{j=m}^{n} \beta^{n-m} \chi_{i}(x_m).$

Suppose that $x_m \ne i$ for $m = 1, \ldots, n$ . By regularity, this event has positive probability. Then

$\mathbb{P}(X_{n+1} = i \mid X_1 = x_1, \ldots, X_n = x_n) = \beta^n \mathbb{P}(X_1 = i).$

This can be done for any n. Since $\beta \gt 1$ , $\beta^n \to \infty$ . Hence, because $\mathbb{P}(X_1 = i) \gt 0$ there is an N such that

$\mathbb{P}(X_{n+1} = i \mid X_1 = x_1, \ldots, X_N = x_N) \gt 1,$

which is impossible. QED

6. Kernels and strategies

Exponential smoothing is an example of a class of updating strategies explored in Berti et al. (Reference Berti, Dreassi, Pratelli and Rigo2021, Reference Berti, Dreassi, Leisen, Pratelli and Rigo2023) and Fong et al. (Reference Fong, Holmes and Walker2023). To study their common structure more generally, we will drop the assumption of having only a finite set of outcomes $\{1, \ldots, t\}$ . Everything I say below applies to this case as well, of course, but the greater generality will allow us to gain insight into connections between different predictive systems and their points of contact with Carnapian inductive logic.

Let $\Omega$ be a Polish space and $\mathfrak{B}$ its Borel sigma-algebra, and let $(\Omega^\infty, \mathfrak{B}^\infty)$ be the corresponding infinite product space ( $\Omega^\infty$ is the the set of all infinite sequences of elements in $\Omega$ and $\mathfrak{B}^\infty$ the corresponding product sigma-algebra). The space $(\Omega, \mathfrak{B})$ is the set of observable outcomes, and $(\Omega^\infty\!,\mathfrak{B}^\infty)$ can be thought of as representing the process of repeatedly observing these outcomes.

A kernel, $\kappa$ , on $(\Omega, \mathfrak{B})$ is a measurable map from $\Omega$ to the set of all probability distributions on $(\Omega, \mathfrak{B})$ , denoted by $\mathscr{P}$ . More precisely,

• • $\kappa(\omega)$ is in $\mathscr{P}$ for all $\omega$ in $\Omega$ , and

• • $\kappa(\cdot)(B)$ is $\mathfrak{B}$ -measurable for every B in $\mathfrak{B}$ .

In the theory of Markov chains, kernels give the probabilities of transitioning from a state to any measurable set of states. In the present context, $\kappa$ can be understood as expressing the effect of an observation x on the probabilities assigned to the next observation. This role will become clearer after we have introduced the notion of a strategy, which goes back to Dubins and Savage (Reference Dubins and Savage1965). Let $\mathfrak{B}^n$ be the sigma algebra of subsets of $\Omega^\infty$ generated by the n-fold product of $\mathfrak{B}$ . Heuristically, $\mathfrak{B}^n$ captures all events about the first n observations. A strategy $\sigma$ is a sequence of measurable mappings $\sigma_n$ from $\Omega^\infty$ to $\mathscr{P}$ . More precisely,

• • $\sigma_0$ is an element of $\mathscr{P}$ and $\sigma_n(\omega^\infty)$ is in $\mathscr{P}$ for each $\omega^\infty \in \Omega^\infty$ , and

• • $\sigma_n(\cdot)(B)$ is $\mathfrak{B}^n$ -measurable for each $B \in \mathfrak{B}$ for $n = 1,2, \ldots$

Taking $(\Omega^\infty, \mathfrak{B}^\infty)$ as a model of an observational process, a strategy $\sigma$ defines a unique predictive distribution $\sigma_n$ that depends only on the first n observations. Thus, a strategy $\sigma$ can be thought of as an inductive logic.

Kernels can be used to define strategies (Berti et al. Reference Berti, Dreassi, Pratelli and Rigo2021, Reference Berti, Dreassi, Leisen, Pratelli and Rigo2023). For example, for any n let $\beta_n \;\colon \;\Omega^\infty \to [0,1]$ be $\mathfrak{B}^n$ -measurable and let $\kappa$ be a kernel on $(\Omega, \mathfrak{B})$ . Then

(17) $\sigma_{n+1}(\omega^\infty)(B) = \beta_n(\omega^\infty) \sigma_n(\omega^\infty)(B) + (1- \beta_n(\omega^\infty)) \kappa(\omega_{n+1})(B)$

defines a strategy, where $\omega_{n+1}$ is the $(n+1)$ st element of $\omega^\infty \!$ . The kernel $\kappa$ expresses the influence of the most recent observation on predictive probabilities and is weighed by $1-\beta_n$ against the immediately preceding predictive probability $\sigma_n$ , which has weight $\beta_n$ . Thus, predictive probabilities are convex combinations in which the influence of the kernel is tempered by past predictive assessments with weights that are allowed to depend on the full history of the process.Footnote ⁴

As noted in Berti et al. (Reference Berti, Dreassi, Pratelli and Rigo2021, Reference Berti, Dreassi, Leisen, Pratelli and Rigo2023), it follows from the Ionescu–Tulcea theorem that for any strategy $\sigma$ there exists a probability measure $\mathbb{P}_\sigma$ on $(\Omega^\infty,\mathfrak{B}^\infty)$ that almost surely agrees with the predictive probabilities $\sigma_n$ for every n. That is, let $X_n (\omega^\infty) = \omega_n$ , where $\omega_n$ is the nth coordinate of $\omega^\infty$ . Then

(18) $\mathbb{P}_\sigma(X_{n+1} \in B \mid \mathfrak{B}_n)(\omega^\infty) = \sigma_n(\omega^\infty)(B)$

for all $\omega^\infty$ in a set of $\mathbb{P}_\sigma$ -probability one and all B in $\mathfrak{B}$ . This allows us to seamlessly switch between two points of view: one where we have a probability over the infinite model of an observational process, and another where we have sequences of conditional probabilities that assign predictive probabilities to the next stage.

If the kernel $\kappa$ satisfies two conditions (see (19) and (20) below), then strategies (17) give rise to a conditionally independent distribution: the random variables $X_1, X_2, \ldots$ in (18) are conditionally independently distributed according to $P_\sigma$ (Berti et al. Reference Berti, Dreassi, Pratelli and Rigo2021, Theorem 5).

Exponential smoothing is clearly an instance of this scheme. To see this, take $\beta_n (\omega^\infty\!) = \beta$ for all n and all $\omega^\infty\!$ , and let the kernel $\kappa$ be given by the Dirac delta function:

${\delta _\omega }\left( B \right) = \left( {\matrix{ 1 \hfill & {{\rm{if}}\omega \in {\rm{B}},} \hfill \cr 0 \hfill & {{\rm{otherwise}}.} \hfill \cr } } \right.$

By Lemma 1 this characterizes exponential smoothing in the finite case (just substitute $\delta$ for the indicator function with $B = \{i\}$ ). The kernel $\delta$ captures the way in which the most recent observation influences the predictive probabilities of future outcomes in perhaps the simplest way possible. It says that only the observation that is actually made matters for prediction. Its influence is hedged against previous predictive probabilities by the weights $\beta$ and $1-\beta$ . Carnapian inductive logic has the same kernel $\delta$ . If we set

${\beta _n} = {{n + \lambda } \over {n + 1 + \lambda }},$

then (17) is equivalent to the Carnapian rule (1) when $\Omega$ is finite, with $\alpha_i = \lambda \sigma_0(\{i\})$ for each i in $\Omega$ . The kernel $\delta$ , together with the weights, expresses something similar to Johnson’s sufficientness postulate: Only the observations actually made matter while their positions in the data stream are irrelevant.

The kernel $\delta$ thus is the common core of exponential smoothing and Carnapian inductive logic. Only the weights $\beta_n$ are different. But there are many epistemic situations where $\delta$ is not a plausible choice of kernel. I already mentioned symmetries like partial exchangeability. There also has been a long debate in inductive logic about the importance of analogy in predictive inference.Footnote ⁵ One type of analogy is the influence an observation may have on the predictive probability of similar outcomes. With a kernel like $\delta$ , there is no such analogy influence. But the kernel framework is very flexible and allows the investigator to express all kinds of relationships between different observations. For several examples of practical significance see Berti et al. (Reference Berti, Dreassi, Pratelli and Rigo2021, Reference Berti, Dreassi, Leisen, Pratelli and Rigo2023) and Fong et al. (Reference Fong, Holmes and Walker2023). In particular, a kernel $\kappa$ can assign a distribution over outcomes that is guided by a similarity relation between outcomes. The only constraints that kernels are usually asked to satisfy are the following (Berti et al. Reference Berti, Dreassi, Pratelli and Rigo2021, 705). First, for any B in $\mathfrak{B}$ ,

(19) $\sigma_0(B) = \int_{\Omega} \kappa(\omega)(B) \sigma_0(d \omega).$

Second, there is a set A in $\mathfrak{B}$ such that $\sigma_0(A) = 1$ and for any $\omega$ in A and B in $\mathfrak{B}$ ,

(20) $\kappa(\omega)(B) = \int_\Omega \kappa(\omega')(B) \kappa(\omega)(d \omega').$

The first constraint (19) is known in Bayesian epistemology as the “weak reflection principle.” If $\kappa$ is considered the future probability assignments of an agent, (19) says that the prior expectation of the future probability of any event is equal to its current probability. The weak reflection principle and its variants are often understood as touchstone principles of rational learning.Footnote ⁶

The second constraint (20) is not as well known as the weak reflection principle. It goes back to at least Blackwell (Reference Blackwell1942), who referred to it as “idempotence.”Footnote ⁷ Idempotence can be thought of as a weak reflection principle for the kernel $\kappa$ itself. From the point of view of each $\kappa(\omega)$ , all possible updates agree with the probabilities assigned by $\kappa(\omega)$ on average. This means that different updates $\kappa$ endorse each other as legitimate. The kernel, as a whole, is internally consistent.

The two principles—weak reflection and idempotence—are very mild. They put almost no limits on what can serve as a kernel. In particular, a large number of analogical influences that observational outcomes may have on one another become expressible in the language of kernels.

In mathematics, the term “kernel” is used to refer to different objects. One is that of a transition probability, which we have been using so far. Another one is being actively investigated in machine learning, especially in the context of the apparent ability of deep learning methods to both interpolate observed data and generalize to new observations by massive overparametrization; this is in contrast to received statistical wisdom, which holds that overparametrization allows interpolation of existing data but generalizes poorly to new situations. Belkin (Reference Belkin2021) provides an in-depth overview and discussion of the salient issues. He focuses on standard deep learning: optimizing parameters of deep neural networks to obtain a predictive function, for example in classification and regression problems. In Bayesian deep learning, Bayesian inference is applied to updating a prior over parameters to posteriors in order to account for uncertainty. In both cases, kernel machines can be used to understand the success of certain deep neural networks. Belkin (Reference Belkin2021) discusses kernels in the setting of standard deep learning. Similar kernels play an important role in Bayesian deep learning (see Wilson (Reference Wilson2014) and Murphy (Reference Murphy2023), Chapters 17 and 18). What they have in common is the idea that a kernel fixes the degree of similarity between data points. This allows inference from the outputs associated with previously observed input data $x_i$ to the outputs associated with a newly observed input datum x.

Kernels, in this strand of the literature, are defined very generally as functions from pairs of data into the real numbers that usually satisfy certain smoothness properties, and not as transition probabilities as in Bayesian predictive inference. But there is a family resemblance between these two types of kernels. Both establish similarity connections between observational outcomes and are versatile tools for making predictive inferences within a broadly probabilistic framework. From the point of view of inductive logic, they can be thought of as expressing analogy influences for predictive inferences, where analogy influences are expressed by fixing a kernel.

Bayesian predictive inference goes significantly beyond the kernels introduced above, in a way that may prove useful for inductive logic and for machine learning. A kernel $\kappa$ determines how the most recent observational outcome influences predictive probabilities. This implicitly assumes that the predictive influence of that outcome is independent of the history of the observational process. This might be a reasonable assumption in many, but certainly not all, situations. However, the approach to predictive inference through strategies already incorporates the conceptual tools to generalize prediction beyond kernels. Instead of a kernel, one can fix a strategy $\alpha$ that captures predictive influences, as in Berti et al. (Reference Berti, Dreassi, Leisen, Pratelli and Rigo2023). More precisely, let $\alpha$ be a strategy and define the strategy $\sigma$ by

(21) $\sigma_{n+1}(\omega^\infty)(B) = \beta_n(\omega^\infty) \sigma_n(\omega^\infty)(B) + (1- \beta_n(\omega^\infty)) \alpha_{n+1}(\omega^\infty)(B).$

In (21), the influence of the most recent observation $\omega_{n+1}$ is not just hedged against the last predictive probability, $\sigma_n$ , as in (17), but also potentially mediated through the observations $\omega_1, \ldots, \omega_n$ .Footnote ⁸ This allows for very fine-grained evaluations of future outcomes. The kernel methodology can be recovered by considering Markovian strategies, where the strategy $\alpha$ depends only on the most recent observation. Many intermediate cases are possible. For example, $\alpha$ may depend on two or three previous observations. This makes room for analogy influences that are considerably more general than the similarity relations between individual data that can be expressed within a kernel framework.

How does the inductive logic approach to exponential smoothing and, more generally, to predictive updating compare to the approach through kernels and strategies? According to the latter approach, the impact of observations on predictive probabilities is expressed by kernels or, more generally, strategies, whereas within inductive logic they are expressed by qualitative postulates for conditional probabilities. The two approaches are complementary. They both improve our understanding of certain predictive regimes. One salient difference between the two lies in the qualitative nature of Carnapian inductive logic. The methodology surveyed in this section requires the choice of a particular kernel, or strategy, that is, a function that takes on real values expressing similarities between data. Carnapian inductive logic, by contrast, considers classes of probability assignments and aims to isolate qualitative principles that give rise to particular families of predictive systems. By qualitative principles I mean relations between unconditional probabilities and conditional probabilities, such as exchangeability, Johnson’s sufficientness postulate, conditional independence, or the invariance constraints for relevance quotients discussed above. As such, Carnapian inductive logic provides a distinct contribution to our understanding of predictive systems.

7. Exponential smoothing and exchangeability

Exchangeability implicitly assumes a kind of stationarity that is absent in exponential smoothing. By de Finetti’s theorem, exchangeable distributions can be represented as mixtures of independent and identically distributed data streams. There appears to be no such inherent underlying structure in exponential smoothing. In fact, exponential smoothing seems to be designed for time-varying learning environments. There are, nevertheless, close connections between exponential smoothing and general conditionally independent distributions, on the one hand, and exchangeable distributions, on the other.

The key insight is that, even if they fail to be exchangeable, conditionally independent distributions are asymptotically similar to exchangeable distributions.Footnote ⁹ There are two main results supporting this insight. Let $(\Omega^\infty, \mathfrak{B}^\infty)$ and $X_n$ be as in the preceding section. Suppose that $X_1, X_2, \ldots$ is conditionally independently distributed according to some measure $\mathbb{P}$ . Consider the empirical measures

${\mu _n}({\omega ^\infty })(B) = {1 \over n}\sum\limits_{m = 1}^n {{\delta _{{X_m}({\omega ^\infty })}}} (B),$

where B is in $\mathfrak{B}$ . The measures $\mu_n$ quantify the average number of times the observational process visits B. Each $\mu_n$ is a random probability measure on $(\Omega,\mathfrak{B})$ . It can be shown that the empirical measures almost surely converge weakly to a random probability measure $\mu$ on $(\Omega,\mathfrak{B})$ . More precisely,

$\mu_n(\omega^\infty)(B) \to \mu(\omega^\infty)(B)$

for every B is in $\mathfrak{B}$ and for every $\omega^\infty$ in a set of $\mathbb{P}$ -probability one. Since the sequence $\mu_n$ is bounded, we have

$\eqalign{{\mathbb E}(\mu (B)\mid {X_1}, \ldots, {X_n}) &= \mathop {\lim }\limits_k {\mathbb E}({\mu _k}(B)\mid {X_1}, \ldots, {X_n}) = \mathop {\lim }\limits_k {1 \over k}\sum\limits_{i = n + 1}^k {\mathbb{P}} ({X_i} \in B\mid {X_1}, \ldots, {X_n}) \\&= \mathbb{P}({X_{n + 1}} \in B\mid {X_1}, \ldots, {X_n}).}$

The last equality follows from the assumption that the sequence of random variables is conditionally independently distributed. The second equality follows since the first n terms of the sum, which are either zero or one, vanish in the limit. Since the sequence of conditional probabilities $\mathbb{P}(X_{n+1} \in B \mid X_1, \ldots, X_n)$ is a martingale, the martingale convergence theorem implies that

$\mathbb{P}(X_{n+1} \in B \mid X_1, \ldots, X_n) \to \mu(B)$

almost surely.

Everything that has been said thus far also holds if the observational process is exchangeable. But in the exchangeable case, something stronger is true. The random measure $\mu$ completely determines the full probability distribution of an exchangeable sequence of random variables. By de Finetti’s theorem, the sequence is distributed independently with common measure $\mu$ , and the uncertainty about $\mu$ is expressed by a prior over the set of all probability measures on $(\Omega,\mathfrak{B})$ .

In the exchangeable case, the measure $\mu$ can be understood as representing the chance setup that generates the observational process. This is the stationarity underlying an exchangeable process. For exponential smoothing and other predictive systems that are conditionally independently distributed but not exchangeable, $\mu$ cannot be interpreted in this way. Rather, from what has been said above it follows that $\mu$ represents the best estimate of the predictive probability measure and that the predictive probability of an observable outcome is, in turn, the expectation of its best estimate.

A second result shows that the connection between exponential smoothing and exchangeability is even closer. Suppose $X_1, X_2, \ldots$ is not exchangeable but is conditionally independently distributed according to a probability $\mathbb{P}$ . Then there exists a sequence of random variables $Y_1, Y_2, \ldots$ that is exchangeable with probability distribution $\mathbb{Q}$ , and

$(X_n, X_{n+1}, \ldots) \to (Y_1, Y_2, \ldots)$

in distribution as $n \to \infty$ . Thus, the probability of $(X_n, X_{n+1}, \ldots)$ being in a certain range of values is approximately equal to the probability of $(Y_1, Y_2, \ldots)$ being in that range of values. This does not guarantee that all probabilistic inferences drawn based on the first sequence are the same as those drawn about the exchangeable sequence, but it does guarantee that bounded random variables that depend on the sequences are evaluated the same asymptotically. As a result, there is a sense in which any sequence of random variables that is conditionally independently distributed is asymptotically exchangeable (even if it is not outright exchangeable).

The connection between exponential smoothing and exchangeability speaks to a point raised by Kuipers (Reference Kuipers and Helman1988) about analogy by proximity in the past. According to Kuipers, it seems doubtful that there are “objective processes” that would give rise to an inductive rule like exponential smoothing. For Carnapian inductive logic there is such a process. As we have seen in section 2, from a parametric Bayesian perspective Carnapian inductive rules derive from independent multinomial distributions, which can be thought of as an underlying objective stochastic process whose parameters are estimated by the inductive rule. By the result described above, exponential smoothing approximates such a process in the long run, even though it is not fully determined by it. This process can be thought of as an objective process that is weakly connected to exponential smoothing.

8. Invariance axioms as an expression of the meaning of data

For a subjective Bayesian, invariance assumptions and probabilistic symmetries are simply the probability judgments of a rational agent (de Finetti Reference de Finetti1974; Skyrms Reference Skyrms1984; Jeffrey Reference Jeffrey1992; Huttegger Reference Huttegger2017). If an agent’s subjective probabilities are exchangeable, they can be represented as a mixture of independent and identically distributed probabilities by de Finetti’s theorem. If they are, in addition, regular and satisfy Johnson’s sufficientness postulate, they give rise to a Carnapian predictive system. The same is true for the invariance postulates that underlie exponential smoothing. Yet, whether subjective probabilities satisfy any of these conditions is a contingent fact about an agent. In particular, there is nothing irrational about probabilities that fail to be exchangeable or to possess some other kind of invariance.

Carnap, at least in his early writings on the subject, had a different view of invariance conditions. In Carnap (Reference Carnap1950) he thought of inductive logic as a branch of formal semantics. On this view, invariance conditions reflect certain features of the underlying language. The language is first-order with relations and individual constants. Exchangeability is a consequence of considering probability as a generalization of logical consequence.Footnote ¹⁰ The probability of a statement given another is thought of as expressing the degree of support the second has on the first. Since logical consequence is invariant under replacing the individual constants of the language, probability should, according to Carnap, have the same invariance.

Carnapian inductive logic takes probabilistic invariance conditions to be determined by semantic features of the underlying logical language. Bayesian inductive logic also has an underlying conceptual framework (usually a set-theoretic and not a language-based one), but probabilistic invariance conditions are additional independent constituents of the inductive system. The two approaches obviously have a lot in common. Some commentators have claimed that Carnap’s views evolved toward an effectively Bayesian position.Footnote ¹¹ There is some evidence for this perspective, but it can also be criticized for good reasons, especially the fact that Carnap himself never seems to have endorsed an interpretation of probability as subjective degree of belief, emphasizing instead the analyticity of probability assignments within inductive logic (Sznajder Reference Sznajder2016). Needless to say, there is a large number of open questions, both historical and systematic, around Carnapian and Bayesian inductive logic, what their points of contact are, and how they potentially differ. This is not the place to embark on the extensive enterprise of discussing these issues. What I can do here is point out a role of probabilistic invariance and symmetry assumptions that I think both Carnapians and Bayesians can endorse.

What I’m going to suggest is not an original point of view. It is implicit in some of the writings of Carnap and de Finetti, and it has been expressed in one form or another by Bayesians who try to engage with the role of priors in inductive inference instead of reflexively dismissing them as arbitrary. Within a sufficiently rich setting, certain features of a prior probability assignment can be understood as “inductive assumptions” in the sense of Howson (Reference Howson2000) and Romeijn (Reference Romeijn2005). In order to explain inductive assumptions, it will be helpful to refer to a distinction between “data” and “evidence” in philosophy of statistics. As explained in Morey et al. (Reference Morey, Romeijn and Rouder2016), data are raw entries in statistical databases, such as sequences of zeros and ones, survey responses, or measurement entries. They acquire the status of evidence—in the sense of statistical evidence—by putting them in contact with statistical hypotheses, $h_i$ , in terms of likelihoods, $\mathbb{P}(d \mid h_i)$ , where d represents a stream of raw data. Together with a prior over hypotheses, this allows inferences about the posterior probability of hypotheses and unobserved data (by Bayes’ theorem). One way to think about this is that d, by itself, is a purely syntactical object. It acquires meaning by specifying a theoretical context regarding how it interacts with statistical hypotheses and how it affects the probability of unobserved data. Evidence is therefore context-dependent:

Here, inductive assumptions about an observational process are expressed by adopting a certain set of hypotheses, likelihoods, and priors. What I would like to point out is that often inductive assumptions can alternatively be expressed by invariance conditions on the probabilistic structure of data streams, directly and without recourse to statistical hypotheses. Consider exchangeability as an example. If observational outcomes are assumed to be exchangeable, then all one can see in a data sequence are frequencies. The same is true, by de Finetti’s theorem, in case one assumes an underlying independent multinomial statistical model, as well as for other types of invariance, like Markov exchangeability, which allows one to see nothing but transition frequencies. In all these cases, invariance assumptions and probabilistic symmetries express the meaning of data. One can look at one and the same sequence of observations and pay attention to different aspects of it, depending on the assumed invariance conditions.

The invariance conditions of exponential smoothing serve the same purpose. Whenever a particular observation is made, it has the same influence on inductive probabilities. The influence is fixed from the start and guides the whole process. This is a consequence of conditional independence in distribution and of invariance of inductive relevance. Thus, an agent who adopts these invariance conditions will also count the number of times an observation was made, but will weigh them according to how recently it was made. Such an agent observes different aspects of a data sequence than an agent who adopts an exchangeable probability. They frame the empirical facts under different inductive assumptions.

The doctrine that inductive assumptions (invariance conditions, statistical models) shape the meaning of data is part of the common ground between Bayesian and Carnapian inductive logic. This leaves many questions unanswered. Most importantly, there is an unresolved gap between the semantic role of inductive assumptions—they express the meaning of data—and their epistemological role in inductive inference. Whether we assume exchangeability, invariance of inductive relevance, or any other invariance condition does not just say something about what we see in the data; it has real consequences about how our probabilities respond to evidence. Carnap tried to resolve these issues within his voluntarist metaphilosophy (the tolerance principle), in which inductive assumptions arise from a merely pragmatic choice of a conceptual framework. De Finetti, as mentioned above, confronted them by regarding inductive assumptions as nothing more than expressions of personal belief. Is there a connection between these two approaches? Do they have the same kind of pragmatism in common? Do these approaches account for the rationality of scientific reasoning? I set these questions aside for now and hope to revisit them in the future.