2. Household decision problem

The model is based on the infinite-horizon stochastic household decision problems in Barnett (Reference Barnett and Barnett1995) and Barnett et al. (Reference Barnett and Su1997).Footnote ⁸ The model is general in that preferences are defined over an arbitrary (finite) number of assets and goods, but the form of the utility function is not specified. Results proven for this model, will hold for more restrictive SMIUF models.Footnote ⁹

Following the earlier papers, $S\left ( s\right )$ , the consumption possibility set for any period $s\in \left \{ t,t+1,\ldots, \infty \right \}$ , is defined to be

(1) \begin{equation} S\left ( s\right ) =\left \{\, \left ( \mathbf {a}_{s},A_{s},\mathbf {c}_{s}\right ) \in Y\Biggm | \begin{array}{l} \sum \limits _{j=1}^{n}p_{js}c_{js}\leq \sum \limits _{i=1}^{k}\left [ (1+\rho _{i,s-1})p_{s-1}^{\ast }a_{i,s-1}-p_{s}^{\ast }a_{is}\right ] \\ +\left ( 1+R_{s-1}\right ) p_{s-1}^{\ast }A_{s-1}-p_{s}^{\ast }A_{s}+\mathit {I} _{s} \end{array} \right \} \end{equation}

where, for each period $s$ , $\mathbf {a}_{s}=\left ( a_{1s},\ldots, a_{ks}\right )$ is a $k$ -dimensional vector of planned real asset balances where each element $a_{is}$ has a nominal holding-period yield of $\rho _{is}$ , $A_{s}$ is planned holdings of the benchmark asset which has an expected nominal holding period yield of $R_{s}$ , $\mathbf {c}_{s}=\left ( c_{1s},\ldots, c_{ns}\right )$ is a $n$ -dimensional vector of planned real consumption of non-durable goods and services where each element $c_{js}$ has a price of $p_{js}$ , $\mathit {I}_{s}$ is the nominal value of income from all other sources, which is non-negative, and $p_{s}^{\ast }$ is a true cost-of-living index defined as a function over some non-empty subset of the $p_{js}$ .Footnote ¹⁰ Prices, including the aggregate price, $p_{s}^{\ast }$ , and rates of return are stochastic processes. $Y$ is the feasible set $ \mathbb {R}^{k+1}\times C$ , where $C$ is the household survival set—a subset of the $n$ -dimensional non-negative Euclidean orthant—and $\mathbb {R}$ has the usual meaning; note that the construction allows for short-selling as asset holdings can be negative.

For the stochastic processes, the information set must be specified. It is assumed that current prices and the benchmark rate of return are known at the beginning of each period and current interest on all other assets is realized at the end of each period. More specifically, for all $i$ , $j$ and $s$ , $ p_{js}$ , $p_{s}^{\ast }$ , $R_{s}$ (and $R_{s-1}$ ) and $\rho _{i,s-1}$ are known at the beginning of period $s$ , while $\rho _{is}$ is not known until the end of period $s$ . Since their returns are unknown in the current period, the assets, $\mathbf {a}_{s}$ , are risky, while $A_s$ is the risk-free asset. Despite the uncertainty, the constraint contains only known variables in the current period, so the consumer can satisfy (1) with certainty.

Again following the prior literature, the (representative) consumer maximizes their intertemporally-additive utility, solving at time $t$ ,

(2) \begin{equation} \sup \left \{ \, u\left ( \mathbf {a}_{t},\mathbf {c}_{t}\right ) +E_{t}\left [ \sum _{s=t+1}^{\infty }\left ( \frac {1}{1+\xi }\right ) ^{s-t}u\left ( \mathbf {a} _{s},\mathbf {c}_{s}\right ) \right ] \,\right \} \end{equation}

subject to $\left ( \mathbf {a}_{s},\mathbf {c}_{s},A_{s}\right ) \in S\left ( s\right )$ for all $s\in \left \{ t,t+1,\ldots, \infty \right \}$ and the transversality condition

(3) \begin{equation} \limsup _{s\rightarrow \infty }E_{t}\biggl [\left ( \frac {1}{1+\xi }\right ) ^{s-t}A_{s}\biggr ]=0, \end{equation}

where the operator $E_{t}[\phantom {x}]$ denotes expectations formed on the basis of the information available at time $t$ , and $0\lt \xi \lt \infty$ is the subjective rate of time preference.Footnote ¹¹ The form of the utility function, $u\left ( \cdot \right )$ , is unspecified, but assumed to satisfy standard regularity conditions. Further, the transversality condition rules out unbounded borrowing at the benchmark rate of return. This decision problem resembles a standard money-in-the-utility-function model, but in this stochastic decision problem the assets can be either monetary or financial assets.

The deterministic version of this decision problem has been the core model underlying monetary aggregation since Barnett (Reference Andrikopoulos1977, Reference Aruoba, Fernández-Villaverde and Rubio-Ramírez1980): See Anderson et al. (Reference Anderson, Jones and Nesmith1997) for a summary. It is also the foundation of most of the stochastic money-in-the- utility function models cited previously. However, Barnett and Wu (Reference Barnett and Su2005) extended the model to relax the assumption of intertemporal additive separability and Barnett and Su (Reference Barnett and Park2019) used this result in an application incorporating credit cards.Footnote ¹² This extension, which includes lagged terms in the utility function, still assumes the existence, uniqueness and differentiability of a solution; the results derived in this paper could readily incorporate this extension. Barnett et al. (Reference Barnett and Liu2021) considers recursive utility, which likely would not be covered by the results here; the relation of that paper to this one will be discussed in the next sections.

3. Expectations and dynamic programming

The goal is to prove that the decision problem defined by equations (1)–(3) has a unique solution and to characterize the solution. As the model is recursive, DP is the natural candidate solution method.Footnote ¹³ DP is especially appealing for stochastic problems where states of the dynamic system are uncertain, because the DP solution determines optimal control functions defined for all admissible states. But, for a stochastic model, care must be taken to ensure that DP is feasible. Bertsekas and Shreve (Reference Bertsekas and Shreve1978), (Chapter 1) and Stokey and Lucas Jr. (Reference Stokey and Lucas1989), (Chapter 9) discuss the requirements for implementing stochastic DP in more detail.

Solving the problem requires that a key gap is addressed. The decision problem defined in equation (2) is not fully specified, as the expectations operator $E_{t}[\phantom {x}]$ is not formally defined. This present lack of definition is intentional as it reflects the SMIUF literature; in particular, Barnett (Reference Barnett and Barnett1995), Barnett et al. (Reference Barnett and Su1997) and the subsequent literature using this model, with the exception of Barnett et al. (Reference Barnett and Liu2021), never formally define the (conditional) expectations operator. Further structure is needed before DP or any other solution technique can be applied. In fact, the specification of expectations is intertwined with whether and how the problem can be solved. The specification of expectations also affects how general the problem is economically. In the rest of this section, different specifications will be discussed, looking particularly at the implications for DP and the economic restrictiveness. First, approaches that limit how risk or uncertainty enters the problem will be briefly discussed. Then, the next subsection will cover the more general specification.

1. Contemporaneous certainty: One simple avenue to solve the model using DP is to specify that all current state variables are known with perfect certainty and only future states are uncertain. In that case, deterministic DP results are applicable. It might appear that the problem given by (2) is deterministic in the current period. But, this is not the case; allowing asset returns to be risky introduces contemporaneous uncertainty as the state variable in the next period is stochastic. If contemporaneous certainty was assumed, it would imply that all asset returns are risk-free. Such an assumption is not appropriate for financial models generally and is nonsensical for the literature expressly seeking to extend monetary aggregation to risky assets. This illustrates, in a particularly strong fashion, how assumptions made to solve a problem can affect the economics of the problem.

2. Finite or countable support: Alternatively, the stochastic processes could be assumed to have finite or countable support. Such an assumption would imply that expectations are weighted sums, rather than integrals against some measure. This specification would imply that there are no issues with measurability in applying the DP algorithm. Consequently, the the ability to solve the problem depends only on the problem’s structure and the results from deterministic DP would apply (Bertsekas and Shreve, Reference Bertsekas and Shreve1978, p. 6). Such an approach is not uncommon in economics; for example, much of the exposition in Ljungqvist and Sargent (Reference Ljungqvist and Sargent2004) restricts the stochastic processes to be Markov chains, so the deterministic results carry over.Footnote ¹⁴ Barnett et al. (Reference Barnett and Liu2021) explicitly assumes Choquet expectations; making this assumption (implicitly) restricts the underlying stochastic processes because Choquet expectations are only defined for finite or countable probabilities. However, assuming countability is stronger than the norm in economics and finance, particularly when risk and financial decision-making are the focus. For example, assuming countability does not meet Aliprantis and Border’s (Reference Aliprantis and Border2006) dictum that “[t]he study of financial markets requires models that are both stochastic and dynamic, so there is a double imperative for infinite dimensional models.”

3. Certainty equivalence: Another common approach is to assume certainty equivalence holds. Going back to Simon (Reference Simon1956) and Theil (Reference Theil1957), the certainty equivalent approach consists of replacing stochastic processes with a deterministic function of the process—most commonly their current (conditional) expectations—and solving the resulting deterministic dynamic program; see Van de Water and Willems (Reference Van de Water and Willems1981) for a general formulation for both discrete and continuous problems. Under certain conditions, the solution for the stochastic problem will be the same as this deterministic version. However, the conditions are restrictive. First, certainty equivalence requires that past and present decisions do not affect either the stochastic processes themselves or the information that the decision-maker would have about the processes going forward (Duchan, Reference Blume, Easley and O’Hara1974). Second, certainty equivalence effectively eliminates the relevance of risk; as Parra-Alvarez et al. (Reference Parra-Alvarez, Polattimur and Posch2021) comment, “if risk matters, breaking certainty equivalence is desired in order to account for the effects of risk.” More explicitly, under perfect certainty: a.) decisions are unaffected by risk that does not change the deterministic function of the processes, and, b.) counterfactually, assets do not exhibit any risk premia (Fernández-Villaverde et al., 2016). To illustrate the first point more concretely, under certainty equivalence, if the conditional expectation is used, then the choices made will not depend at all on variance or any other higher-moment of the stochastic process. The implication is that the agent would take no precautionary responses to hedge against risk in either savings or investment decisions.Footnote ¹⁵ Fernández-Villaverde et al. (Reference Fernández-Villaverde, Rubio-Ramírez, Schorfheide, Fernández-Villaverde, Rubio-Ramírez and Schorfheide2016) point out that the resulting lack of any precautionary response to risk biases any welfare analysis. It is not surprising that the second point follows as assumptions affecting welfare also affect structural asset pricing (Alvarez and Jermann, Reference Aliprantis and Border2004). The general relation with risk is very relevant here. And these issues persist even if certainty equivalence is only used as a linear approximation around a steady state (see, Díaz-Giménez, Javier Reference Díaz-Giménez2001; Aruoba et al., Reference Anderson, McGrattan, Hansen and Sargent2006). But the assumptions needed to ensure certainty equivalence holds have further implications for the economics of the decision problem. Certainty equivalence holds generally if the objective function is quadratic and the dynamics and constraints are linear (Amman, Reference Amman1996). With additional assumptions on the conditional expectation of the stochastic processes, certainty equivalence holds under such a Linear-Quadratic framework with a variety of assumptions on the disturbances, starting with the classic assumption of additive Gaussian noise used in Simon (Reference Simon1956), which is still common (see,Ljungqvist and Sargent Reference Ljungqvist and Sargent2004, pp. 113–115), and under various extensions (e.g.,Speyer and Gustafson Reference Speyer and Gustafson1974; Akashi and Nose, Reference Akashi and Nose1975; Van de Water and Willems Reference Van de Water and Willems1981; Anderson et al., Reference Anderson, McGrattan, Hansen and Sargent1996; Derpich and Yüksel, Reference Binner, Chaudhry, Kelly and Swofford2023,among many others). Restricting the objective function to have a quadratic form and the dynamics to be linear are clearly strong assumptions. For utility functions, the quadratic assumption has very strong and counterfactual implications (Pratt, Reference Pratt1964; Jappelli and Pistaferri, Reference Jappelli and Pistaferri2017); furthermore, fixing the utility function to a particular form is inconsistent with how consumer aggregation theory generally proceeds.

Although each of these approaches finds uses, each limits how risk affects the decision and has further impacts on the economics. Perhaps as a consequence, it is common to use the framework first introduced by Blackwell (Reference Barnett, Han and Zhang1965) and assume that the disturbance spaces are Borel spaces.

4. Measurable selection

The challenge with taking a measure theory approach to stochastic DP is that integration against a measure is not well-defined for all functions.Footnote ¹⁸ Therefore, the measurability of functions will be central to the ability to derive a stochastic DP solution. Furthermore, the measurable space cannot be arbitrary.

Instead of assuming compactness and the necessary and arbitrary ancillary continuity assumptions, a solution is sought within a richer set of functions: universally-measurable functions. Relaxing the measurability restrictions, which expands the class of possible solutions, is more appealing than imposing strong restrictions that limit the class of possible solutions. Section 5.3 discusses these trade-offs further.

The following standard definition will be used repeatedly:

Transition functions are commonly used to incorporate stochastic shocks into a functional equation (see Stokey and Lucas Jr. Reference Stokey and Lucas1989, p. 212). The current treatment is similar, except instead of beginning with transition functions, stochastic kernels are employed.

A stochastic kernel is a special case of a regular conditional probability for a Markov process, defined in Shiryaev(Reference Shiryaev1996), (Definition 6, p. 226). In abstract terms, if $x$ is taken to represent the state of the system at time $t$ and the system is Markovian, the conditional expectation operator applied to an element of $Y$ can formally be viewed as an integral against the stochastic kernel, i.e. $E_{t}\left [ f\left ( x,y\right ) \right ] =\int {f(x,y)\,q\,(dy\,|\, x)}$ (Shiryaev, Reference Shiryaev1996,Theorem 3, p. 226). As this definition requires measurability of the function, a measurability assumption is necessary to define expectations, even before a solution is sought. If a function, defined on the Cartesian product of $X$ and $Y$ , is Borel-measurable and the stochastic kernel is Borel-measurable, then $\int {f(x,y)\,q\,(dy\,|\, x)}$ is Borel-measurable (Bertsekas and Shreve, Reference Bertsekas and Shreve1978, Proposition 7.29, p. 144). The implication is that the conditional expectation is Borel-measurable. Integration defined this way operates linearly and obeys classical convergence theorems. Also, the integral is equal to the appropriate iterated integral on product spaces. These statements will be clarified in the next section.

As the state space for the problem has not been defined, it may appear that the stochastic kernel is limited in that it only represents conditional expectations for Markovian systems. In fact, non-Markovian processes can always be reformulated as Markovian by expanding the state space. In the sequel, the state space for the problem will be formulated so that the process is Markovian. Borel measurability, by itself, is not adequate to prove the existence of the solution. Bertsekas and Shreve (Reference Bertsekas and Shreve1978) address this problem through a richer concept of measurability. Their apparatus includes upper semianalytic functions and measurability with regard to the universal $\sigma$ -algebra. Definitions and relevant results are presented below; proofs can be found in Aliprantis and Border (Reference Aliprantis and Border2006) and Bertsekas and Shreve (Reference Bertsekas and Shreve1978).Footnote ¹⁹ The key relation is that the universal $\sigma$ -algebra includes the analytic $\sigma$ -algebra, which in turn includes the Borel $ \sigma$ -algebra. This implies that all Borel-measurable functions are analytically measurable, and that all analytically measurable functions are universally measurable. Thus the move to universal measurability is, relative to the Borel model, relaxing a constraint, instead of imposing a restriction. By moving to universal-measurability, the information set is enriched and the measurability assumptions are technically relaxed. Such a relaxation does no damage to the economics behind the model. The results are standard and the proofs are omitted.

Borel measurability is not adequate for DP, because the orthogonal projection of a Borel set is not necessarily Borel measurable Specifically, if $f\colon X\times Y\rightarrow \mathbb {R}^{\ast }$ , where $\mathbb {R} ^{\ast }$ is the extended real numbers, is given and $f^{\ast }\colon X\rightarrow \mathbb {R}^{\ast }$ is defined by $f^{\ast }(x)=\sup _{y\,\in Y}f(x,y)$ then for each $c\in \mathbb {R}$ , define the set

(4) \begin{equation} \{\,x\in X\,|\, {f^{\ast }(x)\gt c}\}=\textrm {proj}_{X}\left ( \{\,(x,y)\in X\times Y\,|\, {f(x,y)\gt c}\,\}\right ) \end{equation}

where $\textrm {proj}_{X}(\phantom {x})$ is the projection mapping from $ X\times Y$ onto $X$ . If $f(\phantom {x})$ is Borel-measurable then

(5) \begin{equation} \{\,(x,y)\in X\times Y\,|\, {f(x,y)\gt c}\,\} \end{equation}

is Borel-measurable, but $\{\,x\in X\,|\, {f^{\ast }(x)\gt c}\,\}$ may not be Borel-measurable (Blackwell, Reference Barnett, Han and Zhang1965; Davidson, Reference Davidson1994; Stokey and Lucas Jr., Reference Stokey and Lucas1989).

The DP algorithm repeatedly implements such projections, so the conditional expectation of functions like $f^{\ast }(\phantom {x})$ will need to be evaluated, requiring that the function is measurable. This measurability requirement leads to the definition of analytic sets, the analytic $\sigma$ -algebra, and analytic measurability:

Davidson(Reference Davidson1994), (p. 329) refers to analytic sets as “nearly” measurable because, for any measurable space and any measure, $\mu$ , on that space, the analytic sets are measurable under the completion of the measure. The completion of a space with respect to a measure involves setting $\mu \,(E)=\mu \,(A)$ for any set $E$ such that $ A\subset E\subset B$ whenever $\mu \,(A)=\mu \,(B)$ . Effectively, this assigns measure zero to all subsets of measure zero sets (Davidson, Reference Davidson1994), (p. 39).

Analytic sets address the problem of measurable selection within a dynamic program, because, if $X$ and $Y$ are Borel spaces, and, if $A\subset X$ is analytic and $f \colon X \to Y$ is Borel-measurable, then $f(A)$ is analytic. This implies that if $B\subset X\times Y$ is analytic then $ \textrm {proj}_{X}(B)$ is also analytic. Analytic sets are the smallest groups of sets such that the projection of a Borel set is a member of the group (Aliprantis and Border, Reference Aliprantis and Border2006). Analytic sets are used to define upper semianalytic functions as follows:

The following result is key for the application of the DP algorithm:

Two important properties of upper semianalytic functions are that the sum of such functions remains upper semianalytic, and if $f \colon X\to \mathbb {R} ^{\ast }$ is upper semianalytic and $g:Y\to X$ is Borel measurable, then the composition $f\circ g$ is upper semianalytic. Most importantly, the integral of a bounded upper semianalytic function against a stochastic integral is upper semianalytic. This is stated as a lemma:

Semianalytic functions have one relevant limitation. If two functions are analytically measurable, their composition is not necessarily analytically measurable. This difficulty can be overcome moving to the still richer universally measurable $\sigma$ -algebra:Footnote ²⁰

Definition 7 (Universal $ \sigma$ -algebra). Let $X$ be a Borel space, $P(X)$ be the set of probability measures on $X$ , and let $\mathfrak {B}_{X}(\mu )$ denote the completion of $\mathfrak {B}_{X}$ with respect to the probability measure $\mu \in P(X)$ . The universal $\sigma$ -algebra $\mathfrak {U}_{X}$ is defined by

(6) \begin{equation} \mathfrak {U}_{X}=\bigcap \nolimits _{\mu \in P(X)}{\mathfrak {B}_{X}(\mu ).} \end{equation}

If $A\in \mathfrak {U}_{X}$ , $A$ is called universally measurable, and functions that are measurable with respect to $\mathfrak {U}_{X}$ are called universally measurable.

The universally-measurable $\sigma$ -algebra is the completion of the Borel $ \sigma$ -algebra with respect to every Borel measure. Consequently, it does not depend on any specific Borel measure. Note that every Borel subset of a Borel space $X$ is also an analytic subset of $X$ , which implies that the $ \sigma$ -algebra generated by the analytic sets is larger than the Borel $ \sigma$ -algebra. The fact that analytic sets are measurable under the completion of any measure implies that they are universally-measurable, so $ \mathfrak {B}_{X}\subseteq \mathcal {A}_{X}\subseteq \mathfrak {U}_{X}.$

Universal measurability enables the stochastic DP recursion to be implemented. Of course, if a stochastic kernel is Borel-measurable, it is universally measurable. With universally measurable stochastic kernels, integration operates linearly, obeys classical convergence theorems, and iterates on product spaces. This is stated formally as the following theorem:

Theorem 8. Let $X_{1},X_{2},\ldots$ be a sequence of Borel spaces, $Y_{n}=X_{1}\times \ldots \times X_{n}$ , and $\,Y=X_{1}\times X_{2}\times \cdots$ . Let $\mu \in P(X_{1})$ be given and, for $n=1,2,\ldots$ , let $q_{n}(dx_{n+1}\,|\, y_{n})$ be a universally measurable stochastic kernel on $X_{n+1}$ given $Y_{n}$ . Then for $n=2,3,\ldots$ , there exist unique probability measures $r_{n}\in P(Y_{n})$ such that $\forall \underline {{X}}_{1}\in \mathfrak {B}_{X_{1}},\ldots, \underline {{X}}_{n}\in \mathfrak {B}_{X_{n}}$

(7) \begin{align} r_{n}(\underline {{X}}_{1}\cap \underline {{X}}_{2}\cap \cdots \cap \underline { {X}}_{n})=\int \limits _{\underline {{X}}_{1}}{\int \limits _{\underline {{X}} _{2}}\cdots {\int \limits _{\underline {{X}}_{n}}{q_{n-1}(\underline {{X}} _{n}\,|\, x_{1},\ldots, x_{n-1})}}} \nonumber \\ \times q_{n-2}(dx_{n-1}\,|\, x_{1},\ldots, x_{n-2})\ldots \times q_{1}(dx_{2}\,|\, x_{1})\,\mu \,(dx_{1}) \end{align}

If $f\colon Y_{n}\rightarrow \mathbb {R}^{\ast }$ is universally measurable, and the integral is well-defined, Footnote ²¹ then

(8) \begin{align} \int \limits _{Y_{n}}{f\,dr_{n}}=\int \limits _{X_{1}}{\int \limits _{X_{2}}\cdots { \int \limits _{\,X_{n}}{f(x_{1},\ldots, x_{n})\,q_{n-1}(dx_{n}\,|\, x_{1},\ldots, x_{n-1})}}} \nonumber \\ \times q_{n-2}\left (dx_{n-1} | x_{1},\ldots, x_{n-2}\right )\times \ldots \times q_{1}(dx_{2}\,|\, x_{1})\,\mu \,(dx_{1}). \end{align}

There further exists a unique probability measure $r\in P(Y)$ such that for each $n$ the marginal of $\,r$ on $Y_{n}$ is $r_{n}$ .

The formal definition of the conditional expectations operator is, therefore, the integral of the function versus $r_{n}$ or $r$ . This definition allows universally measurable selection:

Theorem 9 (Measurable Selection). Let $X$ and $Y$ be Borel spaces, $ D\in X\times Y$ be an analytic set such that $D_{x}=\{\,y\,\,|\, (x,y)\in D\,\}$ , and $f\colon D\rightarrow \mathbb {R}^{\ast }$ be an upper semianalytic function. Define $f^{\ast }\colon \textrm {proj}_{X}(D)\rightarrow \mathbb {R} ^{\ast }$ by

(9) \begin{equation} f^{\ast }(x)=\sup _{y\in D_{x}}f(x,y). \end{equation}

Then the set $\, I=\{\,x\in \textrm {proj}_{X}(D)\,\,|\,$ for some $y_{x}\in D_{x}$ , $f(x,y_{x})=f^{\ast }(x)\,\}$ is universally measurable, and for every $ \epsilon \gt 0$ , there exists a universally measurable function $\phi \colon \textrm {proj}_{X}(D)\rightarrow Y$ such that $Gr(\phi )\subset D$ and for all $x\in \textrm {proj}_{X}(D)$ either

(10) \begin{gather} f[x,\phi (x)]=f^{\ast }(x)\quad \text {if }\,x\in I \end{gather}

or

(11) \begin{gather} f[x,\phi (x)]\geq \begin{cases} \,f^{\ast }(x)-\epsilon & \text {if $\,x\notin I$ and $f^{\ast }(x)\lt \infty $} \\ \,1/\epsilon & \text {if $\,x\notin I$ and $f^{\ast }(x)=\infty $} \end{cases} \end{gather}

The selector obtained in Theorem9, $ f[x,\phi (x)]$ is universally measurable. If the function $\phi \left ( \phantom {x}\right )$ is restricted to be analytically measurable, then $I$ is empty and (11) holds. In this case, the selector is not necessarily universally measurable. For Borel-measurable functions $\phi ( \phantom {x})$ , the analytic result does not hold uniformly in $x$ . The strong result given by (10) is only available for universally measurable functions. Similarly, strong results are available for Borel-measurable functions if significantly stronger regularity assumptions are maintained.Footnote ²²The weaker regularity conditions are appealing, as they allow a solution without imposing restrictions on the economics of the problem.

5. Stochastic dynamic programming

There are several issues with simply applying DP to the SMIUF decision problem. First, the functional form of the utility function is not specified, as doing so would restrict the applicability of the results. Second, as previously discussed, there are a number of technical difficulties in applying DP methods in a general stochastic setting. Section 3’s measure theory was developed to overcome these difficulties.Footnote ²³

Three tasks are repeatedly performed in the DP recursion. First, a conditional expectation is evaluated. Second, the supremum of an extended real-valued function in two (vector-valued) variables, the state and the control, is found over the set of admissible control values. Finally, a selector which maps each state to a control that (nearly) achieves the supremum in the second step is chosen. Each of these steps involves mathematical challenges in the stochastic context. An especially important technical concern is that the measurability assumptions not be destroyed by any of the three steps.

The first and second steps require that the expectation operator can be iterated and interchanged with the supremum operator. As shown in Section 4, these requirements are met by the integral definition of the expectations operator, for either the Borel- or universally-measurable specifications. Step two encounters a problem with measurability, because of the issue with projections of Borel sets also discussed in the previous section. Analytic-measurability is sufficient to address this particular problem, but such measurability is not necessarily preserved by the composition of two functions. Using semianalytic functions and assuming universal-measurability solves this problem. Universal measurability also allows measurable selection under mild regularity conditions, addressing the third step without assuming that policy is compact-valued and upper hemi-continuous as is done in Stokey and Lucas Jr. (Reference Stokey and Lucas1989). As all three steps of the DP algorithm can be implemented for the stochastic problem, the existence of an optimal or nearly optimal program can be proven. Furthermore, the principal of optimality holds for the (near) optimal value function.

To show that these results are applicable to SMIUF models, the problem laid out in (2) is mapped into Bertsekas and Shreve’s general stochastic DP model. One adjustment is necessary as Bertsekas and Shreve’s define lower semianalytic functions rather than upper semianalytic functions, because their exposition addresses the finding the infinum of a function. This difference requires careful adjustment of their regularity conditions.

5.1 General Framework

Following Bertsekas and Shreve (Reference Bertsekas and Shreve1978), (pp. 188–189), the general infinite horizon model is defined as follows. Denote

1. X State space: a non-empty Borel space;

2. Y Control space: a non-empty Borel space;

3. S Control constraint: a function from $X$ to the set of non-empty subsets of $Y$ . The set $\Gamma =\{\,(x,y) \,|\, x\in X,y\in S(x)\,\}$ is assumed to be analytic in $X\times Y$ ;

4. Z Disturbance space: a non-empty Borel space;

5. q (dz|x, y)] Disturbance kernel: a Borel-measurable stochastic kernel on $Z$ given $X\times Y$ ;

6. f System function: a Borel-measurable function from $X\times Y\times Z$ to $X$ ;

7. β Discount factor: a positive real number; and

8. g One-stage value function: an upper semianalytic function from $ \Gamma$ to $\mathbb {R}^{\ast }$ .

Then we have

Definition 10 (Stochastic Optimal Control Model). An infinite horizon stochastic optimal control model is the eight-tuple $(X,Y,S,Z,q,f,\beta, g)$ .

The filtered probability space used in the stochastic optimal control model consists of four elements: 1) the (Cartesian) product of the disturbance space with the infinite product of the state and control spaces, $Z\times \left ( \Pi _{i=t}^{\infty }\left ( X\times Y\right ) _{i}\right )$ ; 2) a $ \sigma$ -algebra (generally universally measurable) on that product space; 3) the probability measure defined in Theorem8; and, 4) the filtration defined by the restriction to the product of the state and control spaces that have already occurred, $\left ( \Pi _{i=t}^{s-1}\left ( X_{i}\times Y_{i}\right ) \right ) \times X_{s}$ where it is understood that each space is a copy of the respective space.

Establishing the existence of a solution to a stochastic optimal control model means establishing the existence of an optimal policy for the problem. Specifically, the following definitions from Bertsekas and Shreve (Reference Bertsekas and Shreve1978) are used:

Definition 11 (Policy). A policy is a sequence $\phi =(\phi _{t},\phi _{t+1},\ldots )$ such that, for each $s\in \{t,t+1,\ldots \}$ ,

(12) \begin{equation} \phi _{s}(dy_{s}\,|\, x_{t},y_{t},\ldots, y_{s-1},x_{s}) \end{equation}

is a universally measurable stochastic kernel on $Y$ , given $X\times Y\times \cdots \times Y\times X$ satisfying

(13) \begin{equation} \phi _{s}(S(x_{s})\,|\, x_{t},y_{t},\ldots, y_{s-1},x_{s})=1, \end{equation}

for every $(x_{t},y_{t},\ldots, y_{s-1},x_{s})$ . If for every $s$ , $\phi _{s}$ is parameterized by only $x_{s}$ , then $\phi _{s}$ is a Markov policy. Alternatively, if for every $s$ , $\phi _{s}$ is parameterized by only $\left ( x_{t},x_{s}\right )$ , then $\phi _{s}$ is a semi-Markov policy. The set of all Markov policies, $\Phi$ , is contained in the set of all semi-Markov policies, $\Phi ^{\prime }$ . If for each $s$ and $\left ( x_{t},y_{t},\ldots, y_{s-1},x_{s}\right )$ , $\phi _{s}\left (dy_{s}\mathrel {}\middle |\mathrel {} x_{t},y_{t},\ldots, y_{s-1},x_{s}\right )$ assigns mass one to some element of $Y$ , $\phi$ is non-randomized. If $\overline {\phi }$ is a Markov policy of the form $\overline {\phi }=(\phi _{t},\phi _{t},\phi _{t},\ldots )$ , it is called stationary.

Definition 12 (Value Function). Let $\phi$ be a policy for the infinite horizon model. The (infinite horizon) value function corresponding to $\phi$ at $ x\in X$ is

(14) \begin{equation} \begin{split} V_{\phi }\left ( x\right ) & =\int {\left [ {\sum \limits _{k=0}^{\infty }{\beta ^{k}}g}\left ( {x_{k},y_{k}}\right ) \right ] \,}dr\left ( \phi, \mu _{x}\right ) \\ & =\sum \limits _{k=0}^{\infty }\left [ {\beta ^{k}}\int {g}\left ( {x_{k},y_{k}} \right ) \,dr_{k+t}\left ( \phi, \mu _{x}\right ) \right ] \end{split} \end{equation}

where, for each $\phi \in {\Phi }^{\prime }$ and $\mu \in P(X)$ , $r(\phi, \mu _{x})$ is the unique probability measure defined in equation ( 7 ) and, for every $k$ , the $r_{k+t}(\phi, \mu _{x})$ is the appropriate marginal measure. Footnote ²⁴ The (infinite horizon) optimal value function at $x\in X$ is $V^{\ast }(x)=\sup _{\phi \in {\Phi }^{\prime }}V_{\phi }\left ( x\right )$ .

Note that the optimal value function is defined over semi-Markov policies; this is without loss of generality. Furthermore, Bertsekas and Shreve (Reference Bertsekas and Shreve1978), (p. 216) show that the optimal value can be reached by only considering Markov policies. The advantage of including semi-Markov policies is that the optimum may require a randomized Markov policy, but only need a non-randomized semi-Markov policy. Finally, the Jankov-von Neumann theorem guarantees the existence of at least one non-randomized Markov policy so $\Phi$ and ${\Phi }^{\prime }$ are non-empty.

The following defines optimality for policies:

Definition 13 (Optimal policies). If $\epsilon \gt 0$ , the policy $\phi$ is $\epsilon$ -optimal if

(15) \begin{equation} V_{\phi }\left ( x\right ) \geq \begin{cases} \,V^{\ast }(x)-\epsilon & \text {if $V^{\ast }(x)\lt \infty $} \\ \,1/\epsilon & \text {if $V^{\ast }(x)=\infty $} \end{cases} \end{equation}

for every $x\in X$ . The policy $\phi$ is optimal if $V_{\phi }\left ( x\right ) =V^{\ast }\left ( x\right )$ .

In the next subsection, equation (2) is restated in this optimal control framework. The last subsection addresses what conditions are need to guarantee the existence of a (nearly) optimal policy.

5.2 Restating the Household Problem

Embedding the household decision problem into this framework requires specifying the state and control spaces. Since the spaces in this problem are all finite Euclidean spaces, the state and control spaces will be Borel no matter how defined. For a utility problem, it is natural to generally define ‘prices’ as states and ‘quantities’ as controls, but there is no unique specification required for the DP algorithm. Also, if the utility function demonstrated habit persistence as in Barnett and Wu (Reference Barnett and Su2005), lagged consumption variables would naturally be state variables in the current stage.

Define the period $s$ states by $\mathbf {x}_{s}=\left (\boldsymbol {\alpha }_{s}, \boldsymbol {\psi }_{s}\right )$ , where $\boldsymbol {\alpha }_{s}=\left ( \mathbf {a} _{s-1},A_{s-1}\right )$ and $\boldsymbol {\psi }_{s}$ denotes the vector of interest rates, prices, and other income that were realized at the beginning of period $s$ , normalized by $p_{s}^{\ast }$ ,

(16) \begin{equation} \boldsymbol {\psi }_{s}=\left ( \underset {\text {Interest rates}}{\underbrace {\left ( 1+\rho _{1,s-1}\right ) \frac {p_{s-1}^{\ast } }{p_{s}^{\ast }},\ldots, \left ( 1+\rho _{k,s-1}\right ) \frac {p_{s-1}^{\ast } }{p_{s}^{\ast }},\left ( 1+R_{s-1}\right ) \frac {p_{s-1}^{\ast }}{p_{s}^{\ast }}}},\underset {\text {Prices}}{\underbrace {\frac { p_{1,s}}{p_{s}^{\ast }},\ldots, \frac {p_{n,s}}{p_{s}^{\ast }}}},\frac {\mathit {I }_{s}}{p_{s}^{\ast }}\right ) . \end{equation}

The period $s$ controls are defined to be $\mathbf {y}_{s}=\left ( \boldsymbol {\theta }_{s},\mathbf {c}_{s}\right )$ where $\boldsymbol {\theta }_{s}=\left ( \mathbf {a}_{s},A_{s}\right )$ . The state space $X$ is $\left ( 2k+n+2\right )$ –dimensional Euclidean space, and the control space $Y$ is a subset of $ \left ( k+n+1\right )$ –dimensional Euclidean space. This notation is useful in what follows. Again note that, for $s\lt \infty$ , elements of $a_{s}$ and $ A_{s}$ may be negative, so that short-selling is allowed.

The budget constraint can be used to eliminate one of the controls, because the constraint will hold exactly at every time-period for any optimal solution. Therefore, a redundant control has been specified and the set of admissible controls actually lie in a $\left ( k+n\right )$ -dimensional linear subspace of $Y$ . Besides satisfying the budget constraint, the control variables need to be inside the survival set. By leaving in the redundant control, it is easier to explicitly specify this constraint. The controls set can be written as a function of only period $s$ states and controls as

(17) \begin{equation} S\left ( \mathbf {x}\right ) =\left \{\, \mathbf {y}\in Y\biggm |\sum _{i=1}^{k+1} \left ( y_{i}-\psi _{i}x_{i}\right ) +\sum _{j=k+2}^{k+1+n}\psi _{j}y_{j}-\psi _{k+n+2}\leq 0\,\right \} \end{equation}

where the period subscript $s$ has been suppressed and the other subscripts denote positions in the respective vector. The first key assumption is

Criterion 14. Assume that $\Gamma =\{\,(x,y) \,|\, x\in X,y\in S(x)\,\}$ is analytic in $ X\times Y$ .

The system function has a relatively simple form. It is defined by

(18) \begin{equation} \mathbf {x}_{s+1}=\left (\boldsymbol {\alpha }_{s+1},\boldsymbol {\psi }_{s+1}\right ) =f\left ( \mathbf {x}_{s},\mathbf {y}_{s},\mathbf {z}_{s}\right ) =\left ( \boldsymbol {\theta }_{s},\boldsymbol {\psi }_{s}+\mathbf {z}_{s}\right ) \end{equation}

In words, the first partition of the states evolves according to the simple rule $\boldsymbol {\alpha }_{s+1}=\boldsymbol {\theta }_{s}$ , and the second evolves as a state-dependent stochastic process, according to $\boldsymbol {\psi }_{s+1}= \boldsymbol {\psi }_{s}+\mathbf {z}_{s}$ .Footnote ²⁵ If $\mathbf {z}_{s}$ was a pure white noise process, then $\boldsymbol {\psi _{s}}$ would be a random walk.

The discount factor is defined by $\beta =1/(1+\xi )$ and satisfies $0\lt \beta \lt 1$ . The one-stage value function $g\left ( x_{s},y_{s}\right )$ is simply the period utility function as $g\left ( \mathbf {x}_{s},\mathbf {y}_{s}\right ) =g\left ( \mathbf {y}_{s}\right ) =u\left ( \mathbf {a}_{s},\mathbf {c}_{s}\right )$ ; it is a function of only the controls.Footnote ²⁶ The framework would allow $ g\left ( \phantom {x}\right )$ to be time-varying or to depend on the states, however, this would complicate the derivation of stochastic Euler equations in Section 6. The remaining assumption, which completes the mapping of the stochastic utility problem into the DP model, is:

Criterion 15. Assume that $g(y)$ is an upper semianalytic function from $\Gamma$ to $\mathbb {R}^{+}$ .

The assumptions are mathematically much weaker than the assumptions used in the standard economic text Stokey and Lucas Jr. (Reference Stokey and Lucas1989). More importantly, the assumptions are economically weaker; the key assumption is a richer model of uncertainty, rather than continuity restrictions on the value function, and compactness of constraint set. Furthermore, rather than assume that measurable selection is feasible, or equivalently rather than just assume the measurability challenges with applying DP don’t exist, measurable selection can be shown to be feasible under this richer definition of uncertainty.

5.3 Existence of a solution and the principal of optimality

The existence of a solution to the household decision problem or equivalently the existence of a (nearly) optimal policy can now be proved. First, as it is easier, the optimal value function is shown to satisfy a stochastic version of Bellman’s equation and the Principal of Optimality. The result is most cleanly stated using the following definition:

Definition 16 (State transition kernel). The state transition kernel on $X$ given $X\times Y$ is defined by

(19) \begin{equation} t(B \,|\, x,y)=q\,(\{z \,|\, f(x,y,z)\in B\}\,|\, x,y)=q\,(f^{-1}(B)_{(x,y)}\,|\, x,y). \end{equation}

Thus, $t(B\,|\, x,y)$ is the probability that the state at time $\left ( s+1\right )$ is in $B$ given that the state at time $s$ is $x$ and the control is $y$ . Note that $t(dx^{\prime }\,|\, x,y)$ inherits the measurability properties of the stochastic kernel.

Then the following mapping helps to state results concisely:

Definition 17. Let $V\colon X\to \mathbb {R}^{\ast }$ be universally measurable. Define the operator $T$ by

(20) \begin{equation} T(V)=\sup _{y\in S(x)}\left \{ \,g(y)+\beta \int \limits _{X}V\left ( x^{\prime }\right ) \,t\left ( dx^{\prime }\mathrel {}\middle |\mathrel {} x,y\right ) \,\right \} . \end{equation}

Several lemmas characterize the optimal policies. The following lemma shows that the optimal value function for the problem satisfies a functional recursion that is a stochastic version of Bellman’s equation.

Lemma 18. The optimal value function $V^{\ast }\left ( x\right )$ satisfies $V^{\ast }=T\left ( V^{\ast }\right )$ for every $x\in X$ .

This necessity result implies that an optimal policy is a fixed point of the mapping that is implicitly defined in the lemma. The following sufficiency result implies that a stochastic version of Bellman’s principal of optimality holds for stationary policies.

Lemma 19 (Principal of Optimality). Let $\overline {\phi }=(\overline {\phi }, \overline {\phi },\ldots )$ be a stationary policy. Then the policy is optimal iff $\, V_{\overline {\phi }}=T\left ( V_{\overline {\phi } }\right )$ for every $x\in X$ .

Before examining existence of optimal policies, note that the measurability assumptions already imply the existence of an $\epsilon$ -optimal policy. From Proposition 9.20 in Bertsekas and Shreve(Reference Bertsekas and Shreve1978), (p. 239), the non-negativity of the utility function is enough to assert the existence of an $\epsilon$ -optimal policy using similar arguments as in the previous lemmas.

Lemma 20. For each $\epsilon \gt 0$ , there exists an $\epsilon$ -optimal non-randomized semi-Markov policy for the infinite horizon problem. If for each $x\in X$ there exists a policy for the infinite horizon problem, which is optimal at $ x$ , then there exists a semi-Markov (randomized) optimal policy.

The fact that existence of any optimal policy is sufficient for the existence of a semi-Markov randomized optimal policy is important. The primary concern is with the first period return or utility function. For the initial period, the semi-Markov $\epsilon$ -optimal policy is Markov as clearly $\phi _{t}(dy_{s}\,|\, x_{t},x_{t})=\phi _{t}(dy_{s}\,|\, x_{t})$ . If $ \epsilon$ -optimality is judged to be sufficient, then simply use $g\left ( \overline {\phi }_{t}^{\ast }\right )$ where $\overline {\phi }_{t}^{\ast }$ is the first element of the optimal policy. The principal of optimality would only hold approximately, however. Similarly, if an optimal policy does actually exist, the randomness is not an issue as the optimal policy is non-random in the first element. In that case, the principal of optimality may not hold as equation (2) is only guaranteed to hold for stationary policies. Consequently, minimal additional assumptions are useful.

Before making these additional assumptions, define the DP algorithm as follows:

Definition 21 (Dynamic Programming Algorithm). The algorithm is defined recursively by

(21) \begin{align} V_{0}\left ( x\right ) & =0 & \forall x& \in X \end{align}

(22) \begin{align} V_{k+1}\left ( x\right ) & =T\left ( V_{k}\left ( x\right ) \right ) & \forall x& \in X,\ k=0,1,\ldots \end{align}

Proposition 9.14 of Bertsekas and Shreve (Reference Bertsekas and Shreve1978) implies that the algorithm converges for the problem as stated in the following lemma.

Lemma 22. $V_{\infty }=V^{\ast }$

Unfortunately, the convergence is not necessarily uniform in $x$ . Additionally, it is not possible to synthesize the optimal policy from the algorithm, as is the case for deterministic problems, because $V_{k}$ , while universally measurable, is not necessarily semianalytic for all $k$ .

The regularity assumptions are strengthened by imposing a mild boundedness assumption.

Criterion 23 (Boundedness). Assume that $\forall i$ and $\forall s\in \{t,t+1,\ldots \}$ , $\psi _{i,s}$ $\gt 0$ . Further assume that the single stage utility function contains no points of global satiation.

Assuming Criterion 23 leads to stronger results. First, under this boundedness condition, the DP algorithm converges uniformly for any initial upper semianalytic function not just zero. Furthermore, necessary and sufficient conditions for the existence of an optimal policy are available. Note that the condition would be violated by quadratic utility functions that have a “bliss point” (Jappelli and Pistaferri, Reference Jappelli and Pistaferri2017).

Lemma 24 (Existence). Assume Criterion 23 holds. Then for each $\epsilon \gt 0$ , there exists an $\epsilon$ -optimal non-randomized stationary Markov policy. If for each $x\in X$ there exists a policy for the infinite horizon problem, which is optimal at $x$ , then there exists an unique optimal non-randomized stationary policy. Furthermore, there is an optimal policy if and only if for each $x\in X$ the supremum in

(23) \begin{equation} \sup _{y\in S\left ( x\right ) }\left \{\,g\left (y\right ) +\beta _{X} V^{\ast }\left ( x^{\prime }\right ) \,{t}\left (dx^{\prime } \mathrel {}\middle |\mathrel {} x,y\right ) \,\right \} \end{equation}

is achieved.

Combining this lemma with Lemma 19 implies that there exists an optimal policy if and only if there exists a stationary policy such that $V_{\overline {\phi }}=T\left ( V_{\overline {\phi }}\right )$ for every $x\in X$ . It is clear that the $\epsilon$ -optimal non-randomized stationary Markov policy is the universally measurable selector from Theorem9. If universally measurable selection is assumed to be possible (i.e. the set $I$ defined in Theorem9 is the entire set $D$ ), then the supremum will be achieved. This assumption is weaker than requiring that Borel-measurable selection is possible, as in Stokey and Lucas Jr. (Reference Stokey and Lucas1989). The assumption is also weaker than the regularity conditions needed to solve semicontinuous models in Bertsekas and Shreve (Reference Bertsekas and Shreve1978), which have Borel-measurable optimal plans. The following lemma, which follows from Proposition 9.17 of Bertsekas and Shreve (Reference Bertsekas and Shreve1978) supplies a sufficient condition for the supremum to be achieved.

Lemma 25. Under Criterion 23, if there exists a non-negative integer $\overline {k}$ such that for each $x\in X$ , $\lambda \in \mathbb {R}$ , and $k\geq \overline {k}$ , the set

(24) \begin{equation} S_{k}\left ( x,\lambda \right ) =\left \{\, {y\in S(x)} \mathrel {}\middle |\mathrel {} {g}\left ( {y}\right ) {+\beta \int {V_{k}}}\left ( {{{x}^{\prime }}}\right ) \,{t(d{x}^{\prime } \,|\, x,y)\geq \lambda }\,\right \} \end{equation}

is compact in $Y$ then there exists a non-randomized optimal stationary policy for the infinite horizon problem.

This is a weaker condition than assuming that the constraint sets are compact or that $\Gamma$ is upper hemi-continuous. If the supremum in (23) is achieved for the initial state $x_{t}\in X$ , the boundedness assumption implies that a unique stationary non-random Markov optimal plan exists.

6. Stochastic Euler equations

In this section, Euler equations for the stochastic decision are derived. Although necessary and sufficient conditions for the optimum to exist have been established, the stronger characterization given by Euler equations is often needed and is always useful. The usefulness of stochastic Euler equations is discussed in Stokey and Lucas Jr.(Reference Stokey and Lucas1989), (pp. 280–283).

Bellman’s equation, which has been shown to hold, can be used to derive stochastic Euler equations. Of course, the optimal value function needs to be differentiable. In addition, it must be possible to interchange the order of integration. The interchange is possible, for example, if each partial derivative of $V$ is absolutely integrable (Blume et al., Reference Barnett, Liu, Barnett and Liu1982; Lang, Reference Lang1993; Stokey and Lucas Jr. Reference Stokey and Lucas1989,Theorem 9.10, pp. 266–257). In the present case, it is sufficient to show that the value function is differentiable on an open subset containing $x_{t}$ , because of Criterion 23. Then the value function meets the conditions in Mattner (Reference Mattner2001), particularly the locally bounded assumption, and the interchange is valid.Footnote ²⁷ The envelope theorem then implies the following two results. First, the optimal solution inherits differentiability. Second, the stochastic Euler equations proposed in Barnett et al. (Reference Barnett and Su1997) can be derived. More generally, this derivation demonstrates that stochastic Euler equations can be derived for the class of models.

Proof. Let $\overline {\phi }$ denote the optimal stationary nonrandom Markov policy. Note that at time $s$ , $\overline {\phi }$ is a function of $x_{s}$ . To simplify notation, let $\overline {V}\left ( x\right ) =V_{\overline {\phi } }\left ( x\right )$ . Bellman’s equation implies

(25) \begin{equation} \overline {V}\left ( x\right ) =g\left ( \overline {\phi }\left ( x\right ) \right ) +\beta \int \limits _{Z}\overline {V}\left [ f\left ( \overline {\phi }\left ( x\right ), z\right ) \right ] \,p(dz\,|\, x,y) \end{equation}

holds for any $x_{t}$ . Note $x_{t+1}=f(y_{t},z_{t})$ , so the value function within the integral is being evaluated one period into the future. Let $ x^{0}$ denote the actual initial state. For $x\in N\left ( x^{0}\right )$ , where $N\left ( x^{0}\right )$ is a neighborhood of $x^{0}$ , define

(26) \begin{equation} J\left ( x\right ) =g\left ( x,\overline {\phi }\left ( x^{0}\right ) \right ) +\beta \int \limits _{Z}\overline {V}\left [ f\left ( \overline {\phi }\left ( x^{0}\right ), z\right ) \right ] \,p(dz\,|\, x,y). \end{equation}

In words, $J\left ( x\right )$ is the value function with the policy constrained to be the optimal policy for $x^{0}$ . Clearly, $J\left ( x^{0}\right ) =V_{\overline {\phi }}\left ( x^{0}\right )$ and, $\forall x\in N\left ( x^{0}\right )$ , $J\left ( x\right ) \leq V_{\overline {\phi }}\left ( x\right )$ because $\phi \left ( x^{0}\right )$ is not the optimal policy for $x\neq x^{0}$ . If the original utility function $U\left ( \phantom {x}\right )$ is concave and differentiable then so is $u\left ( \phantom {x}\right )$ and therefore $g\left ( \phantom {x}\right )$ . This assumption implies that $ J\left ( x\right )$ is also concave and differentiable. The envelope theorem from Benveniste and Scheinkman (Reference Barnett, Hinich, Yue, Barnett, Hinich and Yue1979) combined with the fact that the policy $\overline { \phi }$ is optimal uniformly in $x$ then implies that $V_{\overline {\phi } }\left ( x\right )$ is differentiable for all $x\in \text {int}\left ( X\right )$ , As prices and rates of return are assumed to be larger than zero, the only initial conditions for which $V_{\overline {\phi }}\left ( x\right )$ is not differentiable are infinite (positive or negative) initial asset endowments, which can be excluded.

Theorem 27 (Stochastic Euler Equations). If $\,U\left ( \phantom {x}\right )$ is concave and differentiable. Then the stochastic Euler equations for (2) are

(27) \begin{align} {\frac {\partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) }{\partial a_{i}}} & {=\frac {\partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) }{\partial c^{\prime }}-}\frac {1}{1+\xi }\,E_{t}\left [ \left ( 1+\rho _{i,t}\right ) \frac { p_{t}^{\ast }}{p_{t+1}^{\ast }}\frac {\partial u\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) }{\partial c^{\prime }}\right ] \end{align}

and

(28) \begin{align} \frac {\partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) }{\partial c^{\prime }}& {=\frac {1}{1+\xi }}E_{t}\left [ \left ( 1+R_{t}\right ) \frac { p_{t}^{\ast }}{p_{t+1}^{\ast }}\frac {\partial u\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) }{\partial c^{\prime }}\right ] \end{align}

where $a_{t}^{\ast }$ and $c_{t}^{\ast }$ are the controls specified by the non-randomized optimal stationary policy and $c^{\prime }$ is an arbitrary numéraire.

To be clear, these Euler equations are not new, having been derived in Barnett (Reference Barnett and Barnett1995) and Barnett et al. (Reference Barnett and Su1997). However, the derivation here formally establishes their validity under weak measurability conditions. In contrast, prior derivations assumed that Bellman’s equation applied and that it could be differentiated. As discussed extensively in Bertsekas and Shreve (Reference Bertsekas and Shreve1978), assuming that Bellman’s equation applies to stochastic problems is a material assumption. Further the differentiability of a stochastic Bellman’s equation had not been previously established.

The implications of these stochastic Euler equations have been extensively discussed and extended in the monetary aggregation literature cited. Briefly, to provide some intuition on their meaning, substitute from (28) into (27). The linearity of the conditional expectations operator implied by the linearity of the well-defined integral, produces

(29) \begin{equation} \partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) /\partial a_{i}{=}\frac {1 }{1+\xi }E_{t}\left [ \left ( R_{t}-\rho _{i,t}\right ) \frac {p_{t}^{\ast }}{ p_{t+1}^{\ast }}\,u_{c^{\prime }}\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) \right ] \end{equation}

where $u_{c^{\prime }}\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) =\partial u\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) /\partial c^{\prime }$ . The first-order condition for a simple utility maximization problem for consumption goods is $\frac {\partial u\left ( c\right ) /\partial c_{i}}{ \partial u\left ( c\right ) /\partial c_{j}}=\frac {p_{i}}{p_{j}}.$ Footnote ²⁸Similarly, the right-hand side of (29) defines the relevant information for assets’ relative prices; it implies a generalized user cost for risky assets (Barnett et al., Reference Barnett and Su1997). From these stochastic Euler equations, the relative prices or the user costs for assets generally depend on a trade-off between an asset’s rate of return, risk, and the liquidity, or monetary services, the asset provides in terms of its contribution to utility. The three-way trade-off generalizes much of the voluminous asset-pricing literature in finance, where only the two-dimensional trade-off between risk and return is considered. The results in (29) define an asset pricing rule that is a strict generalization of the consumption CAPM asset pricing rule.

As shown by Barnett et al. (Reference Barnett and Su1997), the trade-off simplifies to a trade-off between liquidity and expected returns under risk neutrality, which mirrors the deterministic case. If the marginal utility of consumption defined in the numéraire $u_{c^{\prime }}\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right )$ is independent of the nominal interest rates then

(30) \begin{equation} \partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) /\partial a_{i}{=}\frac {1 }{1+\xi }E_{t}\left [ \left ( R_{t}-\rho _{i,t}\right ) \frac {p_{t}^{\ast }}{ p_{t+1}^{\ast }}\right ]E_{t}\left [u_{c^{\prime }}\left ( a_{t+1}^{\ast },c_{t+1}^{\ast }\right ) \right ] \end{equation}

and the ratio between any two assets would be

(31) \begin{equation} \frac {\partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) /\partial a_{i}}{\partial u\left ( a_{t}^{\ast },c_{t}^{\ast }\right ) /\partial a_{j}} = E_{t}\left [ \frac {\left ( R_{t}-\rho _{i,t}\right )}{\left ( R_{t}-\rho _{j,t}\right )} \right ]. \end{equation}

This ratio is the same as the deterministic case, except that it depends on expected values. From the earlier discussion, under certainty equivalence solving a deterministic problem using expected values replicates the stochastic solution. Therefore, imposing certainty equivalence implies the independence of marginal consumption from interest rates, which illustrates how certainty equivalence makes risk largely inconsequential for stochastic decision problems.Footnote ²⁹