2.1. System primitives of the gradual-impulse control problem

We describe the primitives of the model as follows. The state space is $\textbf{X}$, the space of gradual controls is $\textbf{A}^G$, and the space of impulsive controls is $\textbf{A}^I$. It is assumed that $\textbf{X}$, $\textbf{A}^G$, and $\textbf{A}^I$ are all Borel spaces, endowed with their Borel $\sigma$-algebras ${\mathcal B}(\textbf{X}),$ ${\mathcal B}(\textbf{A}^G)$, and ${\mathcal B}(\textbf{A}^I)$, respectively. The transition rate, on which the gradual control acts, is given by $q(dy|x,a)$, which is a signed kernel from $\textbf{X}\times\textbf{A}^G$, endowed with its Borel $\sigma$-algebra, to ${\mathcal B}(\textbf{X}),$ satisfying the following conditions:

\begin{align*}&q(\Gamma|x,a)\in[0,\infty) \quad \text{for each } \Gamma\in{\mathcal B}(\textbf{X}),\ x\notin \Gamma;\\[3pt] &q(\textbf{X}|x,a)=0, \qquad x\in \textbf{X},\ a\in\textbf{A}^G;\\[3pt] &\bar{q}_x:=\sup_{a\in \textbf{A}^G}q_x(a)<\infty, \qquad x\in\textbf{X},\end{align*}

where $q_x(a):=-q(\{x\}|x,a)$ for each $(x,a)\in\textbf{X}\times\textbf{A}^G.$ For notational convenience, we introduce $\tilde{q}(dy|x,a):=q(dy\setminus\{x\}|x,a)$, for every $x\in\textbf{X}$, $a\in\textbf{A}^G$. If the current state is $x\in\textbf{X}$, and an impulsive control $b\in\textbf{A}^I$ is applied, then the state immediately following this impulse obeys the distribution given by $Q(dy|x,b)$, which is a stochastic kernel from $\textbf{X}\times\textbf{A}^I$ to ${\mathcal B}(\textbf{X}).$ Finally, given the current state $x\in\textbf{X}$, the cost rate of applying a gradual control $a\in \textbf{A}^G$ is $c^G(x,a)$, and the cost of applying an impulsive control $b\in\textbf{A}^I$ is $c^I(x,b,y)$, where $c^G$ and $c^I$ are $[0,\infty)$-valued measurable functions on $\textbf{X}\times\textbf{A}^G$ and $\textbf{X}\times\textbf{A}^I\times\textbf{X}$, respectively. Throughout this paper, we assume that $\textbf{A}^G$ and $\textbf{A}^I$ are compact Borel spaces. Without loss of generality we may regard $\textbf{A}^G$ and $\textbf{A}^I$ as two disjoint compact subsets of a Borel space $\tilde{\textbf{A}}=\textbf{A}^G\cup\textbf{A}^I$. Furthermore, we assume that

(1) \begin{eqnarray}\sup_{a\in\textbf{A}^G}c^G(x,a)<\infty \qquad \forall x\in\textbf{X}.\end{eqnarray}

The system dynamics in the gradual-impulse control problem of interest can be described as follows. In the absence of impulses, the system is just a controlled Markov pure jump process in the state space $\textbf{X}$, where the (gradual) control, selected from $\textbf{A}^G$, acts on the local characteristics of the process, leading to natural jumps. This is conveniently described as a marked point process, which consists of the pairs of subsequent jump moments and the post-jump states (marks). The mark space is thus $\textbf{X}$.

In the gradual-impulse control problem, we will again describe the system using a marked point process. However, when the decision-maker is allowed to apply a finite or countably infinite sequence of impulses from $\textbf{A}^I$ at a single time moment, with each impulse resulting in a post-impulse state, there will be a sequence of states in $\textbf{X}$ at a single time moment. Moreover, the order of the impulses and their resulting states are also relevant. Therefore, the marked point process we use now is in an enlarged mark space. More precisely, each mark contains a sequence of impulses applied at the same time moment, the state before the impulses are applied, and all the states resulting from these impulses. Each jump moment is triggered either by an impulse (or a sequence of impulses), or by a natural jump. A mark in this marked point process is referred to as an intervention. This term is naturally understandable when the mark consists of impulses. That said, we will also allow ‘interventions’ that do not contain any impulses, or that consist of an empty sequence of impulses. This occurs when the decision-maker chooses not to apply any impulse immediately after a natural jump. In the rest of this section, following the method of [Reference Dufour and Piunovskiy7], we will elaborate on this idea and describe rigorously the continuous-time gradual-impulse control problem that we are studying. We begin by stating the precise definition of an intervention in the next subsection.

2.2. Definition and interpretation of an intervention

At the beginning of an intervention, the decision-maker chooses whether to apply an impulse, and which impulse to apply. If the current state is $x\in\textbf{X}$, after an impulse $b\in\textbf{A}^I$ is chosen, the new state, say $y\in\textbf{X}$, is instantaneously realized, following the distribution $Q(dy|x,b)$. Then, based on x,b,y, the decision-maker chooses the next impulse, if any at all, and so on. To be consistent, a cemetery point $\Delta\notin \textbf{A}^I\cup \textbf{X}$ is artificially fixed, which is chosen when the decision-maker decides not to apply any more impulses at the current instant; this leads to the post-impulse state, also denoted by $\Delta,$ which is absorbing, i.e., $Q(\Delta|\Delta,\Delta)\equiv 1.$ Therefore, an intervention is itself a sequential decision process. More precisely, an intervention can be regarded as a trajectory of the following DTMDP, which we refer to as the ‘intervention’ DTMDP model, to distinguish it from several other DTMDP models to appear subsequently.

Let the initial distribution in the intervention DTMDP be always concentrated on $\textbf{X}.$ Then its canonical sample space is $\mathbf{Y}:=\left(\bigcup_{k=0}^{\infty}\mathbf{Y}_k\right) \cup(\textbf{X}\times\textbf{A}^I)^\infty,$ where, for each $\infty>k\ge 1$,

\begin{equation*}\mathbf{Y}_k:=(\mathbf{X}\times \mathbf{A}^I)^k\times(\mathbf{X}\times\{\Delta\})\times(\{\Delta\}\times\{\Delta\})^\infty,\end{equation*}

and $\textbf{Y}_0:=(\mathbf{X}\times\{\Delta\})\times(\{\Delta\}\times\{\Delta\})^\infty.$ Here, if $y\in \textbf{Y}_k$, $\infty>k\ge 0$, then there are k impulses applied in the intervention y. Similarly, if $y\in (\textbf{X}\times\textbf{A}^I)^\infty,$ then there are infinitely many impulses applied in the intervention y.

Now we give the following definition.

With the notation introduced above, we now reiterate, more rigorously than in the beginning of this subsection, the interpretation of an intervention. Given the current state $x\in \textbf{X}$, if the controller decides to use $\Delta$, then this means that no more impulse is used at this instant, and the intervention DTMDP is absorbed at $\Delta$ in the next step; if the controller decides to use an impulse $b\in\textbf{A}^I$, then the post-impulse state follows the distribution $Q(dy|x,b)$. At the next post-impulse state y, if $y=\Delta,$ then the only decision is $\Delta;$ if $y\ne \Delta,$ then the controller decides either to use no impulse, leading to the next post-impulse state $\Delta$, or to use impulse b’, leading to the next post-impulse state, which follows the distribution given by $Q(\cdot|y,b')$, and so on. In other words, an intervention consists of a state and a finite or countable sequence of pairs of impulsive actions and the associated post-impulse states. In particular, no impulse is applied in an intervention if the intervention belongs to $\mathbf{Y}_{0}$; see Figure 1 and its caption for an example. Let

\begin{equation*}\mathbf{Y}^{*}:=\textbf{Y}\setminus \textbf{Y}_0=\left(\bigcup_{k=1}^{\infty}\mathbf{Y}_k\right) \cup (\textbf{X}\times\textbf{A}^I)^\infty\end{equation*}

be the set of interventions where some impulses are applied.

In an intervention, locally, the selection of impulses (including the ‘pseudo-impulse’ $\Delta$) from $\textbf{A}^I_\Delta$ is governed by a strategy in the intervention DTMDP model. This adverb ‘locally’ is understood in comparison with the definition of a policy for the gradual-impulse control problem, as given in Definition 3 below, which governs the selection of impulsive controls as well as gradual controls, and is thus ‘global’. Let $\Xi$ be the set of (possibly randomized and history-dependent) strategies $\sigma$ in the intervention DTMDP. We refer the reader to the appendix for standard terminology related to DTMDPs. The way that a strategy in the intervention DTMDP model is incorporated into a policy in Definition 1 below is through its strategic measure. Let $\beta^\sigma(\cdot|x)$ denote the corresponding strategic measure of a strategy $\sigma$ of the intervention DTMDP, given the initial state $x\in\textbf{X}$. By the Ionescu-Tulcea theorem (see e.g. Proposition C.10 in [Reference Hernández-Lerma and Lasserre13]), the mapping $x\in\textbf{X}\rightarrow \beta^\sigma(\cdot|x)$ is measurable. Let $\mathcal{P}^\mathbf{Y}$ be the collection of all such stochastic kernels generated by some strategy $\sigma\in\Xi$, and $\mathcal{P}^\mathbf{Y}(x):=\{\beta^\sigma(\cdot|x): \sigma\in\Xi\}$ for each state $x\in \mathbf{X}$. Let

\begin{equation*}\mathcal{P}^{\mathbf{Y}^\ast}:=\{\beta(\cdot|\cdot)\in \mathcal{P}^{\textbf{Y}}:~\beta(\textbf{Y}^\ast|x)=1\ \forall~x\in\textbf{X}\},\end{equation*}

and for each $x\in\textbf{X},$

\begin{equation*}\mathcal{P}^\mathbf{Y^\ast}(x):=\{\beta(\cdot|x): \beta(\cdot|\cdot)\in \mathcal{P}^{\mathbf{Y}},~\beta(\textbf{Y}^\ast|x)=1\}.\end{equation*}

2.3 Construction of the controlled process

Let us now describe the promised marked point process $\{(T_{n},Y_{n})\}_{n=1}^\infty$ for the system dynamics of the gradual-impulse control problem, where the mark space is the space of interventions. Then the continuous-time process $\{\xi_t\}_{t\ge 0}$ under control is defined based on this marked point process.

Let $\mathbf{Y}_\Delta:=\mathbf{Y}\cup\{\Delta\},$

\begin{equation*}\Omega_0:=\mathbf{Y}\times(\{0\}\times \textbf{Y})\times(\{\infty\}\times\{\Delta\})^\infty,\end{equation*}

and

\begin{equation*} \Omega_{n}:=\mathbf{Y}\times(\{0\}\times \textbf{Y})\times ((0,\infty)\times \mathbf{Y})^n\times(\{\infty\}\times\{\Delta\})^\infty\end{equation*}

for all $n=1,2,\dots.$ The canonical space $\Omega$ is defined as

\begin{equation*}\Omega:=\left(\bigcup_{n=0}^\infty \Omega_{n}\right)\cup \big( \mathbf{Y}\times((0,\infty)\times \mathbf{Y})^\infty \big)\end{equation*}

and is endowed with its Borel $\sigma$-algebra, denoted by $\mathcal{F}$. We will use the following generic notation for a point in $\Omega$: $\omega=(y_0,\theta_1,y_1,\theta_2,y_2,\ldots).$ Below, unless stated otherwise, the notation $x_0\in\textbf{X}$ will indicate the initial state of the gradual-impulse control problem. Then we put

(2) \begin{eqnarray}y_0:=(x_0,\Delta,\Delta,\dots), \qquad \theta_1\equiv 0.\end{eqnarray}

The sequence $\{\theta_n\}_{n=1}^\infty$ represents the sojourn times between consecutive interventions. Here $\theta_1=0$ corresponds to the fact that we allow the possibility of applying impulsive control at the initial time moment; cf. (5) below.

For each $n=0,1,\dots$, let

\begin{eqnarray*}h_{n}:=(y_0,\theta_1,y_1,\theta_2,y_2,\dots \theta_{n},y_{n})=(y_0,0,y_1,\theta_2,y_2,\dots \theta_{n},y_{n}),\end{eqnarray*}

where the second equality holds because $\theta_1\equiv 0$; see (2). The collection of all such partial histories $h_n$ is denoted by $\mathbf{H}_{n}$. Let us introduce the coordinate mappings

\begin{eqnarray*}Y_{n}(\omega)=y_{n} \qquad \forall~n\ge 0,\\\Theta_{n}(\omega)=\theta_{n} \qquad \forall~n\ge 1.\end{eqnarray*}

The sequence $\{T_{n}\}_{n=1}^{\infty}$ of $[0,\infty]$-valued mappings is defined by

\begin{equation*}T_{n}(\omega):=\sum_{i=1}^n\Theta_i(\omega)=\sum_{i=1}^n\theta_i\end{equation*}

and

\begin{equation*}T_\infty(\omega):=\lim_{n\to\infty}T_{n}(\omega)\end{equation*}

for all $\omega\in\Omega.$ Let $H_{n}:=(Y_0,\Theta_1,Y_1,\dots,\Theta_{n},Y_{n}).$ Finally, we define the controlled process $\big\{\xi_t\big\}_{t\in [0,\infty)}$ by

\begin{eqnarray*}\xi_t(\omega)=\left\{\begin{array}{ll}Y_{n}(\omega) & \mbox{ if } T_{n}\le t<T_{n+1} \mbox{ for } n\ge 1, \\\Delta & \mbox{ if } T_\infty\le t.\end{array}\right..\end{eqnarray*}

It is convenient to introduce the random measure $\mu$ of the marked point process $\{(T_{n},Y_{n})\}_{n=1}^\infty$ on $(0,\infty)\times \mathbf{Y}$:

\begin{equation*}\mu(dt\times dy)=\sum_{n\ge 2}I_{\{T_{n}<\infty\}}\delta_{(T_{n},Y_{n})}(dt\times dy).\end{equation*}

Let

\begin{equation*}\mathcal{F}_t:=\sigma\{H_1\}\vee\sigma\{\mu((0,s]\times B):~s\le t,B\in\mathcal{B}(\mathbf{Y})\}\end{equation*}

for $t\in[0,\infty)$.

We will use the following notation in the next definition. For each intervention $y=(x_0,b_0,x_1,b_1,\dots)\in \mathbf{Y}$, define $\bar x(y):=x_{k}$ if $\infty>k=0,1,\dots$ is the unique integer such that $y\in \mathbf{Y}_k$ (if $k\ge 1,$ then $\bar{x}(y)$ is the state after the last impulse in the intervention y); if such an integer k does not exist, then $y\in(\textbf{X}\times\textbf{A}^I)^\infty$ and $\bar{x}(y):=\Delta.$ The previous equality corresponds to the fact that we kill the process after an infinite number of impulses are applied at a single time moment. An example of a trajectory of the system dynamics in the gradual-impulse control problem is displayed in Figure 1.

Definition 3. A policy is a sequence $u=\{u_{n}\}_{n=0}^\infty$ such that $u_{0}\in \mathcal{P}^{\mathbf{Y}}$ and, for each $n=1,2,\dots$, $u_{n}=\left( \Phi_{n},\Pi_{n},\Gamma_{n}^0,\Gamma_n^1 \right),$ where $\Phi_{n}$ is a stochastic kernel on $(0,\infty]$ given $\mathbf{H}_{n}$ such that $\Phi_n(\{\infty\}|h_n)=1$ if $y_n\in (\textbf{X}\times\textbf{A}^I)^\infty$; $\Pi_{n}$ is a stochastic kernel on $\mathbf{A}^{G}$ given $ \mathbf{H}_{n}\times (0,\infty)$; $\Gamma^0_{n}$ is a stochastic kernel on $\mathbf{Y}$ given $ \mathbf{H}_{n}\times (0,\infty)\times \mathbf{X}$ satisfying

\begin{equation*}\Gamma_n^0(\cdot|h_n,t,x)\in {\mathcal P}^{\textbf{Y}}(x)\end{equation*}

for each $h_n\in\textbf{H}_n$, $x\in\textbf{X}$, and $t\in(0,\infty)$; and $\Gamma^1_{n}$ is a stochastic kernel on $\mathbf{Y}$ given $ \mathbf{H}_{n}$ satisfying

\begin{equation*}\Gamma^1_n(\cdot|h_n)\in {\mathcal P}^{\textbf{Y}^\ast}(\bar{x}(y_n))\end{equation*}

for each $h_n\in \textbf{H}_n$. (The above conditions apply when $y_n\ne \Delta$; otherwise, all the values of $\Phi_n$, $\Pi_n$, $\Gamma_n^0$, and $\Gamma_n^1$ are immaterial and may be put arbitrarily.) The set of policies is denoted by $\mathcal U$.

Let us provide an interpretation of how a policy u acts on the system dynamics. Roughly speaking, an intervention is over as soon as the (possibly empty) sequence of simultaneous impulses is over. Given that the nth intervention is over, the kernel $\Phi_n$ specifies the conditional distribution of the planned time until the next impulse (or next sequence of impulses). The (conditional) distribution of the time until the next natural jump (if there are no interventions before it) is the non-stationary exponential distribution with rate $\int_{\textbf{A}^G}q_{\bar{x}(Y_n)}(a)\Pi_n(da|H_n,t)$. Below, we put $q_{\Delta}(a):=0$ for each $a\in\textbf{A}^G.$ In other words, $\Pi_n$ is the (decision rule of) relaxed gradual control. Given that the nth intervention is over, the next intervention is triggered by either the next planned impulse or the next natural jump; in the former case, the new intervention has the distribution given by $\Gamma^1_n$, and in the latter case the new intervention has the distribution given by $\Gamma^0_n.$ This interpretation will be seen to be consistent with (3) and (4) below, where one can see how a policy u acts on the conditional law of the marked point process $\{(T_n,Y_n)\}_{n=1}^\infty.$ See also the caption of Figure 1.

Suppose a policy $u=\{u_{n}\}_{n=0}^\infty $ is fixed. Let us now present the conditional law of the marked point process $\{(T_n,Y_n)\}_{n=1}^\infty$ under the policy u, which determines the underlying probability measure $\mathbb{P}_{x_0}^u$ on $(\Omega,\mathcal F)$, where $x_0\in\textbf{X}$ is the fixed initial state of the system dynamics. For brevity, we introduce the following notation for each $n\ge 1$, $\Gamma\in \mathcal{B}(\mathbf{X})$, and $h_{n}=(y_0,\theta_1,y_1,\ldots,\theta_{n},y_{n})\in \mathbf{H}_{n}$:

\begin{eqnarray*}\lambda_{n}^u(\Gamma|h_{n},t): = \int_{\mathbf{A}^{G}} \tilde{q}(\Gamma | \overline{x}(y_{n}),a) \Pi_{n}(da | h_{n},t),~\Lambda_{n}^u(\Gamma|h_{n},t): = \int_{0}^t \lambda_{n}^u(\Gamma|h_{n},s) ds.\end{eqnarray*}

Now, for each $n\ge 1$, we introduce the stochastic kernel $G_{n}^u$ on $(0,\infty]\times \mathbf{Y}_{\Delta}$ given $\mathbf{H}_{n}$ as follows. For each $h_{n}=(y_0,\theta_1,y_1,\ldots,\theta_{n},y_{n})\in \mathbf{H}_{n}$,

(3) \begin{eqnarray}G_{n}^u(\{+\infty\}\times \{\Delta\} | h_{n}):= \delta_{y_{n}} (\{\Delta\}) + \delta_{y_{n}} (\mathbf{Y})e^{-\Lambda_{n}^u(\mathbf{X}|h_{n},+\infty)}\Phi_{n}(\{+\infty\}|h_n),\end{eqnarray}

and

(4) \begin{align}G_{n}^u(dt\times dy| h_{n})&:=\delta_{y_{n}} (\mathbf{Y}) \left\{ \Gamma_{n}^1(dy| h_{n}) e^{-\Lambda_{n}^u(\mathbf{X}|h_{n},t)} \Phi_{n}(dt | h_{n}) \right.\nonumber \\[2pt]&\quad \left.+ \int_{\mathbf{X}} \Phi_{n}([t,\infty] | h_{n}) \Gamma_{n}^0(dy| h_{n},t,x) \lambda_{n}^u(dx|h_{n},t) e^{-\Lambda_{n}^u(\mathbf{X}|h_{n},t)} dt \right\}\end{align}

on $(0,\infty)\times \textbf{Y}$. For each fixed initial state $x_{0}\in\mathbf{X}$, by the Ionescu-Tulcea theorem (see e.g. Proposition C.10 in [Reference Hernández-Lerma and Lasserre13]), there exists a probability $ \mathbb{P}^{u}_{x_{0}}$ on $(\Omega,\mathcal{F})$ such that the restriction of $\mathbb{P}^{u}_{x_{0}}$ to $(\Omega,\mathcal{F}_{0})$ is given by

(5) \begin{eqnarray}\mathbb{P}^{u}_{x_{0}} \left( \left(\{y_{0}\}\times \{0\} \times \Gamma \times ((0,\infty]\times \mathbf{Y}_{\Delta})^{\infty} \right)\cap \Omega\right) & = &u_{0}(\Gamma|x_{0})\end{eqnarray}

for each $\Gamma\in \mathcal{B}(\mathbf{Y})$; and for each $n\ge 1$, under $\mathbb{P}^{u}_{x_{0}}$, the conditional distribution of $(Y_{n+1},\Theta_{n+1})$ given $\mathcal{F}_{T_{n}}:=\sigma\{H_{n}\}$ is determined by $G_{n}^u(\cdot | H_{n})$, while the conditional survival function of $\Theta_{n+1}$ given $\mathcal{F}_{T_{n}}$ under $\mathbb{P}^{u}_{x_{0}}$ is given by $G_{n}^u([t,+\infty]\times \mathbf{Y}_{\Delta}| H_{n})$.

The cost associated with an intervention $y=(x_{0},b_{0},x_{1},b_{1},\ldots)\in \mathbf{Y}$ is given by $C^I(y):=\sum_{k=0}^\infty c^I(x_k,b_k,x_{k+1}).$ Here, recall that an intervention consists of the current state, the sequence of impulses applied in turn at the same time moment, and the associated post-impulse states; and each impulse b applied at state x results in a cost $c^I(x,b,z)$ if it leads to the post-impulse state z. (We accept that $c^I(x,\Delta,\Delta):= 0$ for all $x\in\textbf{X}_\Delta$.) With this notation, we now introduce the performance measure considered in this paper:

\begin{eqnarray*}\mathcal{V}(x,u)&:=& \mathbb{E}^{u}_{x} \left[ e^{\sum_{n=1}^\infty \left(C^{I}(Y_n)+ \int_{T_n}^{T_{n+1}} \int_{\mathbf{A}^{G}} c^{G}(\bar{x}(\xi_{s}),a) \Pi_n(da |H_n,s-T_n) ds\right)} \right]\end{eqnarray*}

for each $x\in\textbf{X}$ and policy $u\in {\mathcal U}$. Here we recall that $T_1=\Theta_1\equiv 0$; see (2). To illustrate more explicitly how the policy acts on the impulses, consider the example of only one intervention and null gradual cost $c^G(x,a)\equiv 0$. Then we may write

\begin{eqnarray*}\mathbb{E}^{u}_{x} \left[ e^{ C^{I}(Y_1) } \right]&=&\int_{\textbf{X}\times \textbf{A}^I\times \textbf{X}\times \dots} u_0(dx_0\times db_0\times dx_1\times db_1\times\dots|x)e^{\sum_{k=0}^\infty c^I(x_k,b_k,x_{k+1}) }\\[2pt] &=&\int_{\textbf{X}\times \textbf{A}^I\times \textbf{X}\times \dots} u_0(dy|x)e^{ C^I(y) }.\end{eqnarray*}

More generally, one can compute

\begin{equation*}\mathbb{E}^{u}_{x} \left[ e^{ C^{I}(Y_{n+1}) } \right]=\mathbb{E}^{u}_{x} \left[\mathbb{E}^{u}_{x} \left[ e^{C^{I}(Y_{n+1}) }|H_n \right]\right],\end{equation*}

where

\begin{equation*}\mathbb{E}^{u}_{x} \left[ e^{ C^{I}(Y_{n+1}) }|H_n \right]\end{equation*}

can be written out as an integral similar to that of the $n=0$ case using the conditional laws (3) and (4).

Let the value function ${\mathcal V}^\ast$ be denoted by ${\mathcal V}^\ast(x):=\inf_{u\in {\mathcal U}}{\mathcal V}(x,u)$ for each $x\in\textbf{X}$. A policy $u^\ast$ satisfying ${\mathcal V}(x,u^\ast)={\mathcal V}^\ast(x)$ for all $x\in\textbf{X}$ is called optimal for the following gradual-impulse control problem:

(6) \begin{eqnarray}\mbox{Minimize over $u\in{\mathcal U}:$~}{\mathcal V}(x,u).\end{eqnarray}

In this paper, we will present conditions on the system primitives that guarantee the existence of an optimal policy in a simple form as defined next.

Definition 4. A policy u is called deterministic stationary if there exist some measurable mappings $(\varphi,\psi,f)$ on $\textbf{X}$, where $\varphi(x)\in\{0,\infty\}$ for each $x\in \textbf{X}$, and $\psi$ and f are $\textbf{A}^I$-valued and $\textbf{A}^G$-valued, such that $\Phi_n(\{\infty\}|h_n)=1$,

\begin{equation*}\Pi_n(da|h_n,t)=\delta_{f(\bar{x}(y_n))}(da)\end{equation*}

for all $t\ge 0$, and $u_0(\cdot|x)=\Gamma^0_n(\cdot|h_n,t,x)=\beta^\pi(\cdot|x)$ for some deterministic stationary strategy $\pi$ in the intervention DTMDP model defined by

\begin{equation*}\pi(\{\Delta\}|x_0,b_0,x_1,b_1,\dots,x_n)=I\{\varphi(x_n)=\infty\}\end{equation*}

and

\begin{equation*}\pi(db|x_0,b_0,x_1,b_1,\dots,x_n)=I\{\varphi(x_n)=0\}\delta_{\psi(x_n)}(db).\end{equation*}

In the above definition, $\Gamma^1_n$ was left arbitrary, because, under such a deterministic stationary policy, a new intervention taking place at some $t\in(0,\infty)$ is always triggered by a natural jump.

4.1. Primitives of the hat DTMDP model $\{\hat{\textbf{X}}, \hat{\textbf{A}}, p, l\}$

The state space of the hat DTMDP model is $\hat{\textbf{X}}:=\{(\infty,x_\infty)\}\cup [0,\infty)\times\textbf{X}$, where $(\infty,x_\infty)$ is an isolated point, and the action space of the DTMDP is $\hat{\textbf{A}}:=[0,\infty]\times \textbf{A}^I\times {\mathcal R}$. Endowed with the product topology, where $[0,\infty]$ is compact in the standard topology of the extended real line, $\hat{\textbf{A}}$ is also a compact Borel space. Here, $\textbf{X}$, $\textbf{A}^I$, and $\textbf{A}^G$ are the state, impulse, and gradual action spaces in the gradual-impulse control problem. The transition probability p is defined as follows, using the notation introduced prior to this subsection, e.g., $q_x(\rho_t):=\int_{\textbf{A}^G} q_x(a)\rho_t(da)$ and

\begin{equation*}c^G(x,\rho_t):=\int_{\textbf{A}^G}c^G(x,a)\rho_t(da).\end{equation*}

For each bounded measurable function g on $\hat{\textbf{X}}$ and action $\hat{a}=(c,b,\rho)\in\hat{\textbf{A}}$,

\begin{eqnarray*}&&\int_{\hat{\textbf{X}}}g(t,y)p(dt\times dy|(\theta,x),\hat{a})\\[3pt]&:=&I\{c=\infty\}\left\{ g(\infty,x_\infty)e^{-\int_0^\infty q_x(\rho_s)ds}+\int_{0}^\infty\int_{\textbf{X}}g(t,y)\tilde{q}(dy|x,\rho_t)e^{-\int_0^t q_x(\rho_s)ds} dt \right\}\\[3pt]&&+I\{c<\infty\}\left\{\int_0^c \int_{\textbf{X}} g(t,y)\tilde{q}(dy|x,\rho_t)e^{-\int_0^t q_x(\rho_s)ds}dt\right.\\[3pt]&&\left.+e^{-\int_0^c q_x(\rho_s)ds}\int_{\textbf{X}}g(c,y)Q(dy|x,b) \right\}\\[3pt]&=&\int_0^c \int_{\textbf{X}} g(t,y)\tilde{q}(dy|x,\rho_t)e^{-\int_0^t q_x(\rho_s)ds}dt+I\{c=\infty\} g(\infty,x_\infty)e^{-\int_0^\infty q_x(\rho_s)ds}\\[3pt]&&+I\{c<\infty\}e^{-\int_0^c q_x(\rho_s)ds}\int_{\textbf{X}}g(c,y)Q(dy|x,b)\end{eqnarray*}

for each state $(\theta,x)\in[0,\infty)\times \textbf{X}$, and

\begin{equation*}\int_{\hat{\textbf{X}}}g(t,y)p(dt\times dy|(\infty,x_\infty),\hat{a}):=g(\infty,x_\infty).\end{equation*}

It is known (see e.g. [Reference Costa and Dufour5, Reference Forwick, Schäl and Schmitz10]) that for each bounded measurable function g on $\hat{\textbf{X}}$, the above expressions are indeed measurable on $\hat{\textbf{X}}\times \hat{\textbf{A}}$, and the same also holds for the cost function l on $\hat{\textbf{X}}\times\hat{\textbf{A}}\times\hat{\textbf{X}}$, defined as follows:

\begin{eqnarray*}l((\theta,x),\hat{a},(t,y)):=I\{(\theta,x)\in [0,\infty)\times\textbf{X}\}\left\{\int_0^t c^G(x,\rho_s)ds+I\{t=c\}c^I(x,b,y)\right\}\end{eqnarray*}

for each $(\theta,x),\hat{a},(t,y))\in\hat{\textbf{X}}\times\hat{\textbf{A}}\times\hat{\textbf{X}}$, accepting that $c^I(x,b,x_\infty)\equiv 0.$ Recall that the generic notation $\hat{a}=(c,b,\rho)\in\hat{\textbf{A}}$ for an action in this hat DTMDP model has been in use. The pair (c,b) is the pair of the planned time until the next impulse and the next planned impulse, and $\rho$ is (the rule of) the relaxed control to be used during the next sojourn time. Figure 2 displays the realization of the components $\{(C_n,B_n)\}_{n=0}^\infty$ of the action process in the hat DTMDP model corresponding to the sample path in Figure 1 for the gradual-impulse control problem.

Figure 2.

The realization of the state process in the hat DTMDP model corresponding to the sample path in the gradual-impulse control problem in Figure 1. The time index is discrete from $\{0,1,\dots\}$. The realizations of the components $\{(C_n,B_n)\}_{n=0}^\infty$ in the action process $\{\hat{A}_n\}_{n=0}^\infty$ are indicated above the dashed lines between consecutive states. For example, $(0,b_0)$ next to the state $(0,x_0)$ indicates that the decision-maker applies an impulse $b_0$ immediately, which results in the next state $(0,x_1).$ All the components $x_0,x_1,\dots,$ $x_0'$, $x_1''$, $x_2''$ and $b_1,b_2,b_0'',b_1'',b_2''$ are the same as in Figure 1. The only exception is $(c_3,b_3),$ which does not appear in Figure 1. Nevertheless, $c_3>\theta_2$, because in Figure 1, the first jump in the marked point process therein at the time moment $\theta_1+\theta_2=\theta_2$ is triggered by a natural jump.

The optimal control problem for the hat DTMDP model reads as follows:

(12) \begin{eqnarray}\mbox{Minimize over $\sigma$:}~\hat{\mathbb{E}}_{(\theta,x)}^\sigma\left[e^{\sum_{n=0}^\infty l(\hat{X}_n,\hat{A}_n,\hat{X}_{n+1})}\right]=:V((\theta,x),\sigma),\end{eqnarray}

where $\{\hat{X}_n\}_{n=0}^\infty$ and $\{\hat{A}_n\}_{n=0}^\infty$ are the state and action processes, and the minimization problem is over all strategies $\sigma$ in the hat DTMDP model. We denote by $V^\ast$ the value function of this optimal control problem, i.e.,

\begin{equation*}V^\ast(\theta,x):=\inf_{\sigma}\hat{\mathbb{E}}_{\hat{x}}^\sigma\left[e^{\sum_{n=0}^\infty l(\hat{X}_n,\hat{A}_n,\hat{X}_{n+1})}\right]\end{equation*}

for each $\hat{x}=(\theta,x)\in\hat{\textbf{X}}$, where the infimum is over all strategies. Clearly, $V^\ast(\infty,x_\infty)=1$. It will be seen in Lemma 5 that $V^\ast$ depends on $(\theta,x)$ only through x, and a strategy $\sigma$ is optimal if $V((0,x),\sigma)=V^\ast(x)$ for each $x\in\textbf{X}$. Below, when the context is clear, we often consider the restriction of $V^\ast$ on $\textbf{X}$ but still use the same notation.

Let $\hat{h}_n=((\theta_0,x_0),(c_0,b_0,\rho_0),(\theta_1,x_1), (c_1,b_1,\rho_1),(\theta_2,x_2),\dots,(\theta_n,x_n))$ be the generic notation for an n-history in the hat DTMDP model. A strategy in the hat DTMDP model is a sequence $\sigma=\{\sigma_n\}_{n=0}^\infty$, where for each $n\ge 0,$ $\sigma_n(d\hat{a}|\hat{h}_n)$ is a stochastic kernel on $\hat{\textbf{A}}$ given $\hat{h}_n$, which specifies the conditional distribution of the next action $(c,b,\rho)$ given $\hat{h}_n$. In general, a strategy in the hat DTMDP model can make use of past decision rules of relaxed controls, and the selection of the next relaxed control and that of the next planned impulse time and impulse need not be (conditionally) independent. Therefore, a general strategy in the hat DTMDP model does not immediately correspond to a policy in the gradual-impulse control problem described in the previous section. To relate the gradual-impulse control problem (6) and the hat DTMDP problem (12) (see Proposition 2 below), we introduce the following class of strategies in the hat DTMDP model.

Definition 6. A strategy $\sigma$ in the hat DTMDP model is called typical if under it, given $\hat{h}_n$, the selection of the next action (c,b) and $\rho$ are conditionally independent, and moreover, the selection of $\rho$ is deterministic, i.e.,

\begin{equation*}\sigma_n(dc\times db\times d\rho|\hat{h}_n)=\sigma_n'(dc\times db|\hat{h}_n)\delta_{F^n(\hat{h}_n)}(d\rho),\end{equation*}

where $F^n(\hat{h}_n)$ is measurable in its argument and takes values in ${\mathcal R}$, and $\sigma_n'(dc\times db|\hat{h}_n)$ is a stochastic kernel on $[0,\infty]\times \textbf{A}^I$ given $\hat{h}_n$. One can always write $\sigma_n'(dc\times db|\hat{h}_n)=\varphi_n(dc|\hat{h}_n)\psi_n(db|\hat{h}_n,c)$ for some stochastic kernels $\varphi_n$ and $\psi_n$. Intuitively, $\varphi_n$ defines the (conditional) distribution of the planned time duration till the next impulse, and $\psi_n(db|\hat{h}_n,c)$ specifies the distribution of the next impulsive action given the history $\hat{h}_n$ and the next impulse moment c, provided that it takes place before the next natural jump. Therefore, we identify a typical strategy $\sigma=\{\sigma_n\}_{n=0}^\infty$ as $\{(\varphi_n,\psi_n,F^n)\}_{n=0}^\infty.$ For further notational brevity, when the stochastic kernels $\varphi_n$ are identified with underlying measurable mappings, we will use $\varphi_n$ for the measurable mappings, and write $\varphi_n(\hat{h}_n)$ instead of $\varphi_n(dc|\hat{h}_n)$. The same applies to other stochastic kernels such as $\psi_n$. The context will exclude any potential confusion. Finally, in general, we often do not indicate the arguments that do not affect the values of the mappings in question. For example, if $\varphi_n(\hat{h}_n)$ depends on $\hat{h}_n$ only through $x_n$, then we write $\varphi_n(dc|\hat{h}_n)$ as $\varphi_n(dc|x_n)$.

4.2 Relation between gradual-impulse control and hat DTMDP problems

Each policy u as introduced in Definition 1 induces a strategy $\{(\varphi_n,\psi_n,F^n)\}_{n=0}^\infty$ in the hat DTMDP model as follows, where we need only consider $x_n\in\textbf{X}$, as the definition of the strategies at $x_n=x_\infty$ is immaterial and can be arbitrary. For each $m\ge 1$, and $h_m\in\textbf{H}_m$, there exists a strategy

\begin{equation*}\pi^{\Gamma^1_m,h_m}=\{\pi^{\Gamma^1_m,h_m}_n\}_{n=0}^\infty\end{equation*}

in the intervention DTMDP model such that

\begin{equation*}\Gamma^1_m(dy|h_m)=\beta^{\pi^{\Gamma^1_m,h_m}}(dy|\bar{x}(y_m)).\end{equation*}

Similarly, for each $x\in\textbf{X}$, $t>0$, there exists a strategy

\begin{equation*}\pi^{\Gamma^0_m,h_m,t,x}=\{\pi^{\Gamma^0_m,h_m,t,x}_n\}_{n=0}^\infty\end{equation*}

in the intervention DTMDP model such that

\begin{equation*}\Gamma^0_m(dy|h_m,t,x)=\beta^{\pi^{\Gamma^0_m,h_m,t,x}}(dy|x).\end{equation*}

Finally, there is a strategy

\begin{equation*}\pi^{u_0}=\{(\pi^{u_0}_n)\}_{n=0}^\infty\end{equation*}

in the intervention DTMDP model satisfying

\begin{equation*}u_0(dy|x)=\beta^{\pi^{u_0}}(dy|x)\end{equation*}

for each $x\in\textbf{X}$.

Consider the case $n=0$. Here we define

\begin{align*}\varphi_0(\{0\}|\theta,x)&:= 1-\pi_0^{u_0}(\{\Delta\}|x);\\[3pt] \varphi_0(dc|\theta,x)&:=\pi_0^{u_0}(\{\Delta\}|x)\Phi_1(dc|(x,\Delta,\Delta,\dots),0,(x,\Delta,\Delta,\dots)) \quad \text{on } (0,\infty]; \\[3pt] \psi_0(db|\theta,x,c):=&\frac{\pi^{u_0}_0(db|x)}{1-\pi^{u_0}_0(\{\Delta\}|x)}I\{c=0\}\\&\quad +I\{c>0\} \frac{\pi^{\Gamma^1_1,((x,\Delta,\dots),0,(x,\Delta,\dots))}_0(db|x)}{1-\pi^{\Gamma^1_1,((x,\Delta,\dots),0,(x,\Delta,\dots))}_0(\{\Delta\}|x)}\\& =\frac{\pi^{u_0}_0(db|x)}{1-\pi^{u_0}_0(\{\Delta\}|x)}I\{c=0\}+I\{c>0\} \pi^{\Gamma^1_1,((x,\Delta,\dots),0,(x,\Delta,\dots))}_0(db|x);\end{align*}

and $F^0(\theta,x)_t(da):=\Pi_1(da|(x,\Delta,\Delta,\dots),0,(x,\Delta,\Delta,\dots),t),$ where the second equality in the definition of $\psi_0(db|\theta,x,c)$ holds because

\begin{equation*}\pi^{\Gamma^1_1,((x,\Delta,\dots),0,(x,\Delta,\dots))}_0(\{\Delta\}|x)=0,\end{equation*}

which follows from the requirement that

\begin{equation*}\Gamma^1_n(\cdot|h_n)\in {\mathcal P}^{\textbf{Y}^\ast}(\bar{x}(y_n))\end{equation*}

for all $n\ge 1$ in Definition 3. Also concerning the definition of $\psi_0(db|\theta,x,c)$, note that if the denominator $ {1-\pi^{u_0}_0(\{\Delta\}|x)}$ equals 0, we put

\begin{equation*}\frac{\pi^{u_0}_0(db|x)}{1-\pi^{u_0}_0(\{\Delta\}|x)}\end{equation*}

as an arbitrary stochastic kernel. The reason is that in the expression

\begin{equation*}\frac{\pi^{u_0}_0(db|x)}{1-\pi^{u_0}_0(\{\Delta\}|x)}I\{c=0\},\end{equation*}

equality ${1-\pi^{u_0}_0(\{\Delta\}|x)}=0$ would indicate that the probability of selecting an instantaneous impulse is zero, and so $I\{c=0\}=0$ almost surely. The same explanation applies to the definitions of $\psi_n(db|\hat{h}_n,c)$ below, and will not be repeated there. Note that the right-hand side does not depend on $\theta\in[0,\infty)$, because the initial time moment is always fixed to be $\theta=0$.

The intuition behind the above definition of $(\varphi_0,\psi_0,F^0)$ is as follows. Recall that if the initial system state is $x\in\textbf{X}$, then the intervention $y_1\in\textbf{Y}$ at the initial time in the gradual-impulse control problem is a realization from the distribution

\begin{equation*}u_0(\cdot|x)=\beta^{\pi^{u_0}}(\cdot|x),\end{equation*}

which is the strategic measure of some strategy $\pi^{u_0}=\{\pi_n^{u_0}\}_{n=0}^\infty$ in the intervention DTMDP model. Then $\pi_0^{u_0}(\{\Delta\}|x)$ is the probability that no impulse is applied at the initial time 0 (given the initial system state x) in the gradual-impulse control problem. Consequently, $1-\pi_0^{u_0}(\{\Delta\}|x)$ is the probability of applying an impulse immediately, i.e., of waiting time 0 until the next impulse, and thus $\varphi_0(\{0\}|\theta,x)$. This quantity does not depend on $\theta$, because the initial time is always $0.$ Then for a measurable subset $\Gamma_1\subseteq(0,\infty]$,

\begin{eqnarray*}&&\pi_0^{u_0}(\{\Delta\}|x)\Phi_1(\Gamma_1|(x,\Delta,\Delta,\dots),0,(x,\Delta,\Delta,\dots))\\[3pt]&=&\mbox{$\mathbb{P}$(no impulse at initial time 0 given initial system state \textit{x})}\\[3pt] &&\times \mbox{$\mathbb{P}$(time to wait until next impulse is in $\Gamma_1$ given no impulse is immediately} \\[3pt] &&\mbox{applied at the initial time with the initial state \textit{x})},\end{eqnarray*}

which is equal to

\begin{eqnarray*}&&\mbox{$\mathbb{P}$(no immediate impulse, and the time duration until the next planned} \\[3pt] &&\mbox{impulse is in $\Gamma$)}=\mbox{$\mathbb{P}$(the time duration until the next planned impulse is in $\Gamma$)},\end{eqnarray*}

and thus $\varphi_0(\Gamma|\theta,x)$, where the equality follows because $\Gamma\subseteq(0,\infty].$ (Recall that a planned impulse takes place if no natural jump occurs during the time leading up to it.) Finally, as for $\psi_0(db|\theta,x,c)$, if $c=0$ and $\Gamma_2\in{\mathcal B}(\textbf{A}^I)$, then

\begin{eqnarray*}&&\frac{\pi^{u_0}_0(\Gamma_2|x)}{1-\pi^{u_0}_0(\{\Delta\}|x)}=\frac{\mbox{$\mathbb{P}$(an immediate impulse from $\Gamma_2$ is applied)}}{\mbox{$\mathbb{P}$(an immediate impulse is applied)}}\\[3pt] &=&\mbox{$\mathbb{P}$(an impulse is applied immediately from $\Gamma_2$}\\[3pt] &&\mbox{given that an impulse is applied after time duration 0)},\end{eqnarray*}

which is thus $\psi_0(\Gamma_2|\theta,x,0).$ One can understand $\psi_0(db|\theta,x,c)$ when $c>0$ in the same manner. Very similar intuition guides the definition of $(\varphi_n,\psi_n,F^n)$ below.

Now consider $n\ge 1.$ Let $\hat{h}_n$ be the n-history in the hat DTMDP model. If $\{1\le i\le n: ~\theta_i>0\}= \emptyset,$ then we define $\varphi_n(\{0\}|\hat{h}_n):=1-\pi^{u_0}_n(\{\Delta\}|x_0,b_0,\dots,b_{n-1},x_n);$

\begin{align*}\varphi_n(dc|\hat{h}_n):=&\pi^{u_0}_n(\{\Delta\}|x_0,b_0,\dots,b_{n-1},x_n)\Phi_1(dc|y_0,0,(x_1,b_1,\dots,x_n,\Delta,\Delta,\dots))\\[3pt] &\mbox{on~} (0,\infty];\\[3pt] \psi_n(db|\hat{h}_n,c):=&\frac{\pi_n^{u_0}(db|x_0,b_0,x_1,b_1,\dots,x_n)}{1-\pi^{u_0}_n(\{\Delta\}|x_0,b_0,x_1,b_1,\dots,x_n)}I\{c=0\}\\[3pt] &+I\{c>0\}\frac{\pi^{\Gamma^1_1,(y_0,0,(x_0,b_0,\dots,x_n,\Delta,\dots))}_0(db|x_n)}{1-\pi^{\Gamma^1_1,(y_0,0,(x_0,b_0,\dots,x_n,\Delta,\dots))}_0(\{\Delta\}|x_n)}\\[3pt] =&\frac{\pi_n^{u_0}(db|x_0,b_0,x_1,b_1,\dots,x_n)}{1-\pi^{u_0}_n(\{\Delta\}|x_0,b_0,x_1,b_1,\dots,x_n)}I\{c=0\}\\[3pt] &+I\{c>0\}\pi^{\Gamma^1_1,(y_0,0,(x_0,b_0,\dots,x_n,\Delta,\dots))}_0(db|x_n);\end{align*}

and $F^n(\hat{h}_n)_t(da):=\Pi_1(da|y_0,0,(x_0,b_0,\dots,x_n,\Delta,\Delta,\dots),t).$ Recall the notation that was introduced earlier: $y_0=(x_0,\Delta,\Delta,\dots).$

If $\{1\le i\le n: ~\theta_i>0\}\ne \emptyset,$ then let $m(\hat{h}_n):=\#\{1\le i\le n: ~\theta_i>0\}$ and $l(\hat{h}_n):=\max\{1\le i\le n: \theta_i>0\}.$ When the context is clear, we write m and l instead of $m(\hat{h}_n)$ and $l(\hat{h}_n)$ for brevity. Let $h_m$ be the m-history in the gradual-impulse control problem contained in $\hat{h}_n$. More precisely, $h_m$ is defined based on $\hat{h}_n$ as follows. Let $\tau_0(\hat{h}_n)=0,$ and $\tau_i(\hat{h}_n):=\inf\{j>\tau_{i-1}:~\theta_j>0\}$ for each $i\ge 1.$ Note that $l=\tau_m.$ Then

\begin{equation*}h_m=h_m(\hat{h}_n)=(y_0,0,y_1,\theta_{\tau_1},y_{2},\dots,\theta_{\tau_{m-1}},y_m),\end{equation*}

where $y_0=(x_0,\Delta,\Delta,\dots)$;

\begin{equation*}y_1=(x_0,b_0,x_1,b_1,\dots,x_{\tau_1-1},\Delta,\Delta,\dots);\end{equation*}

if $\theta_{\tau_1}=c_{\tau_1-1}$, then

\begin{equation*}y_2=(x_{\tau_1-1},b_{\tau_1-1},x_{\tau_1},b_{\tau_1},\dots,x_{\tau_2-1},\Delta,\Delta,\dots),\end{equation*}

while if $\theta_{\tau_1}<c_{\tau_1-1}$, then

\begin{equation*}y_2=(x_{\tau_1},b_{\tau_1},\dots,\linebreak x_{\tau_2-1},\Delta,\Delta,\dots);\end{equation*}

$\dots$; if $\theta_{\tau_{m-1}}=c_{\tau_{m-1}-1}$, then

\begin{equation*}y_m=(x_{\tau_{m-1}-1},\dots,x_{\tau_{m}-1},\Delta,\Delta,\dots),\end{equation*}

while if $\theta_{\tau_{m-1}}<c_{\tau_{m-1}-1}$, then

\begin{equation*}y_m=(x_{\tau_{m-1}},\dots,x_{\tau_{m}-1},\Delta,\Delta,\dots).\end{equation*}

For example, if

\begin{eqnarray*}\hat{h}_5&=&((0,x_0),(b_0,0,\rho^0),(0,x_1),(b_1,3,\rho^1),(3,x_2),(b_2,0,\rho^2),(0,x_3),\\[3pt]&&(b_3,2,\rho^3),(1,x_4),(b_4,0,\rho^4),(0,x_5)), \end{eqnarray*}

then $n=5$, $m=2$, $l=4,$ $\tau_1=2$, $\tau_2=4,$ and $h_2=(y_0,0,y_1,3,y_2)$ with $y_1=(x_0,b_0,x_1,\Delta,\dots)$ and $y_2=(x_1,b_1,x_2,b_2,x_3,\Delta,\dots).$ Roughly speaking, the integer $m(\hat{h}_n)$ counts the number of interventions (except $y_0$) contained in the n-history of the hat DTMDP model.

If $0<\theta_l=c_{l-1},$ we define

\begin{align*}\varphi_n(\{0\}|\hat{h}_n):=&1-\pi^{\Gamma^1_m,h_m}_{n-l+1}(\{\Delta\}|x_{l-1},b_{l-1},\dots,b_{n-1},x_n);\\[3pt]\varphi_n(dc|\hat{h}_n):=&\pi^{\Gamma^1_m,h_m}_{n-l+1}(\{\Delta\}|x_{l-1},b_{l-1},\dots,b_{n-1},x_n)\Phi_m(dc|h_m)~\mbox{on~} (0,\infty];\\[3pt]\psi_n(db|\hat{h}_n,c):=&\frac{\pi^{\Gamma^1_m,h_m}_{n-l+1}(db|x_{l-1},b_{l-1},\dots,b_{n-1},x_n)}{1-\pi^{\Gamma^1_m,h_m}_{n-l+1}(\{\Delta\}|x_{l-1},b_{l-1},\dots,b_{n-1},x_n)}I\{c=0\}\\[3pt] &+I\{c>0\}\frac{\pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l-1},b_{l-1},\dots,x_n,\Delta,\dots))}_0(db|x_n)}{1-\pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l-1},b_{l-1},\dots,x_n,\Delta,\dots))}_0(\{\Delta\}|x_n)}\\[3pt] = &\frac{\pi^{\Gamma^1_m,h_m}_{n-l+1}(db|x_{l-1},b_{l-1},\dots,b_{n-1},x_n)}{1-\pi^{\Gamma^1_m,h_m}_{n-l+1}(\{\Delta\}|x_{l-1},b_{l-1},\dots,b_{n-1},x_n)}I\{c=0\}\\[3pt] &+I\{c>0\} \pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l-1},b_{l-1},\dots,x_n,\Delta,\dots))}_0(db|x_n);\\[3pt] F^n(\hat{h}_n)_t(da):=&\Pi_m(da|h_m,t).\end{align*}

Finally, if $0<\theta_l<c_{l-1}$, then we define

\begin{align*}\varphi_n(\{0\}|\hat{h}_n):=&1-\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(\{\Delta\}|x_{l},b_{l},\dots,b_{n-1},x_n);\\[3pt] \varphi_n(dc|\hat{h}_n):=&\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(\{\Delta\}|x_{l},b_{l},\dots,b_{n-1},x_n)\Phi_m(dc|h_m)~\mbox{on~} (0,\infty];\\[3pt] \psi_n(db|\hat{h}_n,c):=&\frac{\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(db|x_{l},b_{l},\dots,b_{n-1},x_n)}{1-\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(\{\Delta\}|x_{l},b_{l},\dots,b_{n-1},x_n)}I\{c=0\}\\[3pt] &+I\{c>0\}\frac{\pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l},b_{l},\dots,x_n,\Delta,\dots))}_0(db|x_n)}{1-\pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l},b_{l},\dots,x_n,\Delta,\dots))}_0(\{\Delta\}|x_n)}\\[3pt] =&\frac{\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(db|x_{l},b_{l},\dots,b_{n-1},x_n)}{1-\pi^{\Gamma^0_m,h_m,\theta_l,x_l}_{n-l}(\{\Delta\}|x_{l},b_{l},\dots,b_{n-1},x_n)}I\{c=0\}\\[3pt] &+I\{c>0\} \pi^{\Gamma^1_{m+1},(h_m,\theta_l,(x_{l},b_{l},\dots,x_n,\Delta,\dots))}_0(db|x_n);\\[3pt] F^n(\hat{h}_n)_t(da):=&\Pi_m(da|h_m,t).\end{align*}

To be specific, we refer to the typical strategy $\sigma=\{(\varphi_n,\psi_n,F^n)\}_{n=0}^\infty$ defined above as the strategy induced by the policy u. The next statement reveals a connection between a policy u and its induced strategy $\sigma$ for the hat DTMDP model.

Proof. One can verify that

\begin{eqnarray*}&&\mathbb{E}_x^u\left[e^{\sum_{i=1}^n C^I(Y_i) +\sum_{i=2}^n\int_{0}^{\Theta_{i}} \int_{\textbf{A}^G}c^G(\overline{x}(Y_{i-1}),a)\Pi_{i-1}(da|H_{i-1},s)ds}\right]\\[3pt]&=&\hat{\mathbb{E}}_{(0,x)}^{\sigma}\left[e^{\sum_{i=0}^{\tau_n-1} c^I(X_i,B_i,X_{i+1})I\{C_i=\Theta_{i+1}\}}\right.\\[3pt]&&\left. e^{\sum_{i=2}^{n}\int_{0}^{\Theta_{\tau_{i-1}}} \int_{\textbf{A}^G}c^G(X_{\tau_{i-1}-1},a)F^{\tau_{i-1}-1}(\hat{H}_{\tau_{i-1}-1})_s(da)ds}\right]\end{eqnarray*}

for each $n\ge 1$. Passing to the limit as $n\rightarrow \infty$ and applying the monotone convergence theorem yields the equality in the statement. The last assertion follows automatically from the first assertion.

To end this section, note that Condition 2 does not imply that the hat DTMDP model is semicontinuous, as defined in the appendix. In fact, the transition probability p, in general, does not satisfy the weak continuity condition, even under Condition 2. The simplest example is as follows.

Example 3. Suppose $q_x(a)\equiv 0$, and $\textbf{A}^G$ and $\textbf{A}^I$ are both singletons. Consider $\hat{a}_n=(c_n,b,\rho)$, where $c_n\rightarrow \infty$ and $c_n\in[0,\infty)$ for each $n\ge 1$, together with the bounded continuous function on $\hat{\textbf{X}}$ defined by $g(t,x)\equiv 1$ for each $(t,x)\in[0,\infty)\times \textbf{X}$, and $g(\infty,x_\infty)=0$. Then

\begin{equation*}\int_{\hat{\textbf{X}}}g(t,y)p(dt\times dy|(\theta,x),\hat{a}_n)=\int_{\textbf{X}}g(c_n,y)Q(dy|x,b)=1\end{equation*}

for each $n\ge 1,$ whereas

\begin{equation*}\int_{\hat{\textbf{X}}}g(t,y)p(dt\times dy|(\theta,x),(\infty,b,\rho))=g(\infty,x_\infty)=0\ne 1.\end{equation*}

One can also construct examples where the transition probability p is not continuous with respect to $\rho\in{\mathcal R}.$