2.1 Waldean Statistical Decision Theory and Quantiles

In this section, we further discuss the proposed criterion introduced in (2.3) and juxtapose it with other possible approaches of how quantiles could be incorporated into a Waldean framework of statistical decision theory.

In a series of pathbreaking contributions, Wald (Reference Wald1945, Reference Wald1947, Reference Wald1950) introduced a framework for statistical decision theory whose main components are a statistical model, an action space, statistical decision rules, a welfare and an expected welfare function, and finally an evaluation criterion. Instead of welfare and expected welfare the presentation could be framed in terms of loss (defined as negative welfare) and risk (defined as expected loss). See Hirano (Reference Hirano2026) for a comprehensive review that also includes contemporary developments.

In the particular context considered here where the DM needs to pick treatment 0 or 1 after observing the i.i.d. sample $w_{N}$ , a minimax regret rule in the Waldean formulation solves

(2.9) $$ \begin{align} \arg\min_{\delta\in\mathbb{D}}\max_{s\in\mathbb{S}}[\max\{\mu_{0},\mu _{1}\}-E_{s}Y_{B(\delta(w_{N}))}], \end{align} $$

where $\mu _{t}=E_{s}Y_{t}$ for $t=0,1$ and $E_{s}$ denotes the expectation operator under $s.$ It is easily shown that $s$ matters only via $(\mu _{0} ,\mu _{1}).$ By the law of iterated expectations,

(2.10) $$ \begin{align} E_{s}Y_{B(\delta(w_{N}))} & =E_{w_{N}}E_{s}(Y_{B(\delta(w_{N}))} |w_{N})\nonumber\\ & =\mu_{0}E_{w_{N}}(1-\delta(w_{N}))+\mu_{1}E_{w_{N}}\delta(w_{N}), \end{align} $$

overall uncertainty can be separated into sampling uncertainty through $w_{N},$ (potential) randomness through the treatment assignment $B,$ and randomness through the potential outcome variables $(Y_{0},Y_{1} )$ ; risk is defined as the average loss over sampling uncertainty.

To adapt the Waldean framework to one that is based on the notion of $\alpha $ -quantile rather than expectation, we propose replacing expectations by $\alpha $ -quantiles in the formulation (2.9), that is, we suggest solving (2.7) which is

(2.11) $$ \begin{align} \arg\min_{\delta\in\mathbb{D}}\max_{s\in\mathbb{S}}[\max\{q_{s,\alpha} (Y_{0}),q_{s,\alpha}(Y_{1})\}-q_{s,\alpha}(Y_{B(\delta(w_{N}))})]. \end{align} $$

Given that there is no equivalent to the law of iterated expectations for quantiles, by doing so, one loses the separation of the regret criterion into loss without sampling uncertainty and sampling uncertainty. Our proposed criterion aggregates joint uncertainty (sampling, treatment assignment, and outcome) before taking the quantile.Footnote ⁶

Instead, to maintain such separation, one could first define regret loss given a realized sample $w_{N}$ as

(2.12) $$ \begin{align} L(\delta(w_{N}),s)=\max\{q_{s,\alpha}(Y_{0}),q_{s,\alpha}(Y_{1})\}-q_{s,\alpha }(Y_{B(\delta(w_{N}))}|w_{N}) \end{align} $$

and then define an alternative regret to (2.6) as the average of regret loss over the sampling distribution $w_{N}$

(2.13) $$ \begin{align} R^{\prime}(\delta,s)=\max\{q_{s,\alpha}(Y_{0}),q_{s,\alpha}(Y_{1})\}-E_{w_{N} }q_{s,\alpha}(Y_{B(\delta(w_{N}))}|w_{N}). \end{align} $$

The first term in (2.12) is a benchmark attainable if one could pick the better treatment in terms of quantile outcome without sampling uncertainty; the second term is the $\alpha $ -quantile of $Y_{B(\delta (w_{N} ))}$ conditional on the realized sample $w_{N}$ . This criterion may be misaligned when the DM cares about tails in the distribution of $q_{s,\alpha }(Y_{B(\delta (w_{N}))}|w_{N})$ over realized samples rather than the average over sampling uncertainty. As yet another alternative, one could consider a regret function defined as the $\beta $ -quantile (for some $\beta $ in the unit interval) of the loss function in (2.12), that is,

(2.14) $$ \begin{align} R^{\prime\prime}(\delta,s)=q_{s,\beta}(L(\delta(w_{N}),s)). \end{align} $$

By the monotone transform identity of quantiles, it follows that

(2.15) $$ \begin{align} R^{\prime\prime}(\delta,s)=\max\{q_{s,\alpha}(Y_{0}),q_{s,\alpha} (Y_{1})\}-q_{s,1-\beta}(q_{s,\alpha}(Y_{B(\delta(w_{N}))}|w_{N})) \end{align} $$

and the interpretation of the criterion is not straightforward.

Manski and Tetenov (Reference Manski and Tetenov2023, Sect. 6.1) suggest an entire class of alternative approaches that also maintain separation. Starting with an arbitrary loss function (for instance, negative welfare) for a given action by the DM (i.e., in our setup, a choice of treatment 0 or 1) and DGP $s,$ the suggestion is to consider, for example, an $\alpha $ -quantile of the loss function over the sampling distribution. In our setup, such an approach could be formulated as

(2.16) $$ \begin{align} \arg\min_{\delta\in\mathbb{D}}\sup_{s\in\mathbb{S}}[\max\{\mu_{0},\mu _{1}\}-q_{s,\alpha}(\mu_{B(\delta(w_{N}))})]. \end{align} $$

One issue with the implementation of the various criteria is that analytical formulas are not generally available.Footnote ⁷

We do not have a strong opinion about which one of the above criteria is generally preferable. However, we find our criterion very natural in situations where a DM applies treatment to an individual (e.g., herself) after observing the sample. Consider, for example, an individual who books a hotel accommodation on an internet platform, picking one of two options after observing a certain number of ratings on each (or picks one of two restaurants after having observed a number of ratings for each on the internet). The outcome is the rating the DM assigns to the hotel she booked, which could be interpreted as a proxy for the welfare the DM received from staying at the hotel. Another example is a medical doctor who picks one of two treatments for a patient after observing a certain number of outcomes on the two treatments.

In that type of example, when repeating the exercise, every observed outcome combines the randomness of the sample, (potential) randomness of treatment assignment, and the randomness of $(Y_{0},Y_{1}).$ If the DM is concerned about quantiles of the outcome distribution, it seems that the criterion proposed in (2.11) is a natural choice.

2.2 Minimax Regret Treatment Rules

Recall that a sample of size $N$ has the general structure $w_{N}=(t,y).$ We consider three different sample designs, namely:

(i) Fixed number of untreated/treated units with $N_{0}\in \{0,\ldots ,N\}$ i.i.d. observations of $Y_{0}$ and $N_{1}:=N-N_{0}$ i.i.d. observations of $Y_{1}$ for some $N\in \mathbb {N}\cup \{0\}$ . Here, the notation for a sample can be simplified by dropping $t.$ We write ${w_{N}=(y(0)^{\prime } ,y(1)^{\prime })^{\prime }\in \, [0,1]^{N}}$ , where $y(0)\in \lbrack 0,1]^{N_{0}}$ contains the $N_{0}$ observations on untreated units and $y(1)\in \lbrack 0,1]^{N_{1}}$ contains the $N_{1}$ observations on the treated units. A treatment rule $\delta \in \mathbb {D}$ is then any mapping $[0,1]^{N} \rightarrow \lbrack 0,1].$ It assigns a treatment probability $\delta (w_{N} )\in \lbrack 0,1]$ after observing the sample.

(ii) Random assignment with $N\in \mathbb {N}\cup \{0\}$ i.i.d. observations, where in the sample, the treatment probability equals $p\in (0,1).$ Footnote ⁸ A treatment rule $\delta \in \mathbb {D}$ is any mapping $\{0,1\}^{N}\times \lbrack 0,1]^{N}\rightarrow \lbrack 0,1].$ The rule $\delta $ assigns a treatment probability $\delta (w_{N})\in \lbrack 0,1]$ after observing a sample $w_{N}=(t,y)$ of treatment statuses ${t=(t_{1},\ldots ,t_{N})}$ and realizations $y=(y_{1},\ldots ,y_{N}),$ where the $i$ -th component of $y,\, y_{i},$ is an independent realization of $Y_{t_{i}}$ .

(iii) Testing an innovation is the case where aspects of the distribution of $Y_{0}$ are known to the DM; in particular, we assume that the $\alpha $ -quantile of $Y_{0}$ is known (but nature can pick arbitrary distributions for $Y_{0}$ subject to that restriction). That is, in this case, the set $\mathbb {S}$ consists of all joint distributions $s$ for $(Y_{0} ,Y_{1})$ with the restriction that the $\alpha $ -quantile of the marginal for $Y_{0}$ equals a certain value $q_{\alpha }(Y_{0}).$ In this case, $N\in \mathbb {N}\cup \{0\}$ i.i.d. observations of $Y_{1}$ are observed. Here again, the notation for a sample can be simplified. We write $w_{N} =y(1)\in \lbrack 0,1]^{N}$ , where $y(1)$ contains the $N$ observations of the treated units. A treatment rule $\delta \in \mathbb {D}$ is any mapping $[0,1]^{N}\rightarrow \lbrack 0,1].$ The rule $\delta $ assigns a treatment probability $\delta (w_{N})\in \lbrack 0,1]$ after observing a sample $w_{N}=(y_{1,1},\ldots ,y_{1,N})$ of $N$ independent realizations $y_{1,i},\, i=1,\ldots ,N$ of $Y_{1}.$

The designs above nest the ones in Stoye (Reference Stoye2009). In contrast to Stoye (Reference Stoye2009), in design (i), we allow for arbitrary numbers of treated and untreated units rather than $N/2$ units each as in “matched pairs” and in design (ii), the treatment probability is any fixed number $p\in (0,1)$ rather than necessarily $p$ = 0.5.

The following statement provides the analog to Proposition 1 in Stoye (Reference Stoye2009) and is the main contribution of this article.

Comments. 1. Proposition 1 establishes that the minimax regret criterion when applied to $\alpha $ -quantiles does not favor data-driven rules. In fact, in the “testing an innovation” case, data-driven rules are strictly dominated by $\delta ^{0}\equiv 0$ when $q_{\alpha }(Y_{0}) >1/2$ and weakly dominated by $\delta ^{1}\equiv 1$ when $q_{\alpha }(Y_{0})<1/2.$ When $q_{\alpha }(Y_{0})=1/2,$ all treatment rules are minimax. This is in stark contrast to the results in Stoye (Reference Stoye2009) about minimax regret treatment rules when the focus is on mean outcomes. Namely, Stoye (Reference Stoye2009) shows that, for example, in the case of binary outcomes, where the sample is obtained as a matched pair, to be minimax regret optimal, the treatment that has more successes in the sample must be chosen with probability one.

2. To provide intuition of the result in (i) and (ii), assume $\alpha \in (0,1)$ and consider first the simplest case where the sample size $N$ is 0. In that case, a treatment rule $\delta $ is simply an element in $[0,1]$ that denotes the probability of assigning treatment. For given $\delta $ , the objective for nature is to find a distribution for $(Y_{0},Y_{1})$ such that the $\alpha $ -quantiles of the marginals have maximal distance (i.e., distance 1) and such that the $\alpha $ -quantile of $Y_{B(\delta )}$ is zero. Let $\delta>0$ first. Assume nature picks $Y_{0}$ and $Y_{1}$ as independent Bernoulli

(2.17) $$ \begin{align} P(Y_{1}=0)=1,\text{ }P(Y_{0}=0)=\alpha-\varepsilon\text{ for some } \varepsilon\in(0,\alpha] \end{align} $$

and, consequently, $P(Y_{0}=1)=1-(\alpha -\varepsilon ).$ Thus, $q_{s,\alpha }(Y_{1})=0$ and $q_{s,\alpha }(Y_{0})=1$ and

(2.18) $$ \begin{align} P(Y_{B(\delta)}=0)=\delta+(1-\delta)(\alpha-\varepsilon). \end{align} $$

Thus, $P(Y_{B(\delta )}=0)>\alpha $ iff $\delta (1-\alpha )>\varepsilon (1-\delta )$ which holds for $\varepsilon $ small enough. Thus,

(2.19) $$ \begin{align} \max_{s\in\mathbb{S}}R(\delta,s)=1 \end{align} $$

for all $\delta \in (0,1].$ If instead $\delta =0,$ then nature chooses $P(Y_{1}=1)=1$ and $P(Y_{0}=0)=1$ which leads to regret of 1.

This result may be surprising at first. For example, if the DM tries to be completely balanced and picks $\delta =1/2$ (which seems reasonable in the no data case), nature can inflict regret equal to one by picking $P(Y_{1}=0)=1$ and, for example, $P(Y_{0}=0)=\alpha -\min \{\alpha /2,(1-\alpha )/2\}.$ This is in stark contrast to the case considered in Stoye (Reference Stoye2009) where the DM cares about expected outcome under $\delta ,$ that is, $\mu _{0}(1-\delta )+\mu _{1}\delta ,$ where $\mu _{t}$ for $t=0,1$ denotes the expectation of $Y_{t}$ under $s.$ Regret for given $\delta $ and $s$ (which only matters via $(\mu _{0},\mu _{1}))$ then equals $\max \{\mu _{0},\mu _{1}\}-(\mu _{0}(1-\delta )+\mu _{1}\delta ).$ When the DM picks a $\delta $ with $\delta \geq 1/2,$ then the maximal regret nature can inflict equals $\delta $ obtained for any $s$ with $E_{s}Y_{0}=1$ and $E_{s}Y_{1}=0.$ Therefore, the DM’s minimax regret choice is $\delta =1/2.$ What explains the different results with quantiles and expectations? What drives the results with quantiles is the discontinuity of the $\alpha $ -quantile of a random variable $X$ with respect to the cdf $F_{X}$ of $X.$ In the above construction, the random variables $Y_{0}$ and $Y_{B(\delta )}$ have cdfs that are uniformly “very close” yet their $\alpha $ -quantiles differ by 1. For example, take again $\delta =1/2$ and consider $\alpha =0.99.$ Then both $Y_{0}$ and $Y_{B(\delta )}$ are Bernoulli with $P(Y_{0}=0)=0.99-0.005=0.985$ and $P(Y_{B(\delta )}=0)=1/2+1/2\cdot 0.985=0.9925.$ Thus, their cdfs are almost identical but their $0.99$ -quantiles differ by 1. Instead the expectations of these two random variables are very close.

Surprisingly, the intuition of the no-data example generalizes to cases with arbitrary sample size $N>0.$ Again, for any given treatment rule $\delta \in \mathbb {D}$ , one constructs an $s_{\delta }\in \mathbb {S}$ for which $\max \{q_{s_{\delta },\alpha }(Y_{0}),q_{s_{\delta },\alpha }(Y_{1})\}=1$ and $u(\delta ,s_{\delta })=0$ and thus $R(\delta ,s_{\delta })=1.$ Again $s_{\delta }\in \mathbb {S}$ can be chosen such that $Y_{0}$ and $Y_{1}$ are independent and both have Bernoulli distributions. That is, $s_{\delta }$ is then fully described by the two parameters $P_{s_{\delta }}(Y_{0}=0)$ and $P_{s_{\delta } }(Y_{1}=0).$ Denote by $0^{N_{0}}$ an $N_{0}$ -dimensional column vector of zeros.

In case (i) of Proposition 1, if $\delta $ is such that $\delta ((0^{N_{0}\prime },v^{\prime })^{\prime })=1$ for all $v\in \{0,1\}^{N_{1}}$ , then nature attempts punishing the DM for always using treatment 1 when no successes are observed for treatment 0, by choosing $P_{s_{\delta }}(Y_{1}=0)=1$ and by choosing $P_{s_{\delta }}(Y_{0}=0)$ “big enough” that only zeros are observed for the untreated individuals “sufficiently often” guaranteeing $Y_{B(\delta (w))}$ has $\alpha $ -quantile 0, but “small enough” that $Y_{0}$ has $\alpha $ -quantile 1. The proof shows that this can indeed be achieved by picking $P_{s_{\delta }}(Y_{0}=0)=\alpha -\varepsilon $ for some small enough $\varepsilon>0.$

If on the other hand $\delta $ is such that $\delta ((0^{N_{0}\prime },v^{\prime })^{\prime })<1$ for at least one $v\in \{0,1\}^{N_{1}}$ , then nature attempts punishing the DM for not using treatment 1 often enough when no successes are observed for treatment 0, by choosing $P_{s_{\delta }}(Y_{0}=0)=1$ and by choosing $P_{s_{\delta }}(Y_{1}=0)$ “small enough” that $Y_{1}$ has $\alpha $ -quantile 1 but “big enough” that $Y_{B(\delta (w))}$ has $\alpha $ -quantile 0. The proof shows that this can indeed be achieved by picking $P_{s_{\delta }}(Y_{1}=0)=\alpha -\varepsilon $ for some small enough $\varepsilon>0.$

A surprising fact about the proof is that exploiting properties of $\delta $ on the set of samples $\{(0^{N_{0}\prime },v^{\prime })^{\prime },\, v\in \{0,1\}^{N_{1}}\}$ alone gives nature enough leverage to inflict maximal regret.

Next, we provide additional insights for this fact. The proof of Proposition 1(i) reveals that for any $\delta \in \mathbb {D,}$ one can find $s_{\delta }\in \mathbb {S}$ with $Y_{0}$ and $Y_{1}$ independent Bernoulli distributions such that $R(\delta ,s_{\delta })=1.$ In fact, the proof reveals that for any treatment rule $\delta $ for which

(2.20) $$ \begin{align} \delta^{0,j}:= {\textstyle\sum\nolimits_{v\in\{0,1\}^{N_{1}},\text{ }\overline{v}=j}} \delta((0^{N_{0}\prime},v^{\prime})^{\prime})<\binom{N_{1}}{j} \end{align} $$

for at least one $j=0,\ldots ,N_{1}$ (where $\overline {v}$ denotes the number of observations in the sample for which $Y_{1}$ equals 1), one can achieve $R(\delta ,s_{\delta })=1$ when $Y_{0}$ and $Y_{1}$ are independent Bernoulli distributions with $Y_{0}=0$ with probability 1. For such a treatment rule $\delta $ , we now discuss regret of $\delta $ as a function of $a\in \lbrack 0,1]$ when nature picks $s_{a}\, \in \mathbb {S}$ and under $s_{a},\, Y_{0}=0$ with probability 1 and $Y_{1}$ is Bernoulli with $P_{s_{a}}(Y_{1}=0)=a.$ We will focus on the case $\alpha =1/2,\, r=0,$ and on two particular treatment rules, namely, $\delta ^{0.5}\equiv 1/2$ (the treatment rule that always picks 1/2 independent of the data $w_{N}$ ) and the empirical success rule $\delta ^{ES},$ defined by

(2.21) $$ \begin{align} \delta^{ES}(w_{N})=I(q_{\alpha}(Y_{1},w_{N})>q_{\alpha}(Y_{0},w_{N} ))+.5I(q_{\alpha}(Y_{1},w_{N})=q_{\alpha}(Y_{0},w_{N})), \end{align} $$

where $q_{\alpha }(Y_{j},w_{N})$ denotes the $\alpha $ -sample quantile of $Y_{j}$ for the sample $w_{N}$ for $j=0,1$ and the $\alpha $ -sample quantile is defined with $r=0$ in (2.5). We will take $N_{0}>0$ and $N_{1}$ even. When $Y_{0}=0$ with probability 1, it follows that $q_{\alpha }(Y_{0},w_{N})=0$ with probability 1 and $q_{\alpha }(Y_{1},w_{N})>0$ occurs if strictly more than $N_{1}/2$ observations on $Y_{1}$ in the sample equal 1. (In that case, $q_{\alpha }(Y_{1},w_{N})=1$ ; otherwise, $q_{\alpha }(Y_{1},w_{N})=0.)$ Note that $\delta ^{0,j}\mathbb {\ }$ in (2.20) equals $0.5\binom {N_{1}}{j}$ for $\delta =\delta ^{0.5};$ it equals $0.5\binom {N_{1}}{j}$ if $j\leq N_{1}/2$ and $\binom {N_{1}}{j}$ if $j>N_{1}/2$ for $\delta =\delta ^{ES}.$

From simple calculations in (A.6) in the Appendix, it follows that, for any $\delta \in \mathbb {D,}$

(2.22) $$ \begin{align} P_{s_{a}}(Y_{B(\delta(w_{N}))}=0)= {\textstyle\sum\nolimits_{j=0}^{N_{1}}} [a^{N_{1}-j}(1-a)^{j}(a-1)]\delta^{0,j}+1. \end{align} $$

Note that $q_{s_{a},\alpha }(Y_{B(\delta (w_{N}))})=0$ if $P_{s_{a}} (Y_{B(\delta (w_{N}))}=0)\geq 0.5$ and $q_{s_{a},\alpha }(Y_{B(\delta (w_{N}))})=1$ otherwise. It follows that regret equals 1 if $a<1/2$ and $P_{s_{a} }(Y_{B(\delta (w_{N}))}=0)\geq 0.5$ and regret equals 0 otherwise.

For $\delta =\delta ^{0.5}$ , it follows that $P_{s_{a}}(Y_{B(\delta (w_{N} ))}=0)=0.5a+0.5.$ Therefore, regret equals 1 if $a<1/2$ and 0 otherwise. For $\delta =\delta ^{ES}$ , we can evaluate $P_{s_{a}}(Y_{B(\delta (w_{N}))}=0)$ given the formulas above. The results do not depend on $N_{0}>0.$ We find that for a given $N_{1}$ regret equals 1 if $a\in \lbrack l(N_{1}),0.5)$ and 0 otherwise, for an $l(N_{1})<$ 0.5. For example, if $N_{1}=10,\, 20,\, 30,\, 40,$ and $50,\, l(N_{1})=0.394,\, 0.416,\, 0.426,\, 0.433,$ and $0.437,$ respectively.

3. In part (iii) of Proposition 1, in the case $q_{\alpha }(Y_{0})<1/2$ , we could not rule out that there are other minimax regret rules besides $\delta ^{1}\equiv 1$ when $N>0.$ When $N=0,$ it is obvious that $\delta ^{1}$ is the unique minimax rule. A minimax regret treatment rule, if it is not unique, may be inadmissible. In the case of “testing an innovation,” $\delta ^{0}\equiv 0$ (when $q_{\alpha }(Y_{0})>1/2)$ is admissible, but we have not determined whether that is true for $\delta ^{1}$ (when $q_{\alpha }(Y_{0})<1/2)$ and for $\delta ^{0.5}\equiv 1/2$ (when $q_{\alpha }(Y_{0})=1/2).$

4. In case $Y_{0},Y_{1}\in (0,1)$ rather than $Y_{0},Y_{1}\in \lbrack 0,1],\, \max _{s\in \mathbb {S}}R(\delta ,s)$ does not generally exist when $\mathbb {S}$ denotes the set of all joint probability distributions over the potential outcomes for $Y_{0},Y_{1}\in (0,1)$ . For example, in cases (i) and (ii), nature choosing only distributions with support on $[\varepsilon ,1-\varepsilon ]$ for some small $\varepsilon>0,$ it can generate regret of $1-2\varepsilon $ and therefore $\sup _{s\in \mathbb {S}}R(\delta ,s)=1$ for the unrestricted space of distributions for $Y_{0},Y_{1}\in (0,1).$ This result can be proven using the exact same proof technique as for Proposition 1. Therefore, if minimax rules are defined with $\sup _{s\in \mathbb {S}}R(\delta ,s)$ (as we do in (2.7)) rather than $\max _{s\in \mathbb {S}}R(\delta ,s),$ the results in Proposition 1(i) and (ii) continue to hold.

5. If $Y_{0},Y_{1}\in \lbrack 0,\infty )$ rather than $Y_{0},Y_{1}\in \lbrack 0,1],$ then $\max _{s\in \mathbb {S}}R(\delta ,s)$ does not typically exist when $\mathbb {S}$ denotes the set of all joint probability distributions over the potential outcomes for $Y_{0},Y_{1}\in \lbrack 0,\infty )$ . For example, in cases (i) and (ii), nature choosing only distributions with support on $[0,M]$ for $M>0,$ it can generate regret of $M$ and therefore $\sup _{s\in \mathbb {S}} R(\delta ,s)=\infty $ when $\mathbb {S}$ denotes the set of all joint distributions for $Y_{0},Y_{1}\in \lbrack 0,\infty ).$ This result can be proven using the exact same proof technique as for Proposition 1. Therefore, if minimax rules are defined with $\sup _{s\in \mathbb {S}}R(\delta ,s)$ (as we do in (2.7)) rather than $\max _{s\in \mathbb {S}}R(\delta ,s),$ then for $\alpha \in (0,1),$ any treatment rule is minimax regret optimal in cases (i) and (ii).

Given the limit experiment results in Hirano and Porter (Reference Hirano and Porter2009), it would also be interesting to study the case where potential outcomes are restricted to be normally distributed. However, in that case, one would need to employ other proof techniques than the ones used in the current article.

Restrictions on nature’s action space: In what follows, we impose various restrictions on nature’s action space and study the implications on the results obtained in Proposition 1. For simplicity, assume $\alpha \in (0,1).$

Comments. 1. Corollary 1(a) is a direct corollary from the proof of Proposition 1. In the proof of Proposition 1(i) and (ii), only Bernoulli distributions are used for nature, while in part (iii), only discrete distributions are used.

2. Corollary 1(b) considers the case where nature is restricted to continuous distributions. We have seen in Proposition 1 that in cases (i) and (ii) any treatment rule $\delta \in \mathbb {D}$ is minimax and $\max _{s\in \mathbb {S}}R(\delta ,s)=1.$ Because without point masses, it is impossible for a random variable to have $\alpha $ -quantile equal to 0, it follows that $R(\delta ,s)$ is always strictly smaller than 1 for any pair $(\delta ,s)$ when $s$ is continuous with respect to Lebesgue measure. The main construction in the proof of Proposition 1 still goes through when one considers a sequence of continuous distributions that converge to the Bernoulli distributions that are used in that proof. As a technical detail, it is important to use $\sup _{s\in \mathbb {S}}R(\delta ,s)$ rather than $\max _{s\in \mathbb {S}}R(\delta ,s)$ in the definition of regret, because $\max _{s\in \mathbb {S}}R(\delta ,s)$ would not exist in the case considered here. One can establish that $\sup _{s\in \mathbb {S}}R(\delta ,s)=1$ for any treatment rule $\delta \in \mathbb {D}\mathfrak {.}$

3. In the case of “testing an innovation,” a restriction on nature’s action space occurs if one assumes that the entire distribution of $Y_{0}$ is known, not just its $\alpha $ -quantile. Namely, in that case, the set $\mathbb {S}$ consists of all possible distributions $s$ for $Y_{1}\in \lbrack 0,1]$ (while the distribution of $Y_{0}$ is given). The analysis of that problem is more difficult. Denote by $F_{Y_{0}}$ and $q_{\alpha }(Y_{0})$ the cdf and $\alpha $ -quantile of $Y_{0},$ respectively.

Take $r=0$ in (2.5) and assume $F_{Y_{0} }(q_{\alpha }(Y_{0}))>\alpha $ . Consider the case $N=0$ in which case a treatment rule $\delta \in \lbrack 0,1]$ denotes the treatment probability. Under these assumptions, the following statements hold.

Denote by $q(\delta )$ the smallest $q\in \lbrack 0,q_{\alpha }(Y_{0})]$ such that

(2.23) $$ \begin{align} \delta+(1-\delta)F_{Y_{0}}(q)\geq\alpha. \end{align} $$

Then, when $q_{\alpha }(Y_{0})<1/2,\, \delta ^{1}\equiv 1$ is the only minimax regret rule and $\max _{s\in \mathbb {S}}R(\delta ^{1},s)=q_{\alpha }(Y_{0}).$ When $q_{\alpha }(Y_{0})=1/2,$ any $\delta \in \lbrack 0,1]$ is minimax regret with maximal regret equal to $q_{\alpha }(Y_{0}).$ Finally, when $q_{\alpha } (Y_{0})>1/2,\, \delta \in \lbrack 0,1]$ is minimax regret iff

(2.24) $$ \begin{align} q_{\alpha}(Y_{0})-q(\delta)\leq1-q_{\alpha}(Y_{0}). \end{align} $$

Given that $q_{\alpha }(Y_{0})-q(\delta )$ is a weakly increasing function in $\delta ,$ if we denote by $\delta ^{\ast }\in \lbrack 0,1]$ an intersection of $q_{\alpha }(Y_{0})-q(\delta )$ and $1-q_{\alpha }(Y_{0}),$ then any $\delta \in \lbrack 0,\delta ^{\ast }]$ is minimax regret and for those $\delta $ we have $\max _{s\in \mathbb {S}}R(\delta ,s)=1-q_{\alpha }(Y_{0})$ .

In the Appendix, we give a proof of the statements made above. The results for the case $q_{\alpha }(Y_{0})>1/2$ are partly in contrast to Proposition 1(iii) where there is a unique minimax regret rule. However, maximal regret is the same here as in Proposition 1(iii). We have not yet generalized the results to arbitrary sample sizes $N>0$ and to the case $F_{Y_{0}}(q_{\alpha }(Y_{0}))=\alpha .$

2.3 Finite Sample Simulation

For the case of “testing an innovation,” Proposition 1(iii) establishes that the minimax regret criterion when applied to $\alpha $ -quantiles does not favor data-driven rules. In this section, we conduct a simulation experiment to juxtapose the pointwise (in $s$ ) regret of the data-driven empirical success rule $\delta ^{ES},$ defined (in the case of “testing an innovation”) by

(2.25) $$ \begin{align} \delta^{ES}(w)=I(q_{\alpha}(Y_{1},w)>q_{\alpha}(Y_{0}))+.5I(q_{\alpha} (Y_{1},w)=q_{\alpha}(Y_{0})), \end{align} $$

where $q_{\alpha }(Y_{1},w)$ denotes the $\alpha $ -sample quantile of $Y_{1}$ for the sample $w,$ with the regret of the minimax regret rule $\delta ^{1}\equiv 1$ (in the case when $q_{\alpha }(Y_{0})\leq 1/2),\, \delta ^{0.5} \equiv 1/2$ (in the case, when $q_{\alpha }(Y_{0})=1/2),$ and the minimax regret rule $\delta ^{0}\equiv 0$ (in the case, when $q_{\alpha }(Y_{0})\geq 1/2)$ .

Obviously, when simulating regret we cannot possibly include all states of nature $s\in \mathbb {S}$ . For the simulation experiment, we create a “sufficiently rich” subset of distributions for $(Y_{0},Y_{1}).$ Namely, for some $n,w\in \mathbb {N}$ , we consider the set of states of nature ${\mathbb {S}}^{E}={\mathbb {S}}^{E}(n,w)$ that consists of all discrete distributions $s$ for $Y_{1}$ supported on a grid

(2.26) $$ \begin{align} \{0,1/n,2/n,\ldots,n/n\} \end{align} $$

with probabilities

(2.27) $$ \begin{align} P_{s}(Y_{1}=j/n) \end{align} $$

being of the form $i/w,\, i=0,\ldots ,w;$ while for the distribution of $Y_{0}$ , we consider two choices, namely,

I) $Y_{0}\in \{0,q_{\alpha }(Y_{0})\}$ with $P(Y_{0}=0)=\alpha -\varepsilon $ and $P(Y_{0}=q_{\alpha }(Y_{0}))=1-(\alpha -\varepsilon )$ for $\varepsilon =0.000001,$ and

II) $Y_{0}$ being continuously distributed on [0,1] with density $f(x)$ equal to $\alpha /q_{\alpha }(Y_{0})$ for $x\leq q_{\alpha }(Y_{0})$ and equal to $(1-\alpha )/(1-q_{\alpha }(Y_{0}))$ otherwise. Note that, in both I) and II), the $\alpha $ -quantile of $Y_{0},\, q_{s,\alpha }(Y_{0}),$ equals $q_{\alpha } (Y_{0}).$ We report results for all choices of $\alpha \in \{0.1,0.5,0.9\}$ , $q_{\alpha }(Y_{0})\in \{0.1,0.5,0.9\},$ sample size $N=30,$ and $(n,w)=(6,12).$ The latter leads to $S^{E}$ having cardinality 18,564.

For the given choices of $n,w,$ and $N$ and for each choice of $\alpha $ and $q_{\alpha }(Y_{0})$ , for each state of nature $s\in {\mathbb {S}}^{E}(n,w)$ and one (of the two possible) choice of distribution for $Y_{0}$ we simulate regret for the four treatment rules $\delta ^{ES},\, \delta ^{1},\, \delta ^{0.5},$ and $\delta ^{0}$ by generating $R=100K$ samples of size $N$ by drawing i.i.d. observations of the distribution of $Y_{1}.$ We use $r=0$ when simulating $\alpha $ -quantiles of the outcome distribution under the various treatment rules. We analytically calculate $\alpha $ -quantiles for $Y_{1}$ for $Y_{B(\delta ^{0.5})}$ , and likewise use the true $\alpha $ -quantile $q_{\alpha }(Y_{0})$ of $Y_{0}$ when calculating regret. For a given treatment rule $\delta $ and a given state of nature $s$ , regret is calculated as $R(\delta ,s)=\max \{q_{\alpha }(Y_{0}),q_{s,\alpha }(Y_{1})\}-q_{s,\alpha }(Y_{B(\delta (w))}).$

We compare the treatment rules along several dimensions, namely,

a) mean regret over all 18,564 states of nature $s\in {\mathbb {S}}^{E}(n,w)$ ,

b) maximal regret over all states of nature $s\in {\mathbb {S}}^{E}(n,w)$ ,

c) minimal regret over all states of nature $s\in {\mathbb {S}}^{E}(n,w)$ , and

d) the proportion of $s\in {\mathbb {S}}^{E}(n,w)$ for which regret for the empirical success rule $\delta ^{ES}$ is smaller than regret for each one of its three competitors.

Just for clarity, in a), for each treatment rule, we sum up its regret over all 18,564 states of nature $s\in {\mathbb {S}}^{E}(n,w)$ and then report that sum divided by 18,564. For a given $s\in {\mathbb {S}}^{E}(n,w)$ , in our simulations, we interpret regret of $\delta ^{ES}$ as smaller than regret of another rule, say $\delta ^{0},$ if the simulated regret of $\delta ^{ES}$ is smaller than the simulated regret of the rule $\delta ^{0}$ minus a threshold of $\xi .$ If instead the simulated regret of $\delta ^{ES}$ falls into the interval [(simulated regret of $\delta ^{0})-\xi ,$ (simulated regret of $\delta ^{0})+\xi $ ], we record regrets of the two rules as equal for that state of nature. We take $\xi $ = 0.00000001 below. Similarly, when programming the empirical success rule in (2.25), the event “ $q_{\alpha }(Y_{1},w)=q_{\alpha }(Y_{0})$ ” is implemented as the simulated $\alpha $ -quantile of $Y_{1}$ falling into the interval $[q_{\alpha }(Y_{0})-\xi ,q_{\alpha } (Y_{0})+\xi ].$

All reported results are rounded to the second digit after the comma and so a reported zero could be as large as 0.004.

In Tables 1 and 2, we do not include results for c) because those results turn out to be equal to zero for all treatment rules and all designs except for $\delta ^{0.5}$ when $q_{\alpha }(Y_{0})\in \{0.1,0.9\}$ in which case minimal regret equals 0.067 for all choices of $\alpha $ and both choices of distributions for $Y_{0}.$

Table 1

Maximal and mean regret over all 18,564 states of nature $s \in {S}^{E}(6,12)$ for four different treatment rules.

Table 1 Long description

The table is divided into two primary sections: Case I and Case II. Each section contains 12 columns representing four treatment rules (delta super E S, delta super 1, delta super 0.5, and delta super 0) repeated for three values of alpha (0.1, 0.5, and 0.9).

* Case I:

- For alpha = 0.1 and q sub alpha (Y sub 0) = 0.1: Max regret is 0.9 for delta super E S, 0.1 for delta super 1, and 0.9 for delta super 0.5 and delta super 0. Mean regret is 0.08, 0.04, 0.14, and 0.1 respectively.

- For alpha = 0.1 and q sub alpha (Y sub 0) = 0.5: Max regret is 0.5 for all rules. Mean regret ranges from 0 to 0.43.

- For alpha = 0.1 and q sub alpha (Y sub 0) = 0.9: Max regret is 0.9 for the first three rules and 0.1 for delta super 0. Mean regret is 0.9, 0.75, 0.75, and 0.

- For alpha = 0.5 and q sub alpha (Y sub 0) = 0.1: Max regret is 0.1 for delta super E S and delta super 1, and 0.9 for delta super 0.5 and delta super 0.

- For alpha = 0.9 and q sub alpha (Y sub 0) = 0.9: delta super 0 shows the lowest max regret at 0.1 and mean regret at 0.04.

* Case II:

- This section follows the same structure as Case I with slight variations in values.

- For alpha = 0.9 and q sub alpha (Y sub 0) = 0.1: Max regret is 0 for delta super E S and 0.1 for delta super 1.

- For alpha = 0.9 and q sub alpha (Y sub 0) = 0.5: Max regret is 0.02 for delta super E S and 0.5 for the others.

- For alpha = 0.9 and q sub alpha (Y sub 0) = 0.9: Max regret is 0.03 for delta super E S and 0.1 for delta super 0.

Bolded values indicate the lowest regret achieved for specific parameters.

Table 2

Proportion (in %) of states $s\in S^{E}(6,12)$ for which regret for the empirical success rule ${\delta }^{ES}$ is smaller than regret of non-data rules $\delta \in \{{\delta }^{1},\delta ^{0.5},{\delta }^{0}\}$ .

Table 2 Long description

The table presents data for Case I and Case II across ten columns. The first column lists the parameters alpha and the quantile function q sub alpha of Y sub 0. The remaining nine columns are grouped into three sets of three, corresponding to alpha equals 0.1, 0.5, and 0.9. Each set contains sub-columns for delta super 1, delta super 0.5, and delta super 0.

* Case I Data:

- For q sub alpha of Y sub 0 equals 0.1: Prop of R of delta super E S, s less than R of delta, s ranges from 0 to 100. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 56.9 to 100.

- For q sub alpha of Y sub 0 equals 0.5: Prop of R of delta super E S, s less than R of delta, s ranges from 0 to 91.7. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 8.3 to 99.5.

- For q sub alpha of Y sub 0 equals 0.9: Prop of R of delta super E S, s less than R of delta, s ranges from 0 to 43.1. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 0 to 66.7.

* Case II Data:

- For q sub alpha of Y sub 0 equals 0.1: Prop of R of delta super E S, s less than R of delta, s ranges from 0 to 100. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 73.0 to 100.

- For q sub alpha of Y sub 0 equals 0.5: Prop of R of delta super E S, s less than R of delta, s ranges from 2.2 to 91.7. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 65.3 to 100.

- For q sub alpha of Y sub 0 equals 0.9: Prop of R of delta super E S, s less than R of delta, s ranges from 43.1 to 99.6. Prop of R of delta super E S, s less than or equal to R of delta, s ranges from 43.1 to 100.

Table 1 provides results for maximal and mean regret over all 18,564 states of nature $s\in S^{E}(6,12)$ for the four different treatment rules. Results for rules that are minimax regret in a given setting are reported in bold.

We first discuss results for maximal regret. The treatment rules that are known to be minimax regret for unrestricted $\mathbb {S}$ also have smallest maximal regret over ${\mathbb {S}}^{E}(n,w)$ (relative to the other treatment rules considered here) for all cases except for case II) with $\alpha $ = 0.9, when $q_{\alpha }(Y_{0})$ = 0.1 (in which case, $\delta ^{ES}$ has smaller maximal regret than the optimal rule $\delta ^{1})$ , when $q_{\alpha }(Y_{0})$ = 0.5 (in which case, it is known that all four rules are optimal but in finite samples for ${\mathbb {S}}^{E}(n,w)$ again $\delta ^{ES}$ does best), and finally when $q_{\alpha }(Y_{0})$ = 0.9 (in which case, $\delta ^{ES}$ has smaller maximal regret than the optimal rule $\delta ^{0})$ . The explanation is of course that ${\mathbb {S}}^{E}(n,w)$ does not contain those states of nature that would inflict the highest regret on $\delta ^{ES}$ (like the particular Bernoulli and discrete distributions for $Y_{0}$ and $Y_{1},$ respectively, that are used in the proof of Proposition 1(iii)). Furthermore, reported finite sample maximal regret over ${\mathbb {S}}^{E}(n,w)$ for the optimal rules matches the theoretical value $\min \{q_{\alpha } (Y_{0}),1-q_{\alpha }(Y_{0})\}$ reported in Proposition 1(iii) except for case II) with $\alpha $ = 0.9 when $q_{\alpha }(Y_{0})$ = 0.5 where the optimal rule $\delta ^{ES}$ has maximal regret smaller than 0.5 (which occurs, again, because ${\mathbb {S}}^{E}(n,w)$ is not rich enough). An open question from Proposition 1(iii) is whether for $q_{\alpha }(Y_{0})\, <$ 0.5 other minimax regret rules besides $\delta ^{1}$ might exist. The simulations for $q_{\alpha }(Y_{0})$ = 0.1 are compatible with the possibility that when $\alpha $ = 0.5 or 0.9 also $\delta ^{ES}$ might be minimax regret.

We next discuss results for mean regret. In most cases where $q_{\alpha } (Y_{0})\in \{0.1,0.9\}$ in which case either $\delta ^{0}$ or $\delta ^{1}$ are minimax regret, their mean performance is also best (or very close to best) among the four treatment rules. On the other hand, when $q_{\alpha }(Y_{0})$ = 0.5 (in which case all four treatment rules are minimax regret according to Proposition 1(iii)) we see huge differences in mean performance across the four treatment rules for a given case I) or II) and $\alpha ,$ but also huge differences in performance for a given treatment rule and $\alpha $ across cases I) and II). With respect to the former point, in case I), when $\alpha $ = 0.1, mean regret for the four rules is in the interval [0,0.43]. With regards to the latter point, for example, for $\delta ^{ES}$ , when $\alpha $ = 0.1, mean regret equals 0.43 and 0.06, in cases I) and II), respectively. Differences across cases I) and II) are also often huge for other quantiles. For example, again for $\delta ^{ES}$ , when $q_{\alpha }(Y_{0})$ = 0.9 and $\alpha $ = 0.1, mean regret equals 0.9 and 0, in cases I) and II), respectively. (Recall that we round results to the second digit. When $q_{\alpha }(Y_{0})$ = 0.9 and $\alpha $ = 0.1, there are not many distributions for $Y_{1}$ in $\mathbb {S} ^{E}(n,w)$ that have an $\alpha $ -quantile that exceeds 0.9.)

We next discuss the results for exercise d) contained in Table 2 where we report the proportion of the 18,564 states of nature $s\in S^{E}(6,12)$ for which the regret of $\delta ^{ES}$ is smaller than the regret of $\delta $ where we consider all $\delta $ from the set of no-data rules $\{\delta ^{1},\delta ^{.5},\delta ^{0}\}$ . The results indicate that for many states $s$ , choices of $q_{\alpha }(Y_{0})$ , $\alpha ,$ and the distribution for $Y_{0},\, R(\delta ^{ES},s)$ , and $R(\delta ,s)$ are very close which leads to numerically unstable results. To deal with the instability, we introduce the buffer $\xi $ as explained above and $Prop(R(\delta ^{ES},s)<R(\delta ,s))$ and $Prop(R(\delta ^{ES},s)\leq R(\delta ,s))$ in Table 2 represent the proportion of states $s$ for which $R(\delta ^{ES},s)<R(\delta ,s)-\xi $ and $R(\delta ^{ES},s)<R(\delta ,s)+\xi ,$ respectively. Those two proportions can be drastically different, suggesting that, in many scenarios, regrets for $\delta ^{ES}$ and $\delta $ are (virtually) identical. According to the measure $Prop(R(\delta ^{ES},s)\leq R(\delta ,s)),$ maybe somewhat surprisingly given the results from Table 1, $\delta ^{ES}$ is to be preferred over the other three rules in all scenarios considered in Case II) (except the case $\alpha =q_{\alpha }(Y_{0})$ = 0.9 for $\delta ^{0})$ and in the majority of scenarios in Case I) (except compared to $\delta ^{0}$ when $q_{\alpha }(Y_{0})$ = 0.5, $\alpha $ = 0.1 and except for most cases with $q_{\alpha }(Y_{0})$ = 0.9 and all three rules $).$ Quite often $Prop(R(\delta ^{ES},s)\leq R(\delta ,s))$ is reported as higher than 95%, but the improvement in regret for many states of nature is minuscule. For example, compared to $\delta ^{0.5}$ in Case I) with $\alpha =q_{\alpha }(Y_{0})$ = 0.1 only in 33.3% of the cases $(R(\delta ^{ES},s)<R(\delta ,s))$ while for 100% of the cases $(R(\delta ^{ES},s)\leq R(\delta ,s))$ implying that the regret of $\delta ^{ES}$ in 66.7% of the cases is at most $\xi $ smaller than the regret of $\delta ^{0.5}$ .