4.1 Instrumental Variables

Angrist et al. (Reference Angrist, Imbens and Rubin1996) introduced the IV estimator for the LATE. Specifically, the IV estimator takes the following form:

$$ \begin{align*}\frac{\widehat{ITT}}{\hat{{\mathbb{E}}}\left(T_i|Z_i=1 \right) - \hat{{\mathbb{E}}}\left(T_i|Z_i=0 \right)}, \end{align*} $$

where $\hat {{\mathbb {E}}}\left (\cdot \right )$ again indicates the sample mean for the specified quantity. Intuitively, this estimator works by taking the estimated effect of assignment to treatment on the outcome and then dividing it by the estimated effect of assignment to treatment on the treatment received. Angrist et al. (Reference Angrist, Imbens and Rubin1996) show that, under the assumptions discussed in the remainder of this section, the the numerator will converge to the LATE multiplied by the fraction of compliers while the denominator will converge to the fraction of compliers. Consequently, the IV estimator will be a consistent for the LATE if several assumptions are deemed plausible.

The first assumption needed for the consistency of the IV estimator is random assignment independent of co-variates.

Assumption 1. Random assignment independent of covariates: Fraction $p \in (0,1)$ of the sample is randomly assigned to treatment ( $Z_i=1$ ) such that $\forall i \, \in \{1\dots N\}$ ,

This assumption requires that the assignment to treatment be independent of the potential outcomes of the treatment received variable and the outcomes. It is standard in the context of completely randomized experiments and will be maintained for the CPW estimator as well.

The next assumption requires that the average effect of assignment to treatment on the treatment received is non-zero.

Assumption 2. Relevance: $\forall i \, \in \{1\dots N\}$ ,

$$ \begin{align*}{\mathbb{E}}\left[T_i(1) - T_i(0) \right] ={\mathbb{E}}\left(\gamma_i\right) \neq 0. \end{align*} $$

Because the treatment received and treatment assigned are both fully observed by the researcher, Assumption 2 is empirically testable under Assumption 1 (Andrews, Stock, and Sun Reference Andrews, Stock and Sun2019).

The next assumption required for the IV estimator is monotonicity. This assumption requires that $T_i(z)$ is increasing with z.

Assumption 3. Monotonicity: $\forall i \, \in \{1\dots N\}$ ,

$$ \begin{align*}T_i(1) - T_i(0) = \gamma_i \geq 0. \end{align*} $$

This assumption is also referred to as the no defiers assumption, since it requires that there be no subjects who receive the opposite of the treatment they are assigned to.

The final assumption needed for the IV estimator is the exclusion restriction, which requires that assignment to treatment only impact the outcome through the treatment actually received.

Assumption 4. The Exclusion Restriction: $\forall i \in \{1\dots N\}$ and $t \in \{0,1\}$ ,

$$ \begin{align*}Y_i(1, t)=Y_i(0,t). \end{align*} $$

Although frequently utilized for analyzing both experiments and quasi-experiments with non-compliance, the assumptions required for the IV estimator are often questioned (Lal et al. Reference Lal, Lockhart, Xu and Zu2023), suggesting there is a need for methods that can estimate the LATE without invoking these assumptions.

4.2 Principal Score Methods

Principal score methods in essence swap the exclusion restriction and monotonicity for a selection on observables strategy. Specifically, they posit that the differences in average outcomes between the different compliance strata can be explained by a set of observed covariates. Following the lead of Feller et al. (Reference Feller, Mealli and Miratrix2017), I present two forms of this principal ignorability assumption. The stronger version requires that, conditional on $X_i$ , average potential outcomes be the same across covariate strata.

Assumption 5. Strong Principal Ignorability: $\forall i \in \{1\dots N\}$ ,

$$ \begin{align*}{\mathbb{E}}\left[Y_i(1, T_i)|X_i=x, \gamma_i = 1 \right] = {\mathbb{E}}\left[Y_i(1, T_i)|X_i=x, \gamma_i = 0 \right]= {\mathbb{E}}\left[Y_i(1, T_i)|X_i=x, \gamma_i = -1 \right] \end{align*} $$

and

$$ \begin{align*}{\mathbb{E}}\left[Y_i(0, T_i)|X_i=x, \gamma_i = 1 \right] = {\mathbb{E}}\left[Y_i(0, T_i)|X_i=x, \gamma_i = 0 \right] = {\mathbb{E}}\left[Y_i(0, T_i)|X_i=x, \gamma_i = -1 \right]. \end{align*} $$

In plain English, this assumption requires that whether a subject complies with their assigned treatment is unrelated to the effect that treatment would have on them after adjusting for covariates. This assumptions is quite strong and likely fails in many applied settings, but it can facilitate the point identification of the LATE without invoking the exclusion restriction or monotonicity.

Consequently, researchers have also considered a weaker form of principal ignorability.

Assumption 6. Weak Principal Ignorability: $\forall i \in \{1\dots N\}$ and $t \in \{0,1\}$ ,

$$ \begin{align*}{\mathbb{E}}\left[Y_i(1, 1)|X_i=x, T_i(1) = 1, T_i(0) = t\right] = {\mathbb{E}}\left[Y_i(1, 1)|X_i=x, T_i(1) = 1\right] \end{align*} $$

and

$$ \begin{align*}{\mathbb{E}}\left[Y_i(0, 0)|X_i=x, T_i(0) = 0, T_i(1) = t\right] = {\mathbb{E}}\left[Y_i(0, 0)|X_i=x, T_i(0) = 0\right]. \end{align*} $$

The first equation in Assumption 6 requires that, given $X_i$ , the average outcome for units assigned to treatment is not dependent on the treatment they would have received if they were assigned to the control. Similarly, the second equation requires that, given $X_i$ , the average outcome for units assigned to the treatment is not dependent on the treatment they would have received if they were assigned to the control. On its own, weak principal ignorability is not sufficient to point identify the LATE, and identification can only be achieved along with monotonicity (Feller et al. Reference Feller, Mealli and Miratrix2017).Footnote ⁴ Although more readily plausible than principal ignorability, weak principal ignorability may fail in a number of applied settings.

There are a large number of estimators that identify the LATE under principal ignorability. The CPW estimator is similar to the weighting approach proposed by Stuart and Jo (Reference Stuart and Jo2015) and further developed by Ding and Lu (Reference Ding and Lu2017). Regression (Bein Reference Bein2015; Joffe, Small, and Hsu Reference Joffe, Small and Hsu2007) and matching (Hill et al. Reference Hill, Waldfogel and Brooks-Gunn2002; Jo and Stuart Reference Jo and Stuart2009) estimators also exist. Porcher et al. (Reference Porcher, Leyrat, Baron, Giraudeau and Boutron2016) provides a more thorough review of the various principal scores methods.

5.1 Assumptions

Along with the standard assumptions of SUTVA, random sampling, positivity, and random assignment to treatment discussed in Sections 3 and 4.1, the CPW estimator will require two additional assumptions.

The first is that the researcher has specified a consistent estimator for the probability that each unit is a complier.

Assumption 7.

$$ \begin{align*}\hat{P}\left(\gamma_i=1|X_i\right) \xrightarrow[N \to \infty]{p} P\left(\gamma_i=1|X_i\right) \end{align*} $$

and $\forall x \in \mathcal {X}$ ,

$$ \begin{align*}0 < P\left(\gamma_i=1|X_i=x\right) \leq 1. \end{align*} $$

There are many potential choices for estimators that will satisfy this condition. For example, a properly specified parametric model will be consistent and can be used if the analyst deems the functional form assumptions reasonable. Non-parametric options are also available. For example, if monotonicity is assumed to hold, $\gamma _i$ can only equal 0 or 1, so any estimator for heterogeneity in the effect of assignment to treatment on the treatment received (i.e., ${\mathbb {E}}\left (\gamma _i|X_i\right )$ ) will also provide an estimate for the probability that $\gamma _i =1$ . The simulations and applications use this approach, relying on the estimator for treatment effect heterogeneity provided by Nie and Wager (Reference Nie and Wager2021). In the case of two-sided non-compliance, the estimator provided by Aronow and Carnegie (Reference Aronow and Carnegie2013) could be used instead. Note that the identification of the conditional probability required by Assumption 7 will almost certainly require additional assumptions that will depend on the precise estimator that is being used and that may be deemed strong in a particular application. For example, the estimator provided by Aronow and Carnegie (Reference Aronow and Carnegie2013) rests on parametric functional form assumptions that may not be plausible in all settings. Even non-parametric estimators, like that provided by Nie and Wager (Reference Nie and Wager2021), still typically require that the distribution of the covariates and treatment effects obey some basic regularity conditions. Identification of $P\left (\gamma _i=1|X_i\right )$ will also almost certainly require restrictions on the dimensionality of $X_i$ (typically that it grow at a rate significantly slower than N) and researchers may find LASSO based methods to be particularly useful in settings where $X_i$ is high dimensional (Belloni, Chernozhukov, and Hansen Reference Belloni, Chernozhukov and Hansen2014).

This assumption also implicitly requires that some measure of the treatment received, $T_i$ , be observed, as the probability that each unit is a complier is otherwise inestimable. This requirement is, of course, shared by IV and other principal score methods as well. Indeed, in practice, this requirement is actually likely weaker for the CPW estimator than it is for IV methods, as the exclusion restriction is unlikely to be plausible in cases of partial compliance (say where non-compliers receiver a weaker dose of the treatment rather than no treatment) or if the treatment received is measured with error, as is likely the case if compliance is self-reported in an exit survey. Indeed, such settings would represent excellent use cases for the CPW estimator as they are not inconsistent with any of the assumptions discussed in this section holding.

It is worth emphasizing that Assumption 7 requires that the chosen estimator converge to the conditional probability that a unit is a complier given the chosen covariates, it does not require that those covariates themselves be strong predictors of compliance type. Indeed, if $X_i$ is completely independent of $\gamma _i$ , such that $P(\gamma _i=1|X_i=x)$ is the same for all values of x, the main result about the asymptotic conservatism of the CPW estimator will still hold as long as the chosen estimator for $\hat {P}\left (\gamma _i=1|X_i\right )$ converges to this constant value as well and the other relevant assumptions are satisfied.

The requirement that $P\left (\gamma _i=1|X_i\right )$ always be greater than zero is worth some discussion as well. In essence, this can be interpreted as something like a positivity requirement, as the effect of the receipt of treatment on the outcome is fundamentally unknowable for covariate strata that contain no compliers. Note, that Assumption 3 implies that this condition holds.

The second assumption needed for the main result to hold is that $\tau _i$ be larger for compliers than non-compliers, averaging across covariate strata.

Assumption 8.

$$ \begin{align*}\int_{\mathcal{X}} f_{X_i|\gamma_i}(X_i=x|\gamma_i=1) \left({\mathbb{E}}\left[\tau_i|X_i=x, \gamma_i=1 \right] - {\mathbb{E}}\left[\tau_i|X_i=x \right] \right) dx \geq 0. \end{align*} $$

In essence, this assumption requires that the effect of assignment to treatment be larger for compliers than in the full sample when the distribution of pre-treatment covariates is fixed at that seen for just compliers. A more interpretable condition that is sufficient for fulfilling this assumption is that the effect of assignment to treatment is, on average, larger for compliers than non-compliers in all covariate strata. I consider it likely that this assumption holds in many applied settings, as it is reasonable to assume that, even if the exclusion restriction fails, the actual receipt of the treatment should be associated with a larger effect than just the assignment to treatment alone.

I also consider a stronger version of Assumption 8 which will require that compliers have a larger magnitude treatment effect across all covariate strata and that average treatment effects are consistently signed for all covariate strata as well.

Note, a. implies Assumption 8 while b. would imply a similar relationship, but when treatment effects are negative.

5.2 Comparison with IV Assumptions and Principal Ignorability

A natural question is how the assumptions presented above compare with the IV assumptions and principal ignorability. This subsection provides such a discussion, first, considering the IV assumptions and then proceeding to principal ignorability.

5.2.1 IV Assumptions

While the exclusion restriction does not naturally imply the CPW assumptions, it can be shown that if treatment effects are positive for all covariate strata, Assumption 4 does imply Assumption 8, as stated in the following proposition.

Proposition 1. If for any $x \in \mathcal {X}$ ,

$$ \begin{align*}{\mathbb{E}}\left(\tau_i|\gamma_i=1, X_i=x\right)> 0. \end{align*} $$

then Assumption 4 implies Assumption 8.

Proof in Section A.1 of the Supplementary Material.

So in situations where the treatment effect is positive for all covariate strata, Assumption 8 will be weaker than the exclusion restriction typically invoked for IV models. Of course, researchers are often unwilling to assume away such backfire effects for all covariate strata, and in such cases, the assumptions needed for the IV estimator will not naturally nest those used for the CPW estimator.Footnote ⁵

Nonetheless, there are still many applied settings where the assumptions needed for the CPW estimator will be more plausible than the IV ones. For example, if compliance is measured with error, then the exclusion restriction will almost certainly fail, but the CPW estimator will still be useful. Similarly, if non-compliers receive a weaker version of the treatment, then the exclusion restriction will not hold, but the CPW assumptions may be plausible. This is effectively the case in the Kalla and Broockman (Reference Kalla and Broockman2022) experiment where non-compliers are defined as respondents who were likely exposed the ad, but did not pay careful attention. The question of what conditions will lead the CPW estimator to outperform the IV estimator is discussed in more depth in Section 6, which uses simulations to explore the performance of both estimators under different assumptions about the underlying data generating process.

5.3 The CPW Estimator

Intuition about the approach the CPW estimator takes can be developed using the following decomposition of the difference between the LATE and ITT estimands.

Lemma 1.

$$ \begin{align*} \text{LATE} - \text{ITT} & = \int_{\mathcal{X}} f_{X_i|\gamma_i}(x|\gamma_i=1) \left[{\mathbb{E}}\left(\tau_i|X_i=x, \gamma_i=1 \right) - {\mathbb{E}}\left(\tau_i|X_i=x \right) \right] dx \\ &\quad + \int_{\mathcal{X}} {\mathbb{E}}\left[\tau_i|X_i=x\right] \left[ f_{X_i|\gamma_i} (x|\gamma_i=1) - f_{X_i} (x) \right]dx . \end{align*} $$

Proof in Section A.2 of the Supplementary Material.

Specifically, this decomposition splits the difference between the ITT estimand and the LATE into two portions: the first represents the difference in the effect of assignment to treatment on compliers versus non-compliers while the second represents treatment effect heterogeneity due to the difference in the balance of covariates between compliers and non-compliers. Note that while this difference is helpful for building intuition about the CPW estimator, the ITT estimand is often a major quantity of interest in its own right and this decomposition should not be understood as a representing a form of bias for the ITT estimator.

The CPW estimator works by choosing weights that will eliminate the second term of the difference in Lemma 1. Specifically, I propose weighting by the estimated probability that unit i is a complier given $X_i$ : $\hat {P}(\gamma _i=1|X_i)$ , which will help eliminate the bias due to covariate mismatch and upweight covariate strata that are more likely to include compliers. Formally, this estimator will take the following form:

$$ \begin{align*}\widehat{\text{CPW}} = \sum_{i=1}^N w_i \left(Z_i Y_i - (1 - Z_i) Y_i \right), \end{align*} $$

where

$$ \begin{align*}w_i = \frac{Z_i \hat{P}\left(\gamma_i=1|X_i\right)}{\sum_{j=1}^N Z_i \hat{P}\left(\gamma_j=1|X_j\right)} + \frac{(1 - Z_i) \hat{P}\left(\gamma_i=1|X_i\right)}{\sum_{j=1}^N (1 - Z_i) \hat{P}\left(\gamma_j=1|X_j\right)}. \end{align*} $$

Interval and standard error estimates for the CPW estimator can be generated using bootstrap methods, as is standard in the principal score methods literature (Feller et al. Reference Feller, Mealli and Miratrix2017).

A useful precursor to my main results is to provide a closed form representation for the asymptote of the CPW estimator and the asymptotic difference between the CPW estimator and the LATE.

Lemma 2. Under Assumption 7,

$$ \begin{align*} \widehat{\text{CPW}} \xrightarrow[N \to \infty]{p} \int_{\mathcal{X}} {\mathbb{E}}\left(\tau_i|X_i=x\right) f_{X_i|\gamma_i}(X_i=x|\gamma_i=1)dx. \end{align*} $$

And,

$$ \begin{align*} \text{LATE} - \widehat{\text{CPW}} & \xrightarrow[N \to \infty]{p} \int_{\mathcal{X}} f_{X_i|\gamma_i}(X_i=x|\gamma_i=1) \left[E\left(\tau_i|X_i=x, \gamma_i=1 \right) - {\mathbb{E}}\left(\tau_i |X_i=x \right) \right] dx. \end{align*} $$

Proof in Section A.3 of the Supplementary Material.

The first result in this lemma shows that the CPW estimator will converge to the average of the conditional average effect of assignment to treatment within covariate stratum, weighted by the probability that the member of each stratum is a complier. The second result represents the asymptotic difference between the CPW estimator and the LATE as the average of the difference in the effect of assignment to treatment on the outcome between compliers and the full sample among covariate strata, weighted by the probability that a unit in each stratum is a complier. Note that, in the likely case that covariate strata with more compliers have a smaller gap between ${\mathbb {E}}\left (\tau _i|X_i=x, \gamma _i=1 \right )$ and ${\mathbb {E}}\left (\tau _i |X_i=x \right )$ , the asymptotic bias of the CPW estimator will be falling with the strength of the relationship between the pre-treatment covariates and compliance type.

While there is not an immediate statistic that can be estimated and used to summarize this bias ( ${\mathbb {E}}\left (\tau _i|X_i=x, \gamma _i=1 \right ) - {\mathbb {E}}\left (\tau _i |X_i=x \right )$ cannot be identified without stronger assumptions), researchers will likely still want to consider how well the available covariates predict compliance type. In many cases, theoretical grounds can be used to justify this belief. For example, in the case of the Kalla and Broockman (Reference Kalla and Broockman2022) application, it is quite reasonable to think that general interest in politics will predict how likely a respondent was to be attentive to the ad being aired. It also worth emphasizing that the strength of the relationship between the covariates and compliance type is an empirical question and that the performance of the estimator chosen for $\hat {P}\left (\gamma _i=1|X_i\right )$ can often be summarized with more general measures of goodness of fit.

The representation of the asymptotic bias of the CPW estimator provided in Lemma 2 also immediately leads to my main result when paired with Assumption 8.

Proposition 2. Under Assumptions 1, 7, and 8,

$$ \begin{align*}\lim_{N\to \infty} P(\text{LATE} \geq \widehat{\text{CPW}}) = 1. \end{align*} $$

The proof follows immediately from Assumption 8 and Lemma 2

The interpretation of this proposition will depend on the sign that is assumed for the LATE. If the LATE is negative, then the CPW estimator will still be more negative than the LATE. Since the coding of treatment and control groups as 1 and 0 is arbitrary, it can be assumed, without loss of generality, that the LATE is positive. However, this may be an unsatisfying result as it does not eliminate the possibility that the CPW estimator converge to a large negative value while the LATE remains positive.

A stronger conclusion then is presented below, which uses Assumption 9 to deliver a result about the magnitude of the LATE and CPW estimator.

Proposition 3. Under Assumptions 1, 7, and 9,

$$ \begin{align*}\lim_{N\to \infty} P(|\text{LATE}| \geq |\widehat{\text{CPW}}|) = 1. \end{align*} $$

Proof in Section A.4 of the Supplementary Material.

Note that Assumption 9 places stronger limitations on the heterogeneity in treatment effects between covariate strata than Assumption 8, so delivering this stronger conclusion does require stronger assumptions about the nature of the data generating process.