2.1 The Setup

Consider a randomized experiment with sample size N and each unit is indexed by $i \in \{1,2, \ldots , N\}$ . Define a treatment assignment vector $\mathbf {T} = (T_1, \ldots , T_N)^\top $ where binary treatment variable $T_i \in \{0, 1\}$ denotes the treatment received by unit i. For example, $T_i = 1$ would mean that unit i receives an informational message, and $T_i = 0$ would mean unit i is in a control group who does not receive any message. Based on the experimental design, treatment assignment probability $\Pr (\mathbf {T}=\mathbf {t})$ is known for all $\mathbf {t} \in \{0,1\}^N$ . Using the potential outcomes framework (Neyman Reference Neyman1923; Rubin Reference Rubin1974), let $Y_i (\mathbf {t})$ denote the potential outcome of individual i if the treatment assignment vector $\mathbf {T}$ is set to $\mathbf {t}$ . Importantly, the potential outcome of individual i is affected not only by her own treatment assignment, but also by the treatments received by others. Thus, we allow for spillover/interference between individuals (Cox Reference Cox1958; Rubin Reference Rubin1980). In our running example, voting behavior of a given individual can be affected not only by whether she directly receives the informational message, but also by messages assigned to her friends as people can share information with one another.

To formalize whose treatment status can affect a given individual, we rely on networks. In particular, consider two networks $\mathcal {G}$ and $\mathcal {U}$ connecting units with different edge sets. For example, people can be connected with Facebook and their offline face-to-face network. Formally, we define an adjacency matrix of network $\mathcal {G}$ to be $A^{\mathcal {G}}$ where $A^{\mathcal {G}}_{ij} = A^{\mathcal {G}}_{ji} = 1$ if unit i is connected to unit j (and zeros on the diagonal). We can also incorporate directed ties when we define $A^{\mathcal {G}}_{ij} = 1$ if unit i “follows” unit j, for example, on Twitter. By assigning different weights to each tie, this framework can accommodate the strength of ties as well. We then define individual i’s neighbors in network $\mathcal {G}$ to be other individuals to whom she is connected in network $\mathcal {G}$ , formally, $\mathcal {N}^{\mathcal {G}}_i \equiv \{j: A^{\mathcal {G}}_{ij} \neq 0\}.$ We define $A^{\mathcal {U}}$ and $\mathcal {N}^{\mathcal {U}}_i$ similarly. For example, neighbors are those with whom individual i is friends on Facebook ( $\mathcal {N}^{\mathcal {G}}_i$ ; neighbors on an online network) or those who individual i meets in person on a daily basis ( $\mathcal {N}^{\mathcal {U}}_i$ ; neighbors on an offline face-to-face network). Importantly, neighbors in the two networks can overlap; we meet some but not all of our Facebook friends on a daily basis. Figure 1 visualizes an example with two networks.

Figure 1

Example of two networks. Note: The same set of people are connected differently by an online network and an offline network. Neighbors in the two networks overlap, i.e.,some, but not all, pairs of people are connected in both online and offline networks.

We explicitly incorporate these two networks into the potential outcomes. In particular, we extend the stratified interference assumption (Hudgens and Halloran Reference Hudgens and Halloran2008) to multiple networks. We assume that the potential outcomes of individual i are affected by her own treatment assignment and the treated proportions of neighbors in networks $\mathcal {G}$ and $\mathcal {U}$ (Manski Reference Manski2013; Toulis and Kao Reference Toulis and Kao2013; Liu and Hudgens Reference Liu and Hudgens2014; Forastiere et al. Reference Forastiere, Airoldi and Mealli2016).

Assumption 1 Stratified Interference with Multiple Networks

For all $i,$

$$ \begin{align*} Y_i (\mathbf{t}) = Y_i (d, g, u), \end{align*} $$

where d is her own treatment assignment, g and u are the proportions of treated neighbors in networks $\mathcal {G}$ and $\mathcal {U}$ , respectively:

$$ \begin{align*} d & = t_i, \ \ g = \frac{\sum_{j \in \mathcal{N}_i^{\mathcal{G}}} t_j}{|\mathcal{N}_i^{\mathcal{G}}|}, \ \ u = \frac{\sum_{j \in \mathcal{N}_i^{\mathcal{U}}} t_j}{|\mathcal{N}_i^{\mathcal{U}}|}, \end{align*} $$

where $t_i$ is the treatment received by individual i. $\sum _{j \in \mathcal {N}_i^{\mathcal {G}}} t_j$ is the total number of treated units among individual i’s neighbors in network $\mathcal {G}$ and $|\mathcal {N}_i^{\mathcal {G}}|$ is the total number of individual i’s neighbors in network $\mathcal {G}$ . $\sum _{j \in \mathcal {N}_i^{\mathcal {U}}} t_j$ and $|\mathcal {N}_i^{\mathcal {U}}|$ are similarly defined.

The potential outcomes depend on the treatment assignment of herself d and the fractions of treated neighbors in each network $(g,u)$ . For example, the potential outcome of unit i is a function of whether she receives the informational message (d), the proportion of Facebook friends who receive the message (g), and the proportion of offline friends who receive the message (u). Although this assumption can only allow for spillover effects through treated proportions in multiple networks, it is more flexible than the conventional assumption of no interference, which requires that the treatment status of neighbors do not change potential outcomes at all. Formally, Assumption 1 can also be viewed as an exposure mapping $f(\mathbf {t}, (A^{\mathcal {G}}_{i}, A^{\mathcal {U}}_{i}))$ set to a three-dimensional vector $(d, g, u)$ (Aronow and Samii Reference Aronow and Samii2017).

Several points about Assumption 1 are worth clarifying. First, because this assumption is made at the individual level, an equivalent statement of Assumption 1 is that the potential outcomes depend on the treatment assignment of herself and the number (rather than proportions) of treated neighbors in each network. The only change is that when defining causal estimands, we should explicitly condition on the total number of neighbors in each network. Second, Assumption 1 is violated when neighbors in the same network have different spillover effects. For example, suppose a given individual i has three Facebook friends $\{F_1, F_2, F_3\}$ , and $F_1$ has a larger spillover effect than the other two. In this case, even when $g = 1/3$ , the potential outcomes of unit i differ depending on whether $F_1$ is treated or one of the other two is treated, which violates Assumption 1. Third and most importantly, unlike the previous literature, Assumption 1 is defined with multiple networks. Therefore, researchers can make the assumption more plausible by specifying networks more precisely. In the example above, suppose $F_1$ has a larger spillover effect than the other two because $F_1$ is also a friend of unit i on Twitter, whereas the other two are not. Then, we can restore Assumption 1 with respect to three networks; the Facebook network, the Twitter network, and an offline network. However, it is important to note that as we consider more networks, it is more difficult to observe all relevant networks, which is the central topic of the paper we further discuss in Sections 3 and 4.

We define networks to be causally relevant if treated proportions in such networks have nonzero causal effects.

For example, if people do not talk about elections at all on Facebook, Facebook would be causally irrelevant because messages received by other Facebook friends would have no effect on voting behavior through Facebook when fixing treated proportions in the offline network.

Finally, as in the standard causal inference setting, we observe only one of many potential outcomes. For all i, $ Y_i = \sum _{(d,g,u) \in \Delta _i} \mathbf {1}\{T_i=d, G_i=g, U_i=u\} \ Y_i(d,g,u)$ (Holland Reference Holland1986).

2.2 Causal Quantities of Interest

Without loss of generality, we assume two networks $\mathcal {G}$ and $\mathcal {U}$ are causally relevant and define our quantities of interest using these two networks.

2.2.1 Direct Effects and Network-Specific Spillover Effects

First, we define the direct effect of a treatment. It is the difference between the potential outcomes under treatment and control, averaging over the distribution of treatment assignment $(G_i, U_i)$ . Formally, we define the average direct effect (ADE) as follows.

Definition 2 Average Direct Effect

(1) $$ \begin{align} \delta \ \equiv \ \frac{1}{N} \sum_{i=1}^N \left\{\sum_{(g,u) \in \Delta_i^{gu}} \{Y_i(1, g, u) - Y_i(0, g, u)\} \Pr(G_i=g, U_i=u)\right\}, \end{align} $$

where the support is defined as $\Delta _i^{gu} = \{(g,u):\Pr (G_i=g, U_i=u)> 0\}.$

It is the weighted average of the causal effects, $Y_i(1, g, u) - Y_i(0, g,u)$ , which hold the proportions of treated neighbors in the two networks constant. Intuitively, this effect quantifies the causal impact of the treatment directly received by herself. Thus, the ADE could represent the direct causal effect of the informational message on voting behavior. The ADE is a simple extension of the expected average treatment effect (Sävje et al. Reference Sävje, Aronow and Hudgens2017) to the two-network settings. We keep the term of the direct effect in order to distinguish it from spillover effects we define next.

Spillover effects describe the causal effects of the neighbors’ treatment status on a given individual. It is the change in the potential outcome when the proportion of treated neighbors goes from a lower proportion to a higher proportion. Formally, it is the difference between the potential outcome for a given individual when $g^H \times 100\%$ of her neighbors in $\mathcal {G}$ are treated and the other potential outcome for the same individual when $g^L \times 100\%$ of her neighbors in $\mathcal {G}$ are treated, holding her own treatment assignment and averaging over the proportions of treated neighbors in $\mathcal {U}$ . Two constants $g^H$ and $g^L$ stand for “Higher” and “Lower” percent. We define the average network-specific effect (ANSE) as follows.

Definition 3 Average Network-Specific Spillover Effect

(2) $$ \begin{align} \tau (g^H, g^L ; d) \ \equiv \ \frac{1}{N} \sum_{i=1}^N \left\{\sum_{u \in \Delta_i^u} \{Y_i(d, g^H, u) - Y_i(d, g^L, u) \} \ \Pr (U_i=u \mid T_i=d, G_i = g^L)\right\}, \end{align} $$

where the support is defined as $\Delta _i^{u} = \{u: \Pr (U_i=u \mid T_i=d, G_i = g^H)> 0 \mbox { and } \Pr (U_i=u \mid T_i=d, G_i = g^L) > 0\}.$ The ANSE for network $\mathcal {U}$ is defined similarly.

It is the weighted average of the spillover effects specific to network $\mathcal {G}$ , $Y_i(d, g^H, u) - Y_i(d, g^L,u)$ , which hold her own treatment assignment and the proportion of treated neighbors in $\mathcal {U}$ constant. Thus, for example, the ANSE $(g^H=0.8, g^L=0.2; d = 0)$ could represent the Facebook-specific spillover effect where we change the treated proportion of one’s Facebook friends (from $g^L = 20$ % to $g^H = 80$ %) while holding constant one’s own treatment assignment ( $d = 0$ ) and averaging over the treated proportion of one’s offline friends (u). Similarly, the ANSE $(u^H =0.8, u^L=0.2; d = 0)$ could represent the offline network-specific spillover effect where we change the treated proportion of one’s offline friends (from $u^L = 20$ % to $u^H = 80$ %) while holding constant one’s own treatment assignment ( $d = 0$ ) and averaging over the treated proportion of one’s Facebook friends (u).

Importantly, the ANSE captures the spillover effect specific to each network separately. Thus, when Facebook (the offline network) is causally irrelevant, as defined in Definition 1, the ANSE in Facebook (the offline network) will be zero. By estimating the ANSE for each network, researchers can unpack the mechanism of spillover effects; which networks do people use to share information and influence each other’s voting behavior? In Appendix B, we additionally show that the ANSE decomposes the total spillover effect, a popular estimand in the literature (Hudgens and Halloran Reference Hudgens and Halloran2008), into each network.

2.3 Estimation

In this section, we study the estimation of the ADE and the ANSE. We begin by introducing a common assumption made in practice; the no omitted network assumption defined below.

For example, in the Facebook mobilization experiment (Bond et al. Reference Bond2012), this assumption of no omitted network means that the Facebook network is the only relevant network and an unobserved face-to-face network is irrelevant; experimental subjects could affect one another through Facebook but not through their unobserved offline interactions. We first examine the estimation under this assumption and then discuss its violation in the subsequent sections.

Following the recent literature (e.g., Hudgens and Halloran Reference Hudgens and Halloran2008; Tchetgen Tchetgen and VanderWeele Reference Tchetgen Tchetgen and VanderWeele2010; Kang and Imbens Reference Kang and Imbens2016; Sussman and Airoldi Reference Sussman and Airoldi2017), we start with design-based, inverse probability weighting (IPW) estimators for the ADE and ANSE (Horvitz and Thompson Reference Horvitz and Thompson1952; Aronow and Samii Reference Aronow and Samii2017). Importantly, researchers can compute the treatment exposure probability $\Pr (T_i = d, G_i = g, U_i = u)$ from the experimental design under the no omitted network assumption.Footnote ¹ Thus, we rely on the following weighting estimators.

Theorem 1 Estimation of the ADE and the ANSE

Under Assumption 2, the treatment exposure probability $\Pr (T_i=d, G_i=g, U_i=u)$ is known for all $(d, g, u) \in \Delta _i$ and all i. Therefore, the average direct effect and the average network-specific spillover effect in $\mathcal {G}$ can be estimated by the following inverse probability weighting estimators.

$$ \begin{align*} \mathbb{E}[\hat{\delta}] = \delta \quad \text{and} \quad \mathbb{E}[\hat{\tau}(g^H, g^L; d) ] = \tau(g^H, g^L; d), \end{align*} $$

where the expectation is taken over the experimental design $\Pr (\mathbf {T}=\mathbf {t})$ , and

(3) $$ \begin{align} \hat{\delta} &\equiv \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{T_i=1\} \tilde{w}_i Y_i - \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{T_i=0\} \tilde{w}_i Y_i \end{align} $$

(4) $$ \begin{align} \hat{\tau}(g^H, g^L; d) & \equiv \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{T_i = d, G_i = g^H\} w_i Y_i - \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{T_i = d, G_i = g^L\} w_i Y_i, \end{align} $$

and weights are defined as

$$ \begin{align*} \tilde{w}_i = \frac{1}{\Pr(T_i \mid G_i, U_i)}, \quad \text{and} \quad w_i = \frac{\Pr(U_i \mid T_i, G_i = g^L)}{\Pr(U_i \mid T_i, G_i)} \times \frac{1}{\Pr(T_i, G_i)}. \end{align*} $$

We provide the proof in Appendix C.1. Although the IPW estimators are unbiased, researchers might want to increase efficiency by incorporating covariate information and some parametric assumptions (Särndal et al. Reference Särndal, Swensson and Wretman1992). In particular, we build on the design-based IPW estimator above and propose a weighted linear regression estimator. This alternative approach reduces standard errors at the expense of some bias due to parametric modeling assumptions. In particular, we consider

(5) $$ \begin{align} Y_i \sim \alpha + \beta T_i + \gamma_0 G_i + \gamma_1 (T_i \times G_i) + \mathbf{X}^\top_ i \theta \mbox{ with weights } w_i. \end{align} $$

The key assumptions are the linearity of $G_i$ and the inclusion of pretreatment covariates $\mathbf {X}_i$ .Footnote ² We emphasize an important tradeoff. On the one hand, both assumptions are parametric in the sense that they are not directly justified by the experimental design. On the other hand, they tend to greatly reduce variance. Especially in settings with multiple networks, treatment exposure probabilities $\Pr (T_i, G_i, U_i)$ are usually small, and in many applied contexts, the standard IPW estimator has large standard errors. The weighted linear regression estimator aims to balance this bias-variance tradeoff in practice. See Toulis and Kao (Reference Toulis and Kao2013) for a Bayesian approach, and Särndal et al.(Reference Särndal, Swensson and Wretman1992); Rosenbaum (Reference Rosenbaum2002) and Aronow and Samii (Reference Aronow and Samii2017) for the design-based covariate adjustment.

4.1 Parametric Sensitivity Analysis

Although the exact bias formula in Theorem 2 does not make any assumption about the unobserved network $\mathcal {U}$ , in order to use it in applied work, it requires specifying a large number of sensitivity parameters. To construct a practical parametric sensitivity analysis method, we rely on a simplifying parametric assumption. In particular, we consider the assumption that the network-specific spillover effect in an unobserved network is linear and additive (Sussman and Airoldi Reference Sussman and Airoldi2017).

Assumption 3 Linear, Additive Network-Specific Spillover Effect in $\mathcal {U}$

$$ \begin{align*} \frac{1}{\sum_{i=1}^N \mathbf{1}\{S_i=s\}}\sum_{i:S_i=s} \{Y_i(d, g, u) - Y_i(d, g, u^\prime)\} = \lambda (u- u^\prime), \end{align*} $$

with some coefficient $\lambda $ for all $(d, g, u), (d, g, u^\prime ) \in \Delta _s$ where $\Delta _s$ is the support of $(d, g, u)$ for i with the neighbors’ profile $S_i=s$ . The neighbors’ profile S is defined such that the probability of treatment exposure is the same for those who have the same neighbors’ profile. Formally, for $i \neq j$ , $\Pr (T_i=d, G_i=g, U_i=u) = \Pr (T_j=d, G_j=g, U_j=u)$ when $S_i=S_j.$ For example, when the Bernoulli or completely randomized design is used, $S_i$ is simply a vector of three values; the number of neighbors in $\mathcal {G}$ and $\mathcal {U}$ , and the number of neighbors common to the two networks.

While this linear additive assumption is strong, it is commonly used in the causal inference literature to derive intuitive, easy-to-use sensitivity analysis methods. For example, the widely used sensitivity analysis methods for unobserved confounders in observational studies rely on similar assumptions (VanderWeele and Arah Reference VanderWeele and Arah2011). We also consider nonparametric sensitivity analysis in the next section under weaker assumptions.

Under Assumption 3, the general bias formula becomes the multiplication of two terms: the network-specific spillover effect in $\mathcal {U}$ , that is, $\lambda $ , and the effect of $G_i$ on $U_i$ given $T_i$ , that is, $ \frac {1}{N} \sum _{i=1}^N \{ \mathbb {E}[U_i \mid T_i=d, G_i=g^H] - \mathbb {E}[U_i \mid T_i=d, G_i=g^L] \}.$ The following theorem shows a simplified bias formula for settings with sparse $\mathcal {G}$ .

Theorem 3 Parametric Sensitivity Analysis

Under Assumption 3, a bias formula is approximated as follows.

(7) $$ \begin{align} \mathbb{E}[\hat{\tau}_B (g^H, g^L; d)] - \tau (g^H, g^L; d) \approx \lambda \times \pi_{GU} \times (g^H - g^L), \end{align} $$

where $\lambda $ is the ANSE in network $\mathcal {U}$ and $\pi _{GU}$ is the overlap, that is, the fraction of neighbors in $\mathcal {U}$ who are also neighbors in $\mathcal {G}$ . Formally, $\pi _{GU} \equiv \sum _{i=1}^N \{|\mathcal {N}^{(\mathcal {G},\mathcal {U})}_i| / |\mathcal {N}^{\mathcal {U}}_i|\}/N$ where $|\mathcal {N}^{(\mathcal {G},\mathcal {U})}_i|$ is the number of unit i’s neighbors common to the two networks. If an experiment uses a Bernoulli design, the approximation is exact regardless of the sparsity of network $\mathcal {G}.$

We provide the proof in Appendix C.3. This simplified bias formula offers several implications. First, the bias is small when $\pi _{GU}$ is small, that is, the overlap of neighbors in $\mathcal {G}$ and $\mathcal {U}$ is small. Hence, the bias is close to zero when the network $\mathcal {G}$ is sparse and neighbors in $\mathcal {G}$ and $\mathcal {U}$ are disjoint. Second, even if two networks $\mathcal {G}$ and $\mathcal {U}$ are independently generated, the bias is not zero because $\pi _{GU} \neq 0$ . We derive a similar parametric bias formula for settings with nonsparse $\mathcal {G}$ in Appendix C.3.

For example, the overlap between the Facebook network and the unobserved face-to-face network is defined to be the fraction of offline friends who are also friends on Facebook. When this overlap is large (small), we expect the bias for the Facebook-specific spillover effect to be large (small).

To use this formula for a sensitivity analysis, researchers need to specify two sensitivity parameters: the network-specific spillover effect in an unobserved network (i.e., $\lambda $ ) and the fraction of neighbors in $\mathcal {U}$ who are also neighbors in $\mathcal {G} $ (i.e., $\pi _{GU}$ ). Once these two parameters are specified, we can derive the bias. Subsequently, because the bias utilizes only sensitivity parameters and $(g^H - g^L)$ , we can obtain a bias corrected estimate by subtracting this bias from $\hat {\tau }_B (g^H, g^L; d)$ . A sensitivity analysis is to report the estimated ANSE under a range of plausible values of $\lambda $ and $\pi _{GU}$ where $0 < \pi _{GU} < 1.$ Note that $\lambda = 0 $ corresponds to the no omitted network assumption.

4.2 Nonparametric Sensitivity Analysis

Now, we provide a nonparametric sensitivity analysis that makes only one assumption of non-negative outcomes. While we introduce our method using a random variable $U_i$ for simplicity, the same method can be applied to a random vector ${\mathbf { U_i}}$ and accommodate multiple unobserved networks. The proposed method is an extension of a sensitivity analysis developed for observational studies with no spillover effect (Ding and VanderWeele Reference Ding and VanderWeele2016) to experimental settings where spillover effects in unobserved networks induce bias.

As the parametric sensitivity analysis, we use two sensitivity parameters: intuitively, the network-specific spillover effect in $\mathcal {U}$ , and the association between $U_i$ and $G_i$ . To capture the network-specific spillover effect in $\mathcal {U}$ , let $\mathrm {MR}_{UY} (g, s)\equiv \max\nolimits_{u} \sum _{i; S_i=s} Y_i(d, g, u) / \min\nolimits_{u}\sum _{i;S_i=s} Y_i (d,g,u)$ denote the largest potential outcomes ratios of $U_i$ on $Y_i$ given $T_i=d, G_i=g$ and $S_i=s$ . For notational simplicity, we drop subscript d whenever it is obvious from contexts. Then, we define $\mathrm {MR}_{UY} = \max\nolimits_{g \in (g^H, g^L), \ s \in \mathcal {S}} \ \mathrm {MR}_{UY} (g, s)$ as the largest potential outcomes ratio of $U_i$ on $Y_i$ over $g \in \{g^H, g^L\}$ and $s \in \mathcal {S}.$ Thus, $\mathrm {MR}_{UY}$ quantifies the largest possible potential outcomes ratio of $U_i$ on $Y_i$ . This ratio captures the magnitude of the network-specific spillover effect in $\mathcal {U}$ . When network $\mathcal {U}$ is irrelevant, $\mathrm {MR}_{UY} = 1$ .

Furthermore, to quantify the association between $G_i$ and $U_i$ , we use $\mathrm {RR}_{GU} (g, g^\prime ,u, s) \ = \ \Pr (U_i=u \mid T_i=d, G_i=g) \ / \ \Pr (U_i=u \mid T_i=d, G_i=g^\prime )$ to denote the relative risks of $G_i$ on $U_i$ for all i with $S_i=s$ . $ \mathrm {RR}_{GU} \ = \ \max\nolimits_{(g, g^\prime ) \in (g^H, g^L), u \in \Delta ^u, s \in \mathcal {S}} \ \mathrm {RR}_{GU} (g, g^\prime ,u, s)$ is the maximum of these relative risks. This risk ratio captures the association between $G_i$ and $U_i$ .

Using these ratios, we can derive an inequality that the ANSE in the observed network $\mathcal {G}$ needs to satisfy as long as outcomes are non-negative. The next theorem shows that we can obtain the bound for the ANSE with two sensitivity parameters, $\mathrm {MR}_{UY}$ and $\mathrm {RR}_{GU}$ .

Theorem 4 Bound on the ANSE

When outcomes are non-negative,

$$ \begin{align*} & \mathbb{E}\biggl[ \frac{\hat{m}(d, g^H)}{B} - B \hat{m}(d, g^L) \biggr] \ \leq \ \tau (g^H, g^L; d) \ \leq \ \mathbb{E}\biggl[ B \hat{m}(d, g^H) - \frac{\hat{m}(d, g^L)}{B} \biggr], \end{align*} $$

where $B = (\mathrm {RR}_{GU} \times \mathrm {MR}_{UY})/ (\mathrm {RR}_{GU} + \mathrm {MR}_{UY} - 1)$ and $\hat {m}(d, g) = \frac {1}{N} \sum _{i=1}^N \mathbf {1}\{T_i=d, G_i=g\} w_i^{\mathtt {B}}Y_i$ for $g \in \{g^H, g^L\}$ .

We provide the proof in Appendix C.4. Note that $\mathrm {MR}_{UY} = 1 $ corresponds to the no omitted network assumption, and $\mathbb {E}[\hat {m}(d, g^H) - \hat {m}(d, g^L)] = \mathbb {E}[\hat {\tau }_B (g^H, g^L; d)] = \tau (g^H, g^L; d)$ under the assumption. B is an increasing function of both $\mathrm {RR}_{GU}$ and $\mathrm {MR}_{UY}$ , implying that the bound is wider when the network-specific spillover effect in the unobserved network $\mathcal {U}$ is larger and the effect of $G_i$ on the distribution of $U_i$ is larger. In fact, the size of the bound is given by $(B - \frac {1}{B}) \mathbb {E}[\hat {m}(d, g^H) + \hat {m}(d, g^L)].$ It is important to note that the bound is not location-invariant because we use mean ratios and risk ratios as sensitivity parameters (Ding and VanderWeele Reference Ding and VanderWeele2016). The bound is valid as far as outcomes are non-negative, but how informative it is can vary. To conduct the sensitivity analysis, one can compute the bound for a range of plausible values of $\mathrm {MR}_{UY}$ and $\mathrm {RR}_{GU}$ . Compared to the parametric sensitivity analysis, $\mathrm {MR}_{UY}$ and $\mathrm {RR}_{GU}$ correspond to $\lambda $ and $\pi _{GU},$ respectively. Finally, as in Section 2.3, we can also use a weighted linear regression estimator for $\hat {m}(d, g)$ instead of the IPW estimator.

Background

To make our simulation as realistic as possible, we rely on the real-world Twitter network studied in Coppock et al.(Reference Coppock, Guess and Ternovski2016). The original authors conducted a political mobilization experiment over the Twitter network to estimate the effectiveness of online mobilization appeals. In particular, they sampled followers of the Twitter account of a nonprofit advocacy organization, the League of Conservation Voters (LCV), and measured a network adjacency matrix among them. Each node of the network is a Twitter user who follows the LCV’s account and a directed edge exists from user i to user j when user i follows user j. Instead of artificial simulated networks, we use this real-world Twitter network as the basis of our simulations after preprocessing as described below.

Simulation design

To make the comparison of sensitivity analysis methods clear, we preprocess the Twitter network of Coppock et al.(Reference Coppock, Guess and Ternovski2016). First, to avoid the influence of outliers, we remove units that are above the 95th percentile of the distribution on the size of Twitter neighbors (also known as the out-degree distribution). To have well-defined treated proportions $G_i$ (explain more below), we also remove those who follow less than five other units. The resulting network contains $2,430$ units with the mean number of neighbors equal to $18.5$ . We simulate an unobserved offline network $\mathcal {U}$ with four different values of the overlap $\pi _{GU}\in \{0.0, 0.2, 0.4, 0.6\}$ where we expect no bias in the case of $\pi _{GU} = 0$ . We use the Bernoulli design to randomly assign treatments with probability $0.5.$

We consider two outcome data generating processes. For the linear additive model, we generate potential outcomes for individual i by $ Y_i (T_i, G_i, U_i) \ = 5 + 2 T_i + G_i + 1.5 U_i + \epsilon _i,$ where $\epsilon _i$ is randomly drawn from a normal distribution with $(\mu , \sigma ) = (0.0, 0.5)$ . $G_i$ and $U_i$ denote treated proportions of neighbors in the observed Twitter network and unobserved treated proportions of neighbors in the offline network, respectively. For the interactive model, we use $ Y_i (T_i, G_i, U_i) \ = 5 + 2 T_i + G_i + 2 T_i \times U_i + 2 G_i \times U_i + 0.5 U_i + \epsilon _i$ . As the main causal estimand, we consider the ANSE, $\tau (0.6, 0.4; 0)$ , where we compare $g^H=0.6$ relative to $g^L=0.4$ when $d=0$ (to have $G_i = 0.6$ and $G_i= 0.4$ both well-defined, we focus on a subset of units who follows at least five units). In the linear additive model, the true ANSE is equal to $0.2$ , and in the interactive model, the true ANSE is approximately equal to $0.4$ (the exact values change according to the overlap $\pi _{GU}$ ).

The offline network $\mathcal {U}$ is unmeasured and thus, we can rely only on a linear regression $Y_i \sim \alpha + \beta T_i + \gamma _0 G_i + \gamma _1 (T_i \times G_i)$ with weights $w_i^{\mathtt {B}} = 1/\Pr (T_i, G_i).$ An estimator, $0.2 \times \hat {\gamma }_0,$ is biased for $\tau (0.6, 0.4; 0)$ due to unobserved network $\mathcal {U}.$ The parametric sensitivity analysis provides a bias-corrected estimate, $0.2 \times \hat {\gamma }_0 - 0.2 \times \lambda \times \pi _{GU},$ with two sensitivity parameters; the overlap $\pi _{GU}$ and the spillover effect in the unobserved network $\lambda $ .Footnote ⁴ The nonparametric sensitivity analysis computes bounds with sensitivity parameters $(\mathrm {MR}_{UY}, \mathrm {RR}_{GU})$ .Footnote ⁵ We evaluate an estimator that ignores unobserved offline networks, the parametric sensitivity analysis, and the nonparametric sensitivity analysis with the $2,000$ Monte Carlo simulations.

Setup

Cai et al.(Reference Cai, De Janvry and Sadoulet2015) designed the experiment with two rounds to estimate the spillover effect. A total of 2,175 households were invited to the first round and they were randomly assigned into one of two information sessions, simple or intensive. The simple session took about 20 min and the PICC agents explained the insurance contract. The intensive sessions took about 45 min and provided all the information given in the simple session, plus an additional detailed explanation of expected benefits and costs of purchasing the insurance. Three days after the first round, a different set of households were invited to the second round and randomly assigned into the simple or intensive sessions. We follow the original analysis and focus on 1,317 households who were invited to this second round and only received information from the simple or intensive session (657 and 660 households, respectively).Footnote ⁶ The main outcome of interest is a binary variable whether each household in the second round takes up the insurance. Table 1 summarizes the relevant aspects of the experimental design. See Cai et al.(Reference Cai, De Janvry and Sadoulet2015) for details and other features of the design.

Table 1

Experimental design.

For participants in the second round, the original authors define the main exposure variable of interest $G_i$ to be the proportion of peers in their financial network who attended the first round intensive session. The direct treatment $T_i$ is defined as an indicator variable taking $1$ if household i receives the intensive session and $0$ otherwise. Following their analysis, we focus on the ANSE specific to the observed financial network and compare $g^H = 0.2$ and $g^L = 0$ under no direct treatment receipt $d= 0$ , formally $\tau (g^H = 0.2, g^L = 0; d = 0)$ , as they show the biggest difference is between no treated peer and one treated peer (table 3 of the original paper). In addition, to satisfy the standard overlap assumption (Imbens and Rubin Reference Imbens and Rubin2015), we analyze units who listed the maximum number of five householdsFootnote ⁷ in the financial network question (1,207 households, more than 90% of all the samples).

To estimate spillover effects, the original authors estimate the following linear regression (model 4 in table 2 of Cai et al.(Reference Cai, De Janvry and Sadoulet2015)) with standard errors clustered at the village level.

(8) $$ \begin{align} Y_i = \alpha + \beta T_i + \gamma_0 G_i + \gamma_1 (T_i \times G_i) + \mathbf{X}^\top_i \theta + \epsilon_i, \end{align} $$

where pretreatment covariates $\mathbf {X}_i$ include the size of neighbors, village fixed effects, and other household characteristics (gender, age, education of household head, rice production area, risk aversion, and perceived probability of future disaster). Thus, if the observed financial network is the only causally relevant network (i.e., the no omitted network assumption holds), an estimator $0.2 \times \hat {\gamma }_0$ is unbiased for the ANSE, $\tau (g^H=0.2, g^L=0;d=0)$ where $\hat {\gamma }_0 = 0.444$ (SE = $0.109$ ) in the original paper. However, as studied in Section 3, when the no omitted network assumption is violated—for example, people might share information through their kinship network, the estimator is biased for the ANSE specific to the financial network.

Sensitivity analysis

We use the proposed sensitivity analysis methods to investigate the robustness of the original findings to unobserved networks. First, for the parametric sensitivity analysis, we consider three values for each of the two sensitivity parameters, the spillover effect in the unobserved network $\lambda \in (0.1, 0.2, 0.4)$ and the overlap $\pi _{GU} \in (0.2, 0.4, 0.6)$ , producing a total of nine scenarios. The three values of $\lambda $ are chosen to represent different scenarios in which we expect spillover effects in the kinship network is much smaller than, smaller than, or similar to the total spillover effect in the financial network ( $\hat {\gamma }_0 = 0.444$ ). Similarly, the three values of $\pi _{GU}$ cover from small to large overlaps. For the nonparametric sensitivity analysis, we also investigate three values for each of the two sensitivity parameters $\mathrm {MR}_{UY} \in (1.3, 1.5, 1.8)$ and the overlap $\mathrm {RR}_{GU} \in (1.3, 1.5, 1.8)$ . Although these are relatively small risk and mean ratios (e.g., see table 1 in Ding and VanderWeele (Reference Ding and VanderWeele2016)), we see below that substantive conclusions change according to different sensitivity parameters. Finally, while the original paper uses a linear regression without weights, we use a weighted linear regression with weights $w_i^{\mathtt {B}} = 1/\Pr (T_i, G_i)$ as proposed in Section 4 for both analyses.

Figure 3 presents results from the parametric (left) and nonparametric sensitivity analysis (right). In the left panel, the leftmost point shows an estimate of the ANSE that ignores the unobserved kinship network with its 95% CI ( $8.87$ percentage point; 95% CI = $[3.31, 14.42]$ ). The nine other points represent estimates from the parametric sensitivity analysis with their 95% CI. Importantly, point estimates are relatively stable over a range of sensitivity parameters and the 95% CI covers zero only when the effect in the unobserved kinship network is large ( $\lambda = 0.4$ ) and the overlap between the observed financial network and the unobserved kinship network is relatively large ( $\pi _{GU} = 0.6$ ). Thus, the parametric sensitivity analysis suggests that, while point estimates of the ANSE specific to the observed financial network are smaller than the original estimates, estimates are still positive and statistically significant at the conventional level of $0.05$ . Although this parametric sensitivity analysis is simple and intuitive, it requires the parametric linear additive assumption as discussed in Section 4.1. We now turn to the nonparametric sensitivity analysis which requires weaker assumptions at the expense of efficiency.

Figure 3

Parametric and nonparametric sensitivity analysis for the field network experiment in China. Note: The left (right) panel reports the parametric (nonparametric) sensitivity analysis with different sensitivity parameters. In both panels, the leftmost point shows an estimate of the ANSE that ignores unobserved networks. In the left panel, the nine other points represent estimates from the parametric sensitivity analysis with their 95% confidence intervals. In the right panel, the thick bars represent nonparametric bounds on the ANSE and the thin bars are their 95% confidence intervals.

The right panel of Figure 3 shows results of the nonparametric sensitivity analysis where the leftmost point reproduces the same ANSE estimate that ignores the unobserved kinship network as a reference point. The thick bars represent nonparametric bounds on the ANSE and the thin bars are their 95% CI. Although lower bounds are positive for small bias scenarios, such as $(MR_{UY}, RR_{GU}) = (1.3, 1.3), (1.5, 1.3), (1.3, 1.5),$ all the 95% CI are inconclusive about the sign of the ANSE. Importantly, this result reveals that the positive ANSE estimates are sensitive to different assumptions about unobserved networks in the nonparametric sensitivity analysis, in contrast to the parametric sensitivity analysis above.

This difference in results from the parametric and nonparametric sensitivity analysis can arise for two reasons. First, the linear additive assumption (Assumption 3) required for the parametric sensitivity analysis could be violated and therefore, the nonparametric bounds are more credible. Second, the nonparametric bounds are in general less efficient than parametric methods and thus, large nonparametric bounds simply indicate insufficient statistical power for detecting the ANSE nonparametrically. In this China experiment, as the original authors detect the nonlinearity in the total spillover effect in the financial network (table 3 of Cai et al.(Reference Cai, De Janvry and Sadoulet2015)), it is more likely that the linear additive assumption is violated and thus, the nonparametric sensitivity analysis is more credible. In practice, it is important to conduct both parametric and nonparametric sensitivity analysis methods as they are complementary and evaluate the robustness of the ANSE estimates under different assumptions.