We sincerely appreciate the interest in our research on the effects of mass public shootings and opinions on gun control. In our article (Newman and Hartman Reference Newman and HartmanForthcoming), we examined how proximity to a mass shooting affected preferences for firearms restrictions. We reported a meaningful and statistically significant increase in support for gun restrictions the closer an individual lived to a mass shooting. We also found that this contextual effect increased in magnitude with the intensity of the event (that is, the number of victims) and dissipated over time (for example, after 10 years). Importantly, we showed that proximity to a mass shooting did not affect a range of treatment-irrelevant policies such as preferences concerning climate change, abortion, same-sex marriage and immigration. For these analyses, we focused much of our efforts on modeling data from the 2010 Cooperative Congressional Election Study (CCES), given its large sample size (n=55,400) and our ability to include many potential mass shooting exposures (n=125 events). We also replicated these results using cross-sectional data from the Pew Research Center, as well as a subsample of the 2010 CCES, which formed the 2010–14 CCES Panel Survey.
In their comment, Barney and Schaffner (Reference Barney and SchaffnerForthcoming) raise questions about our findings from the 2010–14 CCES Panel Survey. They argue that how we coded our ‘treatment’ was problematic and potentially missed important nearby exposures between panel waves. In their re-analysis of the CCES panel data, they conclude that ‘the general effect’ of residing near a mass shooting on gun control attitudes ‘is small and not statistically distinguishable from zero’. They do, however, find evidence of a polarization effect: exposure to a nearby mass shooting leads Democrats to become more supportive of gun restrictions, and Republicans to become less supportive.
What are we to make of these divergent findings? From our perspective, the crux of the matter concerns the definition of who should be counted as treated in the panel data. Barney and Schaffner consider a mass shooting treatment to occur for any exposure within a certain distance threshold between panel waves regardless of pre-treatment exposures.Footnote 1 In contrast, we originally considered a mass shooting treatment to occur when an individual’s nearest event was within 100 miles and occurred between the 2010 and 2012 panel waves for all mass shootings in the database. In retrospect, we do not believe that either approach is the correct way to code event exposures because they both include the possibility of spillover effects from pre-treatment events. We now believe prior event exposures are a serious concern, which is not adequately addressed in the modeling strategy presented by Barney and Schaffner (or our original article). Only by removing prior exposures within the defined treatment area can we get a true estimate of the effect of a mass shooting exposure on gun control attitudes. We will explain this rationale in the next section.
We agree with Barney and Schaffner that the findings from our panel analysis are important not only to those concerned about the substantive effect of mass shootings on policy preferences, but also to scholars interested in using panel data to examine causal relationships. To illustrate the implications of pre-treatment exposure on identifying ‘causal’ effects, we re-analyzed the 2010–14 (3-wave) and 2010–12 (2-wave) CCES panel datasets. On the whole, we find that our original conclusions hold: respondents who live near a mass public shooting (in this case, within 100 miles of an event) are indeed more likely to support gun control than those who do not, provided we account for pre-treatment exposure. These treatment effects are statistically and practically significant: when accounting for pre-treatment exposures in the preceding 5 years, for instance, the effects of the treatment amount to an average increase in support for firearms restrictions of 23 to 41 per cent relative to the untreated respondents (and depending on the type of modeling approach). Ultimately, our new results highlight the importance of properly accounting for pre-treatment exposure when dealing with re-occurring treatments or ‘event chains’, and they contribute to the debate about how best to assess causal effects using panel data.
Re-Analysis and Extension Using the CCES Panel Datasets
We begin by discussing the data processing of the 2010–14 (3-wave) CCES Panel Survey (Schaffner and Ansolabehere Reference Schaffner and Ansolabehere2018),Footnote 2 which we used in our original article (the descriptive statistics are similar when we extend this analysis using the larger sample from the 2010–12 2-wave CCES Panel Study).Footnote 3 We could not provide estimates for 987 of the 9,500 respondents due to missing values on one or more key variables. Twenty-nine respondents were dropped because they did not provide a response to the gun control policy question in one or both panel waves. A further twenty-five respondents were removed because they did not indicate how long they had lived at their current address, which was necessary to accurately account for pre-treatment exposures. Finally, 933 respondents were removed because they appeared to have moved between panel waves, making it impossible to know whether they had been exposed to the key treatment events.
Seventeen mass public shootings occurred between the 2010–12 panel waves (that is, after 7 November 2010 and before 2 October 2012). An additional 125 mass shootings took place prior to the CCES panel start date (from 1966 to 2010). To ensure that respondents were exposed to pre-treatment events, we removed any event that occurred prior to each respondent’s self-reported residency in their current zip code. Thus even though the event database spans several decades, we do not include any mass shootings that occurred prior to a respondent residing at their current address.
We used the ‘zipcode’ package in R that provides longitude and latitude geographic (geodetic) coordinates for each respondent’s self-reported residential zip code (based on its centroid). We then used the ‘distGeo’ function from the ‘geosphere’ package in R to compute the shortest distance between two points on an ellipsoid (a.k.a. geodesic). The advantage of this approach is that it takes into consideration the natural curvature of the Earth, thus providing highly accurate estimates of distance. Between survey waves, the minimum distance to the seventeen mass shooting events is as close as 0.38 miles, while the maximum distance is 4,881 miles (for example, respondents in Hawaii). The median distance is 979 miles, and the mean of this somewhat skewed distribution is 1,132 miles.
The Importance of Properly Accounting for Pre-Treatment Exposure
One complication in attempts to accurately estimate the effect of proximity to a mass shooting on policy preferences is that the panel, like the cross-sectional datasets we previously analyzed, provides a snapshot in time over a 2-year period (or a 4-year period when using the full three waves). Yet prior to this window of time, 125 events occurred before the 2010 CCES panel. How we account for these potential pre-treatment exposures is crucial to accurately estimating the effect of our mass shooting treatment. Gaines and Kuklinksi argue that ignoring prior exposures means that we do not actually estimate ‘the average treatment effect, but, rather, the average marginal effect of additional treatment’ (2011, 450; our emphasis). Notwithstanding this concern, empirical investigations of pre-treatment effects are relatively uncommon in the literature. Druckman and Leeper (Reference Druckman and Leeper2012, 875–76) observe: ‘Despite the potentially grave consequences of pretreatment effects […] there has been virtually no work on the topic’. In short, we must be very careful to properly account for pre-treatment exposure if we want to accurately estimate the effect of gun violence on policy attitudes. This is especially the case for treatments in which it is plausible that the largest effect occurs with the initial treatment and subsequent treatments exert diminishing effects.
To get a sense of how pre-treatment exposure to mass public shootings might obscure any treatment effects between panel waves, we plotted the 17 unique mass shootings in our data that occurred between panel waves (that is, 2010–12) in Figure 1. Panel A shows what we would observe if we ignored pre-treatment exposures and only focused on the 2-year snapshot in time. Panels B and C illustrate just how problematic pre-treatment shooting events are in the 5 and 10 years before the first wave of the CCES panel. Figure 1 demonstrates that few mass shooting events happen in isolation; pre-treatment exposure is a real concern.
Map of mass shootings used in the CCES panel re-analyses
Panel A. Treatment=shootings between 2010–12 panel waves
Panel B. Treatment with pre-treatment exposures (5 years before panel)
Panel C. Treatment with pre-treatment exposures (10 years before panel)
Note: dark red dots indicate mass shootings that occurred between CCES panel waves; light grey dots indicate pre-treatment exposures. Larger dots indicate more victims injured or killed in that event.
Of the 2,592 individuals considered treated by the Barney and Schaffner coding approach (what we label as ‘any exposure’), 70 per cent (n=1,817) of respondents had been exposed to at least one prior mass shooting event in the 10 years before the CCES panel (see Table 1). Since 2005, that proportion is only marginally smaller: 64 per cent of respondents (n=1,664) lived within 100 miles of a mass public shooting prior to the CCES panel interview. It is even more concerning that one out of every three individuals lived within 100 miles of multiple mass public shootings since 2000; it is one out of every four respondents since 2005. These pre-treatment exposures are not hypothetical, as we have tailored each respondent’s exposures to their own length of residency, thus removing any shooting events that occurred prior to when the respondent reported that they moved to their current address. What is more, some of those respondents in the Barney and Schaffner treatment had as many as four pre-treatment exposures (again verified against their residency at that address). To further confound matters, respondents in the ‘control’ condition also suffered from multiple pre-treatment exposures, as evidenced by Table 1. In short, we wonder whether it is wise to think that someone exposed to a mass shooting in an earlier time period would show any change in gun control attitudes during the panel, since we would expect any movement on attitudes to have already occurred.
Allocations to ‘treatment’ conditions (including number of prior exposures among those in the treated and untreated groups)
Notes: Nearest Event=nearest mass shooting for all events (maximum n=142; depends on length of residency at address) occurs within 100 miles and between panel waves; Any Exposure=any mass shooting (n=17) occurs within 100 miles and between panel waves.
Estimation Strategy
In their analysis of the CCES panel data, Barney and Schaffner use a fixed-effects linear probability model on the 3-point ordinal outcome of interest. Although this model is easy to interpret, we have reservations about using this modeling approach, given the ordinal nature of the dependent variable. We instead use two complementary methods to retrieve estimates of the effects of exposure to a mass shooting event on preferences for gun control. First, we report the results from a difference-in-differences (DID) design using ordered logisitic regression for panel data. Puhani (Reference Puhani2012) shows that the desired treatment effect for ordinal outcomes can be estimated as we would with a standard DID framework, and that the treatment effect is simply the interaction coefficient. One potential criticism of this approach is that this DID estimator uses a random-effects regression model, which (as noted by Barney and Schaffner) still allows the possibility of unobserved heterogeneity to affect the results.Footnote 4
To address this issue, in a second step we estimate fixed-effects ordered logistic regression models using a ‘blow-up and cluster’ model (Baetschmann, Staub and Winkelmann Reference Baetschmann, Staub and Winkelmann2015). In cross-sectional data it is relatively easy to use maximum likelihood to estimate ordered logit models for ordinal outcomes. However, in panel data, estimation is quite complicated because ‘unlike in the linear model, no simple transformation (such as first-differencing or within-transformation) is available that would purge the ordered response models from the individual-specific fixed effects’ (Baetschmann, Staub and Winkelmann Reference Baetschmann, Staub and Winkelmann2015, 1). To remedy this issue, the blow-up and cluster approach implements a cluster-robust variance estimator for ordinal data. For example, Dickerson, Hole and Munford (Reference Dickerson, Hole and Munford2014) demonstrate how to estimate a blow-up and cluster fixed-effects ordinal logistic regression model using Stata (‘bucologit’). In sum, our modeling approach should provide accurate estimates of the mass shooting treatment effect given the panel data structure, ordinal scale of our policy measure and potential pre-treatment exposures.
Re-Analysis Using the 2010–2014 (3-Wave) Panel (N=9,500)
We first provide the results from the DID approach using random-effects ordered logistic regression models in Table 2. For our purposes, the estimate of interest is the interaction between the treatment indicators and the panel year: this is the effect of exposure to a mass public shooting on gun control attitudes. The first two columns compare the ‘naïve’ treatment effects from our original article (‘Nearest Event’) and the Barney and Schaffner model (‘Any Exposure’). Not surprisingly, we see a positive and statistically significant interaction from our original coding definition, and a negative insignificant interaction term from the Barney and Schaffner replication. The odds ratios for the DID estimates provide a sense of the size of the effect; for example, those treated in the nearest event model are 29 per cent more likely to support gun control relative to those in the control group (Figure 2).
The effect of exposure to a mass shooting on preferences for gun control (DID estimator)
Notes: DID estimates (odds ratios with 90 per cent confidence intervals) from random-effects ordered logistic regression models with cluster-robust standard errors using the ‘xtologit’ command in Stata 15.
Effect of exposure to mass public shooting, re-analysis of the 2010–14 (3-wave) CCES panel data
Notes: DID analysis of the 2010–12 waves of the 2010–14 Cooperative Congressional Election Panel Survey with 17 mass shooting events between waves. Cell entries are estimates from random effects ordered logistic regression models with cluster-robust standard errors using the ‘xtologit’ command in Stata 15; standard errors are in parentheses; 90 per cent confidence intervals are in brackets. N=8,513. ***p<0.001, **p<0.01, *p<0.05, †p<0.10.
Next we move on to the models that account for pre-treatment exposure in the preceding 5 years (Column 3, Table 2) and 10 years (Column 4, Table 2). Recall that these models capture any treatment exposure during panel waves, excluding individuals who were pre-treated in the previous time period. That is, we use the Barney and Schaffner definition of the treated population but account for prior event exposure. Notice now that the coefficient of the DID estimate accounting for individuals pre-treated in the past 5 years is similar to what we reported in our original article: it is positive, statistically significant at the p < 0.10 threshold, and of similar effect size (individuals exposed to a mass shooting increase their support for gun control by 27 per cent relative to those in the control group). The treatment effect accounting for those pre-treated in the prior 10 years is also positive, though the effect is not statistically significant and its size is much smaller (amounting to just a 5 per cent increase in support for gun control).
Finally, we turn our attention to the fixed-effects ordinal regression models presented in Table 3. As before, we estimate four models that coincide with the different coding definitions of the treatment. Once again, the same patterns from Table 2 emerge: when we account for pre-treatment exposures, residing near a mass shooting has a positive, statistically significant and substantively meaningful effect on attitudes toward firearms restrictions. In fact, using the more conservative fixed-effects approach reveals that the effect of the treatment amounts to a 41 per cent increase in support for gun control, accounting for prior exposures in the preceding 5 years (Column 3, Table 3). We also plotted these treatment effects in Figure 3.Footnote 5
The effect of exposure to a mass shooting on preferences for gun control (fixed-effects estimator)
Notes: estimates (odds ratios with 90 per cent confidence intervals) from fixed-effects ordered logistic regression models with cluster-robust standard errors using the ‘bucologit’ command in Stata 15.
Fixed-effects regression results of the 2010–12 (3-wave) CCES panel data
Note: cell entries are estimates from fixed-effects ordered logistic regression models with cluster-robust standard errors using the user-written ‘bucologit’ command in Stata 15; standard errors are in parentheses; 90 per cent confidence intervals are in brackets. N=1,878. ***p<0.001, **p<0.01, *p<0.05, †p<0.10.
Extension Using the 2012–12 (2-Wave) Panel (N=19,533)
For completeness, we also extend our findings by analyzing data from the larger N 2010–12 (2-wave) CCES Panel Study (n=19,533; Ansolabehere and Schaffner Reference Ansolabehere and Schaffner2018).Footnote 6 The results from the DID random and fixed-effects ordinal regression models are presented in Tables 4 and 5, respectively. Given that these results are very similar to those we presented for the 3-wave CCES panel data, we will not discuss them here. It is relatively straightforward to see how these results compare in Figures 2 and 3.
Effect of exposure to mass public shooting, re-analysis of the 2010–12 (2-wave) CCES panel data
Note: DID events between waves. Cell entries are estimates from random-effects ordered logistic regression models with cluster-robust standard errors using the ‘xtologit’ command in Stata 15; standard errors are in parentheses; 90 per cent confidence intervals are in brackets. N=17,106 ***p<0.001, **p<0.01, *p<0.05, †p<0.10.
Fixed-effects regression results of the 2010–12 (2-wave) CCES panel data
Note: cell entries are estimates from fixed-effects ordered logistic regression models with cluster-robust standard errors using the user-written ‘bucologit’ command in Stata 15; standard errors are in parentheses; 90 per cent confidence intervals are in brackets. N=4,029. ***p<0.001, **p<0.01, *p<0.05, †p<0.10.
Conclusion
There is anecdotal evidence that people exposed to major traumatic events respond to them. For instance, American Psychologist devoted an entire special issue to cataloguing the serious psychological and social damage to individuals exposed to the 9/11 terrorist attacks.Footnote 7 For many people, mass public shootings are also life-changing events. For example, country musician Caleb Keeter survived the 2017 Las Vegas shooting that left fifty-nine people dead and more than 850 people injured (including 422 by gunfire). Immediately following the shooting, Keeter wrote to his Twitter followers: ‘I’ve been a proponent of the 2nd Amendment my entire life. Until the events of last night. I cannot express how wrong I was. Enough is enough […] We need gun control RIGHT. NOW. My biggest regret is that I stubbornly didn’t realize it until my brothers on the road and myself were threatened by it’.Footnote 8
Yet, identifying causal effects without random assignment to treatment and control conditions is fraught with difficulty, especially in a context such as this. There are multiple events that have taken place between panel waves, numerous pre-treatment exposures and media coverage that allows some form of exposure that potentially transcends physical location (which admittedly we do not account for in these analyses). Given these issues, we focused our efforts on the cross-sectional CCES and Pew datasets and were careful to avoid making strong causal claims. Instead, we have argued (and continue to maintain) that there is a modest but significant relationship between living near a mass shooting and preferences for gun control, even when re-analyzing the 2010–12 3-wave and 2-wave CCES panel datasets.
Our re-analysis illustrates the importance of researcher choice when defining the ‘treatment’ in the context of observational data where the treatment of interest may reoccur in time and space (that is, event chains). We find that when recent pre-treatment is not accounted for, proximity to mass shootings appears to exert little effect on attitude change; however, when recent pre-treatment is accounted for, proximity appears to be related to an increase in support for restrictions on guns. Ultimately, our re-analysis and extension contributes to the debate on methodological discussions about investigating observational data.
