BIG DATA CAN HELP US DESIGN BETTER EXPERIMENTS

We begin with what initially appears to be an obvious contradiction between causal inference and big data. Within the causal-inference community, the randomized experiment is generally considered the “gold standard” for the valid identification of treatment effects. Within the big data community, social media such as Twitter often comprise a default example—being familiar, ubiquitous, and undeniably big. Because data generated as the “exhaust” of social media are in every way observational and in no way experimental—that is, no manipulation of treatment, no random assignment of treatment and control—they appear irrelevant for scholars interested in causality. One way to utilize these data for causal inference is to treat them inductively, as a source of suggestive hypotheses that can be tested experimentally.

A recent pair of studies (involving two of this paper’s authors) to determine censorship mechanisms in China illustrates this possibility. In the first study, King, Pan, and Roberts [KPR] (2013) analyzed a corpus of more than 11 million Chinese social media and blog posts that were collected before and after the Chinese government censored them via content filtering.^{Footnote 2} KPR determined whether the posts were censored and then used supervised learning methods to compare the content of censored and uncensored texts. It was surprising that the authors found that Chinese censorship is focused on stopping discussion of collective action while allowing criticism of the state. In other words, Chinese citizens can view vitriolic online criticism of policies and lower-level officials, but posts that praise the government are censored if they discuss issues such as ongoing protests.

In the second study, KPR (2014) conducted a randomized experiment to validate this result, noting the limitations of interpreting the previous result as causal. In this experiment, 1,200 social media posts—which discussed ongoing collective-action and noncollective-action events supportive and critical of the government—were written and submitted with random assignment to 100 of the top social media platforms across China. That is, using another type of big data analysis, the authors validated the theory generated from their observational study with an experiment: they found a significant causal effect on censorship of posts related to collective action and no causal effect on censorship of those supportive or critical of the government. Both KPR studies used big data and big data technology in the service of social science research-design principles.

BIG DATA CAN PROVIDE EMPIRICAL LEVERAGE THROUGH PRECISE SUBPOPULATIONS

Big data is typically diverse data and often advantageous not for enormous sample sizes but rather for providing sufficient sample size on small subpopulations or even individuals. This is crucial in the implementation of methods for causal inference in observational data in which experiments are impractical or unethical. The central principle of methods such as matching is to compare treated and untreated observations that were as similar as possible before treatment. Achieving balance, however, requires pruning the data of observations, creating a tradeoff between bias and variance (King, Lucas, and Nielsen Reference King, Lucas and Nielsen2014). Higher standards for balance will reduce bias but may leave data too sparse for useful inference. The access in big data to fine-grained subpopulations can alleviate the bias–variance tradeoff.

In a recent example, Hersh (Reference Hersh2013) leveraged millions of individual-level observations to understand how people close to the victims of September 11 were influenced by the terrorist attack. From detailed individual-level data, he isolated two treatment groups—families of September 11 victims and neighbors of September 11 victims—and compared them to precisely matched control individuals. He found that, relative to the control group, those who were close to the victims of September 11 became more involved with politics in the years following the terrorist attack and also exhibited a conservative shift in their voting behavior.

In a similar way, the availability of large-scale, fine-grained data can enable tests of complex, rare, or subtle phenomena predicted by formal theory, in the manner of the National Science Foundation–led Empirical Implications of Theoretical Models (EITM) initiative (Granato and Scioli Reference Granato and Scioli2004). Consider, for example, Osorio’s (Reference Osorio2013) formal model of localized drug violence in Mexico. Among other phenomena, the model implies that increased democratization will lead to greater law enforcement and that greater law enforcement will lead to greater violence among drug-trafficking organizations. Testing these predictions requires very detailed data about relatively uncommon events—by municipality, by day, by actor—data that Osorio generated from Spanish-language newspapers and that described a quarter-million events in the Mexican war on drugs during 10 million municipality-days.

In a similar way, Ansolabehere, Hersh, and Shepsle [AHS] (2012) used data about almost 2 million individuals to test subtle predictions about voter registration. A positive correlation between voter registration and age has long been known but attributed to habit or other social psychological explanations. Instead, the AHS model accounts for this through the dynamics of voter mobility. The model can be differentiated from alternative explanations in predictions about the shape of the relationship—a distinction that can be detected only in massive individual-specific data. As in the Osorio study, we observe a theory-driven use of big data, in the spirit of EITM, which otherwise would not have been possible.

BIG DATA CAN REVEAL BEHAVIOR THAT PREVIOUSLY WAS DIFFICULT TO OBSERVE

Another feature of the KPR studies discussed previously is the use of big data to reveal behaviors—that is, censored communications—that are difficult to observe because they are actively hidden. In some cases, “Data” initially becomes “Big” through less strict selection and censoring mechanisms than other data. When this is true, there is potential to observe new behaviors of theoretical interest.

Racism and other forms of out-group hostility are famously difficult to research directly; however, more indirect approaches can be fruitful. Recent work by Stephens-Davidowitz (Reference Stephens-Davidowitz2014a), for example, examined Google searches conducted during the 2008 US presidential race. He found that certain parts of the country were more likely to use racial epithets in conjunction with searches on Barack Obama’s name—patterns that standard social science survey techniques failed to detect. Linking these search results with voter returns, Stephens-Davidowitz found strong evidence that racial hostility cost Barack Obama significant vote shares. In a similar search-based strategy calibrated with Craig’s List ads and social media data, Stephens-Davidowitz (Reference Stephens-Davidowitz2014b) estimated that 5% of American men are gay and that social intolerance keeps 50% to 80% of gay men “in the closet,” a previously unknown statistic. Of course, social media and similar data reflect only the population from which they were extracted (DiGrazia et al. Reference DiGrazia, McKelvey, Bollen and Rojas2013; Nagler and Tucker 2015). As with nonresponse to telephone surveys and opt-in to Internet surveys, the validity of these estimates depends on our ability to convincingly model and calibrate processes of selection to the population of interest. This is one strength of a social science approach to big data. A landmark example of this strength is provided by Lazer et al. (Reference Lazer, Kennedy, King and Vespignani2014), who demonstrated that “Google Flu Trends” is not the valid leading indicator of flu incidence that it initially appeared to be.

Formal theory can point to the conditions under which a selection process may render a behavior unobservable in conventional data but observable in less-constrained data. Much of social choice theory, for example, is built on the fundamental ubiquity and instability of multidimensional preferences. The empirical relevance of this for the study of the US Congress, for example, appears to be muted by ideal point estimates that find voting to be well summarized in the modern era by one dimension (Poole and Rosenthal Reference Poole and Rosenthal1991). Conversely, there are theoretical models that suggest mechanisms (e.g., negative agenda control) will be used by leaders to suppress the consideration of bills along higher dimensions, with the empirical side effect of suppressing the detection of multidimensional preferences in legislative voting data (Dougherty, Lynch, and Madonna Reference Dougherty, Lynch and Madonna2014). Recent work argues that legislative-speech records are less constrained by majority agenda-setters and can reveal higher dimensions, such as distributive preferences induced by electoral systems defined by geographic districts (Monroe, Colaresi, and Quinn Reference Monroe, Colaresi and Quinn2008) or by multidimensional political phenomena such as heresthetical maneuvers of opposition parties (Tzelgov Reference Tzelgov2012). Similar to the findings of Patty and Penn (Reference Patty and Penn2015), in these examples, we observe measurement from big data, informed by and in the service of theory.

Indirect effects in social contexts are invisible in conventional data but can be observed in big data. For example, when studying how friends influence one another, experimental design possibilities for estimating direct effects are well understood; however, it is more difficult to design experiments to estimate indirect effects without partnering with big data. Suppose we want to estimate the effect of a friend casting a ballot on whether another voter will cast a ballot. Whereas it is possible to directly deliver a stimulus about voting to the friend, the researcher must rely on the friendship structure and the size of big data to estimate the indirect effect. In a recent example, Bond et al. (Reference Bond, Fariss, Jones, Kramer, Marlow, Settle and Fowler2012) conducted such an experiment by partnering with Facebook to include 61 million individual users. Their experiment found that, indeed, a Facebook friend may influence another to cast a ballot. Other social experiments have also relied on big data—for example, a large experimental population gleaned from voter registration lists or geographic mapping—to estimate indirect social effects (Nickerson Reference Nickerson2008; Sinclair Reference Sinclair2012).

CONCLUDING REMARKS

It is obvious that big data is here to stay. We believe that the strongest contribution to knowledge will come when we harness both the power of big data and rigorous methods and theories from social science. Given the theme of this symposium, we focus on big data’s ability to help us (1) design better experiments, (2) make better comparisons between precise populations of interest, and (3) observe theoretically relevant social and political behavior that was previously difficult to detect. Beyond the relationship with formal theory and causal inference, big data offers other possibilities to enhance what we have done before or to enable us to do new things (Monroe Reference Monroe2013; Varian Reference Varian2014).

Of course, we are fully aware of the strong critiques suggesting that current big data analyses are driven by atheoretical inquiries, or that many causal claims are being made solely by invoking “big data” without rigorous analysis or an understanding of the underlying assumptions. This criticism undoubtedly is true. It also is true that the public and policy makers have been swept up in the excitement about big data and that many are convinced by claims generated from big data research without sufficient intellectual scrutiny. As social scientists, this obviously is frustrating.

However, rather than dismissing these trends or disengaging, social scientists should view this juncture as a significant opportunity. Big data has the power to transform and expand the universe of answerable social science questions; as social scientists, we can and should shape the direction of big data analysis. Ultimately, much of big data is simply a richer version of social science data—data about how humans behave and interact—for which we already have developed a conceptual understanding. Political scientists have spent decades to hone, test, and perfect the appropriate methods for asking and understanding these types of questions, and we have a central role in integrating these insights into the world of big data.