BIG DATA AS LARGE N

Access to big data in the sense of large n rarely translates into a fundamentally improved ability to make causal inferences. Larger n is helpful for increasing the precision of estimates or the power of hypothesis tests, and it can allow for a wider range of estimation methods that would be unreliable with few observations (e.g., nonparametric methods). However, large n does not automatically remove or even alleviate what is often the most challenging step of empirical research: our ability to estimate consistently the parameters of interest and to make valid and robust statistical inferences.

An estimator that is inconsistent remains inconsistent regardless of how large the number of observations used. Indeed, the definition of an inconsistent estimator is precisely that even after we allow the sample size to go to infinity, the estimator is unlikely to be near the true value of the parameter. For example, no increase in the number of observations, no matter how large, will cause the omitted variable bias in a mis-specified linear regression model to disappear. This is a rather obvious but important point.

Perhaps a more subtle point is that even when consistent estimators or pivotal test statistics are available, increasing the sample size will not necessarily lead to more accurate approximations if we consider all relevant data-generating processes. For example, given a distribution generating the data, confidence intervals will have correct asymptotic coverage for a causal parameter if the probability that these intervals contain the true parameter is arbitrarily close to a predetermined confidence level when n is very large. However, this does not necessarily imply that there will be an n large enough to guarantee approximately correct coverage for all possible distributions that could have generated the data. In other words, even if fixing a data-generating process we can obtain confidence intervals with approximately correct coverage for some large n₀, there is no guarantee that n₀ will be large enough to ensure an equally accurate or better approximation for a different potential data-generating process.

This is the well-known distinction between pointwise and uniform convergence for large n. With only pointwise convergence approximations, increasing the sample size does not guarantee that we get closer to the true value of a parameter or the true coverage of confidence intervals in any meaningful practical sense. Consider, for example, post-model-selection inference. As discussed by Leeb and Pötscher (Reference Leeb and Pötscher2005), with consistent model-selection procedures, the probability of selecting the true model generating the data tends to 1 as the sample size increases. However, this asymptotic result is pointwise, not uniform, which implies that for any sample size—regardless of how large—the probability of selecting the true model will tend to zero when some of the parameters in the true model are sufficiently small. Thus, although it is possible to construct consistent estimators for the finite sample distribution of the post-model-selection estimator, these estimators are of very limited practical use because the lack of uniformity implies that this finite-sample distribution can be arbitrarily far from its pointwise asymptotic limit for any sample size.

BIG DATA AS LARGE P

A related but different understanding of big data centers on the larger universe of models that researchers can explore. The increasing availability of information has led to datasets that, instead of (or in addition to) having a large number of observations (n), have a very large number of variables (p) for each observation. In this interpretation, “big data” refers to datasets with large p, which in turn may refer to cases in which p is either smaller than n but very large relative to it or strictly larger than n.

The fundamental obstacle that we encounter when attempting to make causal inferences is that our units of analysis (e.g., voters, politicians, or businesses) typically choose the courses of action that they expect will be most beneficial. As a result, units that do or do not take certain actions may be very different from one another in potentially unknown but systematic ways. This problem, sometimes referred to as “selection bias,” is particularly pervasive in the social sciences due to the strategic nature of most social interactions.

As argued previously, big data as large n per se cannot solve the problem of self-selection. However, big data as large p brings more information. A large p dataset may contain information on a person’s preferred newspapers, political registration, recent travel history, fitness routine, social media contacts, campaign contributions, business transactions, and blog entries. Such a dataset surely will provide a more complete picture of the individual than a typical “small” dataset with only demographic and socioeconomic information. It is tempting to imagine that such massive quantities of new information will dramatically improve our ability to make causal inferences by allowing us to “control for” a very large set of variables—so large that the systematic preexisting differences among units will all but disappear.

Consider the following example. We want to understand whether the high rates of reelection of incumbent members of the US House of Representatives arise because they are the highest quality candidates or because, once in office, they have access to resources that scare off strong challengers and increase their popularity among voters. To establish which explanation is true, we may seek a measure of incumbency advantage net of intrinsic incumbent quality. A possible strategy is to measure the vote share of incumbent politicians and compare it to that of non-incumbent candidates of the same party. However, what if the incumbent anticipated a defeat and this anticipation prompted his retirement? And what if the anticipation of a tough battle makes it difficult for the party to recruit a strong non-incumbent candidate? The result might be that in some districts in which no incumbent is running, the party’s candidate may be weaker and the electoral circumstances more adverse than in incumbent-held seats—which likely would overestimate the true advantage enjoyed by incumbents.

This is when big data as large p might help. It may be implausible that candidate quality and retirement decisions are exogenous conditional on only party and previous vote share, but what if we had access to big data on politicians and could condition on all previous political speeches, public text messages, press reports, election forecasts, social media interactions, and travel history? Perhaps then we would capture enough information so that after conditioning on it our inferences would be valid. This idea is promising indeed because it implies that big data as large p eventually will allow us to solve automatically the fundamental problem of causal inference by “controlling for” massive amounts of information using sophisticated algorithms, computers, and statistical assumptions—all of which likely would be necessary to address the complications of large p inferences (see the following discussion).

Could an automatic and sophisticated “kitchen-sink” approach eliminate the need to rely on theories and research designs? I suspect not because there is one major catch. When the goal is causal inference rather than prediction or description, inference methods for large p datasets rely on the crucial assumption that the large set of variables included in the model is approximately equal to or, at least, includes all of the variables needed to make exogeneity plausible. In other words, given a large set of baseline variables, high-dimensional methods can be used to perform dimension reduction or to adjust for the potential biases induced by including many regressors. However, these methods will work only under the assumption that the original set of candidate variables includes all variables necessary for exogeneity to hold and excludes all variables affected by the treatment.

And how do we know if our large p dataset contains all relevant controls and excludes all posttreatment variables? We need either a theory about how the treatment affects the outcome of interest and the other variables in the model or a research design that justifies focusing on a particular subset of variables—and, if we want to maximize our chances of accumulating knowledge, we need both.

Moreover, even if we could identify a potentially large subset of our original large p dataset that includes those variables that make exogeneity possible and excludes those affected by the treatment (a big if), crucial restrictions and additional assumptions are needed to avoid the inferential challenges that otherwise would arise. For example, it is well known that when p is smaller than n but very large relative to it, the large number of incidental parameters poses considerable complications. Increasing the number of fixed-effects in nonlinear panel data models may lead to inconsistent parameter estimates (see, e.g., Hahn and Newey Reference Hahn and Newey2004), and even classical heteroskedasticity robust standard errors may be inconsistent in linear panel models (Stock and Watson Reference Stock and Watson2008). Similar issues may arise in cross-sectional linear models with many regressors (see, e.g., Cattaneo, Jansson, and Newey Reference Cattaneo, Jansson and Newey2012; Koenker Reference Koenker1988) and instrumental variable models with many instruments (see, e.g., Andrews and Stock Reference Andrews, Stock, Blundell, Newey and Persson2007). These issues may be addressed by imposing appropriate rate restrictions or bias corrections, but the important point is that increasing p leads to inferential challenges that cannot be ignored.

In addition, recent advances in econometrics have focused on developing uniformly valid methods for causal inference with high-dimensional datasets in which p is allowed to be much larger than n. These approaches begin with an extremely large number of variables, perform model selection to choose only those that are needed, and develop conditions under which valid inferences can be made after the model-selection step (see, e.g., Belloni et al. Reference Belloni, Chernozhukov, Fernández-Val and Hansen2013; Reference Belloni, Chernozhukov and Hansen2014; Farrell Reference Farrell2014).^{Footnote 4} However, these methods crucially require some type of data-reduction structure, such as sparsity—that is, the assumption that only some of the included variables appear in the true model. In other words, although these methods allow for a massive (i.e., larger than n) number of variables in the conditioning set, valid causal inferences cannot be obtained unless we impose restrictions on the way those variables affect the outcome of interest—restrictions that can be justified only by providing a strong research design or a strong theory.

Therefore, the need for theory and research design with big data as large p is not fundamentally altered, even if large p methods provide more flexibility by allowing researchers to use a very large number of variables. Causal inferences based on large p datasets still require the assumption that we have not omitted important variables and have not included posttreatment variables, the same assumption that is required when datasets are “small.”

Returning to our example, is the collection of all previous political speeches, public text messages, press reports, social media interactions, and international travel records sufficient to capture a politician’s inherent quality? What about the way he kisses babies at rallies, how she pronounces vowels, his ability to connect emotionally with voters, and her intelligence? The list could be expanded endlessly, and it is likely that many items on that list would be inherently unobservable. Without a theory and a research design, it is not possible to know when to stop adding to the list.

BIG DATA FOR DESCRIPTION, EXPLORATION, AND HYPOTHESIS GENERATION

That big data is no substitute for theory and research design does not mean that it has no benefits. At the very least, the ability to collect and analyze big datasets creates numerous possibilities for description, exploration, and hypothesis generation, all of which are crucial elements of scientific inquiry.

At the most basic level, big data allows us to systematically and quantitatively analyze phenomena that were previously unavailable. This leads to an increasing availability of “dependent” and “independent” variables that, together with modern machine learning tools, enhances our ability to provide a richer description of phenomena of interest and allows us to uncover empirical regularities previously unobservable. Because the ability to engage in systematic descriptive analysis often is a first step in developing scientific theories and explanations, this is one significant way in which access to big data can contribute to scientific progress.

A related benefit of big data is that as more phenomena become quantifiable, the range of implications of scientific theories that can be tested empirically is expanded significantly. For example, until recently, it was difficult to test systematically those theories that predict shifts in spoken or written political discourse. However, the increasing ability to convert text into data expands the number of predictions that can be tested (see, e.g., Grimmer Reference Grimmer2013). This represents another contribution of big data to scientific progress.

Moreover, the availability of big data—in particular, big data as large p—allows researchers to engage in systematic, large-scale statistical inference (i.e., the simultaneous exploration of hundreds and even thousands of hypotheses) with the goal of detecting a small subset that holds the most promise for a specific scientific explanation (see, e.g., Efron Reference Efron2010). The promise of this type of large-scale exploratory analysis is the ability to accelerate the pace of hypothesis generation. For example, in genetics, researchers may test hundreds or thousands of genes to detect a few that are most likely associated with a particular disease. After this small set of genes has been detected, efforts then can be focused on developing theories and designing specific clinical trials to develop a deeper scientific understanding of the causes of the disease. In the absence of large-scale exploratory analysis, each hypothesis would be studied with the same amount of detail, which may result in a prohibitively costly and slow process.

Finally, for these benefits of big data to materialize, we must ensure that the quality of big data satisfies certain standards, particularly in those cases in which it is created for direct business purposes. Many of the commercial sources of big data (e.g., Google, Twitter, and Facebook) are appealing in that they record detailed information about individuals’ social interactions, preferences, consumption decisions, and so forth—all of which can be valuable for scientific purposes. However, as noted by Lazer et al. (Reference Lazer, Kennedy, King and Vespignani2014), a major challenge in using commercial data is that the companies that produce it constantly make modifications to their data-collection algorithms in order to increase profits and support their business model. As a consequence, commercial data often are endogenously affected by a company’s business decisions rather than exogenously determined—which compromises their validity as a source of information. For example, the number of social media connections per individual might be used as a proxy for a person’s social network. However, if these connections are affected by the company’s algorithms to suggest new contacts and the algorithms evolve over time in ways unknown to the scientific community, this type of data might prove highly unreliable for scientific purposes. These potential challenges must be considered as commercial data becomes more common in scientific research.

To summarize, in combination with theory and research design (and precautions about commercial sources), big data can enhance and deepen our ability to explore and develop scientific explanations. However, the availability of big data per se does not constitute a structural breakthrough in our ability to make causal inferences. This is not because big data is limited but rather because the accumulation of scientific knowledge ultimately requires a theory of how and why phenomena occur as well as a research design to make valid causal inferences about the theory’s empirical implications. There are no algorithmic or automatic shortcuts to scientific discovery. In fact, the need for critical thinking will be stronger the more we become inundated with ever-larger amounts of new information.