BIG DATA: DEFINITIONS AND FORMS

Economic historians accustomed to the problem of comparing prices and quantities over long periods of time will recognize that the “big” in “big data” needs a clear metric. A key characteristic of modern “big data” is that the volume of stored data exceeds human analytic capacity and pushes against the boundaries of currently-available computing power. For that reason, the magnitude of “big” is continually growing. The National Academies of Sciences, Engineering, and Medicine and the National Institutes of Health have described the recent growth of scientific data as a “deluge” (Anderson Reference Anderson1997; National Research Council 2013), though the quantity of printed data produced by governments in the nineteenth century has also been termed an “avalanche” compared with what was available previously (Hacking Reference Hacking1982). The problem of analyzing an ever-growing trove of data is not new in kind, though its scale has changed. To put the issue in perspective, in 1997 the National Academies published a report on Massive Datasets in which the social sciences were represented by a chapter on the computational difficulties of tabulating frequencies and running ordinary least square (OLS) regressions, with equipment available at the time, on a 5 percent sample of the U.S. census from 1990, a dataset of 12.5 million records (Anderson Reference Anderson1997). By contrast, in a recent article in this Journal, Brian Beach, Joseph Ferrie, Martin Saavedra, et al. (Reference Beach, Ferrie and Saavedra2016) report analyses based on the linked records of more than 8 million individuals from complete databases of the 1900 and 1940 U.S. censuses, a considerably more computationally intensive task. It is now becoming common for scholars to work with datasets of 40 million or more records, whether from one source or pooled in a common format from many (Aaronson, Dehejia, Jordan, et al. Reference Aaronson, Dehejia and Jordan2017; Gutmann, Brown, Cunningham, et al. Reference Gutmann, Brown and Cunningham2016). As with “top incomes,” what constitutes “big data” varies over time.

Economic history has traditionally relied on the survival of original sources in manuscript form. Source survival has been both deliberate and accidental. The way in which past actors and societies chose to preserve some records and destroy others in itself provides useful information about their values (Ashplant and Wilson Reference Ashplant and Wilson1988). Storing and preserving records is costly; large-scale record collections that have been preserved reflect what those in the past considered to have continuing value for their own activities or for posterity. These collections tended to be those of powerful individuals or institutions, such as states or churches, both of which have historically collected information about people and their property, though that information originally served the purposes of taxation, governance, and salvation, rather than scientific analysis (Hall, McCaa, and Thorvaldsen Reference Hall, McCaa and Thorvaldsen2000). Economic historians in Britain, Canada, and the United States have made ample use of the population censuses in their research, which is highlighted in this article. Yet in Australia, Ireland, and New Zealand, which shared much in common with these countries (Lloyd, Metzer, and Sutch Reference Lloyd, Metzer and Sutch2013), census authorities sometimes deliberately destroyed the manuscripts after the material had been tabulated. Even in countries where records were routinely preserved, some were lost to war, fire, or neglect after surviving for decades (Blake Reference Blake1996; Dorman Reference Dorman2008).

Historically, the quantity of data available for scholarly research has increased as the costs of storing information have decreased. The falling cost of paper in the nineteenth century facilitated the storage of increasing numbers of records by governments and businesses, requiring in turn a large number of clerical workers to organize, analyze, and physically move the records. In the twentieth century, the cost of reproducing paper records declined, leading to a further increase in the volume of material that could be stored.

Contemporary interest in the scholarly potential of “big data” analysis derives from large reductions in the costs of producing, sharing, and storing electronic information since the late 1990s. As more data are “born digital”—that is, created and stored electronically from the outset—the cost of production, storage, and sharing are further reduced. However, the increasing availability of electronic data engenders some of the same core challenges faced by scholars and archivists in earlier eras when societies chose to store increasing quantities of analog information: organization and preservation in a format that allows useful analysis. Discussing the costs and benefits of processing into the archives 1,500 boxes of records from a former Illinois governor, one archivist lamented in the 1980s that “if it cost nothing to access and preserve records, we could save everything,” but on further examination concluded that “much of the data resemble more the noise and distortions of a badly tuned television set than useful information” (Ham Reference Ham1984). When information storage is costly, the task of distinguishing signal from noise falls to archivists and other data preservation experts. When storage is cheap and more information can be preserved, scholars become increasingly responsible for distinguishing signal from noise, opening opportunities for the discovery of unexpected signals in what might have otherwise been discarded as noise.

So far we have defined “big data” only in terms of logical size and only in relation to computing resources (Gandomi and Haider Reference Gandomi and Haider2015; Ward and Barker Reference Ward and Barker2013). Every author who surveys the landscape of big data agrees that volume of data is a critical part of the definition, but few have discussed the specific ways in which volume of data has increased. Overall, the volume of data available to scholars has increased because more people and other entities of interest are undertaking activities that generate stored data, meaning that more information is being collected for more people and about more activities. It is important to note, however, that some big datasets may be highly self-selected when individuals themselves decide whether or not to participate in the data generating process, particularly in processes that generate born-digital data (Hargittai Reference Hargittai2015), as will be discussed later at greater length.

Volume of data increases not only because more datasets are being created and stored, but also because those datasets include more observations and more characteristics of each observation. More observations can produce an effect akin to increased sample density, or can even produce datasets that contain the entire population of interest (Chetty, Stepner, Abraham, et al. Reference Chetty, Stepner and Abraham2016; Ruggles Reference Ruggles2014). More observations can also mean more frequent data collection, which is often referred to as velocity in discussions of big data. The same tools that allow more people to participate in data generating process, and that allow for higher-density and more frequent samples, such as internet-based surveys and mobile devices that track activity, also make it possible to collect data on more characteristics of the activity or entity of interest, often referred to as variety. We can think of variety as the collection of more variables and velocity as the collection of more observations in the familiar format of a dataset of rows and columns of text and numbers. However, variety also refers to the increasing collection of different data formats, for example, video or images. Still images have become particularly important to the research process in many fields, including economic history, as they allow researchers to work with primary materials away from the data collection site, an aspect of the research process we discuss in the final section. High resolution images require a significant amount of disk space, thus returning us to volume as a defining characteristic of big data.

USES OF BIG DATA IN ECONOMIC HISTORY

Population and the environment are two areas where economic historians are already making use of big data. The collection and analysis of big data are not new activities. Governments have taken complete censuses since the eighteenth century, tabulating and printing the results for the entire population after each one. Researchers have extensively employed these aggregate data particularly for studies of the nineteenth century. In some countries the census tabulated quantities of interest at subnational levels, permitting analysis of geographically grouped data (Easterlin Reference Easterlin1976; Fleisig Reference Fleisig1976; Humphries Reference Humphries1987). Aggregate data on entire populations are typically structured as counts or averages over geographic areas such as counties or municipalities, so their size depends on the administrative geography of a given country. To put their size in perspective, they might be on the order of several hundred (e.g., 659 municipalities in Norway in 1910) to several thousand observations (e.g., 13,565 parishes in England and Wales in 1881). These data are big in the sense of describing large populations, though they reduce individual variation to a relatively small number of observations describing geographical aggregates or averages.

Economic historians have made ample use of individual-level data from high-density samples of complete population data, such as the IPUMS 1 percent and 5 percent samples of the United States census (Ruggles, Genadek, Goeken, et al. Reference Ruggles, Genadek and Goeken2015).Footnote ¹ Such samples have allowed for more precise measures of human behavior, for example, with individual family size replacing aggregate child-woman ratios as a measure of fertility. Since the microdata revolution of the 1970s, more individual-level samples of censuses, social surveys, and civil registration records have become available for many countries, with increasing sample density. The computational resources for analyzing them have also improved. These datasets were large for the time in which they were created; as noted earlier, just 20 years ago, analyzing a 5 percent sample of one census with 12.5 million records stretched the computing capacity of the modal scholar.

Population aggregates and samples of individual-level census data have provided vital sources in economic history, for example, in studies of slavery, Reconstruction, and civil rights in the United States. In the 1950s, Alfred Conrad and John Meyer (Reference Conrad and Meyer1958) used census aggregates for their research; in the 1960s, scholars developed samples of census data to study slavery (Fogel and Engerman Reference Fogel and Engerman1974) and Reconstruction (Goldin Reference Goldin1977). Since the 1990s, the IPUMS has been a critical data source to explain the slow convergence of African American and white economic status (Boustan Reference Boustan2009; González, Marshall, and Naidu Reference González, Marshall and Naidu2017; Johnson Reference Johnson2004; Sundstrom Reference Sundstrom2007).

Today, a prominent form of big population data is the complete universe of individual-level census records. Between 2003 and 2013 the North Atlantic Population Project (NAPP) made available full-count microdata for censuses of Canada (1881); Denmark (1787, 1801); Great Britain (1881, 1911); Norway (1801, 1865, 1900, 1910); Sweden (1880, 1890, 1900); the United States (1880); and Iceland (1703, 1729, 1801, 1901, 1910) (Minnesota Population Center 2015a; Ruggles, Roberts, Sarkar, et al. Reference Ruggles, Roberts and Sarkar2011), totaling just over 100 million individual records. Since 2013 the availability of complete count censuses has expanded dramatically, including every United States census from 1790 to 1940 (except 1890), every British census from 1851 to 1911, and the 1901 and 1911 Irish censuses. Just under one billion historical census records from North America and Europe are now available for research through IPUMS and the NAPP.Footnote ² By 2020 all records will have standardized codes for economically relevant variables such as occupation and industry, nativity, educational attainment, and place of residence. In addition to these historical records, which come with reduced privacy restrictions allowing researchers to see the names of individuals, the complete short- and long-form versions of the United States census since 1960 are now available to researchers in the Census Bureau's Federal Statistical Research Data Centers.Footnote ³ As sample density reaches its logical limit of complete individual records, the quantity of census data an economic historian can analyze currently pushes the limits of computational power for more complex procedures such as record linkage or duration modeling (Ruggles Reference Ruggles2014).

Given that analysis of samples of the census have been so fruitful, what are the scholarly benefits to working with bigger data and specifically complete-count census datasets? Two important advantages of big data for economic historians are precision and the potential to link observations.

At the limit with population data, we can ignore sampling error, which can make it impossible to say anything definitive about less-common phenomena when working with smaller samples, and focus on the substantive interpretation of statistical results. Complete population data make it possible to analyze small subgroups either by themselves or in comparison to other populations. For example, while the 1.9 million Irish and 1.9 million German-born migrants in the United States in 1880 can be adequately studied with a 1 percent sample, important but smaller groups, such as the 125,000 French-born migrants in the same year, are harder to study using even a 5 percent sample. Marginal totals and main effects can be derived, but interactions would evaporate in a “welter of empty cells” (Ruggles and Menard Reference Ruggles and Menard1995). Recent scholarship taking advantage of this increased precision has examined assortative mating among immigrants from specific countries in U.S. cities (Logan and Shin Reference Logan and Shin2012) and interracial marriage in the United States (Gullickson Reference Gullickson2006).

Additionally, complete population data often identify people's location quite precisely, allowing researchers to examine social and economic processes at small geographic scales. Geographic precision in census data derives from the original and fundamental purpose of censuses: to provide data to governments for administration and apportionment of populations to political boundaries. Scholarship that utilizes geographical precision includes work on residential segregation in the United States (Logan and Zhang Reference Logan and Zhang2012; Logan and Parman Reference Logan and Parman2017) and the influence of kin in neighboring households on fertility (Hacker and Roberts Reference Hacker and Roberts2017).

The availability of full-count census data increases the prospects of linking individuals across time to better address questions about behavior and causality. Linkage is possible between complete count censuses (Beach, Ferrie, Saavedra, et al. Reference Beach, Ferrie and Saavedra2016), between a complete count and a sample (Long and Ferrie Reference Long and Ferrie2013), and between a complete census and some other source (Bleakley and Ferrie Reference Bleakley and Ferrie2016; Roberts and Warren Reference Roberts and Warren2017). Linking individuals across time to examine inter- and intra-generational economic mobility, for example, allows scholars to revisit classic questions in American economic history about the degree to which new institutions and environment in the new world allowed individuals to fare better than their parents had done in terms of economic status. The hypothesis that the United States (and similar settler societies such as Canada, Australia, and New Zealand) had a high degree of intergenerational mobility dates to contemporary arguments in the nineteenth century (Turner Reference Turner1893), and generated research in the 1960s and 1970s by economic and social historians who developed longitudinal data on individual economic progress in specific cities and towns (Katz Reference Katz1975; Knights Reference Knights1971; Pearson Reference Pearson1980; Thernstrom Reference Thernstrom1973, Reference Thernstrom1964). In the 1990s, Ferrie created more representative samples using indices (Jackson Reference Jackson1992) to the American census to search over a wider geographic space (Ferrie Reference Ferrie1994, Reference Ferrie1999) for migrants who had moved out of their community of origin. Subsequently, Jason Long and Ferrie (Reference Long and Ferrie2013) have shown using parallel samples of American and British men that the level of intergenerational mobility among these men in the United States was high in the nineteenth century, a contrast to the more pessimistic conclusions of earlier scholars. However, in the twentieth century the relationship reversed, with more mobility in Britain. Thus, big data have helped refine our understanding of the changing relationship between place, migration, and occupational mobility.

Ideally, record linkage would be based on complete population listings from each time point. As an intermediate step to this goal, the IPUMS project developed a set of Linked Representative Samples, in which individuals in the 1 percent samples of censuses for 1850–1870 and 1900–1930 are linked to the 1880 full-count census. Linked data are particularly useful for the study of geographic mobility (Ferrie Reference Ferrie2005), occupational mobility (Abramitzky, Boustan, and Eriksson Reference Abramitzky and Boustan2014; Long and Ferrie Reference Long and Ferrie2013, Reference Long and Ferrie2007), socioeconomic mobility (Long Reference Long2005), and even mobility between census racial categories (Saperstein and Gullickson Reference Saperstein and Gullickson2013), a phenomenon that would be difficult to track with even a relatively high-density sample of the population. Studies of mobility are not limited to the United States. Taking advantage of the availability of full-count census data for Norway in 1865 and 1900 and a dataset for the entire Norwegian-born population of the United States in 1900, Ran Abramitzky, Leah Boustan, and Katherine Eriksson examine the relationship between inheritance and the decision to migrate from Norway to the United States (Reference Saperstein and Gullickson2013), and estimate the economic returns on migration for those who made that decision (Reference Saperstein and Gullickson2012).

Censuses, by definition, record information for full universes of population, usually at the national level, but they are not the only source of universal data. Administrative records can also produce data about complete populations, though usually at a smaller scale. Administrative data often represent continuous rather than periodic collection, allowing scholars to answer questions about demographic processes at a very fine level of detail. For example, using German social security system data for 61.4 million individuals to calculate the daily number of births between 1920 and 1989, Thomas Bauer, Stefan Bender, Jörg Heining, et al. (Reference Bauer, Bender and Heining2013) examine the relationship between births, lunar cycles, and sunspots, finding that the lunar cycle does not affect the number of births but that births and the number of sunspots are positively correlated. Oriana Bandiera, Imran Rasul, and Martina Viarengo (Reference Bandiera, Rasul and Viarengo2013) use administrative data on the full universe of 24 million immigrants who entered the United States through Ellis Island between 1892 and 1924 to re-estimate migration flows in and out of the United States. They find that immigration was considerably more prevalent between 1900 and 1920 than the official statistics suggest. By comparing these administrative records to the number of immigrants found in the censuses of 1900–1920 and accounting for expected mortality, they estimate that out-migration during this period was more than twice as frequent as recorded in official estimates.

Population registers are another valuable and heavily used source of universal administrative data. They were often a product of church and state co-operation, collected to monitor, more or less continuously, the residential location of populations (Bengtsson, Campbell, and Lee Reference Bengtsson, Campbell and Lee2004). Population registers provide today's scholars with a continuous record of the stock and flow of administratively defined historical populations, facilitating the exploration of questions that require large numbers of records or intergenerational linkages. For example, the China Multi-Generational Panel Dataset, publicly available at the Inter-University Consortium for Political and Social Research (ICPSR), includes the records of 370,000 people between 1749 and 1913 (Lee and Campbell Reference Lee and Campbell2016; Lee, Chen, Campbell, et al. Reference Lee, Chen and Campbell2017).Footnote ⁴ Sweden is, perhaps, the country with the longest run of population register data. Swedish data have been used to examine the relationship between childbearing and longevity (Barclay, Keenan, Grundy, et al. Reference Barclay, Keenan and Grundy2016), the effect of birth order on mortality (Barclay and Kolk Reference Barclay and Kolk2015), and the effects on fertility of size of family of origin (Kolk Reference Kolk2014). Digitized population registers have allowed for similar studies in Norway (Grundy and Kravdal Reference Grundy and Kravdal2014, Reference Grundy and Kravdal2010). As these registers continue in computational forms and are automatically updated, they become truly big data (Dribe and Helgertz Reference Dribe and Helgertz2016).

In North America there is no history of continuous population registration.Footnote ⁵ Genealogical projects have produced multigenerational datasets for the populations of Quebec in Canada (Dillon, Amorevieta-Gentil, Caron, et al. Reference Dillon, Amorevieta-Gentil and Caron2017) and Utah in the United States. The BALSAC database, managed by the Université Laval, McGill University, and Université de Montréal, was created by linking marriage, birth, and death certificates in Quebec from the seventeenth century to the present (Université du Québec à Chicoutimi Reference Bailey2017). It currently includes five million individuals over four centuries. The Utah Population Database (UPDB) includes over 7.7 million descendants of those who experienced a vital event on the Mormon Trail and is linked to a host of other medical and administrative records (Smith and Huntsman Cancer Institute Reference Smith2017). It is currently maintained by the University of Utah and is updated annually. In common with population registers, big genealogical databases such as BALSAC and the UPDB are particularly suitable for answering historical questions about intergenerational social mobility or transmission of fertility behavior (Gagnon, Tremblay, Vézina, et al. Reference Gagnon, Tremblay and Vézina2011; Jennings, Sullivan, and Hacker Reference Jennings, Sullivan and David Hacker2012; Maloney, Hanson, and Smith Reference Maloney, Hanson and Smith2014). These genealogies also provide an essential source of data for the study of inheritance of health and disease (Broeckel, Hengstenberg, Mayer, et al. Reference Broeckel, Hengstenberg and Mayer2007; Kerber, O'Brien, Smith, et al. Reference Kerber, O'Brien and Smith2001). Martha Bailey's LIFE-M project is now developing geographically broader population linkages of vital records and census data for the late nineteenth and twentieth century United States (Bailey Reference Bailey2017).

Certainly, economic history has long featured analysis of individual-level data and longitudinal data. The practice of creating a dataset with observations from different time points, or from a variety of data sources, has merely changed in degree, rather than in kind. The Cambridge Population History of England, for example, created linked datasets from parish records, working in conjunction with volunteers in parishes around England to abstract the data from the original manuscripts (Wrigley and Schofield Reference Wrigley and Schofield1981). The continued use of the resulting data attests to their value (Boberg-Fazlic, Sharp, and Weisdorf Reference Boberg-Fazlic, Sharp and Weisdorf2011).

In addition to linking individuals across time, economic historians have created big population datasets by pooling data from multiple censuses, genealogies, or population registers. For censuses, this process was facilitated by the NAPP, described earlier, and the IPUMS-International project, which provides harmonized individual-level census data since 1960 for 82 countries, including a total of 614 million person records. Pooled microdata permit analysis of both within-country and between-country variation, as well as cross-sectional change over time. Recent studies using such data have examined the relationship between socioeconomic status and fertility at the turn of the twentieth century (Dribe, Hacker, and Scalone Reference Bengtsson, Dribe and Quaranta2014) and international variation and changes over time in the living arrangements of the elderly (Ruggles Reference Ruggles2009). Combined data from genealogical databases have been used to examine the relationship between fertility, aging, and mortality across populations (Eijkemans, Van Poppel, Habbema, et al. Reference Eijkemans, Poppel and Habbema2014; Gagnon, Smith, Tremblay, et al. Reference Gagnon, Smith and Tremblay2009). A landmark project using population register data from Belgium, Sweden, China, Japan, and Italy—the Eurasia Project—has explored several demographic processes, including mortality, fertility, and nuptiality, teasing apart biological universals and cultural differences (Bengtsson, Campbell, and Lee Reference Bengtsson, Campbell and Lee2004; Lundh and Kurosu Reference Lundh and Kurosu2014; Tsuya, Wang, Alter, et al. Reference Tsuya, Wang and Alter2010).

“Big population data” most often refers to large collections of individual microdata, but aggregate-level population data can be big as well if they cover a long period of time and aggregate over many relatively small spatial units. Ian Gregory, Jordi Marti-Henneberg, and Francisco J. Tapiador (Reference Gregory, Marti-Henneberg and Tapiador2010) have created a database that points in this direction, mapping regional-level population data for European censuses from 1870 to 2000, facilitating the long-term analysis of population change throughout Europe at the sub-national level. This geographic information systems (GIS) database includes decadal population data from 1870 to the present, interpolated to the boundaries of 562 intermediate-level administrative units currently used by the statistical office of the European Community. While this database does not qualify as “big” in terms of number of observations or computational requirements, it suggests that it is possible to go one step further and to divide aggregate population data into small gridded units, using sophisticated interpolation techniques. One example is the Fourth Version of the Gridded Population of the World, which distributes population over a 30 arc-second grid, representing approximately 1 km square at the equator (http://sedac.ciesin.columbia.edu/data/collection/gpw-v4). These data describe population at five-year intervals from 2000 to 2020; the high resolution of the grid dramatically increases the granularity of spatially-oriented population data of use to researchers. The potential for linkage of small-scale population data to other factors, especially the environment, has also been brought to the forefront by the Terra Populus project which, when completed, will make available yet more high resolution historical population data that are and can be linked to environmental and other data (Minnesota Population Center 2015b).Footnote ⁶

Several recent articles in this Journal demonstrate how big population data are changing the practice of economic history, and will continue to do so. William Collins and Marianne Wanamaker (Reference Collins2015) address a perennial question in American economic history: How the Great Migration improved economic outcomes for African Americans. They begin with a 1 percent sample of the 1910 United States census, from which they link forward 26,829 out of an initial sample of 111,524 southern males aged 0–40. Although a recently published article, some of Collins and Wanamaker's research methods have already aged slightly, showing how quickly big data are changing economic history. Collins and Wanamaker searched digital genealogical indices of the 1930 census to link with the 1910 sample; today linkage between publicly available files of the complete 1910 and 1930 censuses could create a much larger sample. They find a low degree of selection, suggesting wide participation in migration out of the South. However, they also found that white and black movers of comparable skill level and background made significantly different destination choices. As Collins and Wanamaker note in conclusion, longitudinal data are fundamental to understanding migration decisions. Cross-sectional data collected on migrants at their destination cannot show migrants' status and conditions before they departed, which are necessary to understand why people chose to migrate. This article is representative of the frontier of economic history research on late nineteenth and early twentieth century migration, with scholars pursuing similar research strategies in studies of British (Long Reference Long2005) and Norwegian (Abramitzky, Boustan, and Eriksson Reference Abramitzky, Boustan and Eriksson2012) migrants, both domestic and international.

While Collins and Wanamaker show how big data can be used to link people across time, another recent article in the Journal demonstrates how big data can be used to measure concepts with greater precision. Trevon Logan and John Parman (Reference Logan and Parman2017) return to another important topic in American economic history: residential segregation of black and white households. Logan and Parman use the complete individual-level returns of the 1880 and 1940 censuses to construct a new measure of segregation that is based on the race of a household's next-door neighbors. Previous studies of historical segregation had used racial composition at no smaller than the ward level, because wards were the smallest unit for which population by race was consistently reported (Cutler, Glaeser, and Vigdor Reference Cutler, Glaeser and Vigdor1999). Logan and Parman's measure shows that segregation increased substantially between 1880 and 1940. The chance of a black household having a white neighbor declined 25 percent in 60 years, with similar changes across the entire United States. While previous studies with aggregate data (Cutler, Glaeser, and Vigdor Reference Cutler, Glaeser and Vigdor1999) had documented the overall rise in segregation, Logan and Parman are able to show precisely where segregation was more prevalent, and where it grew more rapidly. As Logan and Parman acknowledge, their measure of segregation is simple, relying only on the immediate two neighbors of a household. More complex measures of segregation could use a wider window of households/neighbors or include measures that incorporate indicators of socioeconomic status. As more complete count datasets from population censuses become available, we expect economic historians will become increasingly creative in their construction of new measures of social and economic behavior.

Increased granularity in population data facilitates linkages to environmental data, which have also become available at finer levels of geographic and temporal detail. Data about the environment are well poised to become “big data” because of their potential for large-scale coverage, high frequency through time, and high levels of geographic granularity. Put another way, data about the environment, recorded frequently and divided into relatively small spatial units, are increasingly available for large swaths of territory. Good examples are records of weather through time and across space, which have been recorded systematically for many parts of the world since the nineteenth century, and land cover, which has been systematically documented in the United States by aerial photographs since the 1930s and globally by satellites since the 1970s. Agricultural land use data, which overlap conceptually with land cover change data, also fall into this category, with data collected at the county level for the United States since the mid-nineteenth century and similar data collected elsewhere over various periods of time. Other large collections of environmental data document the distribution of soils and the elevation and slope of terrain, all useful for understanding the environmental context in which economic activity takes place.

As is the case with other types of data discussed here, most historical environment and agriculture data were originally collected and available in analog (paper or film) formats, with an increasing fraction of those converted to digital. An early example is the Parker-Gallman study, based on a sample of data digitized from manuscript records of the 1860 U.S. Census schedules for population, slaves, and agriculture for 405 counties in what would become the Confederate states (Parker and Gallman Reference Parker and Gallman1991; Parker Reference Parker1970). At state and county levels, Michael Haines (Reference Haines2010) has digitized published volumes of the U.S. Census of Agriculture as well as the Census of Population, providing a valuable data source about land cover, crop yields, and livestock rearing. These datasets, though they continue to be well-utilized (Olmstead and Rhode Reference Olmstead, Rhode, Collins and Margo2015), are not what we would term “big.” William Parker and Robert Gallman digitized only a sample of the 1860 Census manuscripts for the counties in question, which was no trivial task given the technology of the period. While the Haines dataset includes all U.S. counties in all censuses, population censuses have been taken only at 10-year intervals (the agricultural censuses since 1920 have been more frequent, generally at five-year intervals), and the United States today contains only 3,144 counties and county equivalents. Neither dataset included all variables in the analog sources from which they were digitized. Historical weather data have also traditionally been digitized from the paper sources in which they were originally published, but here too both print publication of the original sources and the process of digitization have imposed limits on the volume of data that could be collected and processed.

Increasingly, environmental and agricultural data are becoming available at finer levels of granularity with respect to both space and time. The introduction of digital methods of data collection and preservation have facilitated the availability of more detailed environmental and agricultural data, resulting in data that are “born digital” and are limited neither by the constraints of publication nor by those of digitization. The U.S. agricultural censuses are an excellent example. Like other census-type data, they were published as county-level aggregates in books through the 1970s, succeeded by digital versions on CD, and have been available for internet download since the 1980s. Similarly, weather data for the United States are now available in born-digital form, in some cases with frequencies as great as hourly for individual stations (National Centers for Environmental Information 2016).

By definition, virtually all environmental data are spatial; each data cell in a table represents the attributes of some piece of the earth, water, or atmosphere, located in three-dimensional space. Because of those characteristics, and with the assistance of modern GIS technologies, it is possible to manipulate and eventually to subdivide the data. Data that begin, for example, as the attributes of weather at a given moment at a given group of weather stations can be interpolated into the attributes of weather for grid cells of almost any size, often as small as a single square kilometer or a single degree (or less) of latitude and longitude on a side. Those millions or billions of data cells representing, for example, the temperature or precipitation in a one-kilometer grid cell for every minute or hour of an extended period of time, constitute genuinely “big data” (Daly, Gibson, Taylor, et al. Reference Daly, Gibson and Taylor2002; Daly, Halbleib, Smith, et al. Reference Daly, Halbleib and Smith2008). They can be used as they are or re-aggregated in other ways to capture the weather characteristics of a city, county, or some other spatial unit. These processes are analogous to the creation of the gridded population data described earlier.

Spatially defined data also make it possible to integrate various types of information into a single framework, based either on common geospatial definitions (such as counties in the United States) or on a gridded approach. Valuable integrated data sources have recently become available, including the Great Plains Population and Environment Project's data at ICPSR (Gutmann Reference Gutmann2005, Reference Gutmann2007; Parton, Gutmann, Hartman, et al. Reference Parton, Gutmann and Hartman2012), and, even more ambitiously, the global integration of population and environment data in the Terra Populus project at the University of Minnesota (Minnesota Population Center 2015b).

Historians have made a good start on using these sources, as researchers learn about the availability of data and their possibilities. One area where substantial progress has already been made is in the study of land cover change in the context of social, economic, and policy change. Work done by Ken Sylvester and colleagues (Maxwell and Sylvester Reference Maxwell and Sylvester2012; Sylvester, Brown, Deane, et al. Reference Sylvester, Brown and Deane2013; Sylvester, Gutmann, and Brown Reference Sylvester, Gutmann and Brown2016; Sylvester and Rupley Reference Sylvester and Rupley2012) on transitions in land cover in the Great Plains using digitized aerial photographs and satellite remote sensing data is especially notable in this regard. It uses large datasets derived at the pixel or small-scale grid level and demonstrates how the impact of economic and policy change can be directly viewed on the ground. Other notable efforts to understand large-scale land use and land cover change in historical perspective include work by Kees Klein Goldewijk (Reference Goldewijk2001); Jed Kaplan, Kristen Krumhardt, and Niklaus Zimmerman (Reference Kaplan, Krumhardt and Zimmermann2009); and Mingliang Liu and Hanqin Tian (Reference Liu and Tian2010). These environmental datasets are likely to become important resources for economic historians studying agricultural productivity under changing environmental conditions (Olmstead and Rhode Reference Olmstead and Rhode2011).

Weather and climate data also have the potential to play an important role in economic history research. Analysis of high-resolution weather and climate data is already underway in understanding the history of natural disasters and their relationship with demographic and economic outcomes. Studies of drought and its impact, for example, increasingly make use of large-scale data sources. These include work by Glenn Deane and Myron Gutmann (Reference Deane and Gutmann2003) on the drought and dust storms of the United States in the 1930s, Qiang Chen (Reference Chen2015) and Ruixue Jia (Reference Jia2014) on China, Klas Rönnbäck (Reference Rönnbäck2014) on Africa, and Greg Bankoff (Reference Bankoff2007) on more general experiences. Celine Herweijer, Richard Seager, Edward Cook, et al. (Reference Herweijer, Seager and Cook2007) show the potential for extremely long-term studies by examining a millennium of droughts in North America.

High-resolution data also have a strong potential to make a major contribution to economic history research through the detailed simulation of conditions at the intersection of climate, agriculture, and population. Biogeochemical modeling to estimate the historical production of greenhouse gases from agriculture is one way that these approaches have been used, producing high resolution data for future analysis (Parton, Gutmann, Hartman, et al. Reference Parton, Gutmann and Hartman2012) and promising valuable results (Hartman, Merchant, Parton, et al. Reference Hartman, Merchant and Parton2011; Parton, Gutmann, Hartman, et al. Reference Parton, Gutmann, Hartman, Brown, Robinson and French2013; Parton, Gutmann, Merchant, et al. Reference Parton, Gutmann and Merchant2015). Historical agent-based modeling can also take advantage of high-resolution data to examine the ways people have made economic decisions in particular environmental conditions (Sylvester et al. Reference Sylvester, Brown and Leonard2015).

FUTURE OF BIG DATA IN ECONOMIC HISTORY

The datasets available to economic historians studying the 1980s and beyond are increasingly large. While the data creation process varies across domains, an important factor in the production of larger datasets is that the marginal cost of creating and storing a single data point has declined. Researchers continue to make their own datasets from analog sources. Such collections have also grown larger as costs have fallen. Increasingly, economic historians are working with three new types of data: numeric data that have been created and stored organically (Groves Reference Groves2011) in the course of routine economic actions, whether private market transactions or interactions between people and government social and fiscal programs; high-resolution digital images; and digitized texts.