%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Google search data have several major advantages over traditional survey data. First, the high costs of running frequent surveys mean that most survey questions are only asked occasionally making comparisons over time difficult. By contrast, Google Trends provides information on search trends measured weekly. Second, there are many countries where surveys are only conducted sporadically, whereas Google search data are available anywhere in the world where sufficient numbers of people use its search engine. The Google Trends website allows researchers to download data for almost all countries at no cost and to download time series of any search term's popularity over time (provided enough people have searched for it). For these reasons, Google Trends is an attractive data source for social scientists.
Despite these benefits, researchers using search data face a challenge because the meaning of a Google Trends series is ambiguous: an Internet search is not an expressive act, as answering survey responses are, but a practical one intended to achieve a goal. The aim of answering a survey question is to express one's view, whereas the aim of a search is to find some information. Intuitively, a greater proportion of survey respondents answering that the economy is the most important problem facing the country reflects that more people hold this view.Footnote 1 A greater number of people searching for the word “economy” on Google might reflect students reviewing for economics exams, drivers looking for information on fuel economy, or the public's concern with macroeconomic conditions (or some combination of these). Without validating Google Trends against such expressive data, we cannot be sure that these aggregates reflect the underlying attitudes in which we are interested. Past research has validated particular search terms against US survey data, but little work has examined whether these validations hold outside the US context. This article expands the validation approach to two more cases: the United Kingdom and Spain and explores whether similar search terms can effectively capture public opinion trends in the three cases (the United States, United Kingdom, and Spain).
This article argues both that there is a huge potential for using search data to capture trends in public opinion at a fine-grained level and that there is a huge potential to draw false conclusions if the data are not validated. This argument is based on the success in finding good proxies for the salience of five issues in the United Kingdom and two in Spain using Internet search data, on the one hand, and the high proportion of initially plausible search terms that had to be discarded, on the other.
Previous research on Internet data have identified several problems with using Internet data to make claims about the general population. The first problem is representativeness. In most countries, strong differences in Internet usage exists between demographic groups particularly in age and socioeconomic status (Fuchs Reference Fuchs2009). Previous research on Google Trends in the United States suggests that this problem probably does not bias results as much as might be feared (Mellon Reference Mellon2011), although it is an open question whether results in lower-usage countries would be as robust (for a discussion of some of these problems see Gabriel Koehler-Derrick's article, this issue).
A second potential problem is that the meaning of search data is not as self-evident as with survey data (Mellon Reference Mellon2011). When a representative sample of the population is asked “what is the most important problem facing your country?”, the results will show us what proportion of people chose a particular issue as most important at that time. The interpretation is complicated by how respondents actually understand “importance” and “problem” but the results have a clear interpretation. Google Trends are constructed by counting the number of searches that include a particular word or phrase. Consequently, a high number of searches for “Iraq” might reflect the political concern over the conflict but it is also possible that Iraq's soccer team is playing important matches that week. To avoid these problems, we need to check the actual searches reflected in each Google Trends (and hopefully remove the offending terms using a Boolean search).
The third potential problem is that, even if a set of searches are plausibly related to the issue we are trying to track, the patterns in searches may not track general concern about the issue. Any number of factors may affect the probability of an individual translating concern about an issue into Google searches: strength of partisanship, political interest, political knowledge, or strength of interest all could play a role in the link between individual concern and search behavior. These biases were probably observable in the 2012 Republican primaries where Google searches for Ron Paul outstripped those of any other candidate despite the fact that he failed to win a single primary contest (see figure 1 (Google 2012)). There may be scientific value in analyzing trends for candidates,Footnote 2 but the link between them is clearly more complex than a one-to-one relationship with poll results.
The Google Trends of the 2012 Republican Primary Candidates
Here is an analogy to the problem of unvalidated search data: we are handed a large set of surveys tracking interest in a particular issue over time, but we are not told what population had been sampled or how it was weighted. Without knowing these two factors, we consider if it could be a national representative survey, a survey of New York Times readers, or a survey oversampling people with exceptionally high interest in politics. Each of these populations are potentially interesting but we would want to know which of them we are looking at and would be skeptical of research that did not make the distinction.
Returning to Google data, suppose we want to look at whether public interest in an issue predicts policy announcements from government officials. We might correlate searches about that issue with the announcements and find that they have some predictive power. Depending on whether the searches track concern in the general public or the political elite, we should draw different substantive conclusions about the ability of the public to influence policy makers. Unless we test which population the searches are tracking, we cannot use the results to answer our question.
PREVIOUS APPROACHES TO USING GOOGLE TRENDS DATA
Previous research has tried to overcome these ambiguities by comparing Google Trends' data against time series of survey results from pollsters such as Gallup (Mellon Reference Mellon2011; Scheitle Reference Scheitle2011). These articles focused on measuring issue salience using Google Trends. This approach led to the successful measurement of the salience of several issues in the United States with high correlations between Google Trends and the salience of several issues (Mellon Reference Mellon2011). Weekly issue salience series could then be generated by predicting the proportion of respondents picking an issue based on the level of Google searches.
However, the results also demonstrated the danger of not validating the Google data: many plausible Google Trends either turned out not to refer to the issue they were trying to capture or did not correlate with the Gallup trends. For instance, searches for “Iraq War” did not work well as a proxy for concern over the conflict because a high proportion of the searches were for videos of an American contractor being beheaded (Mellon Reference Mellon2011), which seems to reflect macabre curiosity more than political concern. Other plausible terms simply did not correlate strongly with the concern of issues measured by Gallup.
This validation approach has obvious drawbacks as it requires a source of survey data to use for validation. This excludes the use of Google data in many of the contexts in which it would be most useful, including countries where it is hard to conduct survey research and where there is insufficient survey data to use for validation purposes. This means that Internet data are likely to be most useful in the countries where we already have relatively abundant data. An alternative justification for a search term's validity is if some issues can be consistently measured by particular search terms across a variety of contexts. As an example, in the United States, the Google Trends for “jobs” strongly tracks the salience of the issue of “unemployment” (Mellon Reference Mellon2011). If this finding were replicated across many countries we could reasonably use searches for “jobs,” or the local equivalent, to track unemployment concern, even where we do not have long time series of survey data to use for validation.
VALIDATING GOOGLE DATA CROSS-NATIONALLY
This article applies the validation approach to two more countries: the United Kingdom and Spain. This approach has two purposes: first, to discover which search terms can be used as issue salience proxies in the two countries to allow a more fine-grained analysis of temporal patterns in issue salience and second, to explore any regularities in the terms that are valid across countries (such as the “jobs” example in the preceding text). If these regularities exist, we could potentially use search data in contexts without surveys to use for validation. If the regularities are not clear enough to use searches without validation, the analysis helps researchers decide whether search data is likely to be useful by indicating which issues are likely to be measurable and which initial search terms they should try.
In this article, the analysis process closely follows the validation process used in previous work on Google Trends (Mellon Reference Mellon2011). The search terms are tested against survey measurements of issue salience to assess their value as proxies. Each candidate Google Trends series is sequentially tested in four stages: (1) content validity, (2) stationarity, and significance in predicting the relevant issue salience time series (derived from survey results) both (3) with a time trend and (4) without a time trend. Trends are only retained at each stage if they pass the previous stage.
The two countries in this article, Spain and the United Kingdom, provide an interesting contrast in their patterns of Internet usage.Footnote 3 A 2011 Eurobarometer survey showed that 74% of households in the United Kingdom had Internet access, whereas slightly more than half of Spanish households (52%) had access (Eurobarometer 2012). The levels of penetration were much lower, however, at the start of the period, 2005, with 50% of UK households and just 28% of Spanish households having Internet access (Eurobarometer 2006). These cases also provide an interesting test of the level of Internet usage required to use Internet search data (i.e., whether Google Trends data can track any issues when Internet penetration is low).Footnote 4
Survey data from each country are compared with the Google Trends results: in the United Kingdom, a collection of Ipsos MORI polls asking “what do you see as the most important issues facing Britain today?” (Ipsos MORI 2012), and in Spain, Centro de Investigaciones Sociológicas surveys asking for up to three major problems facing Spain (CIS 2012). Time series are created using the survey results for each salient issue in this period, showing what proportion of respondents choose that issue each month. The Google data are available from 2004 to the present but are of poor quality in the first year both in Spain and the United Kingdom, so 2004 is excluded, and the time series run from 2005 to 2012.
For each issue, search terms are chosen that might plausibly track concern about the issue. This task is considerably more difficult for some issues than for others. It is relatively easy to think of terms that might relate to environmental issues (“global warming” and “climate change”) but much more difficult to come up with plausible terms for concern over the “political class” in Spain. To generate Spanish search terms, I asked native speakers which words would be used by someone who was concerned about each issue. Many of these possible terms were discarded as there were insufficient numbers of searches for Google to track the volumes weekly. Several Spanish terms such as “inmigración ilegal” and “inseguridad ciudadana” (for public security) could not be measured weekly.
The first check on the Google search series determines whether the searches correspond to the issue. This content-validity check looks at the top searches that include the search term of interest. This strategy avoids cases such as the “Iraq War” example in the preceding text, where most searches were for politically less-relevant terms. When there are irrelevant searches, we can try to remove them by using a “–” sign in the search string (e.g., “Iraq War–beheading”). If nearly all searches including a word are irrelevant, then the search term is rejected as this indicates the word is primarily being used in a different way (as happened with the Iraq example).
Before the comparison can be made with the survey data time-series properties that can obscure relationships (seasonality) or lead to spurious correlations (nonstationarity) must be accounted for (Mellon Reference Mellon2011).Footnote 5 Google Trends that showed seasonal variationFootnote 6 were adjusted using Seasonal Trend Decomposition by Loess (STL) (Cleveland et al. Reference Cleveland, Cleveland, Mcrae and Terpenning1990). Nonstationary time series (that do not have the same generative mechanism over time) often show spurious correlations with one another. Consequently, several tests need to be performed before the correlations between the series can be tested.Footnote 7
Finally, the remaining searches are correlated against the time series of most important problem responses for the appropriate issue. This correlation is done in two stages: without and with a time trend. First, it is important that Google Trends has predictive power in a bivariate relationship. Because we cannot guarantee that any linear trend will continue outside of the time period, the search series should be able to stand in as a proxy by itself. However, it is also important that the strength of the relationship between the two time series is not simply due to each series' correlation with time. For trends that show a significant association, the strength of the relationship (measured by the R 2 from an ordinary least squares regression between the two time series) can be used to assess how strong a proxy the Google Trends series will be for measuring that issue's salience.
RESULTS
The results of this process are shown in table 1. The first column (search term) shows the keyword used to generate the Google Trends search, and the second column shows the issue it is intended to measure. Black circles show whether a Google Trends series passes the relevant testing stage (or if it needed to be adjusted for seasonality) and white circles indicate that it failed the test. Finally, the goodness of fit (R 2) is shown between the issue salience series and Google Trends series.
Google Trends Issue Salience Validity Results for the United Kingdom and Spain
+ Missing months in the UK MIP series were imputed for the stationarity and criterion validity tests by taking the mean of the two adjacent observations.
* Significance tested at the 10% level using Newey-West robust standard errors to account for autocorrelation and heteroskedasticity
After plausible and trackable terms were chosen, content validity was checked by looking at terms terms used in each Google Trend. Several problematic terms emerged. In the United Kingdom, results for the search term “jobs” included searches for “Steve Jobs,” as well as ones related to finding employment. A similar problem occurred for Spain when looking at searches for the Basque separatist group “ETA.” The word “eta” means “and” in Basque. As a result, a Basque nongovernmental organization supporting children with cancer, Pirritx, Porrotx eta Marmimotots, appears high in the searches. In both of these cases, the confounding term can be removed using the Google Trends interface by excluding searches including the terms “Steve” and “Pirritx,” respectively. In other cases, too large a proportion of the searches are confounding to use the trend. In Spain, Google Trends for “corrupción” mainly consist of searches for “corrupción en Miami,” the Spanish name of the television show Miami Vice. Similarly, searches for “EU,” intended to measure the British concern over the European Union, instead mainly involve searches for the EU World of Warcraft server. In total, four Spanish and eight British search terms are excluded in the content-validity stage.
The next stage checked for time-series problems. None of the issue salience time series examined showed evidence of seasonality, but several of the Google Trends showed considerable seasonal variation. Searches for “jobs” in the United Kingdom, shown in figure 2, show particularly strong seasonal shifts. Panel A shows the unadjusted Google Trends data. Panel B shows the seasonal component extracted using STL. Panel C shows the smoothed trend and panel D shows the remainder component. The trend and remainder components are recombined to create a seasonally adjusted series. Thirteen of the British trends showed seasonality and were adjusted. Surprisingly, none of the Spanish terms showed significant seasonal variation. One Spanish and two British trends were excluded after the stationarity tests.
Seasonal Decomposition of Google Searches for “jobs” in the United Kingdom, showing the original series seasonal, trend, and remainder components
After the time-series problems have been accounted for, it is possible to interpret the association between the series. Thirteen British terms were excluded in the bivariate test, excluding the remaining terms for the issue of housing. One of the terms excluded was “illegal immigration” (a term that showed a strong relationship with immigration concern in the United States). Four additional terms were no longer significant after a time trend was included in the regression.
Two Spanish terms were not significant in the bivariate test, and an additional four were excluded after including a time trend. These exclusions include searches for “ETA,” which were not a significant predictor of the salience of terrorism.
In total, 25 out of 39 UK search trends were excluded in these steps, leaving 14 search trends measuring nine issues. Ten of the 15 initial Spanish search trends were excluded, leaving five terms covering two issues. This result partially reflects the stringent nature of the tests—longer time periods may show that some of the rejected terms were valid.Footnote 8 Nevertheless, the high level of attrition should concern researchers using search data without validation.
For which issues do Google Trends provides good proxies?
Although the remaining search terms in the final column of table 1 all show significant relationships with the relevant issue, their utility as issue salience proxies is dependent on the strength of this relationship.
In both Spain and the United Kingdom, economic issues were the most easily tracked using Google searches. Two terms, “job seekers” and “job seekers allowance” (the British term for unemployment benefits), were strong predictors of economic concern, with the latter searches explaining 86% of the variance in concern. The comparison of Google Trends for “job seekers allowance” and the Ipsos MORI time series for concern over economic issues is plotted in figure 3.
UK searches for “job seekers allowance” (rescaled) compared to the percentage of respondents citing an economic issue as the most important issue
In Spain, the results were similarly strong for economic issues. Four different search terms were related to economic concern. The strongest relationship was “paro” (unemployment) that explained 83% of the variance in concern about economic issues. The comparison between the two time series is shown in figure 4. An alternative translation for unemployment, “desempleo,” explained a similar proportion of variance (R 2 = 0.80). Searches for “trabajo” (work) and “trabajos” (jobs) each also explained more than half the variance.
Searches for “paro” in Spain (rescaled) compared to the percentage of respondents citing economic issues as an important problem
In addition to economic issues, several other search terms showed substantial relationships to issue salience series. Several UK search trends were strong predictors for the public's concern about fuel prices. The term “fuel prices” was most effective, explaining 67% of the variance in concern. However, four other terms were also strong predictors with each explaining between 40% and 60% of the variance. The comparison using searches for “fuel prices” is shown in figure 5. A combination search that combined searches for any of these five terms was also tested. However, the combined index performed poorly compared to the best of the individual terms with a goodness of fit R 2 = 0.56.
UK searches for “fuel prices” (rescaled) compared to the percentage of respondents citing fuel prices as the most important issue
In the United Kingdom, searches for “education” also showed a strong relationship with concern about that issue, explaining 61% of the variance. Another two Google Trends, “pensions” and “global warming” showed strong relationships with the survey results for each issue (R 2 of 0.41 and 0.27, respectively) with the relevant issue but probably not strong enough to justify its use as a proxy in empirical work. Finally, four terms—“Afghanistan war,” “taxes,” “illegal immigration,” and “report crime”—are significant but the percentage of variance explained is low.Footnote 9
Only one noneconomic issue was able to be tracked in Spain. Searches for “inmigrante” explained close to half the variance in concern about immigration (R 2 = 0.47). A visual comparison of the time series (see figure 6) shows that the “inmigrante” series is particularly noisy at the start of the period, possibly because of a relatively low level of overall searches in Spain. It is possible, then, that the underlying relationship between these series may actually be stronger, but a longer time period is needed before this can be tested. Also, note that some of the rejected Spanish terms may be valid (particularly those rejected after including a time trend), but the tests conducted did not reject the claim that any relationship might be spurious. More issues are likely to be captured as the time period increases; many terms were rejected because of low search frequency in the first few years.
Searches for “inmigrante” in Spain (rescaled) compared to the percentage of respondents citing immigration as an important problem
These results show progress toward the first goal of finer-grained issue salience measurement in the United Kingdom and Spain. Weekly time series of concern about five issues—economic issues, fuel prices, education, the environment, and pensions—can be generated for the United Kingdom, although the accuracy of these proxies varies considerably. Similarly, for Spain, weekly time series of concern about the economy and immigration now can be generated. But what about the second goal, to look for generalizations about which search terms are likely to be valid?
WHY DO THESE SEARCH TERMS WORK?
In the trends examined here, issue importance links to the frequency of searches for a particular term in two ways. First, people can be directly affected by a problem and increasingly search for practical information. Second, people can be interested in the issue and seek further information. Overall, searching for practical information seems to be the more common motivation behind the valid Google searches in this article. Perhaps practical searches are more related to issue salience or that it is easier to identify useful practical search terms than ones that reflect the searcher's interest without being obscured by irrelevant factors. Although there are similar motivations behind searches in each country (and previous results in the United States), the similarities are not sufficient to identify terms that would work in other countries without surveys to use for validation.
In the United Kingdom and Spain, economic issues are clearly the most easily tracked using Google Trends as shown in previous work on the United States (Mellon Reference Mellon2011). In each of these cases, the best predictors of concern with the economy are terms that relate to people taking actions as a result of economic conditions (signing up for unemployment benefits and looking for jobs) rather than terms related to general interest in the economy as an issue (for instance, searches for “UK economy” are not significantly related to economic concern). The success of practically motivated searches fits with previous successful uses of Google data in the medical literature. For instance, Google's Flu Trends (Ginsberg et al. Reference Ginsberg, Mohebbi, Patel, Brammer, Smolinski and Brilliant2008) rely on practically motivated searches for flu symptoms and cures. In contrast, the specific aspects of the economic system in each country mean that the precise terms that work still vary. In both Spain and the United States, the word for “jobs” showed a substantial relationship with economic concern. This suggests that concern with economic issues might be generally captured with these searches. In the United States “jobs” searches were by far the strongest predictor of concern about unemployment. However, in Spain much more variance is explained with “paro” than “trabajos,” and “jobs” is not a significant predictor in the United Kingdom after controlling for a linear time trend. All of these results make sense, but picking a term without the aid of survey data would likely lead to a much poorer proxy.
There are also similarities in the other trackable issues. In the United States and the United Kingdom, the salience of fuel prices can be tracked successfully using searches related to price searches: “fuel prices” and “diesel prices,” respectively. This result seems to reflect a similar mechanism to the searches for jobs tracking unemployment concern: when fuel prices are of greater concern, more people will look for information about the price of fuel. Note that “petrol prices” were a substantially weaker proxy than “fuel prices” or “diesel prices” in the United Kingdom because petrol is the more common fuel in Britain (Evans Reference Evans, MacLeay, Harris and Annut2012). Unfortunately, this issue could not be tested in Spain because the surveys do not categorize concern over fuel prices separately.
Immigration concern could be tracked well in both Spain and the United States. In both cases, the search terms that worked reflected an intention to find information about the issue rather than practical steps to deal with it. One term significantly predicted concern over immigration in the United Kingdom, but the degree of explanatory power was trivial, despite the issue's importance on the agenda during this time. One reason for this result may be that an important term in the British discourse on immigration, “asylum seekers,” was rejected at the content-validity stage as it included too many searches looking for information related to asylum charities such as “asylum aid” and “refugee council.” This example reinforces the point that Google Trends can be driven by multiple motivations, especially if terminology is used by both proponents and opponents of a contentious issue.
This discussion shows that, for economic trends, some similarities emerge between the terms that work across countries. However, finding the best term to capture economic concern still requires validation against survey data. The success of some Google Trends in Spain also suggests that even relatively low levels of Internet penetration can generate search data that closely match survey measures for some issues.
The differences in terms used in each country's discourse around immigration, the environment, and terrorism mean that different searches reflect the concern with these issues. We still need to validate Google Trends against existing data to make reasonable inferences about public opinion. Although many of the tens of thousands of possible Google Trends undoubtedly track interesting social phenomena, many more will track nothing at all. Without a strong argument that a suggested Google Trend is in the former, rather than latter, category it is brave to conclude much from the patterns in Internet searches.
One other possible explanation for which search terms work is the level of Internet penetration in the two countries that may affect the trackable searches. Given that fewer than 30% of Spanish households had access to Internet at the start of the study, many searches had to be rejected prior to any testing because of insufficient volume.
Although relatively few countries have sufficient survey data to validate against at present, these data will increase over time. Google provides data from as far back as 2004, so even surveys conducted twice a year will give a sample of 18 to test the Google Trends against, and as the period covered by Google increases, search data will be testable in more countries. Rather than replacing the traditional survey, using Internet data may require researchers to do more survey work, particularly in hard to reach areas. Paradoxically researchers may only be able to unlock the potential of the Internet data being generated today, in retrospect, when they have enough data generated with conventional survey tools to test the validity of search data.
