A. The Supreme Court as a Legitimator

The first line of thought, beginning with Dahl’s (Reference Dahl1957) essay on the majoritarian function of judicial review, suggests that the Supreme Court has the unique capacity among major institutions of American government to confer legitimacy upon policies, causing minority views to lose appeal in the face of those promoted by dominant governing coalitions (see Gibson Reference Gibson1989; Caldeira and Gibson Reference Caldeira and Gibson1992; Mondak Reference Mondak1994; Gibson and Caldeira Reference Gibson and Caldeira2009). Although Dahl was silent on the origins of this special authority, subsequent scholarship has offered a mechanism. Americans perceive the Court to engage in a decision-making process that is both principled and largely disconnected from routine political compromise and the bargaining commonplace in the political branches (Hibbing and Theiss-Morse Reference Hibbing and Theiss-Morse1995). This belief about judging, which Scheb and Lyons (Reference Scheb and Lyons2000) call the “myth of legality,” is a core element of the Court’s legitimacy. Critically, it must be learned. It is believed to be developed through repeated exposure to judicial behavior, where legitimating symbols (e.g., robes, formal legal language, courtrooms) create a positive frame through which people judge the Supreme Court’s proper role in the state and ultimately create a sense of loyalty to the institution (Gibson, Caldeira, and Spence Reference Gibson and Caldeira2003; Gibson and Caldeira Reference Gibson and Caldeira2009). In light of its legitimacy, the Supreme Court is capable of two possible types of effects on public opinion. On one account, the Court is capable of changing support for particular policies. On another, although the Court may not persuade individuals to change their opposition to a policy, it can induce them to accept such a policy (e.g., Gibson Reference Gibson1989).

B. The Supreme Court as a Divider

Whereas there is considerable evidence in support of Dahl’s claim that the Supreme Court enjoys a unique supply of legitimacy, the literature is considerably mixed with respect to the Court’s ability to legitimate particular views of public policy (see Ura Reference Ura2014). A second line of thought sees the Supreme Court not as a source of persuasion in matters of public policy but as an important generator of policy outcomes. As Franklin and Kosaki (Reference Franklin and Kosaki1989, 763) write, “When the Court rules on politically controversial cases, it may establish the law of the land, but it does not put an end to debate.” The Court nevertheless affects public views of policy. In Franklin and Kosaki’s structural-response model, the impact of judicial decisions lies in the way that decisions structure deliberation. Specifically, Supreme Court decisions provide salient topics for discussion groups, where prior beliefs are challenged at times and yet frequently reinforced through repeated interaction with like-minded people. When groups are divided on an issue, far from legitimating policy outcomes, the Supreme Court will polarize. Ura (Reference Ura2014) suggests a similar prediction, relying on the model of thermostatic response applied to the Supreme Court. Like Franklin and Kosaki, Ura views Supreme Court decisions as policy outputs, which can cause the same kinds of public reactions that follow the final decisions of other major policy-making institutions. Under the thermostatic-response model (Wlezien Reference Wlezien1995), the public is able to encourage its representatives to seek preferred policies, because individual reactions to policies, once aggregated, signal desires to continue moving policy in a particular direction or to pull it back. Simply, a policy that is viewed as more liberal (conservative) than ideal is met with demands for a more conservative (liberal) policy. For all but the most liberal or conservative policies implied by Supreme Court opinions, this argument would anticipate polarized discourse. Individuals whose preferences lie to the left of the policy implied by the Court will demand a more liberal policy response; individuals whose preferences lie to the right will demand a more conservative policy response.

C. Empirical Implications

We estimate a number of features of revealed policy opinions in the context of Twitter, focusing here on the issue of same-sex marriage: support/opposition to policies that promote same-sex marriage, as well as the intensity and anger with which these opinions are expressed. As we discuss, expectations about changes in support for the policy are clear, in light of existing models. There has been less work on the emotional effects of court decisions. However, research more generally on emotional features of political interactions with elites has direct implications for the nonpolicy features of public discourse in the wake of judicial decisions. Banks and Valentino (Reference Banks and Valentino2012), for example, show that individuals are more likely to experience anger when they are in situations they dislike and for which they can find a responsible party to blame. Political messaging, such as campaign ads, can trigger significant emotional responses among the public (e.g., Brader Reference Brader2005). Modern media, especially electronic media, have contributed to more emotional reactions among the public to policy debates (e.g., Hibbing and Theiss-Morse Reference Hibbing and Theiss-Morse1998). We believe that emotional responses should largely follow along the lines of the existing models of opinion change, as we discuss below.

What should we expect to observe in the wake of the Supreme Court’s Windsor and Perry decisions? Theoretical expectations are quite clear, given a set of assumptions about the decisions and the context in which they were made. The first issue concerns the kind of policy implied by the decisions. It is important that we can articulate the policy’s ideological nature so that we know what changes in opinion to expect. Windsor found that by defining marriage as a union only between a man and a woman, section 3 of the federal Defense of Marriage Act (DOMA) violated due process and equal-protection principles applicable to the federal government under the Fifth Amendment. Perry found that supporters of California’s ballot initiative banning same-sex marriage lacked standing to appeal an adverse ruling in federal court, the consequence being that California’s Proposition 8 was found to be unconstitutional under the Fourteenth Amendment. Taken together, the decisions reflect a liberal policy; however, it is important to recognize that the Court could have gone further. By failing to reach the equal-protection issue in Perry, the Supreme Court left untouched state laws limiting marriage equality.Footnote ² In Windsor, the Court’s federalism analysis highlighted the states’ traditional role in defining marriage and by so doing raised questions about whether a state could validly restrict marriage to one man and one woman. The critical point is that the decisions resulted in a liberal policy outcome, although not an extremely liberal one.Footnote ³

The second issue concerns the context in which the debate was taking place. At the time of the decision, opinion on the matter was fairly divided, with 50% of the general public supporting same-sex marriage, roughly 42% opposing it, and 8% unsure. Support was estimated to be about 79% among liberals and only 30% among conservatives.Footnote ⁴ The decisions were handed down in a charged and divided policy area.

Given these assumptions, we know what to expect under the theoretical accounts reviewed above. Under a legitimation model, we should observe opinion moving in the direction of greater support for same-sex marriage. Under both the structural- and thermostatic-response models, we should observe polarization. Of course, the natural question concerns polarization among which groups. We consider two, largely because we can identify members of these groups in our sample. The first are ideological groups. We will consider opinion change among conservatives and liberals, who were generally divided on this issue, as reported by nationally representative polls. Second, we consider opinion groups—specifically, groups of individuals who differed with respect to the same-sex-marriage issue prior to the decision. Focusing on this second grouping ensures that we have low within-group variance in opinion compared to between-group variance—the group is constructed to ensure that this is true. If the Court had a primarily polarizing effect on opinion, we should observe that liberals became more supportive of same-sex marriage and that conservatives became more opposed. This should be particularly likely for liberals on the Far Left and conservatives on the Far Right. Similarly, we ought to observe that supporters prior to the decision expressed more support afterward and that those in opposition expressed stronger opposition.

Now, consider the implications for the anger and intensity with which opinions were expressed. Under Franklin and Kosaki’s structural-response model, we ought to observe that liberals and those in support of same-sex marriage should have reacted with less-angry tweets after the decision, whereas conservatives and those in opposition to the policy should have expressed more anger after the decision. Under the thermostatic-response model, both liberals and conservatives should have responded with greater anger, whereas moderates should not have reacted. This “anger at the extremes” prediction is particularly likely on the Far Left and Far Right, precisely where people would have been most frustrated under the thermostatic-response model. Finally, under the legitimation model, all tweeters should have responded with less anger.

Intensity presents an interesting scenario. It is possible that intensity would have changed in ways identical to anger, under each model. That is to say, it is plausible that each model suggests the same implications for intensity and anger. Other reactions are plausible, though. For example, it is possible that liberals (and supporters) became more supportive and yet intense simultaneously in the wake of the decision.

A. What Is a Tweet? Conceptualizing Microblog Data

Scholars have devoted extensive effort to understanding the nature of public opinion, especially what it means for a person to express an opinion (see, e.g., Zaller Reference Zaller1992). Interpreting the meaning of responses to public opinion polls, statements in focus groups, claims about voting in exit polls, and the like have all been carefully studied (e.g., Traugott and Price Reference Traugott and Price1992; Huckfeldt and Sprague Reference Huckfeldt and Sprague1995). As a result, scholars have developed sophisticated understandings of the effects of issue salience, information and media exposure, framing and priming, and myriad other factors that can affect how individuals translate their thoughts on a political topic into revealed statements of opinion (e.g., Achen Reference Achen1975; Iyengar and Kinder Reference Iyengar and Kinder1987). One finding that cannot be escaped, though, is that there is considerable differentiation in how much people know and think about politics.

At the same time, there has been little research on how political statements on microblogs relate to well-conceptualized notions of public opinion (cf. Marwick and Boyd Reference Marwick and Boyd2011). In examining consumer confidence, O’Connor et al. (Reference O’Connor, Balasubramanyan, Routledge and Smith2010), for example, show a correspondence between aggregate sentiment on Twitter and public opinion polls, especially with respect to trends. However, a correspondence between aggregate trends is not the same as a direct connection between individual utterances on microblogs and theoretically relevant quantities in theories of public opinion. Nevertheless, our knowledge about the content of microblogs is growing quickly, and this is promising for understanding the role of social media like Twitter in disseminating information and playing the role of traditional media outlets. Indeed, the mass media have a large presence on Twitter, and scholars have demonstrated the importance of the media for shaping and informing public opinion (e.g., Iyengar and Kinder Reference Iyengar and Kinder1987; Prior Reference Prior2007).

Recent studies have revealed that 85% of Twitter content is related to spreading and commenting on headline news (Kwak et al. Reference Kwak, Lee, Park and Moon2010). Table 1, for example, shows a sample of tweets relating to same-sex marriage posted around the date of the US Supreme Court’s June 2013 decisions on the constitutionality of DOMA and Proposition 8. These tweets represent the range of uses to which Twitter was put in this case. The first few tweets express factual information, mostly retweeting from the mass media. The next few express strong personal sentiments on the issue, from both directions. The last few represent conjectures about national opinion and the ongoing policy debate. In brief, what we see here is that political information and policy opinion is the dominant theme in tweets about same-sex marriage. Together, the extant studies and our illustrative example imply that Twitter has became a portal for the public to express opinions. In the context of politics, Twitter content, together with Twitter users’ information, such as users’ profiles and social networks, has shown reasonable power of detecting users’ political leanings (Conover, Gonçalves, et al. Reference Conover, Gonçalves, Ratkiewicz, Flammini and Menczer2011) and predicting elections (Tumasjan et al. Reference Tumasjan, Sprenger, Sandner and Welpe2010). Although promising, the effectiveness of using Twitter content to measure public political opinions remains unclear. Several studies show a limited correlation between sentiment on Twitter and political polls in elections (O’Connor et al. Reference O’Connor, Balasubramanyan, Routledge and Smith2010).

Table 1.

Examples of Three Dominant Types of Tweets Relating to Same-Sex Marriage

Connecting these observations to social scientific work on opinion formation, we believe that individuals’ statements on Twitter are in many ways conceptually analogous to the model of opinion formation underlying Zaller’s (Reference Zaller1992) theory of public opinion. Tweets are utterances that reflect the thoughts in one’s mind, especially as shaped or primed by the Twitter network in which one interacts. To the extent a tweet contains political information, the tweet reflects one’s ideological and policy views, particularly opinions recently activated by one’s Twitter network. In much the same way as opinion and media choices and ideology all interact simultaneously (e.g., Prior Reference Prior2007), Twitter’s environment is one in which individuals’ tweets are at least in part statements about their policy opinions. As such, we consider Twitter an important example of social expression of opinion.

B. Who Is a Tweeter? Relating Our Samples to the American Public

Certainly, the people who participate in political discourse on Twitter do not constitute a random cross-section of the public, and scholars have set out to assess the demographic correspondence between the Twitterverse and the American public. For example, we know that individuals who tweet tend to be more male and urban than the general public but also that there has been increasing gender diversity and racial diversity on Twitter over time (Mislove et al. Reference Mislove, Lehmann, Ahn, Onnela and Niels Rosenquist2011). At the same time, studies in a variety of contexts have found that the volume and content of political discourse are predictive of mass opinion and that research findings conducted on more clearly defined populations are replicable in the Twittersphere (e.g., Tumasjan et al. Reference Tumasjan, Sprenger, Sandner and Welpe2010; Conover, Gonçalves, et al. Reference Conover, Gonçalves, Ratkiewicz, Flammini and Menczer2011; Bermingham and Smeaton Reference Bermingham and Smeaton2012; Barberá and Rivero Reference Barberá and Rivero2014), in many ways similar to how experimental results are often replicable using convenience samples, such as Mechanical Turk (e.g., Berinsky, Huber, and Lenz Reference Berinsky, Huber and Lenz2012). Nevertheless, because we do not have a better measure of the representativeness of our sample, our analysis will focus on the presence (or absence) of empirical patterns among the tweets of those people participating in political discourse on Twitter. While we anticipate that current research aimed at estimating the demographic and geographic features of tweeters will help future studies build from these findings to further generalize, as we discuss below there are distinct strengths to using these data to study the effects of sudden events, like Supreme Court decisions, on political discourse.

C. How Do We Use Microblogs? Obtaining and Working with Twitter Data

A second challenge to using microblogs to study politics is that, as contrasted with standard tools of the discipline like public opinion polls or legislative voting records, obtaining and working with microblog data presents a new set of technical and logistical challenges. We describe our approach to working with and handling Twitter data. Twitter provides a streaming API to deliver tweets in real time. An API, or application programming interface, is simply a set of protocols for how software interacts with other software (e.g., your own computer’s software and Twitter’s software). Twitter’s API provides a mechanism for accessing and retrieving Twitter data by specifying “filters” to identify which tweets one wants to record. There are several types of filters that can be applied to the API. A language filter selects only tweets written in a particular language. The language of a tweet is determined by Twitter’s language classifier. A keyword/phrase filter can represent topics and help select on-topic tweets. If specified, tweets containing at least one of the keywords or phrases will be delivered. A user filter can be used to track groups of interest. If specified, all tweets authored or retweeted by the particular users will be delivered. If none of the filters is specified, the API will continuously produce 1% random samples of all tweets. When the filters are applied, the API will send all the tweets that match the filtering parameters as long as the volume is lower than 1% of all tweets. Once the percentage of matched tweets is higher than 1%, the API will only return a random sample up to the 1% cap.Footnote ⁵

We outline the architecture of our data collection and analysis in figure 1. First, the Twitter stream is filtered by using topically relevant keywords, language filters, and desired author characteristics. The resulting set of relevant tweets are automatically annotated with a pipeline of topic-specific classifiers, annotating each tweet with estimated author demographic information, topic salience, support, emotional intensity, and other quantities of interest, which are then used for opinion estimation. The tweet text, as well as the original and inferred metadata, is stored in a database. (We describe our coding and classification procedures in greater detail below.) Finally, all the summary and estimated statistics, as well as tools for developing new topics of interest, are made available through a web server, on which we publicly release our data. We describe next the collection of tweets and the construction of relevant measures from those data. In the conclusion, we describe our online methods for data dissemination. While we focus our analysis on our particular case study—same-sex marriage—the framework, architecture, and analytic methods are general. Upon publication, we will deposit all code to replicate our analyses or apply our techniques and tools to any substantive context.

Figure 1.

Architecture of the Twitter-based opinion estimation system

D. Our Study: Same-Sex Marriage and the Supreme Court

As the discussion above suggests, developing a set of keywords to search is more of an art than a science. We began with several keywords that were particularly prominent (“DOMA” and “Prop 8”). We developed keywords and requested a sample of tweets that matched the keywords. The keywords used are “ssm,” “same sex marriage,” “DOMA,” “Prop8,” and “gay marriage.” We also specified the language of sampled tweets to be English only. Our procedure produced over 2,500,000 tweets between March 26 and August 10, 2013, with 87,575 daily tweets on average. On June 26, 2013, the day the Supreme Court decided its two gay marriage cases, we collected 335,399 tweets on the gay marriage topic. The precision (i.e., the fraction of retrieved tweets manually verified to be on topic) was 92.3%. This provides strong support for the accuracy of our topic-filtering techniques based on designing a set of precise keywords for each topic.

In addition to the daily sample of tweets, in early May 2013 we identified 673 frequent tweeters regarding the “same-sex marriage” topic.Footnote ⁶ This panel of tweeters is important to rule out a plausible source of sample selection bias that might emerge by only focusing on the daily random sample of tweets. For example, it could be that support for same-sex marriage (or opinion intensity) is constant over time but that after the Supreme Court’s same-sex-marriage decisions, only those tweeters with extreme views chose to tweet. To be sure, in this scenario the Supreme Court’s decision would have influenced the process by which political views are expressed publicly, which is interesting in its own right; however, such a process would not be consistent with the notion that the Supreme Court can change opinion in any of the ways the theories we evaluate suggest. By focusing on a panel of individuals who tweet often and who tweeted both before and after the decisions, we are able to observe potential changes in support/intensity/anger among the same set of people. To construct the panel, we tracked every tweet of the most frequent tweeters over a 3-month period. Critically, the individuals in our panel do not know they are in a panel; we are simply collecting all of their public utterances on Twitter. We refer to the daily sample of tweets as our “random sample” of tweets, recognizing, though, that they do not represent a national probability sample or anything approaching one. We refer to the set of tweets from our panel as our “panel sample.”

A. Measurement Task 1: Extracting Information from Natural Language

As noted, there exists a suite of tools for quantifying features of natural language and for which the best approach is context-dependent. We illustrate a general approach that is nevertheless customized to our particular setting—tweets about same-sex marriage and the Supreme Court’s decisions in June 2013. To measure the content of the tweets we collected, we adopted classification algorithms specifically designed to detect three features of tweet content we expect to be present and variable in tweets about same-sex marriage: supportiveness, intensity, and anger.Footnote ⁷ Each of the measures was handled by one classifier. For example, to estimate the supportiveness of tweets, we assigned every tweet to one of the following classes: supportive, neutral, and opposing. For intensity, the classes were intense and nonintense. And for anger, the classes were angry, neutral, and happy. As noted above, all code and data will be publicly released upon publication, and future researchers can train our classification algorithms on alternative data.

We began by manually labeling a set of “training” tweets. We developed detailed labeling instructions, hired political science graduate students as labelers, and trained them in person. To train the classifier, we labeled 1,400 tweets, sampled at the rate of 100 per day, over the 2-week periods immediately prior and subsequent to the DOMA and Prop 8 decisions. Coding rules for the research assistant tasks can be found in the appendix. Intercoder reliability, as measured by the Fleiss к statistic, is highest on the “relevance” (95%) and “support” (99%) items and lower on “anger” (93%) and intensity (88%). With the human-labeled tweets in hand, we developed our classification algorithms to automatically label tweets.

Our classification algorithm measures the presence of “features” in each tweet and estimates the relative predictive weight of each feature, given the distribution of those features in our manually labeled tweets. To classify tweets, we developed several groups of features to represent them in feature space:

• Popularity: Number of times the message has been posted or favored by users. As for a tweet, this feature means number of retweets and favorites.

• Capitalization and punctuation: It has been shown that capitalization and punctuation carry valuable signals of emotional intensity in sentiment analysis.

• The text: Unigram, bigram, and trigram of the tweet text.

• Character N-gram in text: Trigram and four-gram of the characters in the tweet text.

• Sentiment score: A score computed via a comprehensive sentiment dictionary, SentiWordNet (http://sentiwordnet.isti.cnr.it/), as well as stylistic features described in Barbosa and Feng (Reference Barbosa and Feng2010).

We experimented with a variety of automated classification algorithms, and for this experiment we report the performance of the Naive Bayes algorithm, which is simple, fast, and shown to be surprisingly robust to classification tasks with sparse and noisy training data (Zhang Reference Zhang2004). The classification performance, using tenfold cross-validation, for supportiveness, intensity, and anger of the tweet text is reported in table 2. Even with the small amount of training data, the classifier is able to accurately identify supportive and neutral tweets, with precision of 73% and 76% respectively, but not the rarer occurrences of opposing tweets. Classifier performance is also acceptable for distinguishing emotionally intense from dispassionate (factual) tweets, of which there was a 69% majority in our sample.

Table 2.

Preliminary Sentiment Classifier Performance for Tweets on “Gay Marriage” Topic

Note.—Tenfold cross-validation, 1,400 tweets.

Indeed, we find that our classifier reliably recovers the levels of support, intensity, and anger among tweets on a daily basis. Figure 2 shows a comparison of the classifier’s estimates of each of these features and the “ground truth”—the labels our research assistants assigned. As this figure demonstrates, we recover both the levels and the trends in sentiment. This evidence demonstrates that the features we select and classification algorithm we adopt reliably and validly predict sentiment, at least as captured by our human-labeled tweets. Importantly, these results demonstrate that we can validly and reliably train a simple classification algorithm to recover sentiment in highly contextual political texts—here, tweets about same-sex marriage. For scholars seeking to use Twitter or similar microblog textual data to study public opinion, our evidence reveals that converting those data into meaningful quantities that we can study with the discipline’s standard tools is simply a matter of manually labeling a small number of documents not unlike common practices and the application of standard classification algorithms to automatically label millions of additional documents.

Figure 2.

Comparison of human-assigned labels and classification algorithm-assigned labels. Triangles show daily averages of human-labeled tweets; asterisks show daily averages of classification algorithm-labeled tweets.

Turning from the manually labeled tweets to our full samples—both the random sample and the panel sample—figure 3 reports the results of our classification. The top panels report the results from our random sample of tweets, and the bottom panels report the results from the relevant tweets among our panelists. The three columns show the daily average value for each of the dimensions we label (the points are scaled to the number of collected tweets each day), along with a smoothed trend (the line shows a LOESS [locally weighted smoothing] smoother with a 95% confidence band).Footnote ⁸

Figure 3.

Trends on three sentiment features among the random sample and panel tweets. The points show daily averages and are sized proportional to the number of tweets collected that day. The lines are fit lines from a GAM smoother; shaded areas show 95% confidence bands.

A number of interesting patterns emerge. Consider first supportiveness. In the random sample, there is a relatively high level of support, although there is some fluctuation in supportiveness over time. Among the panelists, by contrast, there is a lower level of overall support, combined with a slight trend toward neutrality. Indeed, to the extent frequent tweeters on a subject constitute “opinion elites” or at least politically interested citizens, we should expect somewhat more moderate overall opinion, because the mix is more likely to contain tweeters from all political orientations relative to the random sample of tweeters. Further, perhaps surprisingly, while there was an increase in overall supportiveness after the Supreme Court’s same-sex-marriage decisions, it was not immediate; rather, it was not until later in the summer that supportiveness spiked.

Consider next the intensity metric. Here, we find greater variability in the random sample before the decision than in the panel. However, we also see a clearer pattern in the panelists, with a steady decrease in intensity over the late spring, in the run-up to the Court’s decision. However, by early June, well before the decision, intensity had more or less bottomed out and remained low and flat through the Court’s decision. Nevertheless, despite greater variability in the random sample, the Court’s decision does not appear to be associated with any change in intensity. To the extent we might expect the Court’s decision to excite liberals and infuriate conservatives, we also might expect an increase in intensity after the Court rendered its decisions, but we see no such pattern.

Finally, consider anger. Notice there is virtually no variation in anger before the Court’s decision. However, on the day of the decision there is a drop in anger, followed by a jump in anger during the days following the decision. We detect this not only in the random sample but among the panelists as well. And, critically, that increase in anger continues steadily throughout the summer until the end of our sample.

B. Measurement Task 2: Measuring Features of Individual Tweeters

The second measurement task we face concerns collecting information about not just what tweeters say but who they are. There exist several studies in computational science designed to estimate the distribution of demographic groups in social media and on Twitter, but we require information about individual tweeters. There exists a wide range of tools that can be used to estimate a tweeter’s gender, location, or age group, but an important variable for most political science applications is tweeter ideology. Large public opinion polls often use a five- or seven-point self-placement scale. So, ideally, we would administer a survey to each tweeter in our data to develop an index of conservatism/liberalism for each individual. This is not possible, for several reasons. Twitter’s terms of service prohibit us from directly contacting the individuals whose tweets we monitor and collect. However, some users voluntarily provide some indicators of their ideology through their public profiles. For example, some tweeters self-identify as Republican or “conservative.” Indeed, in our panel sample, 176 tweeters indicate their political ideology in some form in their profile. Unfortunately, that percentage is still relatively low, and one might worry that relying on self-reported ideology in a public profile induces substantial selection bias in terms of tweeters for whom we have a measure of ideology.

To overcome these limitations, we develop a variable model to estimate latent ideology for the tweeters. Our model rests on the assumption that who one “follows” on Twitter is a manifestation of latent ideology. Using the structure of tweeters’ social networks to infer their ideology is an idea that has recently been advanced and validated in a variety of Twitter settings (Barberá Reference Barberá2015). The key insight is that once accounting for individuals’ level of activity on Twitter and the extent to which some tweeters are more or less popular, the propensity for any tweeter to follow another is decreasing in their ideological dissimilarity. We model whether one follows another as a function of their distance in some latent space and evaluate whether that space can be interpreted as “ideology.” Of course, we recognize that not all subjects on Twitter may be equally discriminating across the latent dimension; some individuals may be followed by tweeters across the latent dimension, whereas others may be followed only by people very close to them. Our model allows the effect of distance between a tweeter and an individual to be followed to vary across “followed” individuals.

Consider a set of N tweeters and J followed individuals. Let F be an $N\times J$ matrix, where $f_{\mathrm{ij}}=1$ if tweeter in row i follows individual in column j and 0 otherwise. We assume each tweeter and followed individual has a location θ in a latent one-dimensional space. Formally, our model is given by

(1) $\begin{array}{}\text{(1)}& \mathrm{Pr}\left(f_{\mathrm{ij}}=1|\alpha ,{\beta }_j,{\theta }_i,{\theta }_j\right)=\Phi \left(\alpha +{\beta }_j{\left({\theta }_i-{\theta }_j\right)}^2\right),\end{array}$

where Φ is the cumulative normal distribution. Of course, the model given by equation (1) is not identified without further restrictions. We identify the scale by assigning a prior distribution to the unobserved ideal points, such that $\theta \sim N\left(0,1\right)$ . Even with that constraint, the model is still only identified up to a polar rotation (Jackman Reference Jackman2001). As we show below, self-declared political ideology is well correlated with the latent dimension we recover. We therefore select the polarity of the model that makes self-declared conservatives more likely to be at the right end of the dimension and self-declared liberals more likely to be at the left end of the dimension. Finally, we assign diffuse normal priors to the intercept and slope parameters. Specifically, we assume that the intercept has an improper uniform prior and that the slope parameter has a strictly negative uniform prior, enforcing the assumption that increasing distance decreases the propensity to follow someone else: $\alpha \sim U\left(-\infty ,\infty \right)$ and ${\beta }_j\sim U\left(-\infty ,0\right)$ for $j=1$ , … J. We program and estimate our model in R and JAGS (Plummer Reference Plummer2003; R Development Core Team 2009).

We identify the universe of tweeters that each individual tweeter follows. We refer to the tweeters in our panel as “followers” and the individuals they follow as “followees.” We subset the data and retain only those followees in the top 1% of the distribution of the number of followers one has. This results in 1,551 followees remaining in our data. Similarly, we subset the data and retain only those followers in the top 80% of the distribution of the number of followees for each tweeter. This results in 522 followers remaining in our data. (The distribution of followees per follower is extremely right skewed.) We then estimate the latent ideal points, θ, for the followers in our data (as well as for the followees, of course). Of the 2,073 individuals in the matrix, 21 appear as both followers and followees. This suggests that rather than collecting data on tweets among an insular network of individuals talking to each other, we have data on tweets by people who are part of broader Twitter networks.

The estimates we report below are based on a 10,000-iteration simulation (thinned by 10), after a discarded 10,000-iteration burn-in period. Standard diagnostic tests suggest the model converges and mixes within the burn-in period. Figure 4 reports the distribution of posterior mean estimates of θ _i for each of the 522 Twitter followers in our data. We divide the data into three groups—those self-identifying as conservative, those self-identifying as liberal, and those who do not declare a political preference. Conservatives are individuals who self-identify as conservative or Republican in their profile; liberals are individuals who self-identify as liberal or Democrat in their profile. This plot provides strong evidence of the validity of our measurement model. Self-identified liberals are distributed around a mode to one end of our scale, whereas self-identified conservatives are distributed around a mode to the other end of our scale. The average ideal point for self-identified liberals is −0.14, whereas the average ideal point for self-identified conservatives is 0.22, and the t-statistic for the difference between these two groups is 12.3 (147 df).

Figure 4.

Estimated ideal points of tweeters in the panel study. Distribution of posterior mean estimates of latent ideology for individuals in our Twitter panel. Liberals are individuals who identify themselves as either liberals or Democrats in their Twitter profiles. Conservatives are individuals who identify themselves as either conservatives or Republicans in their Twitter profiles. Those labeled “undeclared” did not indicate a political preference in their Twitter profiles. Estimates based on a 10,000-iteration simulation after a discarded 10,000-iteration burn-in period.

Importantly, tweeters who do not self-declare as either liberals or conservatives are distributed bimodally. There is a mode right around the self-identified liberals and another mode right around the self-identified conservatives. This finding is consistent with the interpretation of our estimates as a measure of ideology, in a world in which self-identification as liberal or conservative on Twitter is not necessarily associated with being an ideological extremist; rather, whether one chooses to self-identify as liberal or conservative may be a function of features other than agent ideological predispositions, such as interest in politics, profession, or general public profile.

Finally, it bears mentioning who some of the most liberal and conservative tweeters in our sample are. We focus on the followees, as they tend to be high-profile institutions (they have many followers, by definition), whereas the followers (members of our panel) tend to be private individuals. However, because all of these individuals are scaled in a common dimension, the followees are useful for interpreting what it means to be at one end of the dimension. Also, recall that this panel is constructed of tweeters who often tweet about same-sex marriage. Among the individuals at the far-right end of the dimension—those we interpret as “most conservative”—are Team Santorum KY, a Rick Santorum–led political group; Roaring Republican, a conservative group whose profile says that “#Liberalism is a disease”; Patriot Airborne, a self-described “Proud #NRA member”; and Andrea Silver, a tweeter whose profile says “Christian, Pro-Israel, Pro-constitution.” Among the most liberal members of our group are Big Gay News, a gay media account; SEIU, the official Twitter account of the Service Employees International Union; SenateDems, the Twitter account of the Senate Democratic Policy and Communications Committee; and GOP Unplugged, a Twitter account that mocks Republican politicians. These examples are consistent with Barberá’s (Reference Barberá2015) validation of this type of model as a method for recovering latent ideology.

A. The Effect of Supreme Court Decisions on Supportiveness

To evaluate our first set of predictions—that the Supreme Court decisions either increase supportiveness with respect to marriage-equality policies or polarize opinion among ideological and opinion groups, we first direct attention to figure 3. There, we saw little appreciable change in supportiveness associated with the Supreme Court’s same-sex-marriage decisions in June 2013. In fact, if anything, the supportiveness in tweets about same-sex marriage after the decisions was lower than before. The average level of supportiveness in the random sample before the decision was 1.68 and 1.56 after, $t>57.1$ . The average level of supportiveness in the panel was 1.13 before the decision and 1.11 after, $t>3.3$ . However, while these differences are statistically meaningful simply because of the sheer magnitude of the data, those differences are extremely small in an absolute sense. In any event, they provide no support for the proposition that the Supreme Court’s decisions shifted opinion in favor of same-sex marriage.

Turning to the polarization predictions, figure 5 shows the change in average supportiveness among all members of our panel who tweeted both before and after the Supreme Court’s decisions, as a function of their estimated conservatism (left panel) and their supportiveness before the decision (right panel). In each panel, the points show individual panelists change in supportiveness, and the points are sized proportional to the number of tweets we have from each panelist. The lines are LOESS smoothers with 95% confidence bands. As is clear, we do not observe opinion change consistent with the expectation that the decision polarized the views of liberals and conservatives.

Figure 5.

Change in panelists’ supportiveness after the Supreme Court’s decisions, as a function of conservatism and early support. Left, signed change in panelists’ average supportiveness as a function of their estimated latent conservatism. Right, signed change in panelists’ average supportiveness as a function of their average support before the Supreme Court decisions. All change measures exclude tweets on the day of the Supreme Court’s decisions. Lines are locally weighted GAM regressions with 95% confidence intervals. Points are sized proportional to the number of tweets collected from each panelist.

The right-hand panel of figure 5, which summarizes results for the same set of people studied in the left-hand panel, suggests that opinion does seem to have polarized among competing opinion groups.Footnote ⁹ If there was a polarization of opinion following the Supreme Court’s decision, we should expect to see an upward slope. Such a slope would be consistent with an increase in support for marriage equality among supporters and a corresponding decrease among opposers. This is precisely what we observe. Being more supportive of same-sex marriage before the decision is associated with an increase in supportiveness after the decision; being more opposed to same-sex marriage before the decision is associated with a decrease in supportiveness after the decision.

B. The Effect of Supreme Court Decisions on Intensity and Anger

To investigate how the Supreme Court’s decisions affected intensity and anger in opinion, consider figure 6. This figure reports conditional distributions of anger and intensity over time, for supportive and opposed tweets using a LOESS smoother over time among both supportive and opposed tweets. We interrupt the smoother at the date of the Supreme Court’s same-sex-marriage decisions on June 23 in order to avoid artificially smoothing away sharp jumps that might take place at the decision. (One could, alternatively, estimate a smoother across all of the observations, which we have done. The same basic findings emerge. However, because the effects we predict and detect are both sudden and temporary, they are easiest to see with separate fits to the two periods.) As above, the top panels show the random sample; the bottom panels show the panel.

Figure 6.

Estimated intensity and anger among tweets by estimated supportiveness over time, January 31, 2013, to August 11, 2013. Left, estimated intensity among tweets by day, divided into tweets estimated to be supportive and opposed to same-sex marriage, and those before and after the Supreme Court’s decisions in same-sex-marriage cases. Right, estimated anger among tweets by day, divided into tweets estimated to be supportive and opposed to same-sex marriage, and those before and after the Supreme Court’s decisions in same-sex-marriage cases.

Under the legitimation hypothesis, the Supreme Court’s decisions ought to have decreased intensity and anger, especially among individuals who were opposed to marriage equality. Under the polarization hypothesis, anger should have increased among opposers and decreased among supporters. Intensity may have increased or decreased following the opinion. Consider first intensity. In the random sample (the top panels of fig. 6), among opposed tweets we see a relatively low level of intensity that is constant up until the decision date. Among supportive tweets, by contrast, we see a higher, and increasing, level of intensity over time, leading up to the decision. In the panel, however, whereas opposed tweets are, just as in the random sample, less intense and relatively constant in the level of intensity before the decision, supportive tweets are decreasingly intense over time. That is, intensity is going up among supportive tweets in the random sample but down in the panel. However, in both the random sample and the panel, we see spikes in intensity among both supportive and opposed tweets, followed by a steady, and fast, decrease in intensity in the days following the decision. With respect to intensity, the evidence is clearly inconsistent with a legitimation effect. Of course, the evidence is also inconsistent with a polarizing effect, if what we were to anticipate was that supporters would grow less intense. Yet again, it is entirely possible that major Supreme Court decisions intensify both support and opposition. This is what we observe.

Consider next anger. In both samples, in the immediate days after the Court’s decisions, we see a sharp drop in anger among supportive tweets and a sharp increase in anger among opposed tweets. There is absolutely no evidence that individuals who were initially opposed to marriage equality, either those in our random sample of tweeters or in our panel, exhibited considerably less anger in the way that they expressed their opposition following the Supreme Court’s decision. The results are strongly consistent with the argument that the Court’s decisions polarized opinion groups.

A. Normative Democratic Theory and Big Data

While we believe that microblog data can be used to great effect to study a variety of theoretical problems in political science, we are motivated most directly by theoretical questions about the nature of political discourse and deliberation. One of the key issues in this area of study is the role of engagement with diverse opinions for healthy democratic deliberation. Normative theories of democracy often suggest that effective deliberation and consensus building is associated with exposure to diverse viewpoints (e.g., Barber Reference Barber1984). One line of research suggests that mass media can foster democratic deliberation, because the mass media generally expose one to a greater variety of viewpoints than do interpersonal discussions (e.g., Mutz and Martin Reference Mutz and Martin2001). However, in the context of the modern media environment, especially that of cable news and social media, concerns arise that it has become increasingly easy to self-select into a political echo chamber (see, e.g., Prior Reference Prior2007). Such a trend may raise particularly troubling normative implications in the context of modern social media (Sunstein Reference Sunstein, Fishkin and Laslett2008). Much research on social media, and Twitter in particular, though, suggests that social media do not function as an echo chamber and that individuals are actually exposed to a great variety of political viewpoints on microblogs (e.g., Yardi and Boyd Reference Yardi and Boyd2010).

What might be the consequences of social media qua political echo chamber for political deliberation? Several areas of research suggest possible consequences. For example, research on momentum in politics argues that cognitive responses to hearing consensus opinion can motivate individuals to search for arguments that justify joining the consensus viewpoint (see Mutz Reference Mutz1997 for a discussion). If Twitter is a political echo chamber, one might infer that encountering little variation in opinions on social media indicates a consensus viewpoint and therefore adapt one’s opinions accordingly. However, such cognitive responses are not necessarily unidirectional. Encountering opinions that are counter to one’s prior beliefs may cause one to update one’s opinions about the source of the information and convince oneself that the source of information is wrong (e.g., Huckfeldt and Sprague Reference Huckfeldt and Sprague1995). Our data show that important dynamics in political discourse take place in the context of Twitter. We see evidence of the Supreme Court having an effect on public opinion similar to what has been documented elsewhere—it moves some “opinion moderates” in support of the policy it endorses but has a polarizing effect on the most opinionated among the public. In addition, we see emotional dynamics following a Supreme Court decision not dissimilar to what we might expect in other arenas. Thus, while we do not demonstrate whether Twitter is an echo chamber, we do show trace evidence of meaningful political discourse and deliberation on social media. We believe these are positive findings for scholars seeking to take advantage of the potential for social media to provide empirical leverage on important questions of normative significance—questions previously not amenable to empirical study.

B. Using Social Media Data to Study Politics

As we noted at the outset, modern computational techniques have been brought to bear on social media to study political phenomena. There is much promise in this work, in part because of the facility with which scholars can acquire massive amounts of data about political opinions and behavior. Moreover, not only are those data easy and cheap to access; they provide metrics of potentially more intimate political behavior and beliefs than could be captured in a traditional public opinion survey. Our study has focused on Twitter, largely because it is the social medium that most closely meets the needs of our research interest. However, scholars have relied on, and continue to make use of, many forms of social media data to study politics. For example, studies use Facebook, YouTube, and similar platforms to research political organization and mobilization, political awareness, and political participation, among other political actions (e.g., Bond et al. Reference Bond, Fariss, Jones, Kramer, Marlow, Settle and Fowler2012; King, Pan, and Roberts Reference King, Pan and Roberts2013). However, further development of analytic techniques is still needed. For example, a pressing question concerns what kinds of general conclusions about politics one can draw from social media data. What is the selection process that underlies participation on social media, and how representative are data on social media of the larger polity? Especially if one wants to make use of media such as Twitter to study public opinion, scholars will have to confront these challenges.

C. The Role of Theory in the Big-Data Revolution

Further, we believe the analysis here demonstrates the potential for bringing together big data with theoretical models of politics. As we noted at the beginning of this article, the computational revolution in the study of politics has already begun. However, much of the cutting-edge work is computational in nature, not political. As our example from the introduction about forecasting elections with Twitter illustrates, big-data empirical political science is in need of techniques to more concretely tie the metrics we develop to theoretical concepts. For example, a recent symposium in PS: Political Science and Politics was concerned with the relationship among formal theory, causal inference, and big data, with particular attention being paid to how one uses theoretical models to direct measurement strategy. The considerable analytic potential presented by social media makes the role of theory in modern empirical scholarship even more central, as theoretical interpretation is particularly important in an empirical setting in which there is sufficient power to detect statistically meaningful relationships among virtually any data one may use. We anticipate future research can, and will, continue to address these deep questions about the use of theory in a world of abundant data.