Hostname: page-component-7b7567c8c-lpg2g Total loading time: 0 Render date: 2026-11-15T14:14:50.210Z Has data issue: false hasContentIssue false

Sentiment is Not Stance: Target-Aware Opinion Classification for Political Text Analysis

Published online by Cambridge University Press:  22 April 2022

Samuel E. Bestvater*
Affiliation:
Department of Political Science, The Pennsylvania State University, University Park, PA, USA. E-mail: seb654@psu.edu, burtmonroe@psu.edu
Burt L. Monroe
Affiliation:
Department of Political Science, The Pennsylvania State University, University Park, PA, USA. E-mail: seb654@psu.edu, burtmonroe@psu.edu
*
Corresponding author Samuel E. Bestvater
Rights & Permissions [Opens in a new window]

Abstract

Sentiment analysis techniques have a long history in natural language processing and have become a standard tool in the analysis of political texts, promising a conceptually straightforward automated method of extracting meaning from textual data by scoring documents on a scale from positive to negative. However, while these kinds of sentiment scores can capture the overall tone of a document, the underlying concept of interest for political analysis is often actually the document’s stance with respect to a given target—how positively or negatively it frames a specific idea, individual, or group—as this reflects the author’s underlying political attitudes. In this paper, we question the validity of approximating author stance through sentiment scoring in the analysis of political texts, and advocate for greater attention to be paid to the conceptual distinction between a document’s sentiment and its stance. Using examples from open-ended survey responses and from political discussions on social media, we demonstrate that in many political text analysis applications, sentiment and stance do not necessarily align, and therefore sentiment analysis methods fail to reliably capture ground-truth document stance, amplifying noise in the data and leading to faulty conclusions.

Information

Type
Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s) 2022. Published by Cambridge University Press on behalf of the Society for Political Methodology
Figure 0

Figure 1Geographic distribution of machine-coded sentiment in tweets about the Women’s March.Note: Boxplots of VADER sentiment scores in geocoded tweets referencing Women’s March events in the top six most-referenced cities. Replication data from Felmlee et al. (2020, Figure 7).

Figure 1

Table 1Human-labeled sentiment and stance in tweets about the Women’s March.

Figure 2

Figure 2Geographic distribution of machine-coded sentiment and stance in tweets about the Women’s March.

Figure 3

Table 2Regression analysis: predicting Women’s March approval with ideology.

Figure 4

Figure 3Simulated probabilities: predicting Women’s March approval with ideology.

Figure 5

Table 3Sentiment and stance in mood of the nation responses.

Figure 6

Table 4Classifier performance: mood of the nation responses.

Figure 7

Table 5Regression analysis: predicting trump approval with ideology.

Figure 8

Figure 4Simulated probabilities: predicting trump approval with ideology.

Figure 9

Table 6Sentiment and stance in tweets about the Kavanaugh hearings.

Figure 10

Table 7Classifier performance: Kavanaugh tweets.

Figure 11

Table 8Regression analysis: predicting Kavanaugh approval with ideology.

Figure 12

Figure 5Simulated probabilities: predicting Kavanaugh approval with ideology.

Supplementary material: Link

Bestvater and Monroe Dataset

Link
Supplementary material: PDF

Bestvater and Monroe supplementary material

Appendix

Download Bestvater and Monroe supplementary material(PDF)
PDF 174.5 KB