A fundamental question in a democracy is how competent citizens are at making political decisions and judgments (Achen and Bartels Reference Achen and Bartels2017; Kuklinski and Quirk Reference Kuklinski and Quirk2001). Democratic citizens are perceived to be competent when they hold stable and consistent political preferences that (a) are relatively immune to elite manipulation and (b) are not based on arbitrary considerations (Druckman Reference Druckman2001a).
A large literature from the past three decades indicates that an especially effective way to assess how competent citizens are is to examine their susceptibility to framing effects (Druckman Reference Druckman2001a). In the political domain, a framing effect occurs when presenting the same political issue or problem in a different way alters citizens' attitudes, emotions or behavior (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016). If, for instance, citizens evaluate a new economic program more favorably when it is described as resulting in 90 per cent employment than when it is said to produce 10 per cent unemployment (Quattrone and Tversky Reference Quattrone and Tversky1988), the framing of the issue has affected their evaluations. Alternatively, if citizens are more tolerant of a hate group rally when it is presented as a free speech issue than as a public order issue, the way the event was framed has affected their assessment of it (Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997).
Framing effects studies are considered highly relevant for evaluating citizen competence because widespread effects ‘suggest that distributions of public preferences are arbitrary, and that political elites can manipulate popular preferences to serve their own interests’ (Chong and Druckman Reference Chong and Druckman2007b, 120). Sniderman and Theriault (Reference Sniderman, Theriault, Saris and Sniderman2004, 134) make a similar point by arguing that when citizens are easily moved from one side of an issue to the other via framing, this ‘suggests that the positions they take are far from securely anchored in underlying, enduring principles’.
The claim that framing research can be used to draw inferences about citizen competence calls for a broad account of the prevalence and strength of framing effects across contexts. Yet prior research has only partly achieved that goal. Individual framing experiments have yielded mixed conclusions about citizens' susceptibility to framing effects: some scholars have concluded that framing effects are ‘meaningful and important determinants of public opinion’ (Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997, 224), whereas others have suggested that framing has only a ‘limited influence … on overall evaluations’ (Hopkins Reference Hopkins2018, 702). Past efforts to provide cumulative insights about framing effects have also been insufficient because all have adopted a narrative review approach, which (1) does not consider the entire available evidence and (2) does not combine and analyze actual study results.
In this article, we argue that a systematic meta-analysis of the framing effects literature is a necessary complement to prior individual experiments and narrative reviews. Our meta-analytic approach to the study of framing effects is motivated by five features of this literature. The first is its very large size, which made it impossible for previous narrative reviews to determine how powerful framing effects are across contexts. Secondly, scholars long ago identified two fundamentally different types of influence – equivalency and emphasis framing effects (Druckman Reference Druckman2001a). To date, however, their effects on citizens have not been comprehensively compared. Thirdly, the framing effects literature rarely discusses the magnitude of influence; it focuses almost exclusively on tests of statistical significance. Fourthly, prior discussions often depict framing effects as a general concept and overlook the fact that very different political outcomes have been studied in relation to framing (for example, attitudes vs. behavior). Finally, we are not aware of prior attempts to empirically assess the conditions that augment or attenuate the effects of framing across contexts.
To address these concerns, we have meta-analyzed the literature on framing effects in the political domain. Our findings, which are based on 138 experiments conducted among 64,083 citizens, indicate that framing has statistically significant and medium-sized average effects on citizens' political attitudes (d = 0.41) and emotions (d = 0.47). These effects support the theoretical claim that political and media elites can influence citizens by framing issues (Iyengar Reference Iyengar1991; Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997). They also show that the magnitude of this influence is often not small. However, our meta-analysis also demonstrates that framing effects are much more limited in two important contexts: frames have a weak (even negligible) average impact on citizens' behavior (d = 0.11), and their effect weakens substantially when a competing frame is introduced (d = 0.18). The latter two findings confirm the claim that even though framing can generally influence citizens, its impact is quite weak under more consequential and realistic political circumstances (Hopkins Reference Hopkins2018).
These results suggest that scholars should not take it for granted that framing ‘works’. Instead, any discussion of how strong and widespread framing effects are should consider the political outcome being studied as well as the context in which the frame is presented. From a normative standpoint, the robust effects of framing on political attitudes and emotions reveal that prior concerns about citizens' incompetence and susceptibility to elite manipulation cannot be easily dismissed: framing indeed influences the public. However, our demonstration that framing effects are much weaker in real-world contexts provides reasons for optimism regarding citizen competence. It indicates that the public's political preferences are not as fragile as some scholars believe them to be (for example, Achen and Bartels Reference Achen and Bartels2017), and that even though political elites can influence citizens, their power to do so is highly constrained.
We have also conducted a series of sensitivity and moderator analyses. The sensitivity analyses demonstrate that our results remain robust after accounting for potential publication bias, measurement error and the presence of outliers. The moderator analyses reveal that framing effects are generally similar in size across various subgroups of studies. For instance, some scholars have criticized the use of student samples in experiments (Sears Reference Sears1986), and other researchers have emphasized the theoretical distinction between text-based framing and visual framing (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016). However, we find little evidence of a moderating influence of these factors (and others) on the effect size.
MOTIVATION
Our meta-analysis of framing effects in the political domain is motivated by five features of this literature. The first is its sheer size. In the past three decades, researchers have generated a constant stream of studies examining the political effects of framing (Brugman and Burgers Reference Brugman and Burgers2018). Prior attempts to sort through the extensive framing effects literature have taken a narrative approach, focusing on conceptual tasks such as classifying frames into theoretically distinct types, or distinguishing framing from related concepts like priming and persuasion (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016; Druckman Reference Druckman2001a; Matthes Reference Matthes2009; Scheufele Reference Scheufele1999). While these literature reviews have significantly advanced framing effects research, they are, by definition, unable to determine whether, and to what extent, ‘framing works’ across contexts because they do not analyze actual research results. Statistically integrating the results of previous studies enables hypothesis testing that is both highly generalizable (as it goes beyond the specific stimuli, measures and procedures used in any individual study) and more powerful (since much more empirical evidence is being considered).
The second motivation is the fact that the term ‘framing effects’ in fact encapsulates two different types of influence: equivalency and emphasis framing effects (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016; Druckman Reference Druckman2001a). The former occurs when a different presentation of logically identical information affects people (for example, 90 per cent employment vs. 10 per cent unemployment). The latter occurs when highlighting different considerations regarding a political issue or event influences individuals (for example, presenting a hate group rally as a free speech vs. a public order issue). Although both definitions have been studied extensively in the literature, it remains unclear whether the two types have differential effects on citizens.
The third motivation for our meta-analysis is our observation that framing effects studies tend not to report standardized effect sizes, and focus instead on tests of statistical significance. This approach has serious shortcomings, as a statistically significant effect may have little practical relevance, and a statistically insignificant effect might stem from low statistical power (Borenstein et al. Reference Borenstein2009). A meta-analysis addresses these concerns, as it (a) converts the results of all available studies in the literature into a standardized metric representing the magnitude of influence and (b) takes the statistical power of each primary study into account.
Our fourth motivation is the fact that prior research examines framing effects on a variety of political outcomes. In most cases, these outcomes can be classified into one of three types: (a) citizens' political attitudes, especially their support for different policy measures (for example, Malhotra and Margalit Reference Malhotra and Margalit2010); (b) politically relevant emotional reactions, including negative emotions such as anger and fear, or positive emotions such as hope and enthusiasm (for example, Brader, Valentino and Suhay Reference Brader, Valentino and Suhay2008); or (c) various political behaviors, such as turning out to vote or donating money to a political organization (for example, Gerber et al. Reference Gerber2018). Despite this diversity, prior accounts use the general term ‘framing effects’ without distinguishing among outcomes.
The fifth and final motivation for this study is the need to systematically assess the conditions under which framing effects occur. While early framing studies focused on demonstrating that citizens, in general, are affected by framing (Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997; Zaller Reference Zaller1992), subsequent studies have identified variables that condition framing effects (for example, Lecheler, de Vreese and Slothuus Reference Lecheler, de Vreese and Slothuus2009). Since the unit of analysis in a meta-analysis is a single study (rather than a single person), our focus here is on study-level moderators of framing effects.
THEORY
In the following sections, we review some of the main theoretical considerations relevant to anticipating the effects of framing on different political outcomes. We begin by offering a theoretical perspective on framing effects on citizens' political attitudes. We then discuss the anticipated effects on emotions and behavior. Finally, we discuss the expected effects of framing in the presence of competition.
Framing Effects on Citizens' Attitudes
The political outcome that has been studied most frequently in the context of framing effects is citizens' attitudes. Prior research proposes three main psychological mechanisms by which framing is expected to influence the political attitudes of citizens: availability, accessibility and applicability. These theoretical explanations are discussed in detail elsewhere (for example, Chong and Druckman Reference Chong and Druckman2007a) and are only briefly summarized here. First, at the most basic level, a frame can influence a citizen only if the considerations it emphasizes are already available in that citizen's mind. In other words, if a citizen has no understanding of the concept the frame emphasizes (for example, for a ‘10 per cent unemployment’ frame, if the citizen does not know what unemployment means), he or she will not comprehend the frame, and will therefore not be influenced by it (Chong and Druckman Reference Chong and Druckman2007b). The second explanation for framing effects focuses on accessibility. This perspective posits that frames impact citizens by bringing some considerations to the forefront of conscious thought (Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997, 568). According to this view, framing effects occur because considerations that have become salient in citizens' minds are more likely to be retrieved from long-term memory when people are asked to report their attitude on the issue (Zaller Reference Zaller1992). The third prominent explanation for framing effects, applicability, is only relevant to emphasis framing effects. This perspective holds that in cases where citizens are motivated to consciously evaluate the content of a frame, the frame's relevance to the case at hand (or its strength) is what determines its effectiveness (Chong and Druckman Reference Chong and Druckman2007b).
The three theoretical explanations for framing effects described above all predict that when citizens are exposed to opposite frames, their attitudes should be driven in opposite directions. To date, however, the effects of framing on citizens' political attitudes have only been examined in single studies, not in a meta-analysis. We expect to find that, when examined across contexts, framing will drive citizens' political attitudes in opposite directions (Hypothesis 1).
As we have discussed above, the literature has identified two distinct types of framing effects – equivalency and emphasis. Yet to date, the effects of these two framing types in the political domain have not been comprehensively compared. Based on influential work in the field (Chong and Druckman Reference Chong and Druckman2007b) and on recent meta-analytic evidence from the non-political domain (Nabi et al. Reference Nabi2019), we anticipate that emphasis frames will yield, on average, larger effects than equivalency frames. The reason for this is that emphasis frames vary more elements of the message (Nabi et al. Reference Nabi2019). Equivalency frames are ‘logically identical ways of making the same statement’ (Druckman Reference Druckman2001a, 230); they present the same information in either a positive or negative light (for example, employment vs. unemployment). Emphasis frames, however, vary how the information is presented and its content (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016; Leeper and Slothuus Reference Leeper and Slothuus2015); each emphasis frame highlights a different consideration (for example, free speech vs. public order). Since people often possess a mix of conflicting considerations on political issues (Zaller Reference Zaller1992), emphasis frames can alter the weight they attach to different aspects of an issue (Chong and Druckman Reference Chong and Druckman2007b; Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997). Therefore, they are more likely than equivalency frames to drive attitudes in opposing directions. We thus hypothesize that on average, emphasis frames will have a greater effect than equivalency frames on attitudes (Hypothesis 1a). Note that we compare the effects of emphasis and equivalency framing only on attitudes, as this is the only political outcome that was examined in a large enough number of primary studies.
Framing Effects on Citizens' Emotions
Whereas early framing effects studies focused almost exclusively on citizens' attitudes, other politically relevant outcomes have received growing attention in recent years. One fast-growing line of research examines whether the way political issues are framed elicits differential emotional responses from citizens (Druckman and McDermott Reference Druckman and McDermott2008). Drawing on psychological research, emotions are conceptualized as short-lived, intense mental states that have a strong physiological component and are directed at some external stimuli (Nabi Reference Nabi, Dillard and Pfau2002). Since emotions can facilitate important political consequences, such as policy support and mobilization (Brader and Marcus Reference Brader, Marcus, Huddy, Sears and Levy2013; Panagopoulos Reference Panagopoulos2010), they play an important role in the political arena.
Researchers have studied framing effects on a variety of discrete positive and negative emotions. Positive emotions that have been studied frequently in the framing literature include hope, empathy, happiness and enthusiasm; widely studied negative emotions include fear, anger, disgust and guilt (for example, Brader, Valentino and Suhay Reference Brader, Valentino and Suhay2008; Lecheler, Schuck and de Vreese Reference Lecheler, Schuck and de Vreese2013). Although the empirical evidence on emotions has begun to accumulate, it is not yet clear to what extent framing influences this outcome due to mixed results in the literature. Whereas most past studies have suggested substantial framing effects on citizens' emotions (for example, Aarøe Reference Aarøe2011; Lecheler, Schuck and de Vreese Reference Lecheler, Schuck and de Vreese2013), some studies have observed modest effects or even no direct effects at all (for example, Quick et al. Reference Quick2014).
To explore the extent to which framing influences emotions, we meta-analyze the existing evidence on this topic. Building on prior studies (for example, Lecheler, Schuck and de Vreese Reference Lecheler, Schuck and de Vreese2013; Nabi et al. Reference Nabi2019), we hypothesize that framing influences citizens' emotions when examined across contexts (Hypothesis 2). In addition to this general expectation, we provide a more nuanced account of framing effects on emotions: we expect frames that emphasize positive aspects of a policy or issue to induce positive emotions (Hypothesis 2a), and those that highlight negative aspects to induce negative emotions (Hypothesis 2b).
Framing Effects on Citizens' Behavior
Even though attitudes and emotions play an essential role in politics, it is clear that what matters most for the political process in general, and for political elites trying to influence the public in particular, is what citizens do. In the context of framing effects, this means that the most consequential outcome a frame can have is to alter the behavior of citizens. In recent years, scholars have examined the effects of framing on a variety of political behaviors, including turning out to vote (Gerber et al. Reference Gerber2018), seeking information about political issues (Leeper Reference Leeper2017), adopting environmentally friendly habits (Bolsen, Druckman and Cook Reference Bolsen, Druckman and Cook2014) and many more.Footnote 1
At first glance, one might expect attitudes and behavior to be closely linked; if this is the case, framing should have a similar impact on both outcomes. Yet previous research provides both empirical and theoretical reasons to doubt the intensity of this connection. Empirically, prior meta-analyses indicate that political communications are usually more successful at affecting citizens’ stated attitudes than their behavior. Zoizner (Reference Zoizner2018), for instance, finds that exposure to horserace coverage of politics makes citizens more cynical (an attitudinal outcome) but finds no significant effect of such coverage on political participation (a behavioral outcome). Benoit, Leshner and Chattopadhyay (Reference Benoit, Leshner and Chattopadhyay2007) find that campaign advertising has a much stronger effect on citizens' policy attitudes than on turnout. Strikingly, Kalla and Broockman (Reference Kalla and Broockman2018) show that the average effect of campaign contact on citizens' voting behavior is zero.
From a theoretical point of view, these results are consistent with psychological theories that focus on predicting human behavior. Perhaps the two most relevant perspectives on this issue are the Theory of Reasoned Action (Fishbein and Ajzen Reference Fishbein and Ajzen1975) and its extension, the Theory of Planned Behavior (Ajzen Reference Ajzen1991). Both theories propose a causal model in which people's attitudes predict their behavioral intentions, which, in turn, predict their actions. While these theories have received extensive empirical support, researchers have consistently observed only moderate correlations between people's attitudes and behavior (Hale, Householder and Greene Reference Hale, Householder, Greene, Dillard and Pfau2002). Scholars have suggested that attitudes often do not predict behavior because behavior is influenced by many factors other than attitudes (Fishbein and Ajzen Reference Fishbein and Ajzen1975) or because attitudes are often not strongly held (Howe and Krosnick Reference Howe and Krosnick2017). In line with these theoretical perspectives, O'Keefe (Reference O'Keefe2013) proposed that persuasive effects should weaken as one moves from attitude to behavioral intention to actual behavior. To examine whether this principle applies to political framing effects, we will test the hypothesis that framing effects have a weaker influence on citizens' behavior than on their attitudes (Hypothesis 3).
Framing Effects in Competitive Settings
The most common empirical strategy for studying framing effects on citizens entails conducting an experiment that (a) randomly assigns subjects to groups, (b) exposes each group to a different single frame and (c) compares the means of the groups on some outcome variable(s). A classic example of such a design is Nelson, Clawson and Oxley's (Reference Nelson, Clawson and Oxley1997) seminal framing effects study, which (a) assigned subjects to one of two groups, (b) let each group read a differently framed news article about a planned Ku Klux Klan (KKK) rally (one framed it as a ‘public order’ issue and the other as a ‘free speech’ issue) and (c) compared average tolerance for the KKK rally in both groups.
While studies that use such a single-frame design provide a crucial basis for understanding when and how framing works, researchers have argued that the external validity of such studies is limited. The reason is that studies exposing each subject to only one frame misrepresent citizens' real-world information environment, where they are constantly exposed to conflicting arguments on political issues. This branch of the framing effects literature emphasizes that in real-world scenarios, elite frames are almost always contested, debated and opposed.
Inspired by the early influential studies of Sniderman and Theriault (Reference Sniderman, Theriault, Saris and Sniderman2004) and Chong and Druckman (Reference Chong and Druckman2007a), the past decade has seen a surge of interest in the impact of competitive framing on citizens' attitudes.Footnote 2 A typical competitive framing study follows the same structure as the ‘classic’ framing experiment described above, with one crucial modification: instead of exposing all subjects to a single frame, one experimental group is simultaneously exposed to two conflicting frames on the same issue.
The theoretical logic guiding competitive framing studies is as follows: while a single ‘pro’ frame moves citizens to support a policy and a single ‘anti’ frame moves them to oppose it, a ‘dual’ message presenting both perspectives should result in little attitude change. This effect happens because – assuming that the frequency of exposure and the quality of the frames are held constant – conflicting frames tend to cancel out each other's impact (Chong and Druckman Reference Chong and Druckman2007a). A meta-analysis of the competitive framing literature is needed both because dozens of studies employing frame competition have accumulated in the past decade and, more substantively, because some of these studies have confirmed the theoretical claim that competing frames cancel each other's influence (Arceneaux Reference Arceneaux2012) while others did not (Newman et al. Reference Newman2015). Our fourth and final hypothesis thus posits that the effect of framing diminishes substantially in competitive settings (Hypothesis 4).Footnote 3
METHOD
Locating Studies
To ensure that we meta-analyze all available published studies on framing effects in the political domain, we conducted a thorough literature search consisting of four stages. First, we gathered an initial list of 142 studies mentioned in review articles on framing effects (for example, Brugman and Burgers Reference Brugman and Burgers2018; Druckman Reference Druckman2001a). Secondly, we searched, through June 2018, Google Scholar and Web of Science for additional studies using the following search terms and Boolean operators: ((frame OR framing) AND (polit* OR polic*)). This resulted in 296 additional relevant studies. Thirdly, we scanned the reference lists of all studies deemed relevant for additional sources they may have cited, which yielded thirty-seven more studies. Finally, we contacted by email the authors of all political framing effects studies we had initially located, presented them with our list of studies and initial results, and asked them for additional studies. This resulted in sixty-two more studies. At the end of these four stages, our list contained 537 independent framing effects studies, which were all comprehensively coded for the variables we describe below.
Inclusion Criteria
Since the 537 studies we located and coded explore an enormous variety of frames and outcome variables, they could not be combined in a theoretically meaningful way. Therefore, at this point, we narrowed our dataset by only including studies that met the following criteria. First, we included studies that explored the influence of emphasis or equivalency frames that could be classified as either positive, negative or competitive. Positive frames emphasize the beneficial aspects of the target being framed (for example, policy programs, political events) and are also referred to as ‘pro’ or ‘gain’ frames. In contrast, negative frames focus on the detrimental or harmful aspects of the target being framed and are also called ‘con’, ‘anti’ or ‘loss’ frames. For example, in the context of equivalency frames, one may refer to the potential of a new economic policy to generate 90 per cent employment (positive framing) or 10 per cent unemployment (negative framing). For emphasis frames, one can stress the economic benefits of a policy (positive framing) or its possible environmental costs (negative framing). Competitive frames are those that simultaneously present both positive and negative aspects of the issue or event. All frames focusing on other elements, such as labeling climate-mitigation policies as derived from ‘climate change’ or ‘global warming’ (Schuldt, Roh and Schwarz Reference Schuldt, Roh and Schwarz2015), were excluded.Footnote 4
The second inclusion criterion was that the study's subjects are ordinary citizens (rather than, for example, political elites). Thirdly, we only included studies that adopted an experimental design. Fourthly, the issue being framed in the study had to be political. Finally, the study had to examine framing effects on attitudes, emotions or behavior (we excluded other outcomes, such as message learning). Removing studies that failed to meet any of these criteria reduced our dataset to 138 independent framing experiments. The full list of studies included in our meta-analysis is presented in Appendix A.
Effect Size Calculation
Since we meta-analyze experiments that compare the means of experimental groups on some dependent variable(s), we use the standardized mean difference (Cohen's d) as our effect size measure. In practice, very few studies in the framing effects literature directly report Cohen's d. We therefore had to extract the test statistics each primary study reported and convert them to Cohen's d. To do so, we used the formulas provided by Borenstein et al. (Reference Borenstein2009). For all studies that were deemed relevant but which did not report sufficient statistical information (for example, if it reported the means of experimental groups but not their corresponding standard deviations/standard errors), we contacted the first author and asked for the missing details.
Outcome Variables
To compare the effects of framing across outcomes, we had to reduce the large and highly diverse set of dependent variables examined in the framing literature to a smaller, more conceptually meaningful set of categories. Building on prior meta-analytic literature (for example, Kurdi et al. Reference Kurdi2019; O'Keefe Reference O'Keefe2013), we classified outcome variables as either attitudinal, emotional or behavioral. Attitudinal outcomes mainly included measures of policy support but also some other concepts, such as the perceived consequences of a policy. Emotional outcomes included discrete emotions, which in all cases had either a positive (for example, hope) or negative (for example, fear) valence. Finally, behavioral outcomes included any actual behavior (for example, turning out to vote) or behavioral intention (for example, willingness to participate in politics) studied in relation to framing.
For each study included in the meta-analysis, we analyzed all outcome variables that fall into one of our three categories (attitudes, emotions and behavior). If a single study examined several outcome variables that belong to the same category (for example, policy support and candidate evaluation were both classified as attitudinal outcomes), we aggregated them into a single effect size (Borenstein et al. Reference Borenstein2009). Alternatively, when a single study examined outcome variables from distinct categories (for example, policy support and turnout were classified as attitudinal and behavioral outcomes, respectively), we included each outcome in the respective meta-analysis.
Moderators
To examine whether study-level characteristics moderate the effects of framing, we coded each study for multiple variables whose importance was highlighted in previous framing research. First, as discussed throughout the theoretical section, we coded for framing conceptualization (emphasis or equivalency framing) and type of outcome (attitudes, emotions or behavior). Secondly, researchers have extended the scope of framing effects from text-based only to other modes, such as visual framing (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016) and pointed to the different processes underlying exposure to each medium (Powell et al. Reference Powell2015). We therefore coded for medium type (written text, visual stimulus, a combination of text and visuals, or oral stimulus). The literature also stresses the importance of the actor promoting the frame, emphasizing that citizens react differently to framing depending on its source (Druckman Reference Druckman2001b; Koch and Peter Reference Koch and Peter2017). To account for such differences, we coded for frame source (news media, political source, interest group, unspecified source or other sources, such as experts). We also coded for the country the study was conducted in (US vs. non-US) since much of the knowledge on the effects of political communications is based on the American context, which is a very distinct case compared to other political and media systems (Schmitt-Beck Reference Schmitt-Beck, Esser and Hanitzsch2012).
Next, since researchers have criticized the external validity of studies that use students as experimental subjects (Sears Reference Sears1986), we measured the sample composition of each study (student sample, general population–representative, general population–nonrepresentative or mixed sample). The period in which the study was conducted (election period or non-election period) was another moderator, since framing effects may weaken during election periods, in which citizens absorb more political information and form stronger opinions than at other times (Kalla and Brookman Reference Kalla and Broockman2018). Finally, in studies investigating framing effects on behavioral outcomes, we coded for the type of behavior, actual or intended, as these two types may be viewed as theoretically distinct (Ajzen Reference Ajzen1991).
RESULTS
To test our hypotheses, we estimated a series of random-effects meta-analyses. Table 1 presents one meta-analysis for the effects of positive vs. negative frames on each of the outcome variables (attitudes, emotions and behavior), followed by another meta-analysis for the impact of competitive framing vs. a baseline control group. According to our first hypothesis, framing influences citizens' political attitudes. We find an average effect of d = 0.41 (95 per cent CI [0.36, 0.47], z = 16.10, p < 0.001, k = 161) for attitudes, which allows us to confirm Hypothesis 1. Exposure to opposite frames indeed drives citizens' political attitudes in opposite directions. Following the conventions for interpreting effect sizes (small: d = 0.20; medium: d = 0.50; and large: d = 0.80) proposed by Cohen (Reference Cohen1988), we interpret this average effect as moderately sized.
Framing effects on citizens' attitudes, emotions and behavior
Note: k = number of effect sizes; z = test statistic of z-test. CI = confidence interval. *p < 0.05. **p < 0.01. ***p < 0.001
Our second meta-analysis tests the effects of framing on emotions. Here, too, we find a significant and moderately sized average effect of d = 0.47 (95 per cent CI [0.33, 0.61], z = 6.51, p < 0.001, k = 14), which allows us to confirm Hypothesis 2. We also conducted two separate meta-analyses, one examining whether positive frames induce more positive emotions than negative frames (Hypothesis 2a) and the other examining whether negative frames induce more negative emotions than positive frames (Hypothesis 2b). We find support for both hypotheses: positive frames result in more positive emotions (d = 0.42, 95 per cent CI [0.05, 0.78], z = 2.23, p < 0.05, k = 9) and less negative emotions (d = –0.39, 95 per cent CI [–0.58, –0.20], z = –4.10, p < 0.001, k = 12) than negative frames.Footnote 5
Our third meta-analysis examines the impact of framing on citizens' behavior. We find that the average effect of framing on behavior is statistically significant (d = 0.11, 95 per cent CI [0.06, 0.15], z = 4.83, p < 0.001, k = 26) but substantially weaker than the effect on attitudes (d = 0.41). To formally test the difference between the effects of framing on attitudes and behavior, we conducted three tests. The first is a subgroup analysis (Borenstein et al. Reference Borenstein2009) by type of outcome. The results show a statistically significant difference between the effects of framing on attitudes, emotions and behavior (Q(2) = 90.58; p < 0.001; see Appendix Table B1). Secondly, to directly compare the effects on attitudes and behavior, we performed a meta-regression. These results indicate that framing effects on behavior are significantly smaller than on attitudes (b = –0.28; SE = 0.06; 95 per cent CI [–0.39, –0.16]; p < 0.001).Footnote 6 Thirdly, to ensure that the stronger effects on attitudes are not the result of stronger framing manipulations in the attitudes studies, we conducted a separate meta-analysis of all studies that have used the same framing manipulation and the same sample to examine effects on both attitudes and behavior (k = 15). Since the effect sizes in this meta-analysis are dependent, we conducted a robust variance meta-regression (Hedges, Tipton and Johnson Reference Hedges, Tipton and Johnson2010). We find that studies using the same manipulation and sample to examine framing effects on attitudes and behavior yield significantly smaller effects on behavior (b = –0.18, SE = 0.07, 95 per cent CI [–0.34, –0.03]; p = 0.02). Taken together, these findings provide strong support for Hypothesis 3. Figure 1 plots the meta-analytically derived effect size for each dependent variable (attitudes, emotions and behavior).
Mean effect sizes and 95 per cent CIs for the effects of two framing types on citizens' attitudes, emotions and behavior
Note: point estimates and CIs are based on the unadjusted effect sizes from Table 1.
In addition to directly comparing positive vs. negative framing, we also examined whether framing effects are driven by people's tendency to put more weight on negative information, a phenomenon known as negativity bias (Soroka Reference Soroka2014). We examined this possibility by meta-analyzing the effects of positive and negative frames compared to a control group that received a neutral (or no) stimulus (see Appendix Table B4). The results show that attitudes are equally affected by positive and negative frames, whereas behavior is only affected by negative frames.
Next, we examine whether the effect of framing in competitive settings diminishes substantially compared to non-competitive settings (Hypothesis 4). Since we could only locate two studies that tested the effect of competitive framing on emotions and behavior (one study per outcome), we focus on the impact of frame competition on citizens' attitudes, which was examined in dozens of studies. To test Hypothesis 4, we meta-analyze the average movement of opinions in a competitive framing condition compared to a control group. A neutral control group provides a better comparison for this effect than a single (positive or negative) frame for two reasons. First, whereas single frames vary across studies in their direction and quality (Chong and Druckman Reference Chong and Druckman2007a), a neutral control group remains similar across studies. Secondly, the competition vs. control comparison is the most common analytic strategy in the literature for evaluating competitive framing effects (for example, Druckman and Leeper Reference Druckman and Leeper2012).
Table 1 shows that the average effect size of competitive framing on political attitudes is substantially smaller (d = 0.18, 95 per cent CI [0.14, 0.22], z = 8.74, p < 0.001, k = 34) than the effect of non-competitive framing reported above (d = 0.41). A subgroup analysis reveals that this difference is statistically significant (Q(1) = 48.95, p < 0.001; see Appendix Table B3). These substantial and significant differences, which are also apparent in Figure 1, allow us to confirm Hypothesis 4.
In regards to between-study variance, framing effects on attitudes (Q(160) = 1,273.37, p < 0.01, I 2 = 87.4), emotions (Q(13) = 83.05, p < 0.001, I 2 = 84.3) and behavior (Q(25) = 46.30, p < 0.01, I 2 = 46), as well as competitive framing effects on attitudes (Q(33) = 61.39, p < 0.01, I 2 = 46.2), all yielded significant heterogeneity. This indicates that framing effects may be contingent on study-level factors that moderate the original effect size estimate. In the following section, we report the results of various moderator analyses in which we test for this possibility.
Moderator Analyses
We examined whether conceptual and methodological characteristics of studies in our meta-analysis condition the average effect size of framing. For each moderator, we performed a subgroup analysis using a mixed-effects model (Borenstein et al. Reference Borenstein2009). Appendix B presents the full results of the moderator analyses for positive vs. negative framing (Table B1) and for competitive framing (Table B2).Footnote 7 In the context of positive vs. negative framing effects on attitudes, the results show that none of the moderators we examined significantly influence the effect size. Most interestingly, contrary to our expectation, framing type did not have a moderating effect (equivalency vs. emphasis framing). This result, plotted in Figure 2, leads us to reject Hypothesis 1a; we find no evidence that emphasis framing yields larger effect sizes on political attitudes than equivalency framing. We also find no moderating influence for country, period or type of behavior (actual vs. intended).
Mean effect sizes and 95 per cent CIs for the effects of emphasis and equivalency frames on citizens' attitudes
Note: point estimates and CIs are based on the effect sizes reported in Appendix B.
In line with prior framing research (Kühberger Reference Kühberger1998), we find no difference in average effect size between student samples and other sample compositions (Q(3) = 2.77; p > 0.05). Furthermore, although researchers have called for expanding the scope of framing effects from text-based frames to other types of framing (Cacciatore, Scheufele, and Iyengar Reference Cacciatore, Scheufele and Iyengar2016), we find that written texts are still by far the most dominant way of studying framing effects (k = 133, compared with only k = 8 for visual presentation and k = 16 for oral stimuli). The results for this moderator also indicate no significant difference between medium types (Q(3) = 2.26; p > 0.05).
Sensitivity Analyses
Publication bias
We tested whether our results are robust to publication bias, which is the tendency of studies with insignificant findings to remain unpublished. Such bias raises concerns that the published literature overestimates effect sizes. We assessed the potential impact of publication bias using three methods: trim and fill (Duval and Tweedie Reference Duval and Tweedie2000), p-curve (Simonsohn, Nelson and Simmons Reference Simonsohn, Nelson and Simmons2014) and cumulative meta-analysis (Ioannidis and Lau Reference Ioannidis and Lau2001). In Table 1, we report the average effect sizes adjusted for publication bias according to the widely used trim and fill approach. Whereas the average effect on attitudes remains unchanged, framing effects on emotions (from d = 0.47 to d = 0.29) and behavior (from d = 0.12 to an insignificant d = 0.03) become smaller. This decline suggests that the original effects for these two outcomes might be inflated due to a ‘file-drawer’ problem. Using the p-curve and cumulative meta-analysis approaches, however, we find no evidence of substantial publication bias (see Appendices C and D).
Adjustment for measurement error
Another threat to the accuracy of meta-analytic results is measurement error in the dependent variables in primary studies, which may attenuate the effect sizes they report. To correct for this type of error, we applied Hunter and Schmidt's (Reference Hunter and Schmidt2004) artifact distribution approach and used the scale reliability (Cronbach's alpha) reported in each primary study to correct for attenuation of its effect size estimate. For studies that did not report a reliability estimate, we imputed alpha values by meta-analyzing the reliability scores from the studies that did report it (Reference BonettBonett 2010). Table 1 displays the unattenuated (that is, adjusted for measurement error) overall effects. They are practically identical to the unadjusted estimates.
Outlier detection
Appendix E reports a series of tests examining whether our conclusions are driven by extreme observations. None of our conclusions changes after removing outliers from the analyses.
DISCUSSION
Despite the dominance of framing as a theoretical and conceptual framework for studying elite influence on citizens, several questions about it remain unanswered. For instance, how large and widespread are framing effects? Does the magnitude of influence vary across types of outcomes, frames and other relevant study-level characteristics? Our meta-analysis, which encompasses 138 studies reporting on 237 framing effects (N = 64,083), reveals that, on average, framing has a statistically significant influence on citizens. However, we also find that the magnitude of this influence varies systematically in two important respects: behavior is much less responsive to elite frames than attitudes and emotions, and the presence of a competing frame substantially diminishes framing effects.
Seen as a whole, our findings support the view that framing effects are real, but that their nature is conditional. We find consistent support for the claim that framing influences political attitudes (Chong and Druckman Reference Chong and Druckman2007b; Zaller Reference Zaller1992). This indicates that by simply altering the way they present a political issue or event, elites can substantially influence citizens' support for (and evaluation of) a policy, at least in the short term. Framing also appears to influence citizens' emotions – both positive and negative – quite strongly. This finding is in line with prior accounts showing that communications from political elites can induce powerful emotional reactions from citizens (for example, Panagopoulos Reference Panagopoulos2010).
However, our results also clearly highlight the limits of framing effects. We find that when framing experiments study more politically consequential outcomes (that is, intended or actual behaviors) and when they more closely mimic political reality by incorporating frame competition, the effects they report are much smaller. We interpret the effects on behavior and competition as weak since they are (a) substantially (and significantly) smaller than the effects on attitudes and emotions, (b) considered weak according to known conventions (Cohen Reference Cohen1988) and (c) smaller than the average effect sizes reported in prior framing meta-analyses (Nabi et al. Reference Nabi2019; Zoizner Reference Zoizner2018).
The fact that framing has a much stronger influence on attitudes than on behavior speaks to long-standing discussions about the relationship between these two constructs. Social psychologists long ago identified the problem of attitude–behavior inconsistency and have repeatedly demonstrated that people very often say one thing and do another (Fishbein and Ajzen Reference Fishbein and Ajzen1975). In our context, we find that even though framing is an important determinant of public opinion (Nelson, Clawson and Oxley Reference Nelson, Clawson and Oxley1997), it has a relatively marginal role in shaping political behavior. Recent work, such as Levine and Kline's (Reference Levine and Kline2017), has found that issue framing is a powerful determinant of people's stated support for climate mitigation policies, but it is far less effective at motivating people to spend scarce resources of time, money and attention on communicating that support. Our meta-analysis confirms this principle across a wide variety of political issues. Since ‘frames that influence both public opinion and collective political action in the same direction are likely to have the biggest impact on the political agenda and policy change’ (Levine and Kline Reference Levine and Kline2017, 307), more framing studies should explicitly test effects on both.
Our finding that competing frames result, on average, in little attitude change represents another contribution to the framing literature. Even though researchers have argued that competing frames cancel out each other's influence (Chong and Druckman Reference Chong and Druckman2007a; Sniderman and Theriault Reference Sniderman, Theriault, Saris and Sniderman2004), this has so far only been tested in single studies, which have reported mixed results (Arceneaux Reference Arceneaux2012; Newman et al. Reference Newman2015). Our meta-analysis of competitive framing is based on an aggregation of thirty-four independent experiments, which makes it the most powerful and generalizable test of this hypothesis to date. The small average effect we observe for frame competition further attests to the limits of framing because elite frames are almost always debated and contested in the real world.
The current study presents what we believe is the first comprehensive comparison of emphasis and equivalency framing effects in the political domain. We find that despite the clear conceptual differences between the two types (Druckman Reference Druckman2001a), from a purely empirical standpoint, their magnitude of influence on citizens is indistinguishable. In a recent influential essay, Cacciatore, Scheufele and Iyengar (Reference Cacciatore, Scheufele and Iyengar2016) called for a paradigm shift in framing research, urging scholars to return to a narrower definition of framing based only on equivalency. These scholars have argued that a loose definition of framing that also encompasses emphasis frames ‘has undoubtedly contributed to making framing effects appear as much more widespread and powerful than they actually are’ (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016, 14). Our results do not support that claim: we find that defining framing as either emphasis only, equivalency only or both types combined leads to the same conclusion about frames’ overall capacity to change citizens' attitudes, which is of moderate magnitude in all cases. In other words, framing is similarly powerful regardless of how it is defined. While we find that the impact of these two types of framing is similar in size, our meta-analysis does not enable us to examine whether they have differential mechanisms of influence (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016). We leave this question to future research.
Our study is also the first to quantitatively evaluate, across the literature, the moderators of framing effects in the political domain. We find that framing effects generally remain constant across subgroups of studies. Among these null results, two are particularly revealing. First, we find that sample composition (for example, student samples vs. non-student samples) makes little difference to the results of framing experiments. This result alleviates concerns about the frequent use of non-representative and convenience samples in the literature, and is in line with (1) prior research making similar claims about political science experiments (Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia2011) and (2) the results of a previous meta-analysis of framing effects from psychology (Kühberger Reference Kühberger1998).
Secondly, we examined whether medium type (for example, textual or visual) moderates framing effects. Since visual cues dominate the political information environment and may trigger different reactions than texts (Scheufele and Iyengar Reference Scheufele, Iyengar, Kenski and Hall Jamieson2017), scholars have called for an increased focus on visual framing (Cacciatore, Scheufele and Iyengar Reference Cacciatore, Scheufele and Iyengar2016; Matthes Reference Matthes2009). Our meta-analysis reveals that this call has not yet been answered: text-based frames are still by far the most dominant way of studying framing effects. In terms of effect sizes, even though some previous studies found that visual frames yield stronger effects on citizens' opinions and behavior than texts (for example, Powell et al. Reference Powell2015), we find no evidence that this is the case across the literature. Future research should therefore shed light on the specific conditions under which visuals elicit different public reactions than texts.
The current study is not the first meta-analysis of framing effects to be conducted. Prior efforts include Kühberger's (Reference Kühberger1998) widely cited study from psychology as well as several more recent analyses (for example, Nabi et al. Reference Nabi2019; O'Keefe and Jensen Reference O'Keefe and Jensen2007). These studies, however, do not focus on the political domain and do not compare the effects of framing on different types of outcomes (attitudes, emotions and behavior). One particularly relevant meta-analysis for comparison is the work of Leeper and Slothuus (Reference Leeper and Slothuus2015), who synthesized fifty-five political emphasis-framing experiments and found an average effect size of d = 0.35. Our meta-analysis replicates their results almost precisely, yet we also extend their meta-analysis significantly by examining additional outcomes (emotions and behavior), a broader set of framing effects (that is, equivalency frames) and moderators.Footnote 8
Beyond their empirical and theoretical implications, our results also inform normative discussions of citizen competence and elite influence. It is often claimed that widespread framing effects may be a cause for concern because they suggest that public preferences are arbitrary and can be manipulated (Druckman Reference Druckman2001a; Sniderman and Theriault Reference Sniderman, Theriault, Saris and Sniderman2004). From the perspective of democratic responsiveness, if elites can easily influence the public to support their positions and policies, they have little motivation to respond to citizens’ preferences (Achen and Bartels Reference Achen and Bartels2017). Our results show that these concerns cannot be easily dismissed. Citizens, on average, do respond to elite framing: their opinions and emotions are driven in opposite directions when they are exposed to opposite political frames. However, our results also suggest that normative concerns about citizen competence should not be exaggerated: the capacity of elites to influence citizens by simply altering the way they present issues diminishes considerably when more consequential and realistic political scenarios are examined.
The evidence presented here is limited in at least three ways. First, the studies we meta-analyze are almost exclusively laboratory and survey experiments, which tell us little about framing effects in natural, uncontrolled settings. Secondly, our analyses have been constrained by the unequal distribution of studies: many more studies explore attitudes than emotions or behavior, and there is much more research on emphasis than equivalency framing. The fact that some effects have been estimated with greater precision than others may have influenced our results in unknown ways. Thirdly, there is considerable methodological and conceptual heterogeneity in the framing effects literature. While we have found little evidence of moderating influences, the large proportion of ‘true’ variation between studies was not fully accounted for, suggesting that other study-levels factors might moderate framing effects.
In conclusion, we highlight three directions for future research. First, our literature search reveals that more research is needed on the moderators and mediators of framing effects. We have focused on the type of outcome, competition and framing conceptualization as some of the conditions that might moderate framing effects but note that other moderators, such as source credibility (Druckman Reference Druckman2001b), and mediators, such as belief importance (Lecheler and de Vreese Reference Lecheler and De Vreese2012), were not examined here because we could not locate a large enough number of studies that explored them. We therefore call for both direct replications and theoretical extensions of previously studied moderators and mediators.
Secondly, we believe field experiments that examine framing effects in natural political environments have the potential to complement existing research in important ways. Such designs, which are gaining traction in political science (for example, Gerber et al. Reference Gerber2018), have the potential to overcome the threats to external validity inherent to laboratory and survey experiments.
Thirdly, future research can extend our findings by meta-analyzing the existing evidence on other generic types of framing, such as thematic vs. episodic frames (Iyengar Reference Iyengar1991). An aggregated analysis of additional effects will further our understanding of the influences – and the limits – of framing in the political domain.
Supplementary material
Data replication sets are available at https://doi.org/10.7910/DVN/YJIZBJ and online appendices are available at https://doi.org/10.1017/S0007123420000253.
Acknowledgements
The authors would like to thank Jamie Druckman, Avi Kluger, Jon Krosnick, Tamir Sheafer, Robb Willer, and the editors and anonymous reviewers of the British Journal of Political Science for helpful feedback on this project. We also thank members of the Political Psychology Research Group (PPRG) and the Polarization and Social Change Lab (PASCL) at Stanford University, as well as panel participants at APSA (9/2019), for their valuable feedback.
