INCENTIVES, EFFORT, AND KNOWLEDGE ACCURACY

In this paper, we examine the ability of voters to identify the correct answer to a question where the outcome will not be known until a future, but not very distant, date. The ability to predict the outcome of an event in which initial conditions are knowable but there is some uncertainty about the future is a common, politically relevant task (Lupia and McCubbins Reference Lupia and McCubbins1998). Research demonstrates that incentives for accuracy improve the ability of people to correctly answer political knowledge questions (Feldman, Huddy, and George Reference Feldman, Huddy and Marcus2015; Prior and Lupia Reference Prior and Lupia2008) and financial incentives encourage subjects to update their beliefs about political facts (Hill Reference Hill2017).

We contribute to the literature on incentives and knowledge by studying both certain and uncertain incentives. Research has generally used incentives where the payoffs for correct knowledge occur with certainty; however, Hill (Reference Hill2017) does examine how probabilistic incentives affect learning. In many political contexts, incentives are uncertain and therefore understanding their effects is important to learning about behavior. For example, acquiring accurate knowledge may not change one’s vote with certainty, and even if it does affect a vote, the election outcome and its benefits are uncertain. In either case, the benefit of correct knowledge is uncertain and this uncertainty may affect behavior.Footnote ¹ As elaborated below, we expect both certain and uncertain incentives to improve knowledge accuracy because they both affect the expected benefits of knowledge, which should lead to more correct answers.

Prior research has also focused on situations in which the correct answer to a question can be identified relatively easily online or in a book. However, the questions we use in our experiment (described in the next section) ask respondents to identify the right answer to a question when the outcome will not be known for about 5 weeks. Furthermore, we focus on knowledge in the realm of foreign affairs. Prior research suggests that while it is generally difficult to identify future outcomes in international affairs (Tetlock Reference Tetlock1998, Reference Tetlock1999, Reference Tetlock2006), some people appear able to make accurate predictions (Mellers et al. Reference Mellers, Stone, Atanasov, Rohrbaugh, Metz, Ungar, Bishop, Horowitz, Merkle and Tetlock2015; Tetlock Reference Tetlock1998, Reference Tetlock1992; Tetlock and Gardner Reference Tetlock and Gardner2016). We do not look at individual-level factors that correlate with prediction ability, and instead we focus on whether incentives for accuracy improve people’s ability to make accurate judgments about future outcomes.

In many ways, this is a hard test of responses to financial incentives given that people’s knowledge of and interest in international affairs typically lag behind domestic politics (Converse Reference Converse1964; Delli Carpini and Keeter Reference Delli Carpini and Keeter1996; Holsti Reference Holsti1992; Kinder and David Reference Kinder and Sears1985; Lupia Reference Lupia2015). However, foreign policies can have significant effects on people’s lives and well-being. For example, military conflict, international agreements, international trade policies, immigration policies, membership of international organizations, and economic integration have widespread implications for the economic prosperity and security of domestic populations. The importance of international affairs was highlighted in the 2016 US Presidential election, where immigration and globalization played a significant role in the campaign.

Our basic model of how incentives affect knowledge accuracy is displayed in Figure 1. In the rest of this section, we elaborate on both the direct path by which incentives improve accuracy and the indirect path in which increased effort leads to an increase in knowledge accuracy.

Figure 1

A Theory of Incentives, Effort, and Knowledge Accuracy.

EXPERIMENTAL DESIGN

We recruited 1,016 subjects using Amazon MTurk for the experiment, which we designed to isolate the effects of incentives on effort and accuracy of answers to political knowledge questions. Previous research suggests that the MTurk platform produces acceptable samples for social science research (Berinsky, Huber, and Lenz Reference Berinsky, Huber and Lenz2012; Casler, Bickel, and Hackett Reference Casler, Bickel and Hackett2013; Hauser and Schwarz Reference Hauser and Schwarz2016; Huff and Tingley Reference Huff and Tingley2015; Levay, Freese, and Druckman Reference Levay, Freese and Druckman2016; Mullinix et al. Reference Mullinix, Leeper, Druckman and Freese2015).Footnote ² We expect our theory about incentives, effort, and knowledge accuracy to apply to all people, and therefore there is no reason to expect MTurk respondents to be theoretically inappropriate for our purposes.

The additional advantage of MTurk is that we know subjects are at an internet-connected device and can search for information, and the platform also provides a way to track how long respondents take to complete the task, which we can use as a proxy measure for effort. Appendix C in the Supplemental Material presents descriptive statistics of the sample.

After being recruited via MTurk, but prior to treatment assignment, participants received background information about the experiment and they answered a series of political information questions. We used 10 political information questions that covered a mix of US domestic issues and foreign affairs; the questions are displayed in Section 1.2.2 in Appendix B in the Supplemental Material. After answering these questions, respondents were randomly assigned to the control group or one of the three treatment groups.

In Table 1, we report the average number of correct answers to the pre-treatment political information questions for respondents assigned to each treatment condition. As expected, the averages do not vary across treatment conditions. We measured pre-existing political information so that we can use it to contextualize the magnitude of estimated treatment effects by comparing them to the relationship between prior knowledge and the questions we use as our dependent variable.

Table 1

Descriptive Statistics of Pre-treatment Political Information, by Treatment

The control group received no incentives for accurate answers. We used three different incentive schemes in the experiment and held constant the expected value of a right answer to an outcome question at $0.50. We varied the benefit of a correct answer and the uncertainty associated with receiving the benefit if a question was answered correctly.

In the bonus treatment, participants received a $0.50 guaranteed payment for every correct answer. In the random bonus treatment, we randomly selected one of the five questions and paid respondents $2.50 if they got that particular question correct. In the lottery treatment, each participant earned a ticket for a $50 lottery for each correct answerand one $50 prize was awarded for every 100 correct answers.

We asked comprehension questions about the incentives immediately after exposure to the treatment but prior to our outcome questions. This helped ensure that participants understood the bonuses and their likelihood of receiving a bonus given a certain score (Kane and Barabas Reference Kane and Barabas2019).

To identify the effect of incentives on effort and knowledge accuracy, we chose to examine whether respondents got the right answer to five different questions that asked about an outcome that would not be officially known for about 5 weeks after the respondents were asked to predict the answer to the question.Footnote ³ The values for all five questions changed over the 5 weeks, but because of lumpiness in the response categories, the correct answer changed for only three questions over the time period of the study.

All questions were multiple choice, and the survey required participants to provide a response for each question to continue the survey. The number of possible responses ranged from 6 to 11 responses, which varied according to plausible answers to each question. Table 2 presents the full text of each question, including the number of possible responses. The entire question wording and possible responses are available to view in Appendix B in the Supplemental Material.

Table 2

Knowledge Accuracy Questions and Correct Answers

In all of the conditions, participants were allowed and encouraged to search for information to answer the questions. Making this consistent across conditions allows us to minimize the possibility that subjects in the treatment conditions inferred that we wanted them to search for information, whereas those in the control condition might not make that inference, and therefore our treatment effects would be confounded by subjects’ perceptions of what is expected of them.

These outcome questions represent a combination of correct information and the ability to use it to answer a question about the near-term future. We refer to the ability to answer such question as knowledge following the distinction between information as data and knowledge as the ability to make accurate predictions (Lupia and McCubbins Reference Lupia and McCubbins1998).

Because the correct answer would not be known for 5 weeks, respondents cannot simply look up the answers, but the outcomes also do not occur as far into the future as many of the predictions used in the Good Judgment Project or other forecasting examples.

For Question 1, a participant could search for information about the number of Jihadist terrorist attacks in the USA committed by people who were US citizens or permanent residents at the time of the experiment and this would help them answer the question correctly. Some of the possible answers were either impossible or highly unlikely given the prior number of terrorist attacks, but a correct answer still requires making a judgment about the number of events in the 5 weeks before correct answers were determined. An accurate response may reflect a combination of information search and the ability to make an inference about how the values will change over the course of 5 weeks.Footnote ⁴

The short time between asking the outcome questions and paying subjects means that the correct answers to our questions will necessarily be relatively close to the state of the world when taking the survey. However, the use of these kinds of questions is not wholly different than some used in other studies of predictions. For instance, Tetlock and Gardner (Reference Tetlock and Gardner2016, pp. 125–126) discuss the following example from their work:

Like our questions, this one requires some information about current conditions and the ability to predict into the short-term future, which requires consideration of the trends underlying the current state of the world and projecting them forward. The questions we use are neither as easy as straightforward information questions nor as difficult to answer correctly as predictions with an 18-month time frame.

After the knowledge questions, participants were asked a series of questions relating to effort and information search.Footnote ⁵ The full design, including the treatments, questions, and coding of variables, is presented in Appendix B in the Supplemental Material.

We used two attention checks during the experiment to ensure that our subjects were attentive. Almost 90% of the sample answered both questions correctly.Footnote ⁶

The experiment was launched on October 26, 2018, and participants were paid bonuses on December 1, 2018.