Truthiness, Wikiality, and the American Way

How reliable is all this self-promotion? Does Colbert really influence the campaigns of the political candidates that appear on his show? Colbert already has enough evidence for himself—he feels it in his gut. In this sense, the Colbert bump is “truthy.” According to Colbert, “Truthiness is what you want the facts to be, as opposed to what the facts are. What feels like the right answer as opposed to what reality will support” (Colbert Report October 17, 2005). He uses this term to describe the emotion-based arguments politicians like George W. Bush and media personalities like Bill O'Reilly make to support their points of view.

A concept closely related to truthiness is “wikiality,” which is, “A reality where, if enough people agree with a notion, it becomes the truth” (Colbert Report July 31, 2006). In other words, if something is truthy, it is bound to become a part of wikiality. Colbert particularly uses this concept to refer to the constructed state of reality formed by online polls and open source collections of information on the Internet like Wikipedia. To demonstrate his capacity to influence wikiality, he joked on the show that the world population of elephants was no longer in need of protection because it had tripled in the last decade. Within minutes, the Wikipedia entry for elephant had been changed to highlight this “fact.” He later convinced his fans to vote for him 17 million times in an online competition to choose a name for a new bridge in Hungary. Colbert wryly noted on the Report that this was an impressive feat, given that Hungary has less than 10 million people.

In spite of his appeal to made-up concepts and his small audience (Nielsen reports average viewership of about 1.3 million for 2007), Colbert appears to exert disproportionate real world influence. The New York Times recently reported that publishers were targeting the Colbert Report because of its elite demographic. Said one publisher: “You have a very savvy, interested audience who are book buyers, people who do go into bookstores, people who are actually interested in books” (Bosman Reference Bosman2007). Colbert's own book, I Am America (And So Can You!), recently topped the New York Times bestseller list for several weeks.

In spite of the clear left-leaning nature of his parody, Colbert's popularity has not gone unnoticed by the mainstream media and it has attracted attention from both liberals and conservatives. In 2006 Time Magazine named Colbert one of the 100 most influential people in the world. He was also invited that year to speak at the White House Correspondents' Dinner attended by President Bush. He surprised many when he stayed in character to roast the president (“I believe the government that governs best is the government that governs least—by these standards, we have set up a fabulous government in Iraq”) and lambast the press (after bragging that the Colbert Report was a “No Fact Zone,” he turned to Fox News and warned them that the term No Fact Zone was copyrighted). In spite of the mixed reactions it received, his speech was viewed on YouTube 2.7 million times in less than 48 hours after being posted (The Globe and Mail 2007).

Colbert recently capitalized on his influence by entering the political ring with his own campaign. On October 16, 2007, he told Jon Stewart on The Daily Show that he had only “decided to officially consider whether or not I will announce” his candidacy to be president of the United States. Immediately afterward, Colbert told viewers of his own show: “After nearly 15 minutes of soul searching, I have heard the call.” He later registered as a presidential candidate for the Democratic primary in his home state of South Carolina (the fees to enter as a Republican were too expensive) and began to woo Doritos^TM as a corporate sponsor for his campaign on his show. The Rasmussen polling firm quickly conducted a poll that showed Colbert would win 13% of the vote in a three-way contest with Democrat Hillary Clinton and Republican Rudy Giuliani (Rasmussen Reports October 24, 2007). A separate poll also showed he would actually defeat less popular candidates like Republican Ron Paul and Democrat Dennis Kucinich in head-to-head match ups (Rasmussen Reports October 30, 2007).

In spite of this credibility as a candidate, the Democratic Party decided not to accept Colbert's nomination. The head of the Executive Committee of the South Carolina Democrats, Carol Fowler (no relation to the author), called into the show and tried to sooth the would-be-candidate: “They loved the fact that you sucked up to us all day … we loved the Doritos^TM … some people are still drunk.” Stephen Colbert would have to content himself with influencing the campaigns of others.

Truthy Methods and Selection Effects

The efforts by fans to measure Colbert's influence on the 2006 campaign were not—what's the word?—scientific. They were more truthy than truthful. For example, one method involved averaging the percent of the vote earned by every candidate that appeared on the show. Since 89% of these candidates were incumbents, it is hardly surprising that they beat their opponents by 34 percentage points on average. These amateur statistics fall prey to the problem of selection effects—candidates who are willing to appear on the Report may be much more likely than others to be willing to take the risk of humiliation.

The best way to avoid selection effects is to conduct an experiment with a treatment and a control. That way we wouldn't have to worry about the fact that better-performing candidates are the ones who are more likely to appear on the Report in the first place. Thus, to really control for such effects, it would be best to randomly assign which candidates appear on the show. I floated this idea to Stephen, but he isn't returning my calls. Neither is Congress.

So the next best thing is to figure out a way to clone each candidate who appeared on the program, make each clone run for Congress, and see what happens to their campaigns if they don't go on the program. Unfortunately, my human subjects board wasn't too happy with that approach, either.

Instead, what we can do is try to find similar candidates to compare to those who went on the Colbert Report. Each similar candidate should match the party of the Colbert candidate because Republicans may tend to raise more money than Democrats. Likewise, we should match incumbents with incumbents and challengers with challengers since incumbents are usually better at raising money while challengers are in greater need of money. Finally, the most important items to match are the number of donations and the amount of money actually received immediately prior to the Colbert candidate's appearance on the show. This will help to minimize differences in the quality of the candidates, since we will be matching high-earning candidates with other high-earning candidates (and vice versa). It will also control for the typical ups and downs of fundraising demands during the election cycle, since we will be matching the amount of money raised by both the Colbert candidate and the similar candidate over the same period of time.

To figure out whether candidates receive more donations after appearing on the Report, I acquired data from the Federal Election Commission (FEC) on all individual contributions to U.S. House campaigns between January 1, 2005, and October 30, 2007. There were 1,568 candidates who received at least one donation during this period—806 Democrats, 670 Republicans, and 92 from third parties.

For each candidate who appeared on the Colbert Report's segment Better Know a District, I identified a subset of these candidates matching the party and incumbency status for the nearest election (2006 or 2008) occurring after the show.Footnote ³ I then ranked each candidate in this subset by number of donations received in the 30 days before the candidate's appearance on the show. I also ranked candidates by amount of money received in this same period. I then narrowed the subset to those candidates with the smallest sum of the squared difference in both ranks relative to the Colbert candidate. These rules generated 35 unique matches and 12 matches where more than one candidate from the control group might qualify (under these circumstances, one of the qualifying candidates was chosen by lot). Table 1 shows a list of candidates who appeared on the Colbert Report and their matched counterparts. Colbert might call the group on the left the Hall of Heroes and the group on the right the Hall of Cowards (though who knows if they were even invited on the show?).

Table 1

Candidates Appearing on Colbert Report and Their Matched Counterparts

Note:

D = Democrat, R = Republican, I = Incumbent, C = Challenger

Figure 1 shows the average monthly number of donations and money received by candidates who appear on the Colbert Report compared to their matched counterparts on each day starting 60 days before the appearance and ending 60 days after. Each data point indicates activity for the past 30 days, so the left-most point measures donations made between 60 to 90 days before the show, while the right-most point measures donations made between 30 to 60 days after the show. The value for Day 0 should be close to 0 because we have matched each Colbert candidate to a similar candidate that received nearly the same number and amount of donations in the 30 days prior to the show. Results for Democrats and Republicans are shown in their traditional locations on the left and the right, and results for number of donations and amount of money received appear at the top and the bottom, respectively.

Figure 1

Absolute Differences in Number and Dollar Amount of Donations to Candidates Who Appear on The Colbert Report Compared to Matched Candidates Who Do Not Appear on the Program

These graphs show that Democrats who appear on The Colbert Report enjoy a significant increase in the number and total amount of donations they receive in the next 30–40 days, compared to similar candidates who do not appear on the show. Top panels show the difference in total dollar amount of donations for the past 30 days on each date relative to the candidate's appearance, while bottom panels show similar figures for the total number of donations. Left panels show results for Democratic candidates and right panels show results for Republicans. Open circles indicate points where donations during the past 30 days among candidates who appear on The Colbert Report are significantly different than others (based on nonparametric Wilcoxon Signed Rank tests, p < 0.10).

To be sure that the trends we see are not due to chance we need some statistical tests. Throughout the results below I will use results from these tests to indicate the likelihood that the observed difference could be the result of a random aberration, like getting 10 heads in a row in 10 coin tosses. When I write something like p = 0.05, it means there is a 5% chance that there really isn't any difference in the numbers that are being compared. By convention, when p < 0.05 we say the difference is “significant” though the choice of that number and not some other is essentially arbitrary (one might even call it truthy!).

To evaluate absolute differences between Colbert candidates and others I use a Wilcoxon signed rank test. This test is nonparametric, which is a super-cool term that means I don't assume that a histogram of the data produces a nice, “normal” bell shape. In fact, I know the data doesn't look that way—it looks more like a skateboard ramp, starting high near zero and curving down sharply to become flat. For percentage differences, I use a related nonparametric (so cool) test called the Mann Whitney U. I'm sure Stephen will be pleased that there is a “man” in his statistical test (though what kind of a man calls himself Whitney?).

Democrats in Trouble and the Colbert Bump

First, let's start with Democrats and their fundraising activity before the Colbert Report. Notice in Figure 1 that 60 days prior to appearing on the show, Colbert Democrats are doing about as well as others. However, their luck takes a serious turn for the worse in the next 30 days. At their lowest point 28 days before appearing on the show, donations to Colbert Democrats lag those of similar candidates by 7.7 contributions per month. In dollar terms, Colbert Democrats are receiving $8,449 less than the control group, and this result is weakly significant (p = 0.06).

Poor performance prior to the show suggests one of two possibilities. First, the Colbert Report may be targeting Democrats who are in trouble with their campaigns. If this were true, it might be because they are trying to help them, or because their campaign troubles are salient political news that will be of interest to viewers (this might also give Stephen a better chance to “nail” the candidates during the interview). Second, if the Colbert Report is not selecting Democrats in trouble, it might be they are selecting themselves. Democrats who are in trouble may see an appearance on the show as a way to re-energize their campaigns. Alternatively, Democrats who are doing well may avoid the show, preferring not to take the risk of nationwide public humiliation. Whatever the explanation, by the time candidates appear on the show, their fundraising activity recovers to the same level it was at 60 days earlier, suggesting either that their troubles were temporary or they benefited from some pre-show “buzz” in the media about their upcoming appearance on the Report.

Once Democrats actually appear on the show, we see a dramatic rise in their performance. By the thirty-fourth day after their segment airs, Colbert Democrats are receiving significantly more contributions than matched candidates (p < 0.05) and the difference remains significant through day 40. The biggest difference in this period occurs on day 36, when Colbert Democrats receive 9.2 more contributions than the treatment group (p = 0.03). This compares to the average for all candidates at all points in time in the data of 27.6 donations per month, suggesting a bump of about one-third over the normal number of donations received.

We also see a significant rise in receipts. On day 32, Colbert Democrats receive $8,247 more than the control group and the difference remains weakly significant until day 44 (p < 0.10). In fact, on day 34 the difference becomes strongly significant (p = 0.01), and it remains below the 0.05 confidence threshold until day 42. To give a sense of scale, the average monthly amount received by all candidates at all points in time in the data is $21,107, so we are talking about a bump of roughly two-fifths over the normal rate of receipts.

Finally, notice that the bump is temporary. For both donations and money received, the difference disappears and actually turns negative by day 60, returning candidates to their pre-Colbert levels of fundraising. While there is no guarantee that a second appearance on the Colbert Report would be as good as the first, Stephen would probably point out that this means candidates should appear on his show at least once a month (for best performance, wash, rinse, repeat).

Republicans and the Pre-Colbert Bump

Now let's take a look at the Republicans (the right half of Figure 1). It is important to point out that the results for Republicans are quite tentative since only eight have agreed to appear on Better Know a District. Nonetheless, some of the patterns that emerge are significant. Like their Democratic counterparts, Republicans who appear on the Report raise funds at about the same rate 60 days prior to the show as they do when their show airs. However, in stark contrast to the Democrats, the Republicans featured on the show have typically experienced a burst of activity I call the pre-Colbert bump. Exactly 30 days before appearing on the Report, Republican candidates have received a monthly average of 53.6 more donations than their matched counterparts, yielding an extra $63,357 in campaign funds. The difference between Colbert Republicans and the control group reaches its strongest significance 21 days before appearing on the show (p = 0.02 for the amount and p = 0.04 for the number of contributions) and gradually declines until the show airs.

The pre-Colbert bump suggests one of three possibilities. First, the Colbert Report may be targeting Republicans who are doing well. If this were true, it might be because the show is trying to help them by showcasing their surge in popularity, or because their campaign success is due to salient political news that will be of interest to viewers. It is also conceivable that the show targets well-performing Republicans in an effort to humiliate them. Second, if the Colbert Report is not selecting Republicans based on their performance, it might be they are selecting themselves. Republicans who are doing exceptionally well relative to other candidates may be willing to risk humiliation on the show because they already feel safe in their campaigns. Alternatively, Republicans who are doing poorly may avoid the show, thinking that any potential Colbert bump is too small to justify the risk. Third, it may be that the Colbert bump is actually so strong that it can travel backwards in time. Alas, this hypothesis is untestable (although we might ask for help from Tek Jansen, an animated space hero who appears on the Report).

Looking forward in time, however, Figure 1 shows that there is no evidence of a normal Colbert bump for Republicans. In the days after being on the show, the difference in monthly receipts between Colbert Republicans and the matched candidates actually turns negative, dipping to $4,207 by day 28 (though the difference is not significant, p = 0.46). In fact, Colbert Republicans receive a significantly smaller number of monthly contributions on days 17–20 (p = 0.05), with the difference hovering between 10 and 15 through day 30. Although this is not strong evidence of a “Colbert bust” since the sample is so small, it does argue strongly against evidence for a Colbert bump for Republicans who appear on the show. Ron Paul's legions of hopeful supporters are sure to be disappointed.

Relative vs. Absolute Differences in the Colbert Bump

In Figure 2 I look at the data in a different way. Rather than analyzing differences between Colbert candidates and others in absolute terms, it might make more sense to look at them in percentage terms. Suppose a normal Colbert candidate received 22 donations in the last 30 days and the matched candidate received only 20. The Colbert candidate received two more in absolute terms, but 10% more in percentage terms. In comparison, suppose a Colbert candidate in a competitive race received 220 donations versus 200 for the matched candidate. In this case the absolute difference is much higher at 20, but the percentage remains the same at 10%. Thus, using percentages can help us ensure that the differences in Figure 1 are not purely driven by a couple of high activity candidates in competitive races with large absolute differences who bring up the averages for everyone.

Figure 2

Percent Differences in Number and Dollar Amount of Donations to Candidates Who Appear on The Colbert Report Compared to Matched Candidates Who Do Not Appear on the Program

These graphs show that Democrats who appear on The Colbert Report enjoy a significant percentage increase in the number and total amount of donations they receive in the next 30–40 days, compared to similar candidates who do not appear on the show. Top panels show the percent difference in total dollar amount of donations for the past 30 days on each date relative to the candidate's appearance, while bottom panels show similar figures for the total number of donations. Left panels show results for Democratic candidates and right panels show results for Republicans. Open circles indicate points where the ratio of donations to Colbert and non-Colbert candidates during the past 30 days is significantly different from 1 (based on non-parametric Mann Whitney U tests, p < 0.10).

Notice that the results in Figure 2 are nearly identical to those in Figure 1. Once again, Republicans experience a significant pre-Colbert bump but their performance after the show is lackluster, suggesting even the possibility of a Colbert bust. In contrast, the Colbert bump in monthly number of donations to Democrats is significant (p < 0.05) 32–46 days after they appear on the Report. Within this interval, the difference peaks on day 36, when Colbert Democrats receive 37% more donations than other Democratic candidates. The difference in the amount of the donations is also significant (p < 0.05) 31–49 days after they appear on the Report. Within this interval the difference peaks on day 32 when Colbert Democrats receive 44% more money than the matched candidates. These results are close to the one-third and two-fifths increases reported for absolute differences above, suggesting the findings are robust. To mangle an aphorism: no matter which way you slice it, it comes up Colbert …