Assessing Wikipedia's Accuracy

All of the assessments of Wikipedia cited previously have adopted Nature's approach: The reviewers select a small number of seminal topics within a field and then check the accuracy of every statement in those articles. I call this the “small-n, every-detail” approach. For example, assessments by medical and pharmacological experts have sampled between four (Nicholson Reference Nicholson2006) and 35 (Pender et al. Reference Pender, Lasserre, Kruesi, Del Mar and Anaradha2008) articles. Nature's review was the most ambitious, but even this study sampled only 42 articles (Giles Reference Giles2005). However, the “small-n, every-detail” approach is flawed in both its “small-n” and its “every-detail” aspects. First, when reviewers create their sample, they inevitably choose those articles that deal with the most important issues in their respective fields—but because of their importance, these articles are likely to be the most read, most edited, and therefore most accurate articles in Wikipedia. Second, the “every-detail” approach tends to focus on minor rather than major inaccuracies. Reviewers adopting this approach check every word in their sampled articles for errors, no matter how inconsequential. For example, Nature's review of the Dmitry Mendeleev article identified 19 errors in the entry, but all of them were trivial.Footnote ⁴ Overall, Nature's blind reviewers found a total of 162 errors in the 42 Wikipedia articles they reviewed, but the journal's editors considered only four of these errors “serious.”

I avoid these biases with a “large-n, specific-fact” approach. I identify a specific fact that every article in a category ought to contain and then check every article's accuracy on that fact. For example, every article about a war ought to state the war's beginning and end dates, belligerents, and casualties; every article about an election ought to state the year, office in question, candidates, and vote shares; every article about a politician ought to state his or her offices held and vote shares; and so on. By choosing specific facts in advance and then sampling a large number of articles, I avoid the twin biases of the “small-n, every-detail” approach. My approach is thus best suited to evaluating the accuracy and thoroughness of Wikipedia's objective data (e.g., names, places, biographical information, election results, dates, statistics). I make no effort here to assess whether Wikipedia provides an average voter reading about a candidate on the site with an unbiased review. Rather, my approach allows me to determine whether Wikipedia is a reliable source of objective, verifiable data.

Accuracy of Candidate Biographies

Using this approach, I begin by reviewing Wikipedia's articles about every major-party gubernatorial candidate who ran between 1998 and 2008, examining entries on both prominent candidates (e.g., Arnold Schwarzenegger, Jon Corzine, Eliot Spitzer) and obscure ones (e.g., Jay Lucas, Doug Gross, Mark O'Keefe). One hundred fifty-five gubernatorial elections took place between 1998 and 2008, contested by 246 distinct general election candidates. Of these 246 candidates, 230 (93%) were the subject of a dedicated Wikipedia article. For each of these 230 articles, I checked a specific fact: whether Wikipedia accurately characterized the candidate's previous political experience.Footnote ⁵ I found no errors in these articles at all. Furthermore, this record is even better than it sounds. The experience level of many of the candidates who ran for governor more than once during this period changed between their candidacies. For example, Bobby Jindal ran for Louisiana governor as a newcomer in 2003 and then ran for governor again as a U.S. Representative in 2007. Likewise, Linda Lingle ran for Hawaii governor as Maui's mayor in 1998 and then ran for governor again as chair of Hawaii's Republican Party in 2002. Overall, the sample included 50 occasions in which a previous candidate ran for office again, but with a change in political experience.Footnote ⁶ In every case, Wikipedia provided sufficient detail in its biographies to identify every candidate's political experience at the time of each distinct candidacy.

For the 230 candidates with a dedicated article, then, Wikipedia contained no errors. For the remaining 16 candidates, however, Wikipedia contained no information at all. These 16 candidates were mentioned by name in other relevant articles, particularly those articles on their opponents, but no information other than their names was offered. Wikipedia simply did not contain any information about these candidates. Thus, it was accurate regarding candidates' previous experience, but included no information on 7% of candidates. I will further explore these errors of omission later in the article.

Accuracy of Election Results

I also checked the accuracy of Wikipedia's reported election results. For each election year back to 1976, Wikipedia had an article such as “United States Gubernatorial Elections, 2000”; each of these entries recorded the results of all general elections held that year. For each state, I compared the major-party candidates' vote margins to each state's official results, rounded to one-tenth of a percent. Wikipedia's reported margin was exactly accurate in 61% of the 155 races held between 1998 and 2008. In another 27% of races, Wikipedia was accurate within rounding error.Footnote ⁷ In only four (2.6%) gubernatorial elections was Wikipedia's margin off by more than one percentage point. Wikipedia's most significant error related to the 2002 New Mexico race, accurately reporting that Bill Richardson won 55.5% of the vote but inaccurately claiming that John Sanchez won 34.9% of the vote, instead of his official 39.0%. Thus, Wikipedia overstated the correct margin by +4.1 percentage points. Wikipedia's next largest errors were for the New Hampshire 1998 (−2.2), New York 2006 (+1.2), and New Hampshire 2006 (−1.2) gubernatorial races. On the whole, though, Wikipedia's election results were mostly accurate. There were no errors of omission in these data; Wikipedia had data for every gubernatorial election back to 1998. A statistical analysis based on Wikipedia's reported election results would return essentially the same results as an analysis relying on official data.

Wikipedia's Errors of Omission

It seems, then, that Wikipedia suffers less from inaccuracies than omissions. Scholars in other fields have come to the same conclusion. Pharmacologists report, for example, that “no factual errors were found in Wikipedia, [but] errors of omission were [frequent]” (Clauson et al. Reference Clauson, Polen, Boulos and Dzenowagis2008). Computer scientists have found that an article's length is the best predictor of its overall quality (Blumenstock Reference Blumenstock2008), implying that omissions rather than inaccuracies are Wikipedia's greater flaw. As an online encyclopedia that seeks to contain “the sum of all human knowledge,”Footnote ⁸ those omissions do not result from space constraints; rather, they reflect the limited expertise and interests of contributors. Therefore, we might profitably ask whether any general variables can predict which topics are likely to receive the best coverage. To a large degree, we can probably expect that the same forces that make political topics “salient” to public opinion (cf. Zaller Reference Zaller1992) also make them likely to be addressed in Wikipedia. That is, we should expect Wikipedia to have greater coverage of political topics that are either recent or prominent.

Wikipedia's bias toward recent events is apparent in table 1, which depicts the percentage of midterm-year gubernatorial elections with a dedicated Wikipedia article (such as “California Gubernatorial Election, 2010”). Wikipedia offered a separate article about every gubernatorial election that has been held since the website's rise to prominence in the mid-2000s, as well as articles about many of the gubernatorial elections that occurred during its infancy. However, coverage of elections that occurred prior to its creation in 2001 was extremely poor. Even for elections held as recently as 1998, only 36% had an article. Prior to 1990, coverage dries up completely, with the exception of four states (California, Maine, Minnesota, and South Carolina), for which dedicated volunteers have produced articles going back decades. However, there were virtually no articles about gubernatorial elections in other states prior to the mid-1990s. Consider also Wikipedia's coverage of the U.S. House. All 435 members who served in 2009 had an article, as did all 325 who served in 1887. However, the median article about a member of the 111th Congress was five times longer than the median article about a member of the 50th.

Table 1

Wikipedia's Coverage of Historical Gubernatorial Elections

A better way of describing this “recentness” bias is to say that Wikipedia's volunteer editors are more likely to write about political topics about which they are actively thinking. An event will be more prominent in their minds if it occurred recently than far in the past, of course. More generally, a topic that has received more attention from the public and political observers will be more prominent. For example, the U.S. president receives far more media attention than governors, who receive far more attention than state legislators. Fittingly, then, Wikipedia had an article on every U.S. presidential election since 1788 describing the relevant candidates, issues, and election results. Moving down a notch, the website offered decent coverage of gubernatorial elections of recent years but very little coverage of elections held prior to 1978, as noted previously. Moving down another notch, state legislative elections received almost no coverage at all.

For articles about events (such as an election) or ideas (such as direct democracy), we might expect this sort of “prominence” to be the dominant predictor of Wikipedia's political coverage. For articles about people, however, we might also expect coverage to increase when the subject of the article possesses enough technological savvy to edit his or her own profile. Wikipedia's users are younger and more educated than the general population. In a 2007 study, only 26% of people over the age of 65 claimed to use Wikipedia, but 38% of respondents between the ages of 30 and 49 and 44% of those between the ages of 18 and 29 reported using it. Likewise, only 22% of people with a high school diploma use Wikipedia, but 50% of people with a college degree or higher do so (Rainie and Tancer Reference Rainie and Tancer2007). As such, we might expect coverage to improve for young or highly educated politicians, since these individuals are more likely to contribute to their own Wikipedia profiles.

Thus, we might expect to find two general biases: a bias toward prominent political topics (for all articles) and a bias toward younger, educated politicians (for articles about living people). To quantify the strength of these two biases, I gathered data on 7,361 state legislators serving in late 2009. Of these, 3,512 (48%) were the subject of a Wikipedia article. The median article contained 2,026 characters (1,099 after formatting data and category labels were removed). The shortest article contained a mere 218 characters (77 characters after cleanup).Footnote ⁹ The longest, for New York's Representative Greg Ball, contained 57,978 characters (41,663 after cleanup).

Legislators vary widely in both their prominence and their technological savvy. Within a state, legislators can rise in prominence through leadership experience, tenure, or upper chamber service. Across states, legislators are more prominent in states where average district sizes are larger. New Hampshire's representatives have only about 4,000 constituents each who might write about them on Wikipedia, whereas California's state senators enjoy almost one million constituents each. By a similar logic, legislators may also be more prominent if their chamber is smaller. Alaska's political junkies only need follow 20 senators, but New Hampshire's junkies must track 400 representatives. These various indicators of prominence are outlined along with other variables in table 2.

Table 2

Predicting Wikipedia's Coverage of State Legislators

Note. In Model 1, the dependent variable is a dummy indicating whether each legislator has a dedicated Wikipedia article. In Models 2 and 3, the dependent variable is the logged character count of each article; those without an article are assigned a score of zero. Standard errors in parentheses are cluster-corrected for each of 99 chambers (Nebraska is unicameral).

* p < .05,

** p < .01 (two-tailed).

Using probit analysis, Model 1 predicts whether each legislator will have an article in Wikipedia.Footnote ¹⁰ Legislative leadership has a strong effect on the likelihood of an article existing; a leader's probability of having a dedicated article is 0.20 higher than an otherwise identical non-leader's (see table 3). Logged district population has a similarly strong effect; a one-standard-deviation increaseFootnote ¹¹ results in a 0.23 increase in the predicted probability that a legislator will have an article. As an illustration, Ohio's 33 senators have the fifth-largest districts in the nation and all 33 have separate Wikipedia articles. Ohio's 99 representatives, by contrast, come from average-sized districts, and only 40% have articles. Political experience also has a significant but more modest relationship with Wikipedia coverage. Coverage improves modestly among young, educated legislators, potentially implying that technologically savvy legislators are contributing to their own Wikipedia profiles.

Table 3

Predicted Probabilities of Wikipedia Coverage

Note. Other variables are held at their mean or mode. A 1 SD change in X refers to a shift from 0.5 standard deviations below the mean to 0.5 standard deviations above it.

Table 2 contains two models that estimate each article's length in logged characters, with formatting and category labels removed. Legislators who lack an article are coded as zero. Because this variable is truncated at zero, OLS (Model 2) will tend to underestimate the coefficients. The tobit estimation in Model 3 corrects these biases. Regardless, both Models 2 and 3 tell the same basic story as Model 1: Wikipedia's coverage is heavily skewed toward the most prominent politicians (e.g., leaders, representatives of large districts, longtime politicians), with some additional skew toward technologically savvy politicians. At the margin, for example, OLS predicts that a legislative leader's article will be 120% longer than an otherwise identical non-leader's; in addition, a 1% increase in district population is associated with a 1.355% increase in article length. The tobit estimates in Model 3 are even larger.