MEASURING COMPLEXITY

We analyzed the clarity of all articles published in 2012 from what are generally considered the three leading generalist political science journals: American Political Science Review, American Journal of Political Science, and Journal of Politics. In addition, we analyzed all articles published in 2012 from the leading subfield journals in American politics, comparative politics, international relations, and political theory according to the Robust ISI Impact scores reported by Giles and Garrand (Reference Giles and Garand2007). These journals include American Politics Research, Comparative Political Studies, International Organization, and Political Theory. Our sampling procedure yielded a total of 312 articles. ^{Footnote 2}

Clarity was captured with the commonly employed Flesch Reading Ease (FRE) statistic (Flesch Reference Flesch1948). FRE is calculated according to the following formula:

$${\rm{206}}{\rm{.835}} - {\rm{1}}{\rm{.015}}\left( {{{total\,words} \over {total\,sentences}}} \right) - {\rm{84}}{\rm{.6}}\left( {{{total\,syllables} \over {total\,words}}} \right)$$

FRE scores range from 0 to 100. Texts with FRE scores ranging from 0 to 30 are considered very difficult to read, 31 to 50 are difficult, 51 to 60 are fairly difficult, 61 to 70 are standard, 71 to 80 are fairly easy, 81 to 90 are easy, and 91 to 100 are very easy. The FRE statistic is widely reported in applied research across a variety of disciplines (see, e.g., Coleman and Phung Reference Coleman and Phung2010, Lowrey Reference Lowrey2006, and Terris Reference Terris1949). Moreover, policy makers sometimes require certain documents and contracts to achieve minimum FRE scores to ensure public comprehension. For example, Massachusetts requires insurance contracts to achieve a minimum FRE score of 50, and California requires financial institutions to provide notices to consumers with forms that achieve a minimum FRE score of 50 before they disclose nonpublic information. ^{Footnote 3} The FRE statistic also is calculated as a standard feature in popular word-processing programs (e.g., Microsoft Word), making it easy for scholars to check the clarity of their own writing.

To calculate FRE scores for the articles in our sample, we processed text files through an open-source Java application called Flesh. ^{Footnote 4} FRE scores for the articles in our sample ranged from 12 (very difficult) to 65 (standard). The average FRE score was 33 (difficult), with a standard deviation of 7. We also calculated Flesch–Kincaid Grade Level (FKGL) scores, which correspond to the years of education needed by an individual to understand a text. The average FKGL score for articles in our sample was 13 (i.e., accessible to an individual with one year of college), with a standard deviation of 1. These scores are similar to the average readability of academic research in other disciplines (Hartley, Sotto, and Fox Reference Hartley, Sotto and Fox2004; Loveland et al. Reference Loveland, Whatley, Ray and Reidy1973).

Figure 1 places political science research in context by plotting FRE scores for the average political science article along with other documents and well-known texts. ^{Footnote 5} It is not surprising that the average political science article is not as accessible as material in the New York Times or Reader’s Digest, which tend to be pitched to broader lay audiences. The most similar text to the average political science article is the average judicial opinion. Both types of text are pitched toward educated but diverse audiences. It is also worth noting that the average political science article is substantially less complex than text samples taken from the Uniform Commercial Code (i.e., a set of model laws adopted by each state that govern commercial transactions) or the United States Code—both of which have FRE scores lower than 15.

Figure 1.

Examples of Reading Ease

EXPLAINING VARIATION IN COMPLEXITY

Although the average political science article is classified as difficult, there is considerable variation in text complexity across articles. One possibility is that this variation is essentially random, with differences based on idiosyncrasies in writing, skill, and interest in reaching diverse audiences. However, text complexity also may differ systematically based on coauthorship, methodology, and subfield. The dependent variable is the FRE score of an article. The coauthorship variable is an indicator that has a score of 1 for single-authored articles and 0 otherwise. To examine whether different methodological approaches are associated with differences in textual complexity, we included indicator variables for the use of formal theory; quantitative methods, as indicated by the presence of a statistical test (e.g., linear or nonlinear regression models, a t-test for a difference between means, ANOVA and related methods, or a chi square test); and an experimental research design. Thus, the baseline category captured articles that did not use any of these methods. We captured subfields by examining each article in the sample and then coding it as emphasizing American politics, comparative politics, international relations, political methodology, or political theory. ^{Footnote 6} Comparative politics is the excluded baseline in the model presented here because it was the subfield with the highest average complexity score.

Table 1 presents results from an ordinary least squares (OLS) regression model that explains the text clarity of political science articles from the three leading generalist journals and four subfield journals in 2012. Single-authored articles are not more or less complex than co-authored articles on average. Furthermore, the results suggest that, on average, articles using quantitative or experimental methods are not more difficult to read than those that do not use quantitative, experimental, or formal methods. Articles using formal theory are clearer than the baseline, on average, but the effect size of approximately 3.29 points (with a 95% confidence interval of [0.93, 5.66]) is rather modest. Although articles that use quantitative, experimental, or formal methods require a measure of subject-specific knowledge to fully understand them, the associated text is not substantively different, on average, than articles that do not use those methods.

Table 1

Regression Model of Political Science Article Clarity

Notes: Robust standard errors are in parentheses. * p < 0.05 (two-tailed). The baseline category for the methodological approach is an article that does not use quantitative, formal, or experimental methods. The baseline category for the subfield is comparative politics.

As discussed previously, comparative politics is the subfield with the highest average FRE score. The results presented in table 1 suggest that articles in American politics, international relations, and political theory are all less complex than articles in comparative politics. The effect size is 4.85 [3.23, 6.46] for American politics; 4.98 [1.72, 8.24] for international relations; and 5.80 [1.84, 9.75] for political theory. Other than the differences between these subfields and comparative politics, there were no substantive differences in text complexity across subfields.

COMPLEXITY OVER TIME

Next, we examined whether there have been changes in the textual complexity of political science research over time. To do this, we analyzed a sample of articles published by the American Political Science Review in the past 100 years. Specifically, we used the automated text-analysis procedures described previously to compute FRE scores for each article published in 10-year intervals from 1912 to 2012. ^{Footnote 7} The sample included 319 articles.

The average FRE score for this sample of articles was 36, with a standard deviation of 3. The lowest average FRE score (i.e., most complex) for a year was 32 in 1952; the highest average score (i.e., least complex) for a year was 43 in 1912. Figure 2 plots the data points for each yearly average from 1912 to 2012 (at 10-year intervals) along with a fitted line and 95% confidence intervals. Although the data are somewhat noisy, there is a general trend toward more complex writing throughout the sample period. However, the slope of the line is relatively flat, and a decline of 5 points on the FRE measure translates to less than a grade level of change in reading difficulty.

Figure 2.

Articles Published in the American Political Science Review, 1912–2012