1. Introduction
Elections lie at the heart of contemporary theories of democracy. For the polity, elections are the primary mechanism of democratic control, both ex ante and ex post. In prospect, they enable the people to confer upon officials the mandate to act in their stead. In retrospect, they enable the people to judge the use officials have made of their mandates.
Democratic government depends on popular choice not only in form, however, but also in substance. Elections are not just for show. Ever since the spread of democratic norms in the nineteenth century, autocratic regimes also have resorted to balloting to legitimize their rule, but only apologists endorse the results as “democratic.” In contemporary theories of democracy, that is, democracy requires more than the mechanics of elections. Democracy requires choice, and choice, in turn, requires competition. In the well-known construction of Joseph Schumpeter, democracy is “that institutional arrangement for arriving at political decisions in which individuals acquire the power to decide by means of a competitive struggle for the people's vote.”Footnote 1 Electoral competition is the sine qua non of democratic government.
Even in regimes deemed “democratic,” however, competition has had its abeyances. Single parties have monopolized office holding even while facing electoral challenge, as Japan's Liberal Democrats and Sweden's Social Democrats did for a very long time. Candidates for office—usually lower offices—have frequently won uncontested bids for reelection, as twenty-four Democratic incumbents and six Republican incumbents in U.S. House seats did in 2008.
In the United States, the most extended period of single-party dominance in national and state elections began shortly after the Civil War. The “Solid South” states of the Confederacy voted for the Democratic Party's nominees for president in every presidential contest from 1880 through 1916. Of the 142 gubernatorial elections in the South between 1877 and 1911, Democrats won 138. In the Deep South, from South Carolina in the east to Louisiana in the west, Democrats won at least 80 percent of the vote in a majority of each state's U.S. House seats in more than half of the biennial elections between 1878 and 1910.
Single-party monopolies on political power were not unique to the South, however. The northern tier states in New England, the Midwest, and the Pacific Northwest were no less the property of the Republican Party. Vermont elected a Republican governor in every election between 1855 and 1910.Footnote 2 Between 1862 and 1910, Republicans won twenty-three out of twenty-five gubernatorial elections in Maine, Michigan, Iowa, and Minnesota and twenty-two out of twenty-five in Wisconsin and Kansas. At least half of the time between 1862 (or statehood) and 1910, Republicans controlled every seat in the U.S. House delegations from Maine, New Hampshire, Vermont, Rhode Island, Minnesota, Iowa, Kansas, South Dakota, North Dakota, Idaho, Oregon, and Washington—and spells of GOP monopoly were interrupted as frequently by Greenbacks and Populists as by Democrats. The Republicans did not dominate the vote in the northern tier the way the Democrats did in the South, but they dominated the outcomes of elections hardly less.Footnote 3
In the view of Progressive Era critics, single-party franchises on voter loyalties fostered “boss rule” and prioritized the demands of the wealthy trusts over the needs of the people. Reformers promoted the secret ballot, the civil service, voter registration, nonpartisan elections, and professional municipal management to weaken the grip of the political parties on the choices of voters in elections.
Simultaneously, and more directly, the Progressives sought to introduce democracy into the political parties themselves. The failures of third parties as vehicles for political reform merely proved that two-party competition was not an option for reform in many parts of the country, particularly in the Democratic South and in the Republican northern tier, where party identification was so strongly shaped by the Civil War. The voters' partisan loyalties, reformers learned, were not easily deflected. To break the monopoly of the party bosses, the Progressives concluded, they would have to inject competition and popular choice into the selection of party nominees for federal and state offices through the direct primary.Footnote 4 They concluded, according to the well-known slogan, that “the cure for the evils of democracy is more democracy.”Footnote 5
State governments first stepped in to regulate the party nomination process in California and New York in 1866, criminalizing fraud in elections to nominations in caucuses and conventions. In the last half of the nineteenth century, states gradually extended regulation of nominations by stipulating the use of the secret ballot, assuming control over the printing of ballots, and requiring the use of direct primaries in certain jurisdictions. By the 1890s, many of the southern states gave parties the option of a direct primary, and the Democrats conducted primaries in several states in the twentieth century's first decade. On the initiative of its progressive Republican governor, Robert M. La Follette, Wisconsin in 1903 passed the first law that mandated direct primaries statewide and provided for thorough legal supervision, followed closely by Oregon in 1904. By the end of the Progressive Era, at the entry of the United States into the First World War in 1917, thirty-two of the forty-eight states had implemented mandatory direct primaries for nominations to all state offices and some local offices. Twelve more states had direct primary laws with more limited coverage.Footnote 6
This article offers the first-ever empirical assessment of the extent to which the Progressive hopes of “more democracy” were met in primary elections in the United States. We begin, in Section 2, with a review of the maladies the Progressives diagnosed in the American electoral system and the reasons they might expect the direct primary to be a pro-competitive, democratizing therapy. In Section 3, we examine prior research into the direct primary and electoral contestation, and we describe the never-before-assembled database of primary and general election outcomes that we use to trace competition in primaries for federal and statewide offices. In Section 4, we examine the historical trajectory of competition in primary elections, starting with the first decades after the introduction of the reform and then the succeeding decades.
Consistent with the hopes of reformers, we find primary elections indeed provided a forum for contestation for federal and statewide elections. Although primaries were never broadly competitive, even at the outset, they accounted for about a third of the serious electoral tests faced by statewide office-holders and about a fifth faced by U.S. representatives. The role of primaries as a venue for robust contestation, however, was short-lived, as the competitiveness of federal and statewide primaries decreased sharply starting in the 1940s. The last section of the article explores whether two recent developments in American elections—the extension of two-party competition and the rise in the value of incumbency—conspired to temper the contribution of direct primaries to electoral competition.
2. The Direct Primary and Electoral Competition
The Progressive reformers were political activists, not social scientists, and they did not attempt a careful analytical brief for the direct primary. To them, the nominations reform addressed a political problem, and that problem, in short, was “boss rule.” The “modern political machine,” La Follette declared in an early speech on the subject in 1897 at the University of Chicago:
has come to be enthroned in American politics. It rules caucuses, names delegates, appoints committees, dominates the councils of the party, dictates nominations, makes platforms, dispenses patronage, directs state administrations, controls legislatures, stifles opposition, punishes independence, and elects United States senators. In the states where it is supreme, the edict of the machine is the only sound heard, and outside is easily mistaken for the voice of the people.Footnote 7
The party bosses arrogated decisions to themselves that were properly the people's to make. They presented candidates for office, but they gave the people no choice. The direct primary, the reformers contended, would restore the people's choice. “Under the primary system,” declared the progressive journalist William Allen White, “any clean, quick-witted man in these states can defeat the corporation senatorial candidate at the primary if the people desire to defeat him.”Footnote 8
But we might ask, in turn, what it was that made “machine” control over nominations a problem for democracy, and how it was that the direct primary struck at its roots. By itself, after all, control over nominations ought not to have restricted voters' choices, but even to have enhanced them. Party leaders gain access to resources through their control of government offices, and the fierce logic of competition for office, as described by Anthony Downs, normally disciplines parties to offer centrist candidates attractive to majorities of voters.Footnote 9
Spirited competition for the people's mandate, however, was not what the Progressive Era reformers observed. One reason for its absence, they believed, was the nature of the political parties. Even in the states where the Democrats and Republicans competed on even terms, the struggle between the two parties was a sham competition, the feints of two contenders under the same management. The bosses in both parties chose nominees to satisfy nefarious economic elites of similar type, not the electorate.Footnote 10 Whether fact or fable, the contention was effective political rhetoric. The direct primary addressed the corruption of the party bosses by cutting them out of the nomination process, handing to the Democratic and Republican rank and file the power to designate the candidates they wanted to see in November. It made the party democratic by forcing the party to observe democratic forms.
The second barrier to popular control, in the view of the reformers, was the absence of two-party competition in many parts of the country. “Of what value is a Republican nomination for a state office in . . . Alabama, Arkansas, Florida, Georgia, Louisiana, Mississippi, Oklahoma, North Carolina, South Carolina, Texas, and Virginia?” asked Charles Merriam, rhetorically. “Of what value is a Democratic nomination for a state office in Illinois, Iowa, Maine, Michigan, Minnesota, Oregon, Pennsylvania, Rhode Island, Vermont, Wisconsin? And in many other cases the chances, while a little better, are not worth much as political assets.”Footnote 11 In these states, and in many more locales, the leading political party did not compete for the voters' favor because it did not have to. It was a monopolist, and it acted like one.
At the center of the monopoly was the voters' unwillingness to sample the wares of a competing party. As recent research has shown, political partisanship is a core identity that exercises a very strong influence on the choices voters make in partisan elections. It is durable through the life cycle and even, through socialization, across generations.Footnote 12
By the Progressive Era, it was clear to all that the Civil War and its aftermath had put an indelible stamp on partisan identities, overlaying a regional cleavage onto the earlier and continuing class divisions between the Democratic Party and the Republican and Whig parties. In the North, the national party cleavages disadvantaged local minorities but still gave a foothold to the minority party, usually the Democrats. Substantial Catholic populations, for example, enabled Democrats to win gubernatorial, senatorial, and U.S. House elections in states like Massachusetts, Ohio, Illinois, and Nebraska, which were dependably Republican in presidential elections.Footnote 13 In the South, however, the Republican Party's role in the Civil War and Reconstruction stigmatized it to such a degree that its last prospects for success disappeared with the disenfranchisement of black freedmen by 1900.Footnote 14
The national partisan identities, however, dominated but did not efface other allegiances that divided voters at the local level. In the agrarian South, these were chiefly class differences, often divisions between populist smallholders and plantation and commercial elites, all of them Democrats. In the North, they were sometimes economic divisions, like the differences between Republicans with industrial and agricultural interests, and sometimes ethnocultural divisions, like the differences among Republicans whose lineages touched the British Isles, Germany, and Scandinavia.Footnote 15
The direct primary addressed the problem of voters' partisan loyalties by taking voters as they were. Rather than oppose partisan identification, the direct primary removed it as a consideration, allowing all of the other issues that divided local electorates to come forward as bases for contestation.Footnote 16 It was a pragmatic accommodation to the facts of single-party dominance, “democratic” on the argument that competition for the votes of just the Democrats or just the Republicans is better than no competition at all.Footnote 17
For these reasons, then, Progressives saw the direct primary as a promising vehicle for competition, contestation, and popular choice. The reform addressed the two principal factors impeding electoral contestation: the corruption of the political parties and the deep partisan attachments of the American voters. With the rapid adoption of the direct primary, the reformers' expectations were soon put to the test.
3. Researching Competition in Direct Primaries: Data and Measures
The first studies of competition in direct primaries suggested that the Progressives' hopes for the reform were well founded. Against a backdrop of general elections whose outcomes were never in doubt, V. O. Key Jr. found vigorous—if chaotic—competition for nominations in Democratic primaries in most of the states in the South.Footnote 18 Key suggested that the direct primary fostered competition, the more so when states lacked two-party competition in general elections.Footnote 19 Other studies reached essentially the same conclusion.Footnote 20
Other researchers found cause for skepticism. They discovered that primary elections, like general elections, were less competitive when incumbents ran and more competitive when they did not, suggesting that the pro-competitive effect of primaries was limited.Footnote 21 “The comparative usefulness of the primary as a method for selecting successors for retiring incumbents,” Julius Turner wrote, “does not offset the fact that the primary is not a successful alternative to two-party competition in most parts of the United States.”Footnote 22
How competitive were direct primaries? Until this study, no one has really known even the basic facts. The difficulty in reaching a clear assessment of the contribution of the direct primary to competition in U.S. elections derives from the limited availability of data and the consequent focus of studies on particular places—often the South; on particular periods—frequently the years covered by America Votes; and on particular offices—usually U.S. House seats, U.S. Senate seats, governorships, or state legislative seats.Footnote 23
A thorough assessment of the contribution of the direct primary to competition in U.S. elections requires more comprehensive data. To that end, we have compiled the primary election returns for all statewide and federal offices from 1900 to 2004 using official state election reports, state manuals, newspapers, and almanacs.Footnote 24 We have married the primary election data to general election returns in statewide and federal elections collected in the Harvard State Elections Database.Footnote 25
For general elections, we present three measures of electoral competition: the fraction of contested races, the frequency with which incumbents fall to defeat, and the proportion of the two-party vote received by the winner.Footnote 26
In primary elections, the measurement of competition is less obvious. We calculate the defeat rate for incumbents in primaries straightforwardly, in the usual manner. The measurement differences begin with the assessment of contestation. Primaries are uncontested if just one candidate appears on the ballot, but primaries are also uncontested if no candidate bids for the nomination. We therefore count as uncontested any primary election in which the nomination goes to the only candidate or in which the nomination goes begging. The modal primary is uncontested.Footnote 27
Unlike general elections, in which ballot access laws frequently limit the candidate pool to the two major-party nominees, primary elections allow entry by multiple candidates. In Oklahoma, for example, the average number of candidates for the Democratic nomination for governor over the past one hundred years is just over seven. Variation in the number of candidates in primaries—and the lack of a natural way, like party affiliation, to discriminate among them—means that a two-candidate vote share is not such a obvious metric for the closeness of the election. Even if all primary races involved just two candidates, orienting the data to measure a “normal vote” proves particularly difficult in primaries. Researchers cannot determine the ideological positioning or factional affiliation of candidates from the aggregate election data alone.
We have experimented with different measures of competition in primaries, including the winner's share of all votes, the winner's share of the votes of the top two candidates, the percentage difference between the top two candidates' vote shares, and the incumbent's share of the primary vote. They produce similar results. Accordingly, in our analysis we count as competitive those races in which the winner received less than 60 percent of the total votes cast. Although lower cut points might be chosen, our use of the 60 percent threshold does not have a substantive impact on our conclusions.
4. The Record of Competition in Party Primaries
4.1 Primaries and Competition Soon after Reform: 1910s to 1930s
At the outset, the direct primary appreciably extended the electoral gauntlet for public officials. Figures 1 and 2 present four measures of competitiveness for all statewide and federal offices, aggregated by decade, all coded so that higher values correspond to greater competition. Because of the substantial difference in two-party competition in general elections in the North and South and because of the important differences in the historical conditions of electoral reform, we show the data separately for the northeastern, midwestern, and western states (designated “North” in Figure 1 and hereafter) and the southern states (defined as the eleven secessionist states and designated “South” in Figure 2 and hereafter). Aggregating all of the federal and statewide offices, the four panels present the proportion of all primaries that were contested, the winner's share of the vote subtracted from 1, the fraction of primaries in which the winner received less than 60 percent of the total vote, and the proportion of incumbents defeated in primaries.
Competition in Primaries in the North, 1910s to 2000s.
Competition in Primaries in the South, 1910s to 2000s.
In the North, the contribution of the direct primary to competition for office was important, albeit modest. From 1910 to 1930, just over half of all primaries for nominations to statewide and federal offices were contested, and in over a third of the contested primaries, winners received less than 60 percent of the vote. Incumbents failed to secure renomination in about 3 percent of the primaries they entered. Contestation increased and the fraction of primaries won by less than 60 percent rose in the 1930s.
The direct primary introduced appreciably more electoral contestation into the South. Between 1910 and 1930, nearly two-thirds of all nominating primaries matched at least two candidates, and winners took less than 60 percent of the vote in contested primaries nearly half the time. Roughly 6 percent of the primaries in which incumbents ran were won by their challengers. As in the North, competition for nominations in South increased notably during the 1930s. By any measure, though, the primaries in the South were more competitive than the primaries in the North. As V. O. Key Jr. noted, writing of the same period, the direct primary provided an outlet for competition in polities that lacked the basis for two-party contestation.Footnote 28
The new institution affected the two major parties just about equally. Figure 3 charts the percentage of primaries that were competitive—defined here as the winner having received less than 60 percent of the vote—separately for Democrats and Republicans. It further distinguishes between primaries in states dominated by the Democratic Party or by the Republican Party and in states with two-party competition, designated according to whether the “normal vote”—the underlying partisan division within the state, estimated by averaging each party's vote across all statewide offices, controlling for incumbency and election year—for either the Democrats or the Republicans exceeded 60 percent or fell within the “competitive” range of 40 to 60 percent.Footnote 29
Competition in Primaries, by Party.
For the first three decades of the direct primary, Figure 3 shows, just under half of both Democratic and Republican Party primaries were competitive in states in which the party was either dominant or viable in general elections. Republican primaries in Republican states were particularly spirited in the 1910s, reflecting the GOP's split between the Theodore Roosevelt progressives and the William Howard Taft conservatives in those years. Similarly, the surge in primary competition in the 1930s, so notable in Figures 1 and 2, was principally a phenomenon within the Democratic Party. The party's rising fortunes emboldened Democrats to put themselves forward, and the popularity of Franklin Delano Roosevelt and his New Deal inspired his followers to challenge Democratic leaders of the earlier day, even (or especially) in Republican states. In general, however, the low value of the subordinate party's nominations discouraged minority-party primary competition in states loyal to the Democrats or the Republicans.
In these first three decades of Progressive election reform, then, the direct primary contributed significantly to the total stock of competition in U.S. elections, in the North and in the South, in the Democratic Party and in the Republican Party. The contribution of the direct primary in electoral competition differed, however, depending upon the office.
Figure 4 shows the percentage of races for election in which the winners faced a competitive challenge—again defined as the winner having received less than 60 percent of the vote—in one or both of the two stages of the electoral process, the primary election and the general election.Footnote 30
Competition in Elections for Statewide Offices and the U.S. House.
From the 1910s through the 1930s, it shows, nearly three quarters of the winning candidates survived a significant trial in either the primary or the general election or in both elections for offices elected statewide, that is, in races for the U.S. Senate, for governor, and for a host of other state executive offices. About a quarter of the time, the only competitive contest was the race for nomination in the primaries.Footnote 31
Even at the start, however, the direct primary contributed far less to competition in elections to the U.S. House of Representatives. Between 1910 and 1938, about 60 percent of House members faced a competitive challenge in at least one stage of the election process. Just one in eight, though, encountered a challenge only in the primary, and only one in six faced the double jeopardy of a competitive race in both the primary and the general election. Even in the first thirty years of the Progressive electoral system, House members most often ran the gauntlet only in the general election—if they ran it at all.
Overall, though, the effect of the direct primary in the first three decades of its implementation was to add to the electoral challenge faced by candidates for statewide and federal offices. The direct primary introduced more contestation into systems that also had vigorous two-party competition. And although competition in primary elections was nowhere a substitute for two-party competition in general elections, the direct primary injected contestation into systems that lacked effective competition in general elections. In races for statewide office, the primary was the only competitive stage about a quarter of the time, and in races for the U.S. House the primary was the only competitive stage about an eighth of the time.
In fact, the direct primary redressed all or much of the decline in two-party competition that motivated the reform in the first place. Figure 5 shows the proportion of general election contests in which the winner took less than 60 percent of the vote, beginning in the 1870s, and above it, the proportion of elections in which the victor earned less than 60 percent of the vote in either the primary or the general election, starting in the 1900s, separately for statewide offices and the U.S. House. Both panels document the large and permanent decline in interparty electoral contestation at the turn of the nineteenth into the twentieth century, which Schattschneider called the “System of 1896.”Footnote 32 Remarkably, the Progressive reform made up the whole difference in the contests for the statewide offices. In its first three decades, the direct primary boosted the proportion of competitive election sequences to equal the high levels of the late nineteenth century. It had a less substantial impact in the races for seats in the U.S. House, but even so, the direct primary compensated for about half of the difference. At least at first, “more democracy” made up for “less.”
Competition in Elections for Statewide Offices and the U.S. House, Before and After the Direct Primary.
This is not to say that party voters acting in direct primaries disciplined officials with any greater frequency than party elites acting in caucuses and conventions. We (and others) have little information about the extent of contestation and competition in the caucuses and conventions that selected nominees before the adoption of the direct primary, although a study of turnover in the U.S. House finds no obvious difference in the likelihood that incumbents were denied renomination before versus after the advent of the primary nomination system.Footnote 33 The difference, rather, was in who had the ability to exercise discipline. Direct primaries forced elected officials into an additional competition for the favor of the electorate—frequently, where nomination was tantamount to election, for the blessing of the overwhelming majority of the electorate—and as we have shown, the contest was rigorous a significant fraction of the time. Direct primaries gave at least some part of the electorate the opportunity to influence elected officials, an opportunity previously denied. In the first decades after adoption, the direct primary vindicated the claims of its advocates for its power to promote competition and choice in U.S. elections.
4.2 Primaries and Competition in a Mature System: 1940s to 2000s
The power of the direct primary as an instrument for contestation declined rapidly after 1940, in the North and the South, in the Democratic Party and the Republican Party, in statewide elections and U.S. House elections.
In the North, the percentage of contested primary elections dropped from about 50 percent in the 1940s to about 40 percent in the 1960s, and the percentage of primaries in which the winner achieved less than 60 percent of the vote fell from about a third to less than a quarter (see Figure 1). By the 1960s, only about one in one hundred incumbents who sought reelection were denied renomination by the party in the electorate. Competition rose slightly during the 1970s and again during the 1990s, falling back in each succeeding decade. Even in the 1990s, however, only one in five nominees for statewide and federal offices in the North won primaries with less than 60 percent of the vote.
In the South, where competition in primaries was initially more vigorous, the decline of primary competition was even more dramatic. In the 1940s, almost 60 percent of party primaries were contested; in the 1960s, around 40 percent (see Figure 2). In the 1940s, about 30 percent of nominees won primaries with less than 60 percent of the vote; in the 1960s, about 20 percent. In the 1940s, more than 3 percent of incumbents failed to secure renomination; in the 1960s, less than 3 percent; and in the 1990s, less than 1 percent. By century's end, on every measure, primary election competition in the South came closely to resemble primary election competition in the North, which meant there was not much of it.
Of the two parties, the Republicans, distinctly in the minority following the New Deal, suffered the more rapid decline in competition in their primaries. In the states where the Republicans had hopes of victory, the proportion of winners of GOP primaries for statewide and federal offices who gained nomination with less than 60 percent of the vote fell from around half before World War II to only a quarter by the 1950s. (See Figure 3.) Competition held up better in the Democratic Party primaries, though only slightly: in the states where the Democrats could compete in November, the proportion of competitive Democratic primaries declined from around half before the war to around a third in the 1950s.
Although competition in Democratic and Republican primaries never returned to the levels of the first decades of the direct primary, both parties experienced a resurgence in contestation toward the end of the twentieth century. For the Democrats, the percentage of nominees who won with less than 60 percent in states where the party was competitive rose by nearly 10 points in the 1970s and 1980s, a time of readjustment following the empowerment of some Democratic constituencies and the disillusionment of others by the Second Reconstruction in civil rights. For the Republicans, the revival of competition came later, and concentrated in states the GOP could not expect to win. Between the 1970s and the 2000s, the proportion of competitive Republican primaries in Democratic states tripled. (The proportion of competitive Democratic primaries in Republican states, meanwhile, remained at a dismal 15 percent.) The competitiveness of GOP primaries in the states more favorable to the Republicans increased as well, by around 10 percentage points. The party's improved prospects encouraged Republicans to enter the ring, and the appeal of Ronald Reagan and his program inspired “movement conservatives” to mount challenges for party leadership. For a second time, a realignment within the political parties worked itself out through the primary election process.
Still, with the long-term decline in primary election competitiveness came a long-term decline in the contribution of the direct primary to the competitiveness of the entire electoral process. In statewide elections during and after World War II, overall electoral competitiveness dropped only slightly, from around 75 percent in the 1910s to 1930s to around 67 percent in the 1940s to 1950s to around 63 percent in the 1960s to the 1980s, rebounding to around 70 percent in the 1990s and 2000s. (See Figure 4.) The contribution of primary elections to the trials faced by officeholders, however, withered nearly away. The percentage of statewide officeholders who encountered competition only in the primary declined from about 25 percent to about 17 percent to about 9 percent. Meanwhile, the percentage of statewide officeholders who experienced competition only in the general election increased from roughly 23 to 32 to 40 percent. The rise in interparty competition in statewide races nearly exactly compensated for the decline in intraparty competition.
In races for seats in the U.S. House, primary elections already played a smaller role in testing prospective officeholders, but their role diminished still more after 1940. Between the 1910s to 1930s and the 1940s to 1950s, the percentage of House elections that were competitive in at least one of the two stages declined by 10 percentage points, from roughly 62 percent to 51 percent, and the retreat from competitiveness in the primaries is implicated in the entire decline. (See Figure 4.) After 1960, the competitiveness of the House election process declined even more, in primary elections by a little and in general elections by a lot.
This is not to say that primary elections no longer make any contribution to the competitiveness of the electoral process in the United States. Historically, as Figure 5 shows, throughout the direct primary's career, nominating contests propped up the competitive level of elections for statewide offices and slowed the decline of competition in elections to the U.S. House. Even now, primaries by themselves account for about one-eighth of the experience statewide and House officeholders have with electoral competition. That share has declined rather markedly over one hundred years, but still, if primaries did not exist, politicians—and voters—would feel the lack.
5. The Limiting Conditions of Primary Competition
All the same, the postwar experience with the direct primary is a disappointment and a riddle, relative to the hopes its proponents invested in it. In its earliest years, the reform worked—for the most part—as anticipated. Primaries tested public officials, putting them before the judgment of the electorate, even in areas of single-party monopoly. Within a few decades, though, the effectiveness of the primary as a competitive venue dissipated. What happened? What has changed since the implementation of the direct primary to temper so dramatically its contribution to electoral competition in the United States?
There are many possibilities, of course, and with these data, we can evaluate two of the most important: first, the increase in the number of states with effective interparty competition and, second, the growth in the power of incumbency.
Among the most significant developments of the last fifty years in the politics of the United States is the extension of two-party competition. In the first four decades of the twentieth century, only 49 percent of all statewide officers served in states in which the division of the state “normal vote” was closer than 60 to 40. In the last four decades of the twentieth century, 64 percent of all statewide elections occurred in states that were two-party competitive. Single-party franchises on political power are now a thing of the past in most parts of the country. Reversing Key, perhaps interparty competition substituted for intraparty competition: as general elections offered an outlet for contestation, primary competition fell as a result.
As plausible and as attractive as this possibility is, however, it does not find support in our evidence. Above, in Figure 3, we graphed the incidence of primary competition for Democrats and Republicans in states each party dominated and in states they contested. If rising competition between parties sapped competition within parties, competition should decline in primaries in two-party competitive states but hold steady in dominant-party primaries in single-party states.
Figure 3 shows something different: the percentage of winners with less than 60 percent of the primary vote declined in both the two-party competitive states and the single-party dominant states. More importantly, the erosion in primary competition in two-party and one-party states was virtually the same magnitude—15 to 20 percentage points—and virtually in lockstep. According to this evidence, competition in general elections did not take away from competition in primary elections.
But perhaps the aggregation by party masks regional variation in the relationship between competition in primaries and general elections. After all, the most breathtaking increase in two-party competition, the phenomenon that has remade American electoral politics in the last half century, stemmed from the rise of the Republican Party in the South. In presidential elections, the Solid South made its last appearance in the Democratic column in 1944 and its first appearance in the Republican column (of many) in 1972. In congressional elections, about three-fifths of the House candidates from the South ran unopposed in the 1940s and 1950s, about a third in the 1960s and 1970s, and about a sixth in the 1990s. Perhaps Key was right still, but only about the South.
To assess this possibility, Figure 6 displays the percentage of election sequences in which the winners overcame a competitive challenge—having received less than 60 percent of the vote—in either the primary election, the general election, or both, for statewide offices and the U.S. House, divided by region. If the minority party's rising fortunes channeled electoral contestation out of primary elections and into general elections, then the increase of competition in general elections should have occurred simultaneously with the decline of competition in primary elections, or led the decline of competition in primary elections, but should not have trailed it.
Competition in Elections for Statewide Offices and the U.S. House, by Region.
Of the four histories in the display, the only one that possibly met the expectation was elections for statewide offices in the North. The competitiveness of general elections started to rise in the thirties, just as the competitiveness of primary elections started to fall, and the competitiveness of general elections continued to rise—albeit slightly, in the overall—as the competitiveness of primary elections continued to fall. A sharp drop in two-party competition in the 1970s, however, had no counterpart increase in primary competition. And competitiveness fell in both stages of elections for the U.S. House of Representatives from the North, turning lower in primary elections about two decades before trending lower in general elections.
In the South, though, the decline in primary competition preceded the rise in two-party competition both in statewide elections and in U.S. House elections, by a decade or two in each case. If the growth of the Republican Party in the South drew political contestation away from the Democratic primary and into the November election, it was a long time in anticipation.
The weight of our electoral evidence suggests, then, that the rise of competition in general elections was separate from the decline of competition in primary elections. In the North, interparty competition increased in statewide elections beginning in the 1930s and decreased in U.S. House elections beginning in the 1950s. In the South, it increased in both, starting in each case in the 1950s and 1960s. The downward path of competition in party primaries, on the other hand, began in the 1940s, in parallel, in both North and South. Whatever caused the rise in two-party competition in the North (in statewide races) and in the South (in statewide and U.S. House races) was evidently specific to each region—most probably the New Deal realignment of voter loyalties in the North and the civil rights realignment in the South. Whatever caused the decline in primary competition, however, was apparently common to both North and South.
One important candidate for the common cause is the rise in the electoral value of incumbency. A sizable research literature documents a substantial increase in the incumbency advantage in U.S. House elections beginning in the 1950s; other research finds a significant rise in the incumbency advantage in elections for statewide offices.Footnote 34 The electoral privileges that incumbents came to enjoy were not easy to anticipate during the era of Progressive electoral reform, when officeholders' fortunes were subject much more to partisan forces beyond their control.
Figure 7 examines the impact of incumbency on the effectiveness of the direct primary in promoting electoral competition. As in earlier figures, primary and general elections count as competitive if the winner received less than 60 percent of the vote. The key comparison is the evolution of competition in elections with and without incumbents. If the rising importance of incumbency contributed to the decline in competition in nominating primaries, the decline should be concentrated in elections that were contested by incumbents.
Competition in Elections for Statewide Offices and the U.S. House, with Open Seats and Seats Contested by Incumbents.
And so it was. In contests for seats in the House that were opened by death or retirement, over 85 percent of the winners faced a stern challenge in either the primary or the general election, both in the direct primary's first years of operation and later. The contribution of the primary by itself to the total experience of competition also changed hardly at all.
In contrast, 60 percent or so of the election sequences contested by House incumbents were competitive at either stage of the process in the first three decades of Progressive election reform, and the incidence of competition fell to around 30 percent after 1970. Competition slumped in the general elections and in the primaries, but proportionately more in primaries. After 1950, only one in twenty incumbents turned back a strong intraparty challenger on the way to reelection.Footnote 35 The direct primary's contribution to electoral competition in U.S. House elections remained as strong as ever in contests for open seats, but it diminished almost to insignificance in races run by House incumbents.
A closer look at competition in races for statewide offices yields the same conclusion. In contests for statewide offices vacated by death, retirement, or term limitation, over 80 percent of the winners bested competition in either the primary or the general election. The contribution of the direct primary to the total experience of competition declined somewhat: first-time winners of statewide offices were particularly less likely to have been tested only in the primary. The loss of competition in open-seat primaries, though, was compensated by a gain in two-party competition in general elections.
In contrast, the steady increase in two-party competition in general elections for statewide offices contested by incumbents did not overcome a sharp decrease in intraparty competition in primary elections. In the direct primary's first decades, two-thirds of incumbents overcame significant electoral challenges at one stage or both in their successful bids for reelection, and primaries contributed to their trials over half the time. After 1960, however, only about 55 percent of reelected incumbents fought tough races, and primaries added to their ordeals only a quarter of the time.Footnote 36
The declining efficacy of the direct primary in fostering competition in U.S. elections relates to the rising effectiveness of incumbents.Footnote 37
Conclusion
The direct primary was a pro-competitive, democratizing reform in statewide and federal elections in the United States in the twentieth century. It introduced contests for the favor of any and all voters who wished to participate in nominations for public office, opening the political parties to the direct influence of the people, an outcome the Progressives saw as a good in and of itself. It also extended opportunities for contestation into areas of the United States that had no prospects for two-party competition, giving voters in single-party states a new instrument for accountability and choice. To be sure, intraparty competition was not the same as interparty competition. From the beginning, primaries were run before a partial electorate, often, but not always, with lower voter participation, with the object of selecting nominees, not (directly) officeholders.Footnote 38 It has never been certain that intraparty competition is a worthy objective and, to this day, activists and scholars debate whether democracy is better served when voters control nominations or when party elites do.Footnote 39 One fact, however, should not be overlooked: in its historical context, the direct primary fostered competition where it did not exist before.
In the very first decades of the reform, we have shown, the direct primary contributed significantly to the competitive rigors that tend to keep officials honest, active, and responsive. Later, it contributed less to an electoral process that itself became less competitive.
On one level, our evidence indicates, the Progressive experiment in election reform addressed key problems in the operation of American elections. The direct primary enabled contestation on the issues that divided the electorate locally, whether they divided the parties nationally or not, respecting the partisan loyalties of the electorate, not working against them. Equally, the direct primary limited the power of party elites to control nominations and thereby the actions of elected officials. In the first decades of its implementation, the direct primary added a new layer of constraint on the actions of elected officials, a layer that was particularly important in locales that lacked significant two-party contestation.
On another level, our evidence also indicates, ironies abounded. Even as they accommodated themselves to the realities of voters' partisan loyalties, Progressive reformers urged citizens to “vote for the man, not for the party.” As partisan loyalties weakened, however, voters hearkened to the advice—to the benefit of incumbents, who had the advantages of experience and exposure over their challengers. The Progressives also battled the spoils system, the political patronage that enriched party leaders and entrenched party regulars in office. As they succeeded in reform, though, they released resources from the parties' grasp—to the benefit, again, of incumbents, who expropriated for their own electoral purposes the public resources that had once been directed to the party's benefit. The rising incumbency advantage, our evidence shows, eventually weakened the contribution of the direct primary to electoral competition.
From the very beginnings of the republic, with the deliberate construction of the constitutional order of checks and balances, Americans have shown a distinct propensity to find solutions to political problems in institutional engineering. In accordance with the American tradition, at the turn of the nineteenth century to the twentieth, Progressive reformers urged the direct primary to “cure the evils of democracy” with “more democracy.” The experience of a century with the direct primary shows that while it is quite possible to engineer electoral competition, it is much more difficult to sustain it.
Appendix
Elected Offices and Data Sources for Each State
LG = Lieutenant Governor; SS = Secretary of State; AG = Attorney General; Tr = Treasurer; Au = Auditor, Controller, Comptroller; E = Commissioner of Education, Superintendent of Schools, etc.; Ag = Commissioner of Agriculture, Agriculture & Industry, Dairy, etc.; Rg = University Regent, Trustee; PU = Public Utility Commissioner, Public Service Commissioner, Railroad Commissioner, Railroad & Public Utility Commissioner, etc.; Co = Corporation Commissioner; Cm = Commerce Commissioner; I = Insurance Commissioner; Lb = Commissioner of Labor; Ld = Land Commissioner, Surveyor, Inpector, Geologist; M = Commissioner of Mines, Mine Inspector: Gm = Game Commissioner; Pn = Prison Commissioner; Tx = Tax Commissioner, Tax Collector; CC = Charities & Corrections Commissioner; Pr = Printer; St = Statistician; El = Elections; Hy = Highway Commissioner; PW = Board of Public Works; Ck = Court Clerk, Court Reporter; Aj = Adjutant General; J = Supreme Court Justice, Appeals Court Judge.
