Introduction
When Richard Simeon's seminal piece was published in 1976, policy studies in Canada had yet to establish itself as a recognized subfield in political science. To be sure, some Canadian political scientists were producing informative work in public policy, most notably in the neo-Marxist stream of political economy, but these scholars did not identify themselves as policy scholars (for example, Panitch, Reference Panitch1977). Most scholars identifying as such were then working in policy schools and several of them had backgrounds in disciplines other than political science, often in economics. Simeon was ahead of his time in the 1970s when most political science departments did not recognize the study of public policy as a stand-alone subfield. It is fair to say that Simeon's call for making policy studies a proper subfield of political science has largely materialized in Canada. Many Canadian Ph.D. programmes in political science now offer specialization in public policy, with comprehensive exams in the subfield. The Canadian Journal of Political Science regularly publishes policy studies and positions in public policy are a core part of faculty recruitment for political science departments. In itself, this special issue testifies to the vitality of policy studies in Canadian political science.
Simeon's article not only called for the development of policy studies within political science, it also promoted a specific vision whereby political science knowledge can inform public policy. Concerned by the proliferation of idiosyncratic prescriptive case studies in the nascent field of policy studies, Simeon's vision argued in favour of producing more comparative policy research that aims at explaining the scope of policy, the choice of instruments and their distributive outcomes. Simeon foresaw a public policy subfield in which methods enable theory building and testing through systematic observation of political factors, alongside the economic factors that were more commonly included in the policy analyses at the time. Simeon was not alone in promoting such a vision. He participated in a broader movement reacting to behavioural studies, which focused on individuals and decision making and overlooked political inputs (which were included in the “funnel of causality” covered by Wilder in this special issue) and governmental outputs (Anderson, Reference Anderson1971; Cyr and deLeon, Reference Cyr and deLeon1975; Feldman et al., Reference Feldman, Heidenheimer, Heclo, Adams, Smith, Hayward and Watson1978; Hofferbert, Reference Hofferbert1974; Leichter, Reference Leichter1977; Lindblom, Reference Lindblom1968; Rose, Reference Rose1973a, Reference Rose1973b). As Leichter (Reference Leichter1979: ix) wrote during this period, “students of comparative politics had come to realize that the analysis of political systems will remain incomplete as long as the questions of what government does, why and with what consequences remain unanswered.” This comparative movement influenced the subfield and Simeon's article became a point of reference for Canadian policy scholarship and has remained so today (Atkinson, Reference Atkinson2013; Bennett, Reference Bennett1990; Lemieux, Reference Lemieux2002; Taylor and Eidelman, Reference Taylor and Eidelman2010; Wilder and Howlett, Reference Wilder and Howlett2014).
Developments over the last 40 years indicate that policy studies have indeed evolved in the direction suggested by Simeon and other scholars of the 1970s (Engeli and Rothmayr Allison, Reference Engeli and Allison2014). Baumgartner (Reference Baumgartner2016: 5–6) argues that the most important development over the last decades consists of “the creation of an integrated intellectual community across national borders,” which allows for “exploring the reasons for difference as well as the impact of different institutional, cultural, or political factors on important policy outcomes.” At the same time, the diversity and complexity of public policy as a subfield of political science has increased significantly. There is no single theoretical approach or tradition that unifies the subfield. Policy scholars continue to pursue different objectives, some preferring to inform policy, others to develop explanatory theories. Various disciplines engage in policy analysis, and political science is only one among several. Lastly, interpretive streams of policy research, also referred to as critical, discursive, argumentative or deliberative, have also emerged in those same decades and all challenge theory building as understood by positivist thinkers (Dryzek, Reference Dryzek1990; Fischer and Forester, Reference Fischer and Forester1993; Fisher, Reference Fisher2003; Fisher et al., Reference Fisher, Douglas Torgerson, Durnová and Orsini2016; Hajer and Wagenaar, Reference Hajer and Wagenaar2003; Smith and Orsini, Reference Smith and Orsini2007; Stone, Reference Stone1997; Yanow Reference Yanow2007).
Over the past several decades, policy scholarship has evolved and this evolution is reflected in scholarly works and journal articles. Adams and colleagues (Reference Adams, Infeld, Minnichelli and Ruddell2014: 119) argue that editors and reviewers of public policy journals act as “gatekeepers” in selecting what they consider exemplary policy research. In deciding what type of research to publish, they influence what are to be considered important theoretical, methodological and empirical developments. In this article, we examine publication trends among Canadian public policy scholars to better understand how the subfield has positioned itself over time with respect to the issues raised by Simeon. We focus our analysis on the entire set of articles that were published in five leading public policy journals between 1980 and 2015: The Journal of Public Policy, The Journal of European Public Policy, Governance, Policy Studies Journal and Policy Sciences. While an analysis such as the one that we present here cannot account for all relevant scholarship in public policy, it provides important insights about the subfield's representation in the most competitive publication venues.
This article does not attempt to causally relate the evolution of the discipline, as observed in our analysis of journal articles, to Simeon's article. As emphasized above, Simeon's article was part of a broader international movement in which the influence of Simeon would be difficult to isolate. We use Simeon's call for comparative research as a point of reference on the importance of theory-oriented comparative approaches to policy studies. Other similar points of reference exist in the policy literature, but, as just suggested, Simeon's article stands out as particularly important in Canada at a time when policy studies was emerging as an autonomous area of study. Simeon's article provides us with a useful reference to compare aspirations for policy studies in the 1970s to the evolution of Canadian publications ever since and allows us to assess whether policy studies have become as comparative and theory-oriented as advocated by Simeon in the 1970s.
Our analysis shows that scholarly journals have over time integrated several of the elements of Simeon's vision for the policy subfield. In most instances, journals have integrated these elements, albeit slowly and perhaps modestly. Comparative analysis has indeed become more common in recent years, but much room remains in policy journals for non-comparative studies. Likewise, explanations have become more common than in the past, but there is still important space for descriptive studies and journal articles have included lengthier presentations of methods than in the past, but methods widely vary. In other words, publication trends in policy studies, especially among Canadian scholars, are consistent with Simeon's vision, but the policy subfield is probably as diversified today as it has ever been.
We start with a discussion of the evolution of the policy subfield, drawing on key issues raised by Simeon in 1976. We then move to the presentation of the details of the content analysis that was performed on five policy journals and discuss our results in light of the main challenges that were brought forward in Simeon's article. We conclude with a discussion of the current state of policy studies as a subfield of political science in Canada.
Simeon's Challenges for Public Policy Study
Simeon emphasized several challenges for policy research in his 1976 piece, of which we selected two. These two challenges are not only salient in his article, they still draw the attention of Canadian policy scholars today. The first challenge addresses the scope and unit of comparisons. Comparisons, he wrote, can be across countries, units within countries, periods or policy sectors, as long as policy scholars compare with the aim of building theories: “We need to look at the broad evolution of patterns of policy over long periods within countries, provinces, and other units, in the ways they deal with similar problems as a first step toward the primary goal of explaining the differences” (Simeon, Reference Simeon1976: 550–51). While open to all kinds of comparisons, Simeon lamented that policy research in the 1970s too often resulted in descriptive single-case studies. Indeed, the extent to which Canadian scholars make their knowledge amenable to comparison became one of Simeon's major preoccupations later on (see White et al., Reference White, Simeon, Vipond and Wallner2008). The second challenge concerns theory building. Simeon's leaning toward comparison was largely justified by his vision of policy studies as yielding cumulative knowledge. Drawing on these two challenges, our analysis assesses whether, how and to what extent comparative policy research has evolved over time.
Have Canadian policy studies become more comparative overtime?
Simeon was hard on single-case studies. He wrote:
Individual case studies tend to be isolated and unique, each looking at different issues, using different methods, and asking different questions. This makes comparison extremely difficult. Their focus has often been on the details of the policy itself, rather than on using the policy to generalize about politics. Cumulative knowledge and theory cannot simply grow automatically by piling case studies on top of each other. Case studies have also a tendency not to focus on the “normal” but on the unique, exotic, or important, so insights gained from them may actually be misleading. Moreover, in focusing on a specific decision or piece of legislation, case studies tend to ignore those issues or alternatives which simply do not come up for debate. It is easy to get submerged in the minutiae of the issue itself, and therefore to miss what might be much broader factors influencing the outcome. (1976: 551)
While he acknowledged that individual case studies could be useful—such as in providing rich details and enabling the formulation of new hypotheses—he nevertheless observed that these potential benefits rarely materialized. Rather than focusing on methodological improvements as a way of overcoming the difficulties of individual case studies, Simeon (551) suggested substituting comparisons, even “very simple” ones, for them. Comparing requires thorough thinking that goes beyond the description of the singularities of a particular case; comparative objects of study call for a level of abstraction and explanation that transcends the individual case. In comparisons, policies cannot only be described thickly, they must be presented as “variables” that enable the categorization or the measurement of government action.
To encourage such overarching lines of inquiry, Simeon (559) suggested conceptualizing policy both as means used by governments to obtain compliance with their decisions, as well as scope, which is the span of social and economic activities in which governments decide to become involved. Simeon argued that scholars engaged in the comparative analysis of means or scope are, almost naturally, led to think along general categories and measures that can travel across cases. Thinking in terms of general categories can also be achieved in case studies, something that Simeon later acknowledged, when he and colleagues argued that the comparative turn refers more to a body of general knowledge to which empirical studies must be linked than to comparative methodology (White et al., Reference White, Simeon, Vipond and Wallner2008). In short, in 1976, Simeon argued in favour of comparisons because they require thinking in terms of variables and thinking in terms of variables is a precondition for the production of cumulative knowledge.
Both the desirability and the sharpness of the so-called comparative turn has animated Canadian political science in recent years (Montpetit et al., Reference Montpetit, Blais and Foucault2008; Turgeon et al., Reference Turgeon, Papillon, Wallner and White2014; White et al., Reference White, Simeon, Vipond and Wallner2008). Reasons for the turn are, however, persuasive. First is the fact that the politics and policies of small countries like Canada have less appeal than those of influential international players. To strengthen the appeal of their research, policy scholars from small countries are more likely to turn to comparisons. Secondly, the Canadian scholarly community is relatively small in comparison with the American and European ones. American scholars who have devoted their entire career to the study of Medicare, for example, find a large home-based audience and intellectual community to discuss their work, while a single-case study on Canada might be of interest to only a handful of actors and scholars who work on the specific policy domain. Simeon himself made this point about the study of Canadian politics (Reference Simeon, Shugarman and Whitaker1989). Without a contribution to theory and/or a comparison with other countries, scholarly work on Canadian public policy may prove difficult to publish in reputed international journals.
However, some scholars have disputed Simeon's contention that general explanations arising from comparisons can be as relevant, if not more, as descriptive single-case studies. Alan Cairns (Reference Cairns, White, Simeon, Vipond and Wallner2008) features among the most prominent sceptics of the comparative turn in Canada (see also Fourot et al., Reference Fourot, Sarrasin and Holly2011; Noël, Reference Noël2014). For Cairns, making a descriptive case study into a comparative one might very well increase the relevance of the research outside the circles of individuals and scholars directly concerned with the case, but it might equally decrease the relevance of research inside this circle. In such circumstances, individuals concerned with concrete policy problems are likely to complain about scholarly work being increasingly disconnected from society's concerns. The world of practice, the argument goes, needs informed insights from scholars, not theoretical abstractions (Kristof, Reference Kristof2014).
Furthermore, the debate about the use of case studies for theory building has greatly evolved. Knowledge on matters such as case selection was already sophisticated in comparative politics in the 1970s (for example, Lijphart, Reference Lijphart1971), but not so in the subfields of political science relying on single cases. Recent advances in new methodological tools—process tracing, for example—that apply to single cases have expanded generalization opportunities for single-case studies (Collier, Reference Collier2011; George and Bennett, Reference George and Bennett2004; Gerring, Reference Gerring2007). As Gerring argues, a critical case study can provide “the strongest sort of evidence possible” (2007: 115). In fact, Simeon was less a critic of single-case study than he was a critic of atheoretical studies, that is, studies that do not aspire to draw inferences beyond the phenomenon under study (Levy, Reference Levy2008: 3). In the contemporary methodological debate, a growing number of scholars argue in favour of acknowledging the limitations of any given method and, when possible, drawing from a mix of methods that compensates for each other's weaknesses instead of placing a specific method on a pedestal.
Lastly, interpretive streams of policy research have also emerged in those same decades and all challenge theory building as understood by positivist thinkers (Dryzek, Reference Dryzek1990; Fischer and Forester, Reference Fischer and Forester1993; Fisher, Reference Fisher2003; Fisher et al., Reference Fisher, Douglas Torgerson, Durnová and Orsini2016; Hajer and Wagenaar, Reference Hajer and Wagenaar2003; Smith and Orsini, Reference Smith and Orsini2007; Stone, Reference Stone1997; Yanow Reference Yanow2007). Unlike Simeon, critical policy study tends not to elevate comparison above single-case study. If anything, critical scholars worry about attempts to systematize, categorize and measure that are behind comparative ventures. Individual case studies deprived of any ambition to contribute to positivist objectives, the argument goes, can still be useful to policy actors as exemplary cases. They are also useful to reflective scholars interested in the variety of ways whereby power is subtly practised in democratic and less democratic societies (Flyvbjerg, Reference Flyvbjerg2001). For critical thinkers, scholars who are taken by an individual case will often be more transparent and honest about their political engagement than positivist comparativists, whose pretention to objectivity blinds them to power relations and with which they unavoidably become involved (Stone, Reference Stone1989). All this is to say that Simeon's article did not settle the issue of whether it is preferable to conduct comparative analysis over single-case studies. If anything, debates over this issue have remained as lively as they were in the 1970s.
Is the comparative turn yielding theory building?
As emphasized above, Simeon believed that comparison encourages researchers to think conceptually so as to make observations across several units amenable to measurement, categorization and generalization. Measurable variables and typologies, he argued, are necessary for theory building. The testing of theories, he further suggested, requires comparisons. On theory building, he writes:
Despite this complexity [of policy], it is possible to summarize and simplify, and to abstract from the whole range of government activities some dimensions especially relevant for political scientists. The chief criterion for selecting these dimensions should be what aspects of policy are most relevant to the study of politics. The dimensions should also be relevant to the normative concerns of politics, such as equality and participation. And, they should enable us to be comparative. Finally, we should, at least in principle, be able to measure them. (1976: 557)
Simeon's ontological posture in his 1976 article is positivist in the sense that it values the accumulation of knowledge in theories that offers potential for generalization. Although he did not directly address the issue of objectivity—and despite an acknowledgement of normativity in the choice of topics—his call for the use of measurable variables and typologies suggests that he valued some form of objectivity. As mentioned above, following the argumentative turn in the 1990s, an important part of the policy literature has rejected such ambitions for policy (for example, Smith and Orsini, Reference Smith and Orsini2007). Yet, some policy scholars also began promoting frameworks and theories in the 1980s and the 1990s that offer the opportunity to fulfill Simeon's theory building goals. Sabatier's (Reference Sabatier1987) Advocacy Coalition Framework (ACF), Ostrom's (Reference Ostrom1990) Institutional Analysis and Development Framework (IADF), Baumgartner and Jones's (Reference Baumgartner and Jones1993) Punctuated Equilibrium Theory and applications of historical institutionalism (Hall and Taylor, Reference Hall and Taylor1996; Pierson, Reference Pierson2004;) feature among the most prominent of these frameworks (Sabatier, Reference Sabatier2007; on more recent theories, see also Schlager and Weible, Reference Schlager and Weible2013). Each of them puts forward its own worldview of policy making and supplies abstract concepts that enable comparison, as well as measurement or categorization into variables. Some go as far as providing testable causal theories that make sense of relationships between concepts and variables. To be sure, none of these frameworks is entirely consistent with Simeon's world view of policy making, which he described using the metaphor of the “funnel of causality” (1976: 556). The funnel image suggests that a range of factors is processed within narrowing sets of political considerations (see also Hofferbert, Reference Hofferbert1974). Nonetheless, recent theoretical frameworks encompass to different degrees the factors that Simon stressed as important using the funnel metaphor. It can be said for example that belief systems that divide actor coalitions in the ACF are one way of accounting for ideological conflicts, a factor deemed important by Simeon. Likewise, rational choice and historical institutionalism offer powerful ways to operationalize institutions, which also are reflected in Simeon's discussion of funnel of causality, while the concept of “image” in the Punctuated Equilibrium Theory speaks to the importance of culture, yet another important factor in Simeon's view (Elkins and Simeon, Reference Elkins and Simeon1979).
Since the late 1980s, there is an increasing supply of theoretical frameworks specific to policy studies that enable cumulative knowledge and theory building in ways that Simeon had envisioned in 1976. This said, the presence of these frameworks has not made policy studies into a unified theory building project, if only because such a project has become vigorously contested in the critical/argumentative streams of policy studies. But it might be argued that these frameworks offer scholars more explicit opportunities to participate in various theory building projects. Although the frameworks mentioned above were devised in the United States, several Canadians contributed to their development. Examples include Mark Sproule-Jones's contribution (Reference Sproule-Jones1993) to the IADF through his studies of watershed management; Andrew Stritch's contribution (Reference Stritch2015) to the ACF with his work on disclosure requirements for trade unions; Cashore and Howlett's contribution (Reference Cashore and Howlett2007) to Punctuated Equilibrium Theory with their study of forestry; and Béland and Waddan's contribution (Reference Béland and Waddan2015) to institutionalism through work on social policy. While opportunities to contribute to theory building were few in 1976, they have multiplied since and several Canadian policy scholars have seized the opportunities. Yet, the multiplication of these opportunities has further diversified the policy subfield rather than mainstreaming it, as often feared by critical thinkers (for example, Flyvbjerg, Reference Flyvbjerg2001).
Content Analysis of Articles Published in Five Policy Journals
To examine our two central research questions we selected the five generalist public policy journals with the highest H-index in the 2013 public administration ranking produced by Thomson Reuters.Footnote 1 The journals are Policy Sciences, The Journal of Public Policy, The Policy Studies Journal, The Journal of European Public Policy and Governance. The five journals have a solid anchorage in political science, one key element of Simeon's vision for policy studies. A large majority of the articles published in these five journals are authored by political scientists, and political scientists are in the majority on their editorial boards.Footnote 2 To avoid any potential bias, we left aside journals that pertain to a specific policy sector, even when the journal enjoys a high H-index (for instance, Climate Policy).
Policy Sciences has been published since March 1970 and The Policy Studies Journal since September 1972 and therefore they cover the entire 1980–2015 period. The Journal of Public Policy’s first issue appeared in 1981. The other two journals are more recent: the first issue of Governance was published in January 1988 and The Journal of European Public Policy appeared for the first time in June 1994. Our initial intention was to code all five journals since their first issue, but we quickly realized that the enterprise was unrealizable for the two oldest journals. In fact, most issues published in the 1970s are composed of atypical articles by today's standards. They are shorter, written by practitioners rather than scholars and they are primarily descriptive. As Simeon lamented in his 1976 article, policy studies was then atheoretical, focused on bureaucracies and the efficacy of the policies for which these bureaucracies had responsibility. The journals began to change late in the 1970s and therefore we decided to begin our analysis in 1980. For our analysis we use regular research articles as the unit of analysis, of which 4,097 were found in the five journals from 1980–2015.Footnote 3
We then subjected the articles in these five major policy journals to content analysis using a coding scheme designed to address the two questions presented in the previous section: (1) Have Canadian policy studies become more comparative over time? (2) Is the comparative turn yielding theory building? Five student coders were asked to examine the content of each article and determine whether or not it had comparative content and of what nature. Basic information such as the title of the article, the name of the author(s), the author's country of origin and the date of publication was also collected.Footnote 4 Coders also coded the article's objective of explaining rather than just describing, as well as the method(s) employed, if any. The author's country of origin was then used to focus on scholarship published by Canadian public policy scholars in comparison to scholars from other countries, particularly Australia.
While Simeon's article was part of an international movement to establish and transform policy studies as a subfield, his article reached Canadian scholars first and foremost, all the more so because the article was published in The Canadian Journal of Political Science. Simeon's article was a relevant point of reference mostly to Canadian scholars and we therefore focus the presentation of the results of our content analysis on the articles published by Canadian scholars. However, to get some perspective on these publications, we compare them with the articles of Australian scholars. Australia is used here as a yardstick against which Canadian publication patterns are examined. Points of reference for Australian policy scholars surely are different from those of Canadian policy scholars and therefore we expect different publication patterns between the two countries. It is these differences that will provide some perspectives on patterns of Canadian publications. We selected Australia for the comparison because of similarities with Canada in terms of scholarly culture and country size. Australia also provides for a fairer comparison with Canada than would the United States or the United Kingdom.
Between 1980 and 2015, 180 research articles feature at least one scholar associated with a Canadian institution, as opposed to 101 articles that feature a scholar from an Australian institution (labeled “Canadian” and “Australian” for reader-friendliness in the presentation of results, this without any assumption about citizenship). The percentage of articles with at least one co-author affiliated with a Canadian institution (regardless of the order of authorship) has increased over the years. While it was under 2 per cent of the research articles in the 1980–1989 period, it reached 4 per cent in the 2010–2015 period. The largest contingent comes from the United States, whose scholars are indicated as authors (sole or co-author) on about half of the articles. More comparable to Canada is Australia, whose scholars authored 2.9 per cent of the articles between 1980–2015.
Figure 1 shows the distribution of research articles in the five journals per period. In a two-sample t test, we found Canadian scholars to be just as likely as those from Australia to author articles with empirical content, as opposed to articles that are uniquely conceptual, theoretical or methodological. More than 15 per cent of the articles published by Governance in the journal's first two years of existence were by Canadians, a proportion that declined from period to period to reach fewer than 5 per cent in the 2010–2015 period.Footnote 5 This proportion is comparable to the proportion of Australians publishing in Governance since 1990. Canadians and Australians also produce roughly 5 per cent of the content of The Journal of Public Policy since 1990. While in the 1990–2009 period Canadians produced almost 10 per cent of the content of Policy Sciences, the proportion declined to 5 per cent after 2010, a percentage under that obtained for Australian research articles. Canadians, however, have increased their presence in the Policy Studies Journal since the 1980s to reach almost 5 per cent after 2010, a figure above that for Australians. Understandably, Canadians and Australians policy scholars are least present in The Journal of European Public Policy. In short, the article outputs of Canadians and Australians are similar in many ways.
Articles Authored by Scholars Affiliated with Canadian and Australian Institutions
Results of the Content Analysis
The comparative turn
Although Simeon stressed the crucial relevance of comparison, he did not place country comparisons above any other form of comparison. Comparisons, he argued, could also involve subunits within countries, different policy sectors or periods of time. While it is largely unproblematic to identify articles with country and subunit within-country comparisons, we defined articles with sector comparisons when at least two distinct government programmes or policy domains were clearly identified. We defined articles with time comparisons when clear time periods, with distinct beginning and ending dates, were specified. Simeon (Reference Simeon1976: 550–51) remained relatively vague about how comparisons across time should be carried out, simply saying that policy studies should embrace long periods. In keeping with the value accorded to systematization in his vision, we assume that comparisons across time require, at a minimum, clear beginning and ending dates.
Overall, our content analysis shows a steady increase in the proportion of research articles employing comparisons over the time periods examined. The percentage of articles by Canadians and non-Canadians that present country, federated unit (states or provinces) or local government comparisons grew from 13 per cent in the 1980–1989 period to 20 per cent in 1990–1999 to 24 per cent in 2000–2009 and to 26 per cent in 2010–2015. As shown in Figure 2, the proportion of articles by Canadian scholars that feature territorial comparisons is constantly above these figures across time. It has increased from 20 per cent in the 1980–1989 period to stabilize at around 30 per cent between 2010 and 2015. Interestingly, a country's small size and light international weight does not always increase the propensity of its policy scholars to produce more territorial comparisons than would be the case in the United States and other large countries. In fact, the proportion of territorial comparisons by Australians reached 25 per cent in the 1980s, dropped to 17 per cent in the 2000–2009 period and rose again up to 22 per cent in the 2010–2015 period. Overall, Canadian scholars seem more likely than Australian scholars to include a country, subnational or local government comparison in their empirical outputs. While 29 per cent of the research articles authored by at least one Canadian scholar have comparative content, only 21 per cent of the articles by Australians employ comparisons. In a two-sample t-test, we found that Canadians are significantly more likely than Australians to conduct territorial comparisons.
Proportion of Territorial Comparisons by Scholars Affiliated to Canadian and Australian Institutions
Our overall statistics for scholarship published by authors from all countries also show an increase in time comparisons—several of which also feature a comparison of territorial units—which have about doubled from 1980 to 2015. In contrast, sector comparisons have decreased over the last five years. As shown in Figure 3, Canadian policy scholars have similarly increased the proportion of their publications featuring time comparisons, but Australians did not. In fact, in a two-sample t-test, Canadians are significantly more likely to resort to time comparisons than Australians. The same does not hold true for sector comparison. A two-sample t test failed to find Canadians any more likely to resort to sector comparisons than Australians (that is, the probability that Canadians employ more sector comparisons than Australians is only of 61 per cent). The breakdown by time period (Figure 3) nevertheless shows distinctive trends for sector comparisons between Canadians and Australians over time. Lagging at about 20 per cent in the 1990–1999 period, sector comparisons among Canadian scholars reached above 30 per cent in the 2010–2015 period. Meanwhile, sector comparisons among Australian scholars followed the reverse pattern.
Proportion of Time and Sector Comparisons by Scholars Affiliated to Canadian and Australian Institutions
Together, Figures 2 and 3 reveal that Canadian public policy scholars have become more comparative in their approach to policy studies between 1980 and 2015. Despite similarities in size and scholarly cultures between Canada and Australia, Australian scholars have not embraced the comparative path to the same extent as Canadians.
Although Simeon did not mention anything specific about the importance of including Canada in comparative studies in his 1976 article, Canada does feature relatively frequently in small-N comparisons, that is, comparisons involving ten countries or fewer. While the United Kingdom and the United States are overall more frequently included in small-N comparisons, Canada does well, given the relative size of the country, with 71 small-N comparisons. Australia appears only 39 times. Interestingly, Canada is most likely to appear in the small-N country comparisons published in Governance and the Policy Studies Journal. The country rarely features in comparisons published in The Journal of European Public Policy, which compares mostly European countries. Canada features more frequently in small-N comparisons than in single country studies. In fact, only 39 single country studies on Canada and 33 single country studies on Australia were identified in the five journals over the entire period, in comparison with above 1000 studies on the United States. While the proportion of single country studies on the United States has radically dropped over time, it might be argued that beside the United States, policy journals tend to provide increasing space to comparative articles over single country studies.
Although policy scholars from small countries might be expected to have more comparative aspirations, the propensity to carry comparative work differs between Canadians and Australians. Australians are less likely than Canadian scholars—as well as scholars from other countries—to publish research with a comparative content in the five journals analyzed. Canadian and Australian policy scholars seem to have made distinct epistemological choices, as far as comparison is concerned. The Canadian epistemological orientation is more consistent with that proposed by Simeon in his 1976 article but also with the general international trend in the direction of more comparisons. As shown in Figures 2 and 3 however, there is still significant space for non-comparative studies in policy journals.
Theory building
The second question of whether policy studies have become more cumulative or amenable to theory building proves a challenging one to answer. Our coding scheme distinguishes between articles that propose a description, a prescription, an explanation, an evaluation or a conceptual discussion, either theoretical or methodological. We make the assumption that an increase of the volume of articles oriented toward theory building and cumulative knowledge over time is reflected in an increase in the volume of articles proposing an explanation and a decrease in the volume of articles offering a description. If we look at the entire body of articles that we coded, this is precisely the trend that we observe: explanatory articles have increased from 45 to 62 per cent while descriptive papers have decreased from 24 to 15 per cent. The increase in explanatory outputs and decrease in descriptive outputs have been steady over the entire period. As shown on Figure 4, Canadian trends are consistent with these overall trends. Explanation-focused articles increased from 50 per cent of the Canadian publications in the 1980s to 65 per cent in the 2010–2015 period, while description was at the highest in the 1990s (23 per cent) and went down to 8 per cent in the 2010–2015 period. Figure 4 reveals contrasting patterns in publication by Australian policy scholars. While explanation peaked at 65 per cent in the 1990s, description reached their highest in the most recent period (30 per cent).
Proportion of Explanations and Descriptions by Scholars Affiliated to Canadian and Australian Institutions
The increasing number of articles focused on explanation should not too quickly be taken as proof that policy studies have lived up to Simeon's vision elaborated in his 1976 article. While theory building goes always with the aim of developing an explanation for a general phenomenon, not all explanations have necessarily such endeavours. A number of articles that feature explanations only try to make sense of idiosyncratic events, hence the importance of looking at other indicators of theory building and knowledge accumulation. Most research articles that aim at theory building and knowledge accumulation put forward a general explanation informed by systematic observations. In turn, systematic observations require methods. Therefore, it might be expected that articles that do not explicitly present a method—including articles that present unsystematic literature reviews as a source of empirical confirmation—are in decline to the extent that theory building and knowledge accumulation gain popularity among policy scholars.
We pointed out above that Simeon was seemingly less against case studies than against studying policy in atheoretical fashion. Since 1976, single-case methodologies progressed tremendously and they have been employed by several scholars for the explicit purpose of advancing cumulative knowledge. Nevertheless, single-case studies, whether they pertain to a country or another entity, are still frequently stigmatized as focused on the description or explanation of idiosyncratic events. Since Simeon clearly called for fewer single cases in 1976, we believe it is useful to examine trends in the publication of articles featuring just one case. Figure 5 highlights the evolution of Canadian and Australian articles that do not include any explicit indication about method and/or rely on single cases. Both Canadian and Australian policy scholars have published fewer and fewer articles that are silent about methods over time, although the decline is sharper in Canada. The results are more surprising when it comes to single cases. In fact, the number of single-case studies authored by Canadians was low in the 1980s, a finding at odds with Simeon's complaint of the previous decade. The proportion climbed a little over 30 per cent in the 2000–2009 period and fell to 23 per cent during the most recent period. With the exception of the 1980s, the proportion of single-case studies authored by Australians was similar to the Canadian proportions.
Proportion of Articles by Canadians and Australians without an Explicit Method or Based on a Single Case
Taken alone, any indicators of a shift toward theory building and knowledge accumulation may prove misleading. Assessed in conjunction, however, we are relatively confident that these data provide a helpful indication about the place of theory building and knowledge accumulation in policy journals. Consistent with broader trends, Canadian publications provide less description and more explanation than in the past. Articles authored by Canadians that skip any explicit methodological presentation or rely on single-case studies are less numerous over the last five years than they were previously. The picture for Australians is more complex: while explanations are decreasing and descriptions are increasing, papers thin on methodology or featuring a single-case study are both in slight decline. While Canadian trends are consistent with Simeon's vision put forth in 1976, Australian trends possibly abide by a logic specific to Australia's policy scholarship.
Conclusion
Simeon (Reference Simeon1976) adopted a critical stance on the state of policy studies in the 1970s, depicting it as insufficiently anchored in political science knowledge, too focused on descriptive single-case studies, and overly concerned with the efficacy and efficiency of policies administered by specific bureaucracies. Simeon's article offered prescriptions about what policy studies should be, promoting a rather comprehensive vision for the subfield. Forty years after the publication of Simeon's article, we proposed here an empirical analysis of the evolution of the publications in public policy, enabling us to take stock of where we are in relation to Simeon's vision put forth in 1976. Our systematic empirical analysis of top-ranked journals contributes some important insights and knowledge about the past and the current state of Canadian policy studies. Such knowledge, we believe, is a starting point for future debate about developments and trajectories in policy studies.
The analysis presented here shows that Simeon's criticism no longer holds and that it is less valid for Canadian policy scholarship than it is internationally. According to Thomson Reuter's Social Citation Index, public policy journals rank among the top political science journals, showing that policy studies have established themselves at the core of the discipline.Footnote 6 As our content analysis of articles published between 1980 and 2015 reveals, the proportion of single-country studies has been in decline while comparative studies have been on the rise. In the most recent period, a majority of the single country studies is focused on the United States, undoubtedly an artefact of the country's size and of the number of policy scholars affiliated to American institutions. Canadian policy scholars also increasingly embrace comparative studies, which were already relatively popular in the 1980s. Our results further highlight that over the last 35 years, Canadian public policy scholars have increasingly used social science methods of qualitative, quantitative or mixed nature for systematic observation, in view of contributing to explanations rather than descriptions. Canadians appear particularly prone to follow this path, at least in comparison with Australians. Canadians public policy scholars have thus published fewer articles of the type decried by Simeon since the publication of his article in 1976, a larger proportion of their output having the characteristics that he valued. In other words, trends for Canadian public policy publications are in the direction of Simeon's vision.
Moreover, our observed increases in comparative content are based on conservative estimates; readers should keep in mind that the mere existence of The Journal of Comparative Policy Analysis is evidence that comparison has gained importance in policy studies. The supply of comparative content by The Journal of Comparative Policy Analysis from 1998 on further adds to the comparative turn.
We do not want to suggest that the policy subfield has become more unified behind Simeon's vision for theory building and comparative analysis. If anything, the subfield has gained in diversity since 1976 in Canada as elsewhere. There were fewer comparative studies of territorial units produced by Canadian policy scholars in the 1980s than in the first half of the 2010s. The increase, however, is of just 10 percentage points, leaving significant space for other types of studies. There are fewer articles published by Canadians that do not present a method of observation, but methodological diversity has, if anything, gained in importance as suggested by our discussion of new methods for single-case studies. Moreover, Simeon's article was part of a broader movement and it is complicated to distinguish its specific influence on Canadian scholarship from the larger movement, as well as from other important changes that have taken place in the subfield of policy studies. Simeon's article contributed to orienting policy studies in Canada toward comparison and theory building, but the subfield today is diversified, both theoretically and methodologically, a fact that our content analysis captures only partially. Over the last decades, the multiplication of specialized journals with a focus on specific policy sectors or methodological orientations has changed the publication landscape in public policy. Several scholars prefer other publication venues, such as journals with clear theoretical orientation—Critical Policy Studies for example—or books.
The mobilization of theories, Cairney (Reference Cairney2013) argues, does not naturally translate into the general synthesis sought by some proponents of knowledge accumulation. The theories and concepts used by policy scholars often belong to different intellectual traditions. Therefore, they are rarely amenable to synthesis. Competing theories tend to produce contrasted, or even incompatible, outlooks on the policy process. Therefore, various policy theories and multiple methods add to the subfield's pluralism rather than reducing it through falsification or synthesis. Our observations certainly do not support a view of policy studies as defined by a single vision—whether consistent with Simeon's vision or not—toward which scholars from Canada and elsewhere would converge.
