POLLING THE POLLSTERS

This study is based on an online survey of directors of ASOs. ^{Footnote 1} The list of ASOs and their contact information was created using a variety of sources including the Academic Association of Research Organizations (AASRO) and the list of Academic and Not-for Profit Survey Research Organizations (LANSRO). ^{Footnote 2} After identifying a number of organizations that had closed, we were left with a list of 136 organizations that we believe might conduct survey research. ^{Footnote 3} Only 115 organizations opened the email invitation to the survey (two of these immediately clicked the opt-out link). We used multiple email reminders and follow-up phone calls to boost the response rate. Ninety-four organizations started the survey and 80 completed the entire questionnaire. The 59% response rate is similar to other surveys targeting academic survey organizations (e.g., the surveys conducted by the AASRO). The survey was implemented between April 26, 2015 and June 1, 2015. Given that we attempted to reach all members of the population of ASO directors, this study is more akin to an enumeration or a canvas than a survey. True population parameters do not exist for the population of ASO directors. We are unable to determine precisely if those who did not complete the instrument are substantially different from those who did.

WHO LEADS ACADEMIC SURVEY ORGANIZATIONS?

Tables 1 and 2 present demographic information about the leaders of academic survey organizations. Nearly half (48%) of the directors who answered the survey had an academic background in political science. Sociology (10%), then psychology (9%) were the next most common disciplinary backgrounds. The remaining directors came from a wide variety of backgrounds: social science (5%), business (4%), public administration/affairs (4%), survey methods/statistics (4%), criminal justice/criminology (3%), and education (3%), with 9% mentioning some other field.

Table 1

Field of Highest Degree

Table 2

Characteristics of Directors of Survey Organization

Most ASO leaders are male (58%) and almost all are white (90%). Notably, none of the organizations that responded to our survey had a responding director who identified as black or African American. Directors typically have 13–15 years of experience in the field of survey research. Turnover seems relatively low for director positions given that most have worked between seven and nine years with his or her current ASO. Prewitt (Reference Prewitt, Rossi, Wright and Anderson1983, 142) notes that “it is not unusual to find survey organizations directed by individuals reluctant to be directing.” The results here do not generally support such a claim, but perhaps the reluctant directors (and new directors) were also reluctant to take part in the survey or more difficult to reach.

As we note later in this paper, training and educating undergraduate and graduate students is often an important element of a survey research organization. Our survey found that nearly three-fourths of the directors teach courses at their institution. Approximately half (52%) were tenured or tenure-track faculty, 22% non-tenure track faculty, 14% were non-faculty administrators, 10% were university/college staff members, and 3% held some other type of position. Female directors were far less likely than male directors to hold tenure-track or tenured positions (31% vs 67%). In contrast, 44% of female directors held staff or non-faculty administrator positions compared to 11% of male directors. Similarly, 83% of male directors taught semester length courses compared to 56% of female directors.

WHAT ARE THE METHODOLOGICAL NORMS OF ASOS?

Conducting survey research is arguably among the most labor intensive methodologies in all of the social sciences. In fact, Prewitt (Reference Prewitt, Rossi, Wright and Anderson1983) suggests survey research is more labor intensive than most methods used in the natural sciences. It requires a variety of tasks, including recruiting and training quality interviewers, maintaining software and equipment, storing, processing, and analyzing survey data, and disseminating results to multiple, diverse audiences. Kennedy, Tarnai, and Wolf (Reference Kennedy, Tarnai, Wolf, Marsden and Wright2010, 589) suggests “survey research mixes science, engineering, and art” and managers of survey organizations frequently must draw on a range of different techniques to collect valid and reliable data. Many of the organizations that we surveyed have been conducting survey research for over two decades (median age of the ASOs was 23 years, with the youngest being only a year old and the oldest having been in practice for 91 years) and the directors who manage these organizations typically have over a decade of experience in the field of survey research.

Over a half-century ago, Campbell (Reference Campbell1953, 228) advised that managing a research organization requires “an adaptability to changes in external circumstances.” We find that ASOs remain adaptable to a rapidly changing environment. ASOs currently employ a variety of methods and techniques to conduct survey research projects ranging from small projects costing $2,000 to larger projects costing $500,000. Table 3 provides an overview of some of the methodological norms among ASOs.

Table 3

Survey Methodology

WHAT ARE THE GOALS OF ACADEMIC SURVEY ORGANIZATIONS?

Academic survey organizations often have multiple, overlapping, and sometimes conflicting goals and missions. ASOs are in the business of conducting rigorous, social scientific research. However, this goal often requires attracting contract work, especially for the ever increasing number of organizations that are being required to be self-funded (Moore Reference Moore2014). In addition, the academic organizations work with faculty and students. An educational mission is consequently often explicitly or implicitly attached to the survey organization.

Organizations that “conduct research as a service at the specification of outside contractors” must balance the requirement of meeting a client’s needs and the desire to conduct research that addresses more traditional academic research goals, such as hypothesis testing (Stahler and Tash Reference Stahler and Tash1994). Furthermore, Campbell (Reference Campbell1953) notes that research organizations embedded in a “highly centralized hierarchy” [like that of a university] might face additional constraints. Academic organizations frequently must meet a higher threshold regarding the protection of the well-being of human subjects (e.g., Institutional Review Boards).

Directors were given a list of goals that some organizations may have. They were asked to state whether they considered the goal to be very important, important, somewhat important or not important (see table 4). Interestingly, although all of the organizations are “academic” organizations, training and educating undergraduate or graduate students and generating data for peer-reviewed research were not the highest rated goals. Instead, “promoting your university’s name, brand, or reputation” had the highest number of directors saying such a goal was very important (52%). This was followed by 46% rating very important a similar promotional goal of “producing results for state and local media organizations.” Prewitt (Reference Prewitt, Rossi, Wright and Anderson1983) notes that “prestige” is a common goal for organizations because it facilitates the achievement of other goals.

Table 4

Importance of Academic Survey Organizational Goals

WHAT CHALLENGES DO ACADEMIC SURVEY ORGANIZATION SEE FOR THEIR PROFESSION?

As noted above, survey research can be an extremely labor intensive method of collecting data; furthermore it is a method facing a rapidly changing technological environment (i.e., cell phones, Internet). These features create interesting challenges to those who direct academic survey organizations. Our survey instrument asked directors an open-ended question: “What do you think is the greatest challenge faced by public opinion survey research organizations in the United States today?” One of the most commonly mentioned challenges is the problem of declining response rates, especially in regards to telephone surveys. Related to declining response rates is the increasing costs of conducting rigorous survey research, combined with the reality of an uncertain funding environment. Another concern mentioned was the changing technological environment and the use of Internet surveys. In addition, directors voiced concern over the growth in the number of low quality surveys which compete for attention for the products produced by ASOs. Table 5 details responses about the challenges facing the academic survey research field.

Table 5

Challenges Faced by Survey Organizations

HOW DO ACADEMIC SURVEY ORGANIZATIONS VIEW POLL AGGREGATORS?

In response to our question about challenges facing ASOs, 10% of directors mentioned the challenge of communicating survey results to the public. Others mentioned the growing skepticism the public has of survey research. Although no one mentioned polling aggregators such as Fivethirtyeight or Huffington Post, specifically, as challenges facing ASOs, we would argue that the popularity of these websites is related to the challenges of communicating results to an ever skeptical public. Blumenthal (Reference Blumenthal2014) noted a USA Today headline that read: “Election aftermath: How’d pollsters like Nate Silver do?” The headline conflates Silver, an aggregator of polls, as a pollster. This highlights that not only does the public not always understand what survey organizations do, but journalists might also have misperceptions of survey research. Blumenthal also hints that individuals in the survey research profession might begin to resent poll aggregation because aggregators receive more attention from journalists than do the survey organization that actually collected the polling data. As one member of the profession suggested, “it is much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others’ polls (p. 297).”

We asked directors a series of questions about poll aggregators and found a diverse set of opinions (see table 6). A majority (55%) of directors agreed the poll aggregators increased interest in survey research among the public and the media, an opinion suggesting a win-win for both aggregators and ASOs. However, some 49% also believed that aggregators led to an increase emphasis on “horse race” questions, which are not always seen by political scientists as of great interest to serious scholars. Many directors, 43%, agreed that such websites shifted attention from individual surveys to the aggregator (thus why journalists might consider Nate Silver a pollster). Some (34%) also agreed poll aggregators helped to give low quality surveys legitimacy.

Table 6

ASO Leaders Opinion of Poll Aggregators

Blumenthal (Reference Blumenthal2014, 298) seems to acknowledge this as well and observes that most of the “fuel for polling aggregation…..comes from inexpensive automated polls whose producers face a different set of costs and incentives than traditional pollsters.” However, these poll aggregators do score or weight the surveys they use in their aggregations. Therefore, such websites may actually help identify ASOs that conduct rigorous survey research. Nearly a quarter of directors seem to agree with this logic and believe poll aggregators enhance the perceived value of high quality surveys. Furthermore, the success of poll aggregators in predicting elections may also increase the confidence the public has in surveys (23% of directors agreed with this statement while 22% disagreed and 55% neither agreed nor disagreed).

SUMMARY

ASOs are more akin to research centers than academic departments. As such, they are relatively new organizations in the scope of the long history of universities (Stahler and Tash Reference Stahler and Tash1994). Converse (Reference Converse1987) estimated only about eight academic survey organizations existed in the United States in 1960 and 20 by 1970. By 1987 Sudman and Bradburn (Reference Sudman and Bradburn1987) found approximately 50 ASOs. And today our study has generated a list of 136 academic survey organizations. Guterbock (Reference Guterbock2010) argues that in no other country are academic survey centers as numerous as in the United States.

Yet, these survey organizations face numerous challenges. This study highlights several of these challenges including declining response rates, rising costs to obtain representative samples, an uncertain fiscal environment, a changing technological landscape, and overlapping and competing missions and goals. But ample reasons remain for optimism about the future of ASOs. The demand for high-quality survey data is increasing alongside the complexity of the methodologies used to collect such data. Academic survey organizations are clearly well-positioned to meet this demand. Moore (Reference Moore2014) argues that these organizations are such valuable research partners because they combine expertise in survey methodology with infrastructural capacity. Although ASOs are typically small compared to other research centers at universities, their small size provides a certain nimbleness allowing flexibility, innovation, and experimentation (Guterbock Reference Guterbock2010). Given the rapidly changing environment that public opinion scholars are facing today, the nimbleness and flexibility of ASOs are exactly what is needed in the fields of survey research and political science.

SUPPLEMENTARY MATERIAL

To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S104909651600144X.Footnote *