Research Background

This article is based in part on a web survey of 286 organizations from 50 countries that attended at least one Conference of the Parties of the United Nations Framework Convention on Climate Change (UNFCCC) between 1997 and 2001. The entire population of organizations was surveyed and the final response rate was 52.4%. A list of the accredited organizations and the names of their representatives sent to each of the negotiations is available on the Internet at the UNFCCC web site (www.unfccc.int). Email addresses for representatives who had attended the UN negotiations were found by searching the web sites of the organizations or by calling the organizations directly.

The survey was posted on the Internet using Hosted Survey, a company that provides web-hosted software applications and provides confidential services to protect the integrity of the study. Once the questions were written and pre-tested in traditional pencil-paper format, actually publishing the survey on the Internet took approximately three hours.

Using Hosted Survey, a personalized email invitation was sent to each of the recipients with an embedded link that took the respondent directly to the survey, which opened in a new window. The invitation outlined the purpose of the study, indicated the institutional affiliation of the principal investigator, and provided contact information in case of questions. Each recipient received a unique password that was part of the link, which was used for security purposes and to track respondents. The first page of the web survey was the Human Subjects Information sheet. The survey was three pages long and was comprised of 29 questions including a mix of multiple-choice, open-ended, likert-scale, and matrix-style questions. The survey questions addressed a range of topics including: views on climate change and the Kyoto Protocol, organizational structure, and activities at the United Nations Conference of Parties. The survey took approximately 12–15 minutes to complete, comparable to the pencil-paper version.

After two and a half weeks a reminder email was sent only to those who had not yet taken the survey, again including the embedded link. Finally, a hard copy of the survey was sent to 100 organizations that had not responded to the web survey; seven of these hard copy surveys were returned for a final response rate of 52.4% (150 out of 286).

Developing a Sampling Frame

While sampling bias is still a significant problem for general population web surveys, these concerns do not apply to all research populations. For populations that are highly connected to the Internet such as interest groups, government agency staff members, university students, faculty, teenagers, activist organizations, and elected officials, web surveys can be an efficient way to collect information.

Numerous studies have used non-probability sampling to recruit survey respondents through listserves, discussion forums, and newsgroups (Anderson and Ganseneder 1995; Schmidt 1997; Kaye and Johnson 1999). Search engines such as Google can be used to direct interested respondents to the survey based on the topic (Schmidt 1997). Other options for publicizing an open survey include submitting the web site to search engines (using a tool such as http://www.submitit.com or http://www.addme.com) or advertising in a banner or pop-up advertisement on web pages that might be of interest to your population.

Recruiting subjects in these ways requires careful planning. Contacting respondents via newsgroups requires careful observation of “netiquette.” In particular the survey must be relevant to the newsgroup topic and its intentions must be clearly spelled out. Newsgroup participants can be quite hostile if they believe that the topic is not relevant to the discussion (Schmidt 1997; Kaye and Johnson 1999). Respondents can be screened through an initial page of questions before they can access the questionnaire. It is important to remember that anyone may access this type of survey; any restrictions on participation (such as gender or residency) should be clearly indicated.

While increasingly common, recruiting volunteers from online newsgroups, message boards, and listserves is a questionable practice as it introduces sampling error. Because undefined volunteer samples are non-representative and are not randomly selected, the results can not be generalized beyond these respondents. It is also not possible to calculate sampling error to estimate the precision of the sample. While nonprobability sampling can be used in cases where the population is unknown or not easily identified (i.e., studies of deviance), in other cases it should be used with extreme caution if at all (Singleton and Straits 1999).

Survey Design

It is no longer necessary to know how to write programming code or learn HTML in order to design a web survey; in many cases it is as simple as filling out an online form. Despite the simplicity, web surveys require special design consideration compared to paper surveys because of the introduction of visual and interactive features.

In a one-page web survey, respondents can look over the entire survey before answering questions, while in an interactive survey (multiple pages or even one question per page) respondents only see a limited number of questions at a time. Combining questions on a single screen may reduce the time it takes to complete the survey, as it limits the number of times that respondents must orient themselves to the question answer format. Respondents answer more quickly once they become familiar with the answer categories (i.e., high, medium, low) (Couper et al. 2001). As well, the more screens there are in a survey, the longer the download time for the respondent, which may encourage abandonment of the survey. Dillman (2000) argues in favor of the “one screen survey” for these reasons.

If a multi-page format is chosen, respondents may not know how much further they have to go, and this may result in a number of incomplete surveys. One solution is to have a “progress bar” on the page that indicates what percentage of the survey has been completed. Another option is to indicate page numbers (e.g., 1 of 3) in a corner of the screen. Couper et al. (2001) tested the hypothesis that providing a progress indicator would reduce abandonment, however the difference was not statistically significant. Regardless of abandonment concerns, providing some kind of progress indicator is a courtesy to busy respondents.

Public email accounts such as Hotmail, Yahoo, and AOL now have sophisticated filters set up to try to eliminate or reduce spam, and as a result a survey may end up in a junk mail folder or blocked from delivery. One way to work around the spam issue is to send out an invitation by post with the web site. Alternatively, a letter or postcard could be sent to raise awareness about a forthcoming email invitation. The drawbacks of this are that it requires additional money and the researcher must have both a postal and email address for each respondent.

The introductory email message must be clearly written and should include at minimum the following information: a professional subject heading, institutional affiliation, study purpose, the purpose of the survey, estimated time to completion, and contact information. Subject headings that include words that are not recognizable by a standard dictionary may bounce back due to junk mail filters. My pre-test had the subject heading “UNFCCC survey” which was rejected by 20% of the email servers; however, “United Nations Climate Change Survey” was accepted by all servers. In the electronic community, using all capital letters is equivalent to shouting and should be used sparingly both in the subject heading and in the message text. Capital letters are also difficult to read on many computer screens.

While web surveys have the advantage of being simple to design with a broad array of graphic elements, visual features can also detract from the survey. Fonts can be difficult to read, colors may contrast making it difficult to focus on the text, extensive animation may distract from the questions, and pages can easily become cluttered. It is important to design a web survey with “professional good taste,” meaning limited gimmicks and an emphasis on clarity.

There are a few simple design guidelines that should be taken into account. Each page should have the same consistent design in terms of fonts, colors, and logos. The only exception could be the introductory page. White or very light backgrounds are best for readability, although white is preferred, and the text color should contrast well with the background. This is particularly important if you may have visually impaired subjects. The most readable fonts on a screen are Verdana or Arial. While Times New Roman is very readable on paper, it is difficult to read on the screen. Other good web fonts are Georgia and Trebuchet. More than three fonts on a page can lead to an untidy appearance, and it is wise limit the use of unusual fonts such as scripts as they can be distracting and hard to read. Different font sizes can be used with great effect to provide visual breaks or highlight information. Italics should be avoided, as they can be difficult to read on the screen (Couper 2002; Duyne et al. 2002).

Survey Implementation

When asked why they preferred web surveys, respondents to the International Climate Change survey provided the following responses: “quick and easy,” “can't get lost on my desk,” “no pressure to answer certain questions,” “can simply delete if not interested,” “they are usually well-organized,” “easy to answer,” “limits paper waste,” “no postage,” “less intrusive,” and “easy to submit.” Approximately half (48%) of the respondents to this open-ended question indicated that they preferred Internet surveys because they were faster to respond to than other modes.

The list of companies providing web surveys is growing rapidly, with many universities now offering these services.²

• Cost—Is cost based on the number of invitations, response rates, length of the survey, or the amount of time the survey is active? If the service is free, does this include email invitations, unlimited survey questions, technical assistance, uploading to a statistical program such as SPSS, conversion of responses to data entry codes, or special design features?
• Security—What are the security provisions? How often is the data backed up? How often is the survey unavailable due to maintenance?
• Contact—Whose return address is on the email? In most cases having a university affiliated return address will add legitimacy to the invitation.
• Design—What design features are available? These may include fonts, colors, graphics, skip logic, logos, progress indicators, animation, audio, or video.
• Flexibility—Can you make changes to the survey once it is activated? Can you add additional subjects at a later date?
• Data—How is the data accessed? How is it retrieved?

Once a host has been chosen, a series of implementation decisions must be made. While pre-testing is as important for web surveys as it is for mail back surveys, it is also important to test the visual and interactive elements of the survey using a variety of computer screens, browsers, and Internet connections, as well as through both PC and Macintosh systems.

One unexpected effect of Internet surveys is a higher degree of connection between the researcher and the respondent. I received numerous personalized replies both positive and negative from people who had received the survey. Fifteen asked to be sent a hard copy of the results, others commented on the research topic, and one person threatened legal action for sending out spam. Another respondent responded by email to say that they would not participate in a survey sent by an American institution because the U.S. had withdrawn its support from the Kyoto Protocol, and that if they were contacted again they would pursue legal action. Apparently this degree of interaction is not uncommon. Goodson et al. (1999) referred to “office rage” in their survey of APSA members' views on political science publishers, commenting on the hostile responses they received from 0.5% of their respondents “who apparently would like to kick us out of the discipline for using their e-mail addresses to conduct this survey.”

The Internet provides a high degree of anonymity and while this may be an asset to survey research, it also raises concerns about data validity. It is important to review the data, particularly if obtained though a public survey; in order to check for “spoof responses” that do not take the survey seriously or attempt to compromise the research. One Internet survey on drug dealers (the respondents were found using Usenet groups on drugs) had five spoof responses (Coomber 1997).

Like all Human Investigations Committee approved surveys, it is important to include consent or information sheet and contact information. For most surveys this can be the introductory page of the survey. Respondents simply click on a button at the bottom of the page that says something to the effect of “I Accept” or “I Agree” to the terms outlined in this release, which then takes them to the first page of questions.

Survey companies may automatically send a thank you email when a respondent completes the survey. For those who are interested in the results of the survey, a summary can be posted on another web page, and a link to that web page can be included in the thank you message. After submitting the survey the respondent should be directed to an acknowledgement page that confirms the receipt of their survey and thanks them for their participation. This will help prevent duplicate submissions from the same respondent.

One of the recent developments in web survey technology is that the responses can be instantly downloaded into a statistical analysis program such as SPSS, without having to do any data entry or coding by hand. During the design phase, researchers have the option of choosing whether they want actual responses or codes (e.g., yes = 1, no = 2) as the final product. This reduces the chances of data entry or coding errors, as well as saves time. The responses can be checked at any time, thereby enabling the researcher to make changes to the survey if necessary partway through.

Improving Response Rates

A major concern with all surveys is response rates. The International Climate Change Survey asked respondents: “What would stop you from completing a web survey.” This question yielded varied answers: “too long,” “boring questions,” “biased questions,” “too much other work to do,” “topic not of interest,” “unclear purpose,” “too many questions,” “questions too personal,” “slow connection,” “unknown or suspicious entity conducting the survey as I only answer university surveys,” “ease of form filling,” “if there's no room to make comments,” “anonymity,” “security,” “computer system crashes,” and “technical problems.”

A recent University of Michigan study (2001) of student drug and alcohol use tested response rates in web surveys compared to mail surveys using two randomly selected samples of 3,500 undergraduate students each. The mail survey had an overall response rate of only 40%, compared to 63% for the web survey (Crawford et al. 2002). Cook et al. (2000), in their overview of 68 web and email surveys, found that response rates were most influenced by the number of contacts, personalized contacts, and pre-contacts. Given the ubiquity of spam and the ease of deleting email messages, pre-contact may be particularly important for web-based surveys that involve email invitations. Pre-contact by email should clarify the institutional affiliation, survey purpose, estimated time to complete, and the date that the survey will begin. All email contact with subjects should come from an email address that is clearly a professional organization such as a university (i.e., survey@university.edu).

The International Climate Change survey had a few significant response rate issues: language barriers, differing degrees of interest in the issue, difficulty in locating email addresses, organizations that had ceased to exist, and sporadic Internet access in developing countries. However, as the follow-up mail back version of the survey indicated (seven out of 100 returned), the 52.4% response rate could have been significantly lower if only a mail back survey had been used.

It has been shown that response rates can be improved by offering potential respondents incentives such as including a dollar, or even “a penny for your thoughts” (Babbie 2001, 256) with the survey, however this is difficult to do for an email invitation web survey. One option is to provide a list of charities for the respondents to choose from and the researcher would donate a specified dollar amount to that charity. Another option is to provide a code for a gift certificate at an online retailer with broad appeal. Obviously this becomes more complicated with an international survey given the increase in shipping costs for international orders. Increasing the value to cover international shipping is one option, or providing a choice of retailers such as Amazon.com (United States), Amazon.co.uk (United Kingdom), Amazon.ca (Canada), and Amazon.co.jp (Japan). Alternatively, respondents could be entered into a drawing for an electronic gift certificate or a cash prize.

Knapp and Heidingsfelder (2001) looked at nine unrestricted self-selected surveys and found that higher drop out rates were due to longer surveys, sensitive topics, and a lack of incentives. Numerous studies have illustrated the importance of multiple contacts to improve response rates (Scott 1961; Goyder 1985) (Dillman 2000; Babbie 2001), and this holds true for web surveys as well.

Dillman (1978) emphasized the importance of personalization for mail surveys, and this is probably even more important for web surveys given the high incidence of spam email and the ease with which respondents can delete the message. This is also why it is important to address each email personally to the respondent, rather than sending out a mass mailing to a distribution list. Most survey companies do this automatically from spreadsheet information that you upload, or you can send each email by hand. While doing it by hand is somewhat time consuming, the effort and cost is far less than collating, folding, and stuffing for a mailed survey.