RESEARCH DESIGN

We set out to test the effects of systematically incorporating news magazines and discussions about politics in the school-based curriculum on students' levels of political knowledge, internal efficacy, and media use. The first step in our research project involved working with New Jersey's Department of Education to compile a list of potential research sites within about a 40-mile radius of our campus. In compiling this list, we included schools in communities with similar socioeconomic backgrounds, yet also with large enough enrollments to support treatment and control groups at the schools. We finished with a list of 34 high schools, and from that list we worked to recruit schools to participate in the research. We initially contacted schools in the fall of 2006 with an eye toward conducting the research in winter and spring 2007. The response to our initial recruitment was disappointing, however, and we retooled the research design and added incentives for participation. (A point we will return to in lesson #8 on offering adequate incentives to motivate students and parents to participate.) We tried again in fall 2007 to recruit schools for winter and spring 2008, successfully generated more interest, and selected four schools as research sites for the project.

Twenty-seven social studies classes across the four schools participated. We randomly assigned classes to one of three groups: a treatment group where students read two or three articles about politics in Time magazine each week for eight weeks, then discussed the articles at home with their parents and the next day in class; a treatment group where students read the same articles but discussed them only in class; and a control group that did not read the articles nor engage in discussions about politics. All participating students completed a questionnaire that measured levels of media use, political knowledge, and internal political efficacy before the eight-week intervention; a second questionnaire with identical measures of media use, knowledge, and efficacy at the end of the eight weeks; and a third questionnaire containing the same measures six weeks after the intervention ended. Parents of some of the students also completed a telephone survey that had the same measures of media use, political knowledge, and political efficacy.

Each week we sent by overnight mail copies of Time magazine for the treatment groups, as well as a discussion guide for the teachers that identified the articles that the students should read and the discussion questions that they should consider for their conversations at home and in the classroom. We also visited all of the participating classes three times to administer the surveys, and, after the experiment ended, we returned a fourth time to distribute incentives, debrief the students, and present preliminary results.

The project yielded promising findings. For example, we found that the combination of reading news articles and discussing them at home is related to increases in internal political efficacy, information-seeking, and political knowledge, but student aptitude and parental characteristics also come into play. Another key finding, and one that we did not fully appreciate when we designed the project, was that conducting research in high schools presents unique methodological and logistical obstacles.

We entered this project enthusiastic about the work but admittedly naïve about the challenges. In the interest of helping other scholars to learn from our experience, we offer advice for conducting political science research in high schools. The suggestions appear in approximate chronological order so that researchers can consider this a step-by-step, although not necessarily comprehensive, guide.

1. PLAN YOUR PROJECT WITH AN EYE ON THE HIGH SCHOOL CALENDAR

Research projects involving high school students require approval at multiple levels in public schools, from the superintendent (and in some cases even school boards) to high school principals to curriculum coordinators and, finally, the teachers. Authors of studies of the effects of health education programs in middle and high schools recommend beginning the process of recruiting schools or school districts six months to a year before data collection (Aarons et al. Reference Aarons, Rose, Walker, Lyles, Jenkins and Raine2001; Piper, King, and Moberg Reference Piper, King and Moberg1993).

The timing of the initial contact is also important. Harrington et al. (Reference Harrington, Binkley, Reynolds, Duvall, Copeland, Franklin and Raczynski1997) found that school principals were most receptive to proposals during the summer months when schools were not in session. Launching a research project that requires parental participation and consent also tends to be smoother at the start of the academic year, when materials about research can go out with student orientation packets (Johnson et al. Reference Johnson, Bryant, Rockwell, Moore, Straub, Cummings and Wilson1999).

We found that few school administrators in our area were readily available during the summer months, however, and that it made sense to have recruiting materials ready to hit their mailboxes in late August. Researchers should be prepared, at minimum, to spend most of the fall following up, answering questions, and meeting with teachers and administrators to convince them to grant you access to their students.

You could try beginning even earlier in the spring before schools adjourn for the summer, but we found that teaching assignments and enrollments can be in flux during the summer, and administrators prefer to wait until the fall to make commitments involving research projects. Also, depending on the state in which you are conducting the research, late spring may be the season for state-mandated standardized achievement tests, as well as Advanced Placement tests. Add to that the usual crush to cover a body of material by the end of the year, and you may find that high school teachers and administrators are not eager to discuss new research initiatives during this busy time, even if the project is sufficiently further down the road.

2. FIND INTERMEDIARIES WHO WILL ADVOCATE FOR YOU DURING THE RECRUITMENT PROCESS

Elementary, middle, and high schools are popular places for conducting research, particularly when it comes to testing new curricula about healthy living; diet; tobacco, alcohol and drug use; and sexuality (see, for example, Aarons et al. Reference Aarons, Rose, Walker, Lyles, Jenkins and Raine2001; Blinn-Pike, Berger, and Rea-Holloway Reference Blinn-Pike, Berger and Rea-Holloway2000; Goodman et al. Reference Goodman, Smith, Dawson and Steckler1991; Harrington et al. Reference Harrington, Binkley, Reynolds, Duvall, Copeland, Franklin and Raczynski1997; Johnson et al. Reference Johnson, Bryant, Rockwell, Moore, Straub, Cummings and Wilson1999; L'Engle, Pardun, and Brown Reference L'Engle, Pardun and Brown2004; Markham et al. Reference Markham, Tortolero, Escobar-Chaves, Parcel, Harrist and Addy2003; O'Donnell et al. Reference O'Donnell, Duran, San Doval, Breslin, Juhn and Stueve1997; O'Donnell et al. Reference O'Donnell, Stueve, Agronick, Wilson-Simmons, Duran and Jeanbaptiste2005; Piper, King, and Moberg Reference Piper, King and Moberg1993; Severson and Biglan Reference Severson and Biglan1989). Research fatigue is possible. Like many large organizations, schools also can be insular, and gaining access to research participants may be difficult for outsiders. Researchers suggest that finding a credible and familiar intermediary to introduce you and your project to administrators, teachers, and parents can be effective in recruiting schools to participate in the project. Harrington et al. (Reference Harrington, Binkley, Reynolds, Duvall, Copeland, Franklin and Raczynski1997) recruited school nutritionists to help advocate on their behalf when they sought to present and test the effects of a healthy eating curriculum. Goodman et al. (Reference Goodman, Smith, Dawson and Steckler1991) relied on school health coordinators and health educators at local health departments to convince school administrators to support a tobacco prevention curriculum in public schools in North Carolina. Goodman et al., referring to this practice as “priming the pump,” also solicited support from the state department of education, which mailed a letter to school administrators on behalf of their project that was timed to arrive while local intermediaries were making contact (Reference Goodman, Smith, Dawson and Steckler1991, 376).

We followed a similar approach, first contacting the social studies coordinator for secondary education at the New Jersey Department of Education, who endorsed the project and spoke about it informally at gatherings of social studies coordinators from high schools around the state. Our initial letter of recruitment to superintendents, principals, and social studies coordinators at the high schools also included a reference to the state coordinator's endorsement of the research. As we got further into the recruitment process, the high school social studies coordinators acted as advocates with their principals, teachers, and, in one school district, the board of education, which had to approve the research.

3. INVOLVE TEACHERS EARLY, AND MINIMIZE THE PROJECT'S DEMANDS ON THEM WHEREVER POSSIBLE

Piper and colleagues (Reference Piper, King and Moberg1993), who conducted research on a healthy lifestyles curriculum in middle schools, recommend that researchers involve teachers as early as the recruitment meetings with the administrators, or at minimum, make a presentation at a faculty meeting or in-service gathering. “If teachers do not ‘buy in’ during the recruitment phase, problems will surface during the implementation phase—often very serious problems that jeopardize the continuation of the program at a particular site,” the authors point out (Reference Piper, King and Moberg1993, 178).

Although we were able to meet with teachers before the research began, we still noticed an enthusiasm gap between administrators and teachers at some of our research sites. As we conducted the research, we quickly saw why. High school teachers are often juggling five or six classes during a school day with only one or two periods free for planning, grading, and communicating with parents, colleagues, and administrators. Administrators may see participating in research as an excellent active learning opportunity for students, but teachers may see it as one more obligation in an already overcrowded workday.

The teachers with whom we worked were unfailingly gracious and professional, but we learned that the success of our project relied, in part, on ensuring that the research did not impose unnecessary burdens on them. School administrators helped in persuading teachers of the value of handing over part of their instructional time for the project. But it also was useful to take some time to get to know the teachers and answer their questions. Each survey would take a full day to administer across the participating classes, allowing us an opportunity to spend time with teachers at lunch or during planning periods. Thus, we were able to build good working relationships as the project advanced, which was important during the data collection phase. A handful of students were absent on the days when we administered the questionnaires, and we had to leave questionnaires for those students to complete with the teachers. We either returned promptly to collect the straggling questionnaires, or left behind stamped, self-addressed envelopes to get the surveys returned through the mail.

The overriding lesson here is that time is precious in high school classrooms, and you have to be well prepared so that you take only the time that is absolutely necessary for your research. You also need to plan to spend time building working relationships with the teachers. Doing this will go a long way toward making data collection go smoothly.

4. ALLOW ADEQUATE TIME FOR REVIEW OF YOUR PROJECT BY THE INSTITUTIONAL REVIEW BOARD (IRB) AT YOUR COLLEGE OR UNIVERSITY

Unless you are planning to study only high school seniors who have turned 18, most of your participant pool will consist of minors. This adds layers of consideration and scrutiny to your IRB submission. The rules surrounding informed consent vary somewhat for minors compared to adults. Also, if your research involves an experiment requiring any form of deception, the justification for the deception and the proposed debriefing of participants after the experiment need to be carefully explained in your IRB submission.

Our project qualified for an expedited review from our university's IRB, which meant that two board members and administrative staff could handle the application as opposed to submitting it to the full board for its monthly meeting. The review process, however, was still lengthy and detailed and consisted of extensive dialogue and multiple revisions over several months.

5. KNOW AND BE PREPARED TO SATISFY THE REQUIREMENTS FOR INFORMED CONSENT FOR MINORS PARTICIPATING IN RESEARCH

Under federal law, researchers typically must obtain informed consent from participants before the research can occur (Code of Federal Regulations, Title 45, Public Welfare, Part 46, Protection of Human Subjects). When the participants are minors, consent must be obtained from parents or guardians. The complexity of this process depends on the nature of the research.

The acquisition of informed consent can take one of two forms, commonly referred to as either active or passive consent. An active consent procedure requires that, on receipt of a letter (detailing the research process, the risks and benefits, issues of confidentiality, etc.), the potential research participants (or their parents) indicate with their signatures whether they consent to participate. Each individual's status as a research participant is determined by the receipt of that written consent form. In contrast, under a passive consent procedure, potential participants and their parents receive a letter with the aforementioned details and parents are asked to contact the researchers if they do not consent for their children to participate. As long as the researchers do not receive written documentation otherwise, individuals in the population under study can be considered research participants.Footnote ⁷

Researchers prefer to use passive consent when recruiting minor participants because response rates tend to be higher than when active consent is required (Severson and Biglan Reference Severson and Biglan1989). Lower response rates result in smaller samples of students and reduce the statistical power of data in subsequent analyses. Also, lower response rates can translate into biased samples, particularly when researchers are measuring sensitive topics, such as attitudes toward alcohol and drug use. Kearney et al. (Reference Kearney, Hopkins, Mauss and Weisheit1983) found that requiring active parental consent for elementary, middle, and high school students to complete questionnaires related to a drug and alcohol prevention curriculum resulted in an overrepresentation of white students and underrepresentation of African American and Asian American students. No bias existed regarding student gender, and evidence of bias in terms of academic measures was mixed.

Because our project posed minimal risk to participants, we received IRB approval to use passive consent. As a further safeguard, however, our IRB also required us to have students sign a statement of assent when we went into the classroom to administer the first survey. If students declined, they in effect opted out of the research on their own. Five students, out of a total pool of 633 eligible students at the four schools, opted out on their own.

Clearly, passive consent is the preferred option when obtaining consent for large numbers of research participants. This option may not be available to you, however, depending on the nature of your research. In addition to federal law, your project also might come under the purview of state law(s) regarding informed consent and research involving minors. Design your project with those laws in mind. The IRB at our university made us aware of a state law that governed the use of surveys in public schools. The statute stated that we would have to acquire active consent from parents if we asked students to list their political affiliations on our questionnaires (New Jersey Statutes Annotated 18A:36–34, 2002). The law did not define political affiliation as applied to minors, but in any event the information was not relevant to our research. We could forego that data if it meant being able to use passive consent as opposed to active consent. Yet it was important to know the legal terrain before we proceeded with the study. Researchers ought to consult with their IRB and their state department of education about the consent requirements for studying minors in school settings before completing their research design.

Two other helpful resources are the Council of American Survey Research Organizations (CASRO) (http://www.casro.org/index.cfm) and the National Conference of State Legislatures (NCSL) (http://www.ncsl.org). CASRO tracks changes in state laws pertaining to survey research and related issues, such as informed consent, and NCSL has a searchable database of pending and recent changes in state laws broken out by policy areas.

6. EVEN AFTER A THOROUGH IRB REVIEW, PARTICIPATING SCHOOLS HAVE THE FINAL SAY

Even if the relevant federal and state statutes allow passive consent, the school district(s) that you are working with might still require active consent as a matter of policy. This was the case with two of the four school districts that participated in our project, and it reduced our yield of participants from those schools. Although this is not necessarily fatal to a research project, it is an added consideration that researchers need to keep in mind when executing the project and analyzing data.

In addition to the letter of explanation and consent form approved by our IRB, two schools attached their own letter to parents indicating that their school required written parental permission before engaging in the research process. To some extent, this was understandable given issues of privacy and litigation that schools face today. Nevertheless, it made an already cumbersome process even more complicated and, we suspect, was a major cause of lower response rates at those schools.

At the two schools that required active parental consent, 33% and 45% of students eligible for the research participated. At the two schools with passive parental consent, 87% and 93% of eligible students participated. No parents contacted us to have their children excluded from the study in the schools that offered passive consent. Student participation at the two schools with passive parental consent was less than 100% primarily due to students being absent on the days when we conducted our three surveys (one preintervention and two postintervention). We were able to reach most, but not all, of those students by leaving behind surveys for the students to complete when they returned to class.

It is difficult to fully assess whether the schools that required active consent ended up with biased samples of students because we do not have information about the students whose parents did not provide active consent. Based on the students who did receive permission to participate, however, the pools of participants from the passive and active consent schools were not significantly different in terms of the distribution of the race and ethnicity of the students. There was a statistically significant gender difference, however, in that a larger percentage of students who participated from the active consent schools were female (54% compared to 42% female in the passive consent schools, chi-square = 4.26 with 1 degree of freedom, p = 0.04). Because this was an experiment, as opposed to a survey with results extrapolated to a larger population, we relied on random assignment to treatment and control groups at the teacher level to control for differences between classes and schools. To be safe, however, we also controlled for gender along with other variables in our analyses.

7. US MAIL IS MORE RELIABLE THAN “BACKPACK MAIL,” BUT EVEN THEN …

Reliable communication with parents may be among the biggest challenges when conducting research involving minors. In developing a budget for your project, you might be tempted to save money by asking schools to send letters and reminders home with the students, as opposed to mailing the information directly to the parents. “Backpack mail” and conventional mail however, each have their drawbacks, and researchers should consider those challenges when planning their projects (see Johnson et al. Reference Johnson, Bryant, Rockwell, Moore, Straub, Cummings and Wilson1999 and L'Engle et al. Reference L'Engle, Pardun and Brown2004 for discussions of the most effective way to get information home to parents of research participants).

As any parent will tell you, students' backpacks can have a “Bermuda Triangle” quality that results in important information disappearing long before students get home. The logical alternative is to mail the information to parents. Schools, however, may worry about privacy and may be hesitant to provide a mailing list to researchers. One compromise that we reached was to provide the material to the school and have the school mail it to parents with reimbursement to the school for copying, postage, and labor. This approach, while logical, also can have unintended consequences. One of our participating schools, as a cost-savings rule, always used double-sided photocopying for correspondence. The school photocopied two pieces of mail—our one-page cover letter and one-page consent form—onto two sides of one sheet of paper. The cover letter had our contact information, which parents needed to retain as a condition of our IRB approval. Yet the cover letters inadvertently came back to us when parents signed and returned the consent form that was on the back. We had to resend the contact information to each parent to comply with our approved IRB protocol. The devil is in the details, and in this case clear communication between the researchers and the school could have saved us all some headaches.

8. OFFER ADEQUATE INCENTIVES TO BOOST PARTICIPATION

With IRB approval and grant money in hand, and a group of schools willing to work with us, we proceeded to recruit parents and students to participate in our experiment. Although we had carefully reviewed the political science literature concerning our areas of substantive interest—media use, political knowledge, and political efficacy among teens and their parents—we did not come across much information in the political science literature regarding the use of incentives to convince high school students to participate in an experiment. As a result, we incorporated only a school-level incentive and learned perhaps the most important lesson of all. Although your project may be fascinating and compelling to you and your colleagues, the public may view participating in research as an obligation with little payoff. Incentives are crucial to attracting enough participants to give your experimental data statistical power.

In the first incarnation of our research project, we offered each participating school a year-long subscription to Time magazine (the news magazine we used for our project) as a token of our appreciation and as an inducement to participate. We offered no incentives to teachers, parents, or students. We also opted for active consent forms for all of the participating schools.

After a few weeks of distributing letters explaining the project and consent forms to students and parents, we had a return rate of completed consent forms ranging from 12% to 38% per school. The low response rates came despite sending consent forms home with students and via the US mail, along with reminders. Because some of the responses consisted of parents saying that they and their children would not participate, the actual number of participants in the research was probably lower.

We initially conducted our recruiting during the fall of 2006, with the target of running our experiment beginning shortly after students returned from the winter break in January 2007. We planned to devote most of the spring term to completing the experiment and debriefing the participants. Given the paltry response rates, however, we decided to request a no-cost extension on our grant and return to the drawing board with the aim of redesigning the experiment and conducting it in the following academic year.

In the process of retooling, we looked beyond the political science literature and focused on health education, in which school-based studies are more common. The health education literature demonstrated that incentives are crucial to persuading students, parents, and teachers to return consent forms and to participate in surveys (Blinn-Pike, Berger, and Rea-Holloway Reference Blinn-Pike, Berger and Rea-Holloway2000; Markham et al. Reference Markham, Tortolero, Escobar-Chaves, Parcel, Harrist and Addy2003; O'Donnell et al. Reference O'Donnell, Duran, San Doval, Breslin, Juhn and Stueve1997; O'Donnell et al. Reference O'Donnell, Stueve, Agronick, Wilson-Simmons, Duran and Jeanbaptiste2005).

The research presented in this literature offered examples of the use of modest incentives in the recruitment process. In their health survey of urban youths, O'Donnell et al. (Reference O'Donnell, Duran, San Doval, Breslin, Juhn and Stueve1997), for instance, provided $25 gift certificates to teachers who collected completed consent forms from parents of at least 90% of students in their classes. Each school that collected completed consent forms from parents of at least 90% of students also received a $250 incentive from the researchers (O'Donnell et al. Reference O'Donnell, Duran, San Doval, Breslin, Juhn and Stueve1997).

In encouraging parental and student participation in survey research regarding a program promoting abstinence, O'Donnell et al. (Reference O'Donnell, Stueve, Agronick, Wilson-Simmons, Duran and Jeanbaptiste2005) provided parents a $15 incentive for returning completed consent forms (either indicating consent or refusal) and $25 for completing the surveys. Students in the study received $5 for returning completed consent forms and $5 for each survey they completed. In their survey research regarding sexual risk-taking among urban youth, Markham et al. (Reference Markham, Tortolero, Escobar-Chaves, Parcel, Harrist and Addy2003) also reported providing students a $5 gift certificate for returning completed consent forms and another $5 gift certificate for completing a survey.

In sum, the health education literature offered examples of the use of incentives ranging from $5 for students to $15 to $25 for parents for return of consent forms and completion of surveys, as well as incentives for teachers and schools to collect completed consent forms from parents of at least 90% of students targeted for studies. In each instance, scholars stressed that it would have been virtually impossible to gather data without incentives. Even with incentives, the return rate for completed consent forms in the two studies reported by O'Donnell and colleagues was 87% to 88% (O'Donnell et al. Reference O'Donnell, Duran, San Doval, Breslin, Juhn and Stueve1997; O'Donnell et al. Reference O'Donnell, Stueve, Agronick, Wilson-Simmons, Duran and Jeanbaptiste2005).

In light of these findings, as well as our prior experience, it did not seem feasible to achieve acceptable rates of response and participation without offering some type of incentive. We received permission from the IRB and our funder to redesign the project while staying within our original budget so that we could provide a healthy package of gift card incentives for students, parents, and teachers and pizza parties for classes in which at least 80% of parental consent forms came back completed. The incentives are described in table 1.

Table 1

Incentives for Research Study

Although redesigning the project kept our experiment out of the field for one academic year, the delay paid off. The average response rate for all classes was 69%, and more than 80% of students in 11 of the 27 participating classes returned completed consent forms from their parents.Footnote ⁸

9. DO NOT UNDERESTIMATE THE TIME THAT YOU NEED TO SPEND AT YOUR RESEARCH SITES

As you budget time for the project, err on the side of having everything take longer than expected, beginning with the number of visits needed to recruit administrators and teachers to commit to the project. If you are collecting data on multiple dates, expect to make some visits in between to deal with administrative issues. Allow time for a thorough debriefing with students, teachers, and administrators at the end of the research because that is where the promised educational payoff occurs for your participants.

Maintaining a consistent presence at your research sites allows you to build rapport with teachers and administrators, and you can acquire a greater appreciation for the context in which your research is occurring. This may prompt you to tweak your research design early in the project, or it may help as you sort through your data and interpret your results at the end of the process.

As you plan a budget for your research, factor in mileage for all site visits. Each of our four research sites was an average of 26 miles from our home institution. We estimate that we put a total of 2,100 miles on our cars during the project.

10. EVEN WITH THE MOST CAREFUL PLANNING, THE UNEXPECTED WILL OCCUR

As with most primary research, you will encounter things you had not anticipated. Draw on your reserves of patience and diplomacy and the intellectual resources of your peers and your institution when you encounter surprises.

The surprises can be small, such as the school that balked at providing pizza as an incentive for students returning completed consent forms. There was concern that the incentive encouraged the use of junk food as a reward, running counter to the school district's increasing focus on students developing healthy eating habits. We left it to the discretion of the teachers as to what food to provide if pizza was unacceptable.

Or the surprises can be significant, such as the participating student who raised questions about our methodology and the wording of the questions on our first survey. The student communicated those concerns in an e-mail to the IRB at our university, setting off a series of conversations and meetings with the IRB staff about how to respond. Our proposed study had successfully undergone a thorough blind peer review when we sought and won the grant for our work, so we were confident that we had designed a scientifically sound project. Working with the IRB, we drafted a letter to the student in which we addressed each of the student's questions substantively and respectfully, and we conducted an individual debriefing with the student in addition to the debriefing with the student's class at the end of the project. We satisfied the student's concerns, but the episode was a potent reminder that study participants are human beings who may have legitimate questions about the research process, and those questions must be treated with care and respect during and after your study.

ACKNOWLEDGMENT

The authors wish to thank the Center for Information & Research on Civic Learning and Engagement at Tufts University, which provided financial support for this research.