IS THERE A NEED FOR REPORTING STANDARDS?

Some basic information is essential for evaluation and communication of a study’s research design and findings. Reporting guidelines can serve as a tool to assist scholars in their efforts to write papers that do not inadvertently leave out important details that would undermine the value of their studies’ contributions. In this spirit, the Standards Committee’s guidelines request that authors report a basic and common-sense list of items. However, if these guidelines are essentially a codification of existing practice, they would be innocuous but of little use. We investigated the possibility that reporting is already exemplary by surveying the reporting in a sample of articles that use experimental methods selected from the discipline’s leading journals. We find that reporting is generally good, but even in work at the apex of the academic pyramid there are significant reporting gaps. We did not investigate all journals or include unpublished manuscripts, but it is our impression that the level of reporting omissions in these other outlets is likely to be similar or worse.

To get a rough idea of the extent to which political science experimental research already follows the reporting standards we recommend, we had two coders independently evaluate a random sample of published research. To form the population from which we sampled, we began by first identifying every article that employed some sort of experimental manipulation over the past nine years (2005–2013) in 16 highly regarded general interest and field journals.Footnote ¹ Articles were included if they used any of the following words in their abstracts: “experiment,” “randomly assigned,” “treatment,” “control,” “treatment group,” or “control group.” Articles were then excluded if it was clear from the context of the abstract that the study was not referring to an experiment in the sense of a researcher-controlled manipulation.Footnote ² This search yielded 490 articles in total. Each article’s experiment(s) was then classified as field, survey, or laboratory based on its abstract (and text when necessary).Footnote ³ Field experiments were defined as the random assignment of subjects in a naturalistic setting (e.g., voters) to treatment and control conditions. These studies proved relatively easy to classify. The distinction between survey and lab experiments was more ambiguous, as a number of experiments that are conducted in a laboratory setting follow identical procedures to survey experiments with the exception of the location in which the survey was conducted (i.e., surveys conducted in a lab setting that might have without substantial alteration been administered face-to-face, over the phone, or via the Internet). Because of this similarity, we defined survey experiments broadly to include not only those administered via a traditional survey mode, but also any experiment carried out in a laboratory that could have been conducted in an identical fashion through one of the common survey formats.Footnote ⁴ As such, “laboratory experiments” were limited to those studies conducted in a laboratory environment that involved viewing video, any interaction of subjects with other participants, multiple waves (e.g., a pre-test survey two weeks before the experimental manipulation of interest), and/or a cognitive distractor task that could not have been completed through an alternative surveying method (e.g., math problems calculated by hand).

From the collection of the 490 experimental articles we identified, we drew a random sample of 60 for coding. We employed stratified sampling based on two factors in addition to laboratory, field, or survey. To see whether reporting standards improved as experiments became more common in the discipline, we classified articles as “early” (2005–2009) or “late” (2010–2013). A second distinction was “general interest” versus “field” journals, which permits the determination of whether reporting differs between what are sometimes considered to be the top three general interest journals (American Political Science Review, American Journal of Political Science, and Journal of Politics) and those that tend to be more specialized in nature (the other 13 journals examined). This results in 2 × 2 × 3 distinct cells and is well-suited for comparison across time period, journal type, and experiment type.Footnote ⁵

Starting with the full proposed reporting guidelines (see Appendix 1), we developed a coding scheme to measure the completeness of reporting. We boiled the guidelines down to 19 reporting items that we took as representative of the information that should be reported in any experimental research report. Table 1 presents a brief description of how these 19 items were translated into coding standards. The items range from things that seem quite important, such as a complete description of the treatments, to items that would be useful to know and are indications of the level of care in reporting exercised by the authors, such as reporting the response rates to any surveys the researchers administer. For each article, adherence to each of the 19 reporting items was coded on a three-point scale (0–2) that denoted the extent and quality of the reporting vis-à-vis the ability to adequately evaluate the research design and analyses. A score of “0” is awarded when no pertinent information is provided, a “1” corresponds to some information but important omissions, and a “2” signifies either trivial or no omissions. Given the meaning assigned to each value, the coding scheme is somewhat lenient in awarding a “1” (as opposed to a “0”) but rather stringent with the assignment of a “2.” Thus, a “1” corresponds to a wide range of missing information (in terms of severity) while “0” and “2” are more narrowly defined as completely missing and (almost) completely reported, respectively.Footnote ⁶

Table 1

Reporting Guidelines Used in Coding Analysis

Results

Table 2 shows the percentage of articles with missing information (i.e., those coded as a “0” or “1”) for the 19 reporting items, both for the overall sample (column 8) and broken down by various sampling strata. The results are based on coding the sample of 60 articles. Five articles were drawn at random from each of the 12 cells (experiment type × journal type × time period); two articles were assigned to coder 1, two to coder 2, and the fifth to both coders to provide an estimation of the consistency between their efforts. This design thus yields 72 observations on the reporting standards of 60 experimental articles (48 independently analyzed and 12 coded by both coders). In the results presented in Table 2, a jointly coded article is considered to have missing information only if both coders come to this conclusion.Footnote ⁷

Table 2

Percent of Articles Receiving Any Downgrade by Category

Note: Cell entries for each coding category are the percentage of articles that received a score of zero or one with frequencies presented below. Frequencies differ when categories were coded as not applicable for certain articles.

Although most articles in the discipline’s top general interest and field journals report most items, the columns reveal substantial missing information in reporting across all types of experiments, in both general and specialized journals, and in both the early and late periods. Before we discuss the prevalence of missing information, it is useful to understand what sort of events triggered such a coding. The final column of Table 2 describes, for each item, an example of incomplete reporting found in the sample. Omissions regarding the subject selection that triggered a coding less than “2” included leaving out an explanation of how subjects were recruited, omitting details about the subject characteristics (such as year or major for college students), or excluding details about the timing of subject recruitment and conduct of the experiment (such as what year or time of year the experiment took place). Omissions regarding subject characteristics included failure to report any sample means for some variables used in the analysis and failure to report baseline covariate levels or balance across experimental groups. Omissions regarding the treatments involved failure to provide scripts or images for all the treatments. Omissions regarding the outcome variable involved failure to report the exact question or response options used to code the outcome. Omissions regarding attrition included failure to report attrition by experimental group or to provide any explanation for why some subjects are missing from the statistical analysis calculating treatment effects.

Column 8 of the data in Table 2 lists the rates of missing information in reporting for the overall sample. There are several main findings. First, it appears that the use of random assignment and some description of the assignment process were almost universally reported. Second, authors frequently do not provide basic data tables, such as pretreatment means for important covariates or mean outcomes by experimental group following treatment. Third, although a basic description of the treatments is always provided, important details are frequently missing, and complete materials are not provided in the paper or a linked appendix a quarter of the time. Fourth, it is often difficult to learn the response rates for surveys (when used), the number of subjects initially assessed for eligibility, the number of subjects assigned to each experimental group, the number of subjects in each group that is missing outcome data, and the reasons why subjects that were assigned to treatment groups are missing from the statistical analysis.

Table 2 also presents the breakdown of reporting omissions by type of experiment, journal type, and time period. Although there is some interesting variation across type of experiment, we restrict our comments to journal type and year of publication. First, the reporting patterns for the general interest and the specialized journals (columns 4 and 5 of Table 2) are quite similar. A priori, it might have been expected that the general interest journals in our sample would adhere to higher reporting standards because of stricter standards imposed by peer review. Alternatively, we may have expected some of the specialized field journals to adhere to higher reporting standards because certain fields have a longer history of using experimental methods. Neither of these propositions, however, is borne out by the data. The largest difference in coding scores between general and specialized journals is 25 percentage points in the case of reporting the response rates from surveys, with an average difference of about 8.6 points across coding categories.Footnote ⁸ Higher rates of missing information occur on 10 standards in general journals and six standards in specialized journals (with rates being the same on the remaining two standards). As such, we cannot conclude that experimental studies published in either general or specialized journals adhere to stricter reporting standards.

Second, and somewhat surprisingly, reporting appears similar for the 2005–2009 and 2010–2013 periods (columns 6 and 7 of Table 2). For only four of the 19 standards does the difference in the percentage of articles with missing information exceed 10 points. The largest disparity between the early and late periods is about 22 percentage points for the two measures concerning the use of private survey firms (not surprising, given the much smaller sample sizes for which these categories were relevant), with an average difference of less than eight points across coding categories. Once again, we find that there is no general sense in which studies in the early or late time periods are adhering to more stringent reporting standards. In eight of the 19 coding categories, a greater percentage of studies from the late period have important omissions, while the same is true for eight of the 19 categories for studies from the early period (with scores in the remaining three categories tied).

Table 3 considers the sample by article rather than by reporting category. It appears that the omissions reported in Table 2 are not all confined to a minority of papers that report very little. Rather, the substantial majority of papers leave out useful information. Approximately one-third of all articles have three or more items coded 0, signifying complete or nearly complete omission of a reporting item. We see that there is very little difference across journal type (general versus field) or time period (early versus late), but substantial difference across experiment type (more omissions in survey and lab than field experiments). Overall, 88.3% of the articles had three or more items and 35% had six or more items with more than trivial reporting omissions (i.e., coded 0 or 1).

Table 3

Percent of Articles with Omissions

Note: Cell entries are percentage of articles with omissions, with frequencies presented below. Major omissions are standards coded as a “1,” while complete omissions are standards coded as a “0.” For jointly coded articles, the average score is rounded up if necessary (i.e., an article’s standard assigned a “1” by one coder and a “2” by another (for an average score of “1.5”) is treated as a “2” and thus not considered a substantial omission, while an article’s standard assigned a “0” by one coder and a “1” by another (for an average score of “0.5”) is treated as a “1” and thus a major but not complete omission). Only one article (a lab experiment in a general journal from the late time period) exhibited six or more complete omissions.

To summarize the findings, most items are reported most of the time. However, there are substantial omissions, and these omissions occur in all types of experimental research, in both general interest and field journals, and reporting quality does not appear to be improving over time.

THE COMMITTEE AND PROCESS

To assist authors and readers, the Standards Committee composed reporting guidelines. The process used to draft the reporting guidelines was designed to represent the major experimental methodologies and to permit both interaction among the committee members and also feedback from the section membership. The committee consisted of six members: Alan Gerber (Chair, Yale University), Kevin Arceneaux (Temple University), Cheryl Boudreau (University of California, Davis), Conor Dowling (University of Mississippi), Sunshine Hillygus (Duke University), and Thomas Palfrey (Caltech). The committee held three conference calls and exchanged numerous e-mails from early 2011 to early 2012, after which they submitted the proposed guidelines to the section president. The guidelines were also posted on the section website and section members were informed of this, told there would be a comment period, and asked for comments. Following the comment period, the committee reviewed whether, in light of the comments, any changes to the proposed reporting standards were needed. After this review, the guidelines were finalized. The version of the guidelines that emerged from this process appears at the end of this report (Appendix 1).

CREATION OF THE INITIAL GUIDELINES

During the first meeting, the committee divided up the territory according to committee expertise. Arceneaux created a list of proposed reporting guidelines for field experiments; Boudreau and Palfrey for laboratory experiments; Hillygus for survey experiments. Prior to the second meeting, Dowling and Gerber assembled the various lists, along with reporting standards from three other fields—(1) medicine (CONSORTFootnote ⁹ ), (2) economics (e.g., Econometrica Footnote ¹⁰ , Palfrey and Porter (Reference Palfrey and Porter1991); American Economic Review Footnote ¹¹ ; Journal of Political Economy Footnote ¹² ), and (3) psychology (American Psychological Association, JARS (2008))—for the committee to discuss.

During the second meeting, the committee reviewed the various reporting standards (both our internally created documents and those from the other three fields) and, because it was felt there was sufficient overlap across types of experiments, we decided to craft a single set of reporting standards. After this second meeting, Boudreau consolidated the work of the committee members into a single document, which then went to each committee member for additional changes and comments. During the third meeting, the committee discussed this newly created document (any and all changes, whether additions or subtractions, were considered). Based on discussions held during this meeting, Dowling and Gerber made minor changes to the document and circulated the revised document to the committee for comments. The document, “Recommended Reporting Standards for Experiments (Laboratory, Field, Survey),” was unanimously approved by the committee and then posted to the section for comment.

COMMENT PERIOD AND REVISIONS

During the comment period we received a comment that took issue with one guideline. Professor Diana Mutz (University of Pennsylvania) questioned a provision that was listed under a subsection of Part C, which recommended, “If random assignment used, provide evidence of random assignment.” A bullet point under this heading recommended, “If demographic or other pretreatment variables were collected, [provide] a table (in text or appendix) showing baseline means and standard deviations for demographic characteristics and other pretreatment measures by experimental group.”

Mutz objected that examination of group means, contrary to the implication of this guideline, is not a method of determining whether the experiment used random assignment. In particular, as detailed in a paper forwarded along with her comment (Mutz and Pemantle, Reference Mutz and Pemantle2013), she objects to formal balance testing in which researchers evaluate the integrity of the randomization procedure by inspecting whether there are statistically significant differences in the means of pretreatment variables across treatment groups. First, she argues that, apart from the rare blunder in implementation, if the experiment did employ random assignment then this was, in fact, the procedure employed. The hypothesis that the groups have been randomly assigned is not an open question to be adjudicated by statistical analysis, since differences in covariate balance can arise by chance in any given random sample. Mutz does agree that formal tests or their rough ocular equivalents may be useful to detect errors in the implementation of randomization, but contends that these errors are uncommon. Of more concern to Mutz is the possibility that formal balance tests may lead researchers to make inadvisable modeling decisions, such as including covariates that may reduce the precision of the treatment effect estimates. For more detailed discussion of these issues, we recommend that readers consult Professor Mutz’s paper.

We considered Mutz’s arguments and although the cautions raised are well taken, we decided that on balance the value of the information contained in a table of pretreatment means and standard deviations outweighs the dangers Mutz highlights. First, and sufficient for our recommendation that researchers be asked to report covariate sample statistics, observing unexpectedly large differences in group means or variances can signal problems in data collection or analysis. Detectable imbalances can be produced in several ways (other than chance). These include, but are not limited to, mistakes in the randomization coding, failure to account for blocking or other nuances in the experimental design, mismatch between the level of assignment and the level of statistical analysis (e.g., subjects randomized as clusters but analyzed as individual units), or sample attrition.

Although the usefulness of the sample statistics for purposes of error detection is sufficient for their inclusion in the guidelines and the basis of our judgment, we add in passing that, for some readers, there may be other uses of summary statistics for covariates for each of the experimental groups. For instance, if there is imbalance, whether statistically significant or not, in a pretreatment variable that is thought by a reader to be highly predictive of the outcome, and this variable is not satisfactorily controlled for, the reader may want to use the baseline sample statistics to informally adjust the reported treatment effect estimates to account for this difference. Finally, we note that the guidelines do not counsel any particular modeling response to the table of covariate means that we ask the researcher to provide. As stated earlier in this report, our guiding principle is to provide the reader and the reviewer the information they need to evaluate what the researcher has done and to update their beliefs about treatment effects accordingly. As such, the guidelines try to “break ties” in the direction of asking authors to show a table that might be useless, rather than omit information that could be of use to some reasonable share of researchers.

To avoid any confusion about our intentions, we decided to amend the guidelines in light of Professor Mutz’s concerns by including a brief preface to this reporting item to elaborate a bit on the rationale for the table of sample means. We now write that:

“If random assignment used, to help detect errors such as problems in the procedure used for random assignment or failure to properly account for blocking, if demographic or other pretreatment variables were collected, provide a table (in text or appendix) showing baseline means and standard deviations for demographic characteristics and other pretreatment measures (if collected) by experimental group.”

SUPPLEMENTARY MATERIAL

To view supplementary material for this paper, please visit http://dx.doi.org/10.1017/xps.2014.11.