Systematic reviews are among the most reliable information on which to develop guidelines and base treatment policy. However, such reviews need to contain a high proportion of all relevant evidence, requiring the need to find all applicable trials and the need to analyze data on all participants (7). Missing or incomplete data are common in meta-analytic studies. Deciding how to handle missing data from identified studies of systematic reviews is an unavoidable decision faced by all reviewers. Investigators who have conducted systematic reviews on weight loss interventions have contacted authors directly for additional data, which can result in more detailed information (2;14;18). Alternatively, if authors are not prompted for information or have been contacted but are unable to provide missing data, reviewers exclude the studies (1;3;6;10–13), impute values or derive estimates from existing data (4;5;8;9;15;16), or use point estimates when data are presented in graphical form (6;16). Because commonly used methods for missing predictors in models of effect size often yield biased estimates (17), gaining complete data directly from authors avoids these drawbacks. The purpose of this descriptive study was to examine mode of contact and response levels of authors who had been asked to provide missing or incomplete data for a systematic review on diet and exercise interventions for weight loss.
METHODS
We conducted an extensive review of the diet and exercise literature, which included a computerized search of the MEDLINE database (U.S. National Library of Medicine, 1966 through 2003), manual searches of bibliographic references, and extensive cross-referencing of ninety-two review articles. From this systematic review, we located 1,646 articles on diet restriction and/or exercise for weight loss. Of 1,646 articles, 604 studies met selection criteria for the meta-analytic study, which required that intervention trials include (i) ≥18-year-old overweight or obese (body mass index [BMI] ≥25 kg/m2) participants; (ii) one group assigned to a weight-loss program involving energy restriction, exercise (aerobic or weight training), or both energy restriction and exercise; (iii) intervention duration for weight loss >24 hours; and (iv) English language only. Case studies, crossover trials, and exploratory studies were excluded.
Of the 604 studies that met selection criteria, 241 (39.9 percent) of the retained articles did not include essential data (e.g., sample size, age, gender, caloric restriction, exercise amount or duration, and so on). For most variables, we contacted authors only if they indicated that the particular variable was collected as part of their study (e.g., authors reported baseline BMI but failed to indicate end of study BMI). We contacted authors by letter, electronic mail (e-mail), or a combination of both letter and e-mail. Authors whose studies were published >10 years ago were not contacted and were excluded in any calculations for this particular study. Each request to authors provided a description of the meta-analysis (on university letterhead and principal investigator signature for letter requests), complete article citation, specific information needed in spreadsheet format, and our contact information. To enhance responses by letter, we provided a self-addressed stamped return envelope. If authors failed to respond within approximately 4 weeks from the initial date of requests, then they were contacted again by letter or by e-mail (if available).
Statistical Analysis
Survival analyses were performed with the Kaplan–Meier method to determine differences in the proportion of responders over time among the modes of contact and to determine whether response rates differed between authors from United States and those from other countries. A log-rank test was used to determine a significant difference at the 0.05 level. If significance was detected, pair-wise comparisons were made using log-rank tests with a Bonferroni correction. Pair-wise hazard ratios were also constructed with associated 95 percent confidence intervals. Logistic regression was used to determine whether the number of items requested and publication date influenced the likelihood of response. Mean and standard deviation were calculated for all variables of interest. All analyses were performed using PC-SAS (version 8.2, Cary, NC).
RESULTS
Response
One hundred forty-six authors from nineteen countries were contacted to provide additional data (Table 1). Only forty-six authors (31.5 percent) responded to our requests for information. The number of requests per author inquiry averaged more than two. However, the number of items requested per contact did not influence the probability of response (p=.63). Table 2 lists the specific reason(s) for contacting authors and the number of studies for each request.
Response rates differed by mode of contact (letter, 24 percent; e-mail, 47 percent; and combination of letter and e-mail, 73 percent). The method to contact authors influenced the likelihood of response. Of the two methods, e-mail was the most likely method to elicit a response compared with letter requests (hazard ratio [HR]=2.5; 95 percent confidence interval [CI]=1.3–4.0). Combining letter and e-mail had a higher proportion of response but was not statistically different from e-mail alone (73 percent vs. 47 percent, p=.36). Furthermore, the combination of methods (i.e., letter plus e-mail follow-up) rather than multiple contacts using the same method (e.g., letter alone or e-mail alone) was more effective for eliciting a response from the author, because multiple contacts using the same method did not increase the likelihood of response (p=.46). These relationships held true even after controlling for publication date.
Article publication date also influenced the likelihood of response (p<.05). The older the article the less likely the author would respond. Fifty percent of the authors responded for articles less than 5 years old, 50 percent responded for articles 5 to 10 years old, 12 percent responded for articles 10 to 15 years old, and 15 percent responded for articles older than 15 years. There was a 78 percent (CI=0.107–0.479) reduction in the odds of response if the article was 10 years old or older.
Duration
The duration between the information request and the author's response averaged 14 ± 22 days (median=6 days). The time between the request and the author's reply was shortest for e-mail (3 ± 3 days; median=1 day), followed by the combination of e-mail plus letter (13 ± 12 days; median=9 days), and was longest for letters only (27 ± 30 days; median=10 days) (p<.05). For e-mail, we received no further response from authors after 11 days, whereas we continued to receive letter responses up to 111 days after our initial requests for information.
Country
Of the 146 authors to contact, 68 authors were from the United States with the remaining 78 authors residing in 18 different countries. The average rate and time to respond was not statistically different for U.S. authors compared with authors from other countries. Examining the average time to response by method showed that, for both the e-mail and combination (e-mail and letter) methods, there was no difference between the United States and countries outside the United States. Letter response time was 17 days longer (102 ± 40 days) for authors outside the United States compared with U.S. authors (85 ± 42 days).
DISCUSSION AND CONCLUSIONS
During the systematic review process, we found that many studies had missing or incomplete data on important variables, such as subjects' age and sample size. Contacting authors by letter or e-mail can result in more detailed information, yet it was very difficult to acquire essential information from authors after a study had been published, with almost 40 percent of authors not locatable and less than one-third responding to our requests. In addition, efforts to contact authors were both time-consuming and required additional resources. However, it does appear that either e-mail or a combination of e-mail and letter are better at increasing response rates compared with letter alone. Although meta-analysis may be a useful statistical approach for systematic reviews on diet and exercise for weight loss, we believe that the reporting quality of weight loss research studies can be improved significantly.
When interpreting the results from the present study, it should be pointed out that the method by which authors were contacted was not approached systematically. The purpose of the study was to describe author response tendencies to help improve future studies in which contacting authors is a necessity. Specifically, we were interested in describing how far back authors are willing to respond to request for information and which methods of contacting these authors provide the greatest likelihood of response. The data presented here provide insight into these questions and can help improve future studies.
CONTACT INFORMATION
Cheryl A. Gibson, PhD (cgibson@kumc.edu), Research Associate Professor, Department of Internal Medicine, University of Kansas School of Medicine, 3901 Rainbow Boulevard, Mail Stop 1020, Kansas City, Missouri 66160
Bruce Wayne Bailey, PhD (bruce.bailey@umb.edu), Assistant Professor, Department of Exercise and Health Science, University of Massachusetts Boston, 100 Morrissey Boulevard, Boston, Massachusetts 02125-3393
Michael J. Carper, PhD (carperphd31@yahoo.com), Postdoctural Fellow, Health and Performance Enhancement, Pennington Biomedical Research Center, 6400 Perkins Road, Baton Rouge, Louisiana 70810
James D. LeCheminant, PhD (jlechem@siue.edu), Assistant Professor, Department of Kinesiology and Health Education, Southern Illinois University Edwardsville, Box 1126, Edwardsville, Illinois 62026
Erik Paul Kirk, PhD (ekirk@im.wustl.edu), Postdoctural Research Fellow, Department of Internal Medicine, Washington University School of Medicine, 660 South Euclid Avenue, Box 8031, St. Louis, Missouri 63110
Guoyuan Huang, PhD (ghuang@usi.edu), Assistant Professor, Department of Physical Education, University of Southern Indiana, 8600 University Boulevard, Evansville, Indiana 47712
Katrina Drowatzky DuBose, PhD (dubosek@ecu.edu), Assistant Professor, Department of Exercise and Sport Science, East Carolina University, Greenville, North Carolina 27858
Joseph E. Donnelly, EDD (jdonnelly@ku.edu), Professor/Director, Center for Physical Activity & Weight Management, University of Kansas, 1301 Sunnyside Avenue, Lawrence, Kansas 66045
This work is supported by National Institutes of Health NIH DK49181, Joseph E. Donnelly, Ed.D., Principal Investigator. Preliminary results were accepted as an abstract at the 2003 annual conference of the American College of Sports Medicine.
