Hostname: page-component-74656f9884-rdgjk Total loading time: 0 Render date: 2026-11-09T11:03:45.469Z Has data issue: false hasContentIssue false
  • English
  • Français

Thinking About Data, Research Methods, and Statistical Analyses: Commentary on Sijtsma’s (2014) “Playing with Data”

Published online by Cambridge University Press:  01 January 2025

Irwin D. Waldman  and
Scott O. Lilienfeld
Irwin D. Waldman*
Affiliation:
Emory University
Scott O. Lilienfeld*
Affiliation:
Emory University
*
Correspondence should be made to Irwin D. Waldman and Scott O. Lilienfeld, Department of Psychology, Emory University, 475 PAIS Building, 36 EagleRow, Atlanta, GA30322 USA. Email: psyiw@emory.edu and slilien@emory.edu
Correspondence should be made to Irwin D. Waldman and Scott O. Lilienfeld, Department of Psychology, Emory University, 475 PAIS Building, 36 EagleRow, Atlanta, GA30322 USA. Email: psyiw@emory.edu and slilien@emory.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

We comment on Sijtsma’s (2014) thought-provoking essay on how to minimize questionable research practices (QRPs) in psychology. We agree with Sijtsma that proactive measures to decrease the risk of QRPs will ultimately be more productive than efforts to target individual researchers and their work. In particular, we concur that encouraging researchers to make their data and research materials public is the best institutional antidote against QRPs, although we are concerned that Sijtsma’s proposal to delegate more responsibility to statistical and methodological consultants could inadvertently reinforce the dichotomy between the substantive and statistical aspects of research. We also discuss sources of false-positive findings and replication failures in psychological research, and outline potential remedies for these problems. We conclude that replicability is the best metric of the minimization of QRPs and their adverse effects on psychological research.

Keywords

questionable research practicesreplicabilityminimizing false positives

Information

Type
Original paper
Information
Psychometrika , Volume 81 , Issue 1 , March 2016 , pp. 16 - 26
Copyright
Copyright © 2015 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Abelson, R.P. (1995). Statistics as principled argument. Hillsdale, NJ: Erlbaum.Google Scholar
Asendorpf, J.B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J.J., & Fiedler, K., et al. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119.CrossRefGoogle Scholar
Bertamini, M., & Munafò, M.R. (2012). Bite-size science and its undesired side effects. Perspectives on Psychological Science, 7, 67–71.CrossRefGoogle ScholarPubMed
Button, K.S., Ioannidis, J.P., Mokrysz, C., Nosek, B.A., Flint, J., & Robinson, E.S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.CrossRefGoogle ScholarPubMed
Cohen, J. (1962). The statistical power of abnormal-social psycological research: A review. Journal of Abnormal and Social Psycology, 65, 145–153.CrossRefGoogle Scholar
Cohen, J. (1969). Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Erlbaum.Google Scholar
Cohen, J. The earth is round \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(p<. 05)$$\end{document} (1994). American Psychologist. 49, 997–1003.CrossRefGoogle Scholar
Collins, A.L., & Sullivan, P.F. (2013). Genome-wide association studies in psychiatry: What have we learned. British Journal of Psychiatry, 202, 1–4.CrossRefGoogle ScholarPubMed
Cumming, G. (2014). The new statistics why and how. Psychological Science, 25, 7–29.CrossRefGoogle Scholar
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidenceintervals, but can’t make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119–126.CrossRefGoogle Scholar
Fisher, R.A. (1925). Statistical methods for research workers. London: Oliver & Boyd.Google Scholar
Frank, M.C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600–604.CrossRefGoogle ScholarPubMed
Gigerenzer, G. (1998). We need statistical thinking, not statistical rituals. Behavioral and Brain Sciences, 21, 199–200.CrossRefGoogle Scholar
Giner-Sorolla, R. (2012). Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science, 7, 562–571.CrossRefGoogle ScholarPubMed
Gray, K., & Wegner, D.M. (2013). Six guidelines for interesting research. Perspectives on Psychological Science, 8, 549–553.CrossRefGoogle ScholarPubMed
Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum.Google Scholar
Harzing, A.W. (2011). The publish or perish book: A guide to the software. London: Tarma Software Research.Google Scholar
Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.Google Scholar
Hersh, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 16569–16572.Google Scholar
Huff, D. (1954). How to Lie with Statistics. New York: W.W. Norton & Company.Google Scholar
Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.Google Scholar
Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124.CrossRefGoogle ScholarPubMed
Ioannidis, J., Munafò, M.R., Fusar-Poli, P., Nosek, B.A., & David, S.P. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241.CrossRefGoogle ScholarPubMed
John, L.K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.CrossRefGoogle ScholarPubMed
Kahneman, D. (2011). Thinking: Fast and slow. New York: Farrar, Straus and Giroux.Google Scholar
Kuhn, T.S. (1970). The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.Google Scholar
Lehrer, J. (2010). The truth wears off. The New Yorker, 52–57.Google Scholar
Lykken, D.T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 1151–1159.CrossRefGoogle ScholarPubMed
Meehl, P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834.CrossRefGoogle Scholar
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biomettika, 20A, 175–240, 263–294.Google Scholar
Nickerson, R.S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241–301.CrossRefGoogle ScholarPubMed
Nosek, B.A., Spies, J.R., & Motyl, M. (2012). Scientific Utopia II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.CrossRefGoogle ScholarPubMed
Nosek, B.A., & Lakens, D. (2014). Registered reports. Social Psychology, 45, 137–141.CrossRefGoogle Scholar
Oakes, M. (1986). Statistical inference: A commentary for the social and behavioral sciences. New York: Wiley.Google Scholar
Pashler, H., & Wagenmakers, E.J. (2012). Editors’ Introduction to the Special Section on replicability in psychological science. A crisis of confidence?. Perspectives on Psychological Science, 7, 528–530.CrossRefGoogle Scholar
Ripke, S., et al. (2013). Genome-wide association analysis identifies 13 new risk loci for schizophrenia. Nature Genetics, 45, 150–159.CrossRefGoogle ScholarPubMed
Rosenthal, R. (2001). Meta-analytic procedures for social research. (2nd ed.). Beverly Hills, CA: Sage Publications.Google Scholar
Rosnow, R.L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.CrossRefGoogle Scholar
Rozeboom, W.W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57, 416–428.CrossRefGoogle ScholarPubMed
Schmidt, F.L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist, 47, 1173–1181.CrossRefGoogle Scholar
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105(2), 309–316.CrossRefGoogle Scholar
Simons, D.J. (2014). The value of direct replication. Perspectives on Psychological Science, 9, 76–80.CrossRefGoogle ScholarPubMed
Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefGoogle ScholarPubMed
Simonsohn, U., Nelson, L.D., & Simmons, J.P. (2013). P-Curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143, 534–537.CrossRefGoogle Scholar
Smith, M.L., Glass, G.V., & Miller, T.I. (1980). The benefits of psychotherapy. Baltimore, MD: Johns Hopkins University Press.Google Scholar
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.CrossRefGoogle ScholarPubMed
Vrij, A., Granhag, P.A., & Porter, S. (2010). Pitfalls and opportunities in nonverbal and verbal lie detection. Psychological Science in the Public Interest, 11, 89–121.CrossRefGoogle ScholarPubMed
Wicherts, J.M. (2014, May 24). The power paradox and the myth of the failed study. Paper presented at the Annual Convention of the Association for Psychological Science, San Francisco, California.Google Scholar
Wicherts, J.M., Bakker, M., & Molenaar, D. (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS One, 6, e26828.CrossRefGoogle Scholar