Evaluating Experimental Research

doi:10.1017/9781108684354.012

Chapter 11 - Evaluating Experimental Research

Published online by Cambridge University Press: 19 December 2019

Henry L. Roediger and

Jeremy K. Yamashiro

Edited by

Robert J. Sternberg and

Diane F. Halpern

Show author details

Robert J. Sternberg: Affiliation:
Cornell University, New York
Diane F. Halpern: Affiliation:
Claremont McKenna College, California

Book contents

Get access

Summary

Experiments allow researchers to determine cause and effect relations between variables. As such, they are a critical component in the advance of scientific psychology. In this chapter, we discuss the theory behind the design of good experiments, and provide a sample study for evaluation. We outline three important types of replication, and give an overview of historical events that led to a renewed vigilance regarding replicability. Finally, we discuss generalizability of research in terms of four factors: Subjects (or participants), materials used in the experiment, dependent measures, and the experimental situation. Effects that generalize across these sets of factors are robust. We end the chapter with a set of 18 critical-thinking questions that should be borne in mind while reading and evaluating experimental research. Referring to these questions will help to sharpen critical thinking skills about experimental research.

Keywords

subjects systematic replication variable within subject design

Information

Type: Chapter
Information: Critical Thinking in Psychology , pp. 249 - 279

DOI: https://doi.org/10.1017/9781108684354.012 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Aarts, A. A., et al. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI:10.1126/science.aac4716Google Scholar

Bacon, F. (1620/2000)The new organon. Eds. Jardine, L. & Silverthorne, M.. Originally published as Novum Organum, sive indicia vera de Interpretationes Naturae in 1620. Cambridge: Cambridge University Press.Google Scholar

Banaji, M. R., & Crowder, R. G. (1989). The bankruptcy of everyday memory. American Psychologist, 44, 1185–1193. DOI:10.1037/0003-066X.44.9.1185CrossRef Google Scholar

Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. DOI:10.1037/a0021524CrossRef Google Scholar PubMed

Bones, A. K. (2012). We knew the future all along: Scientific hypothesizing is much more accurate than other forms of precognition – A satire in one part. Perspectives on Psychological Science, 7(3), 307–309. DOI:10.1177/1745691612441216CrossRef Google Scholar PubMed

Boring, E. G. (1929). A history of experimental psychology. New York and London: Century.Google Scholar

Camerer, C. F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature: Human Behavior, 2, 637–644. DOI:10.1038/s41562-018–0399–zGoogle Scholar

Craik, F. I. M., & Tulving, E. (1975). Depth of processing and the retention of words in episodic memory. Journal of Experimental Psychology: General, 104(3), 263–294. DOI:10.1037/0096–3445.104.3.268Google Scholar

Cummings, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.Google Scholar

Deese, J. (1959). On the prediction of occurrence of particular verbal instructions in immediate recall. Journal of Experimental Psychology, 58, 17–22. DOI:10.1037/h0046671Google Scholar

Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: It’s all in the mind, but whose mind? PloS ONE, 7(1), e2908. DOI:10.1371/journal.pone.0029081Google Scholar

Freyd, J. J., & Gleaves, D. H. (1996). “Remembering” words not presented in lists: Relevance to the current recovered/false memory controversy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 811–813. DOI:10.1037/0278–7393.22.3.811Google Scholar

Ganley, C. M., Mingle, L. A., Ryan, A. M., Ryan, K., Vasilyeva, M., & Perry, M. (2013). An examination of stereotype threat effect on girls’ mathematics performance. Developmental Psychology, 49(10), 1886–1897. DOI:10.1037/a0031412Google Scholar

Gallo, D. A. (2010). False memories and fantastic beliefs: 15 years of the DRM illusion. Memory & Cognition, 37, 833–848. DOI:10.3758/MC.38.7.833Google Scholar

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, 26, 1–104.Google Scholar

Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Online. http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf Google Scholar

Hagger, M. S., et al. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573. DOI:10.1177/1745691616652873Google Scholar

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of P-hacking in science. PloS Biology, 13(3), e1002106. DOI:10.1371/journal.pbio.1002106CrossRef Google Scholar PubMed

Ioannidis, J. P. A. (2005). Why most published research findings are false. PloS Medicine, 2(8), 0696–0701. DOI:10.1371/journal.pmed.0020124Google Scholar

Jenkins, J. J. (1979). Four points to remember: A tetrahedral model of memory experiments. In Cermak, L. S. & Craik, F. I. M. (Eds.), Levels of processing in human memory (pp. 429–446). Hillsdale, NJ: Erlbaum.Google Scholar

Kahneman, D. (2012). A proposal to deal with questions about priming effects. Online. https://tinyurl.com/yyj8v7nx Google Scholar

Kantowitz, B. H., Roediger, III, H. L., & Elmes, D. (2015). Experimental psychology (10th ed.). Belmont, CA: Wadsworth.Google Scholar

Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. DOI:10.1207/s15327957pspr0203_4CrossRef Google Scholar PubMed

Klein, R. A., et al. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45(3), 142–152. DOI:10.1027/1864-9335/a000178Google Scholar

McDermott, K. B., & Naaz, F. (2014). Is recitation an effective tool for adult learners? Journal of Applied Research in Memory and Cognition, 3, 207–213. DOI:10.1016/j.jarmac.2014.06.006Google Scholar

Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387. DOI:10.1037/0003-066X.38.4.379CrossRef Google Scholar

Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530. DOI:10.1177/1745691612465253Google Scholar

Piaget, J. (1962). Play, dreams, and imitation in childhood. New York: Norton.Google Scholar

Popper, K. (2002). The logic of scientific discovery. New York: Routledge.Google Scholar

Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5), 653–656. DOI:10.1177/0956797614553946CrossRef Google Scholar

Roediger, III, H. L. (2012). Psychology’s woes and a partial cure: The value of replication. The APS Observer, 25(2), 9, 27–29.Google Scholar

Roediger, III, H. L., & Gallo, D. A. (2002). Processes affecting accuracy and distortion in memory: An overview. In Eisen, M. L., Goodman, G. S., & Quas, J. A. (Eds.), Memory and suggestibility in the forensic interview (pp. 3–28). Mahwah, NJ: Erlbaum.Google Scholar

Roediger, III, H. L., & Gallo, D. A. (2016). Associative memory illusions. In Pohl, R. F. (Ed.), Cognitive illusions: Intriguing phenomena in thinking, judgment and memory (2nd ed.) (pp. 390–405). New York: Psychology Press.Google Scholar

Roediger, III, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17, 249–255. DOI:10.1111/j.1467–9280.2006.01693.xGoogle Scholar

Roediger, III, H. L., & McCabe, D. P. (2007) Evaluating Experimental Research: Critical Issues. In Sternberg, R. J., Roediger, III H. L., & Halpern, D. F. (Eds.), Critical thinking in psychology (pp. 15–36). New York: Cambridge University Press.Google Scholar

Roediger, III, H. L., & McDermott, K. (1995). Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 803–814. DOI:10.1037/0278–7393.21.4.803Google Scholar

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. DOI:10.1177/0956797611417632Google Scholar

Spellman, B. A. (2015). A short (personal) future history of Revolution 2.0. Perspectives on Psychological Science, 10(6), 886–899. DOI:10.1177/1745691615609918Google Scholar

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175–184. DOI:10.1016/S0022–5371(67)80092–6Google Scholar

Tulving, E. (1985). Memory and consciousness. Canadian Psychology, 26, 1–12. DOI:10.1037/h0080017CrossRef Google Scholar

Underwood, B. J. (1965). False recognition produced by implicit verbal responses. Journal of Experimental Psychology, 70, 122–129. DOI:10.1037/h0022014Google Scholar

Accessibility standard: Unknown

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

Accessibility compliance for the HTML of this chapter is currently unknown and may be updated in the future.