Questionable research practices (QRPs) in social psychology and other disciplines have been the target of efforts dedicated to improving empirical social psychology; however, a focus on improving empirical practices can be more effective by linking it with a simultaneous focus on improving theoretical and interpretive practices. While mature sciences are generally characterized by broad theoretical consensus (i.e., paradigms; Kuhn Reference Kuhn1962), these perspectives are often in conflict, and efforts to resolve these discrepancies foment theoretical advancement and a more precise understanding of scientific phenomena (Popper Reference Popper1959). When social psychology – or any scientific discipline – adopts a singular ideological worldview, this serves to quell sources of potentially generative conflict. It also leads to additional (and justifiable) concerns that researchers are motivated to produce and interpret findings in a manner that is consistent with a given research perspective.
Duarte and colleagues identify several instances of questionable interpretive practices (QIPs) that may have resulted from ideological homogeneity (Jussim et al., in press b). These are instances where researchers used proper research methods for gathering data, but engaged in potentially problematic interpretations of that same data. These practices are unlikely to be unique to research areas touched by political ideology, and we believe that naming them and giving them a more general description will help researchers be vigilant for these practices more broadly:
-
1.
Premature theoretical closure: This is the practice whereby a finding is treated as firmly established when all of the necessary conditions for claiming that the finding is supported have not yet been tested. For example, Duarte et al. highlight work that suggested right-wing authoritarians are more likely to make hypocritical political judgments, when these judgments were only tested for a very limited range of issues.
-
2.
Imprecise naming: There is an incentive to name constructs with as much breadth as possible so that the construct studied can be thought to extend to a wider array of situations. Duarte et al. highlight research where the original authors made claims about “unethical decisions” broadly, although they primarily measured decisions contrary to liberal values.
-
3.
Begging the question: In some cases, a particular research question or method is framed in such a way that the only possible result confirms the researcher's hypothesis. Duarte et al. discuss research where the original investigators built their conclusion (denial of realities) into the name of their measure of environmental attitudes.
There are theoretical problems in (social) psychology that go beyond questionable interpretations of the data and include problematic theoretical practices at several stages of the theoretical process. We suggest that the QIPs identified by Duarte et al. are part of a more general category of questionable theoretical and interpretive practices (QTIPs). Although there are many potential QTIPs, we think that three are worth briefly describing and adding to the list offered by Duarte et al. in the target article and by Jussim et al. (in press b).
-
4.
Déjà vu constructs: The incentives in social psychology are to produce novel theories, leading to a proliferation of theories in our major journals. However, oftentimes new theoretical constructs are merely old constructs with new branding (Hagger Reference Hagger2014). Instead of ego-depletion, for example, much of the work could fit under the much older label of mental fatigue.
-
5.
Homophone constructs: Psychologists study many everyday behaviors and phenomena leading us to name our constructs with everyday words. This also leads to a situation where we give the same name to a variety of different phenomena that are not interchangeable. For example, intentions can mean many different things, and only by specifying the precise type of intentions (e.g., implementation intentions, continuation intentions; see Hagger Reference Hagger2014) is it possible to make precise predictions as well as comparisons across studies and literatures.
-
6.
Naturalistic fallacy: Social scientists are particularly prone to providing empirical support for emerging social trends, reifying an apparently emerging status quo with data suggesting that these desired end-states are the way things naturally are. Historically, both liberally tinged (biological origins of sexual preference) and conservatively tinged (effects of illegal drugs) research programs have produced effects that appear motivated to justify a given social order.
QTIPs are different from QRPs. For example, Duarte and colleagues connect their contribution to discussions about QRPs in social psychology. QRPs lead to inflated Type I errors, and produce results that appear robust but are in fact false positives. If the dearth of non-liberals in academic social psychology leads to QRPs, presumably in the favor of a liberal perspective, then one would predict that research agendas conducive to a liberal perspective (e.g., conservatives are rigid; Jost et al. Reference Jost, Glaser, Kruglanski and Sulloway2003) would be more likely to produce false positives compared to research agendas conducive to a conservative perspective (e.g., the effect of grit on successes in life; Duckworth et al. Reference Duckworth, Peterson, Matthews and Kelly2007). However, it is not at all clear that more or less liberal research agendas produce different levels of false positives, and, indeed, neither the conservatives-are-rigid nor the grit-leads-to-success research agendas appear to be false positives. What is in doubt is the interpretation and scope of the reliable effects produced from various research agendas. This is not due to QRPs, but rather may be the result of QTIPs.
Rather than focusing on who is biased against whom (we know people are generally biased against those they disagree with; Brandt et al. Reference Brandt, Reyna, Chambers, Crawford and Wetherell2014), the focus should be on how to improve social psychological theory. By naming and describing QTIPs, we hope to move the discussion from who is biased to a focus on how we can improve the theoretical foundations of social psychology. Even if the analysis by Duarte et al. is entirely incorrect, pushing social psychologists to think about their QTIPs in addition to the QRPs is a valuable contribution.
Questionable research practices (QRPs) in social psychology and other disciplines have been the target of efforts dedicated to improving empirical social psychology; however, a focus on improving empirical practices can be more effective by linking it with a simultaneous focus on improving theoretical and interpretive practices. While mature sciences are generally characterized by broad theoretical consensus (i.e., paradigms; Kuhn Reference Kuhn1962), these perspectives are often in conflict, and efforts to resolve these discrepancies foment theoretical advancement and a more precise understanding of scientific phenomena (Popper Reference Popper1959). When social psychology – or any scientific discipline – adopts a singular ideological worldview, this serves to quell sources of potentially generative conflict. It also leads to additional (and justifiable) concerns that researchers are motivated to produce and interpret findings in a manner that is consistent with a given research perspective.
Duarte and colleagues identify several instances of questionable interpretive practices (QIPs) that may have resulted from ideological homogeneity (Jussim et al., in press b). These are instances where researchers used proper research methods for gathering data, but engaged in potentially problematic interpretations of that same data. These practices are unlikely to be unique to research areas touched by political ideology, and we believe that naming them and giving them a more general description will help researchers be vigilant for these practices more broadly:
1. Premature theoretical closure: This is the practice whereby a finding is treated as firmly established when all of the necessary conditions for claiming that the finding is supported have not yet been tested. For example, Duarte et al. highlight work that suggested right-wing authoritarians are more likely to make hypocritical political judgments, when these judgments were only tested for a very limited range of issues.
2. Imprecise naming: There is an incentive to name constructs with as much breadth as possible so that the construct studied can be thought to extend to a wider array of situations. Duarte et al. highlight research where the original authors made claims about “unethical decisions” broadly, although they primarily measured decisions contrary to liberal values.
3. Begging the question: In some cases, a particular research question or method is framed in such a way that the only possible result confirms the researcher's hypothesis. Duarte et al. discuss research where the original investigators built their conclusion (denial of realities) into the name of their measure of environmental attitudes.
There are theoretical problems in (social) psychology that go beyond questionable interpretations of the data and include problematic theoretical practices at several stages of the theoretical process. We suggest that the QIPs identified by Duarte et al. are part of a more general category of questionable theoretical and interpretive practices (QTIPs). Although there are many potential QTIPs, we think that three are worth briefly describing and adding to the list offered by Duarte et al. in the target article and by Jussim et al. (in press b).
4. Déjà vu constructs: The incentives in social psychology are to produce novel theories, leading to a proliferation of theories in our major journals. However, oftentimes new theoretical constructs are merely old constructs with new branding (Hagger Reference Hagger2014). Instead of ego-depletion, for example, much of the work could fit under the much older label of mental fatigue.
5. Homophone constructs: Psychologists study many everyday behaviors and phenomena leading us to name our constructs with everyday words. This also leads to a situation where we give the same name to a variety of different phenomena that are not interchangeable. For example, intentions can mean many different things, and only by specifying the precise type of intentions (e.g., implementation intentions, continuation intentions; see Hagger Reference Hagger2014) is it possible to make precise predictions as well as comparisons across studies and literatures.
6. Naturalistic fallacy: Social scientists are particularly prone to providing empirical support for emerging social trends, reifying an apparently emerging status quo with data suggesting that these desired end-states are the way things naturally are. Historically, both liberally tinged (biological origins of sexual preference) and conservatively tinged (effects of illegal drugs) research programs have produced effects that appear motivated to justify a given social order.
QTIPs are different from QRPs. For example, Duarte and colleagues connect their contribution to discussions about QRPs in social psychology. QRPs lead to inflated Type I errors, and produce results that appear robust but are in fact false positives. If the dearth of non-liberals in academic social psychology leads to QRPs, presumably in the favor of a liberal perspective, then one would predict that research agendas conducive to a liberal perspective (e.g., conservatives are rigid; Jost et al. Reference Jost, Glaser, Kruglanski and Sulloway2003) would be more likely to produce false positives compared to research agendas conducive to a conservative perspective (e.g., the effect of grit on successes in life; Duckworth et al. Reference Duckworth, Peterson, Matthews and Kelly2007). However, it is not at all clear that more or less liberal research agendas produce different levels of false positives, and, indeed, neither the conservatives-are-rigid nor the grit-leads-to-success research agendas appear to be false positives. What is in doubt is the interpretation and scope of the reliable effects produced from various research agendas. This is not due to QRPs, but rather may be the result of QTIPs.
Rather than focusing on who is biased against whom (we know people are generally biased against those they disagree with; Brandt et al. Reference Brandt, Reyna, Chambers, Crawford and Wetherell2014), the focus should be on how to improve social psychological theory. By naming and describing QTIPs, we hope to move the discussion from who is biased to a focus on how we can improve the theoretical foundations of social psychology. Even if the analysis by Duarte et al. is entirely incorrect, pushing social psychologists to think about their QTIPs in addition to the QRPs is a valuable contribution.