Duarte et al. document the types of distortions that can creep into a scientific field when a particular political ideology takes hold and alternative viewpoints are largely absent. We agree with their analysis and offer additional support for a key theme: Ideological homogeneity can nurture threats to the validity of research conclusions and can be especially damaging to external and construct validity.
An example is meritocracy, a measure used by Napier and Jost (Reference Napier and Jost2008) to test the conclusion that conservatives are better at “rationalizing inequality – for example, by seeing it as emerging from a fair, legitimate, and meritocratic system” (pp. 568–69). However, their single-item measure of meritocracy is comparable to items in personal efficacy scales (asking if success in life is due to luck or hard work). Other research shows that this item is significantly related to personal agency but not to system justification (Schlenker et al. Reference Schlenker, Chambers and Le2012). “Meritocracy” is compatible with the liberal view that conservatives rationalize injustice, whereas “personal agency,” a label better supported by data, is inconsistent with a darker portrayal of conservatives. This example illustrates how the labeling of constructs guides interpretations.
Researchers should be skeptical when interpreting scale results: look carefully at the items themselves, make their own judgments about relevant concepts, and rely on empirical justification for conclusions. Ultimately, construct validity is determined by examining a measure's place in the nomological net of similar and dissimilar constructs. Political homogeneity can distort this process by allowing ideologically compatible values to influence interpretations (e.g., by using a biased, limited selection of other constructs for assessing convergent and discriminant validity), minimize skepticism, and create premature consensus.
Another example is the system justification (SJ) scale, which was designed to measure “the rationalization of the status quo” to avoid acknowledging the injustice of the system (Kay & Jost Reference Kay and Jost2003, p. 825). Conservatives score higher on SJ than liberals, seemingly supporting the interpretation that the former are more defensive, fearful, and motivated to distort reality. However, SJ is positively related to personal control, optimism, self-esteem, agreeableness, moral commitment, and work ethic, and negatively related to depression, neuroticism, and cynicism (Schlenker et al. Reference Schlenker, Chambers and Le2012). The nomological net into which this measure fits would usually be regarded as indicating positive adjustment and mental health and seems inconsistent with a darker view of conservatives. As with meritocracy, using a wider range of measures for the net – ones having positive as well as negative connotations – yields a better appreciation of what might actually be measured and why a different set of conclusions might be appropriate.
Embedding liberal values within theory and method is especially evident in the extensive literature linking political views with personality and prejudice. A major confound has plagued this research: Studies focused on attitudes toward minority, primarily Left-leaning social groups (e.g., atheists, homosexuals, Blacks) and failed to include social groups across the entire ideological spectrum (Chambers et al. Reference Chambers, Schlenker and Collisson2013). When Right-leaning social groups (e.g., Christians, business people, military personnel) were included, liberals expressed as much prejudice toward those groups as conservatives expressed toward the Left-leaning ones. In other words, both conservatives and liberals express prejudice toward groups whose values and goals conflict with their own. The restricted range of prior targets permitted misleading generalizations that have questionable external validity.
Another construct validity problem applies to measures that are widely assumed to be antecedents of prejudice and markers of intolerance and bigotry: right-wing authoritarianism (RWA), modern and symbolic racism (MR/SR), and social dominance orientation (SDO). Each scale contains items that comprise important value components of political ideologies. For example, the RWA scale includes items that reference religion and traditional values, which are embraced by conservatives more than liberals, and the SDO scale includes items that assess preferences for equality (receiving equal outcomes regardless of inputs) rather than equity (receiving outcomes commensurate with one's inputs), which are favored by liberals more than conservatives (Schlenker et al. Reference Schlenker, Chambers and Le2012). Similar concerns about measures of prejudice have been raised by others (e.g., MR/SR: Sniderman & Tetlock Reference Sniderman and Tetlock1986; Zuriff Reference Zuriff2002), but these critiques have largely been ignored and these measures continue to be widely utilized.
To illustrate the problem, we found that scores on these measures (e.g., RWA, MR/SR, SDO) were negatively related to evaluations of Left-leaning groups but positively related to evaluations of Right-leaning groups. In other words, these relationships again show prejudice on both sides, with conservatives (high scorers) and liberals (low scorers) each favoring groups who shared compatible values (Chambers et al. Reference Chambers, Schlenker and Collisson2013).
In two other studies, we manipulated both the race (Black or White) and ideological position (conservative or liberal) of the target and assessed participants' scores on MR and anti-black racism (Chambers et al. Reference Chambers, Schlenker and Collisson2013). If these measures assessed racial bigotry – as they are purported to do – they should predict negative attitudes towards Black targets and positive attitudes towards White targets, regardless of the target's ideological position. However, we found that they predicted attitudes based on the target's ideological position and not its race. Higher MR scores, for example, predicted negative attitudes (i.e., greater prejudice) towards both Black and White liberal targets, and positive attitudes (i.e., lower prejudice) towards both White and Black conservative targets. In other words, these measures seem to be tapping differences in core ideological beliefs and values. Ironically, they failed to predict the very thing they are supposed to predict – racial prejudice.
Although we focused on construct and external validity, internal and statistical conclusion validity can also be compromised. An example is the relationship between a measure of social inequality (Gini index) and happiness, which was used by Napier and Jost (Reference Napier and Jost2008) to conclude that, unlike conservatives, liberals “lack ideological rationalizations that would help them frame inequality in a positive . . . light” (p. 571). However, data re-analyses showed a major confound (between Gini index and time) and a failure to control for variables that were included in their other work (church attendance). When these were taken into account, the social inequality effect disappeared (Schlenker et al. Reference Schlenker, Chambers and Le2012).
Keep in mind that threats to validity can creep into any research, but whenever a particular ideological position dominates, corrective activities are more likely to be suppressed. The beauty of science is that corrective activities usually occur; the question is how long these might take.
Duarte et al. document the types of distortions that can creep into a scientific field when a particular political ideology takes hold and alternative viewpoints are largely absent. We agree with their analysis and offer additional support for a key theme: Ideological homogeneity can nurture threats to the validity of research conclusions and can be especially damaging to external and construct validity.
An example is meritocracy, a measure used by Napier and Jost (Reference Napier and Jost2008) to test the conclusion that conservatives are better at “rationalizing inequality – for example, by seeing it as emerging from a fair, legitimate, and meritocratic system” (pp. 568–69). However, their single-item measure of meritocracy is comparable to items in personal efficacy scales (asking if success in life is due to luck or hard work). Other research shows that this item is significantly related to personal agency but not to system justification (Schlenker et al. Reference Schlenker, Chambers and Le2012). “Meritocracy” is compatible with the liberal view that conservatives rationalize injustice, whereas “personal agency,” a label better supported by data, is inconsistent with a darker portrayal of conservatives. This example illustrates how the labeling of constructs guides interpretations.
Researchers should be skeptical when interpreting scale results: look carefully at the items themselves, make their own judgments about relevant concepts, and rely on empirical justification for conclusions. Ultimately, construct validity is determined by examining a measure's place in the nomological net of similar and dissimilar constructs. Political homogeneity can distort this process by allowing ideologically compatible values to influence interpretations (e.g., by using a biased, limited selection of other constructs for assessing convergent and discriminant validity), minimize skepticism, and create premature consensus.
Another example is the system justification (SJ) scale, which was designed to measure “the rationalization of the status quo” to avoid acknowledging the injustice of the system (Kay & Jost Reference Kay and Jost2003, p. 825). Conservatives score higher on SJ than liberals, seemingly supporting the interpretation that the former are more defensive, fearful, and motivated to distort reality. However, SJ is positively related to personal control, optimism, self-esteem, agreeableness, moral commitment, and work ethic, and negatively related to depression, neuroticism, and cynicism (Schlenker et al. Reference Schlenker, Chambers and Le2012). The nomological net into which this measure fits would usually be regarded as indicating positive adjustment and mental health and seems inconsistent with a darker view of conservatives. As with meritocracy, using a wider range of measures for the net – ones having positive as well as negative connotations – yields a better appreciation of what might actually be measured and why a different set of conclusions might be appropriate.
Embedding liberal values within theory and method is especially evident in the extensive literature linking political views with personality and prejudice. A major confound has plagued this research: Studies focused on attitudes toward minority, primarily Left-leaning social groups (e.g., atheists, homosexuals, Blacks) and failed to include social groups across the entire ideological spectrum (Chambers et al. Reference Chambers, Schlenker and Collisson2013). When Right-leaning social groups (e.g., Christians, business people, military personnel) were included, liberals expressed as much prejudice toward those groups as conservatives expressed toward the Left-leaning ones. In other words, both conservatives and liberals express prejudice toward groups whose values and goals conflict with their own. The restricted range of prior targets permitted misleading generalizations that have questionable external validity.
Another construct validity problem applies to measures that are widely assumed to be antecedents of prejudice and markers of intolerance and bigotry: right-wing authoritarianism (RWA), modern and symbolic racism (MR/SR), and social dominance orientation (SDO). Each scale contains items that comprise important value components of political ideologies. For example, the RWA scale includes items that reference religion and traditional values, which are embraced by conservatives more than liberals, and the SDO scale includes items that assess preferences for equality (receiving equal outcomes regardless of inputs) rather than equity (receiving outcomes commensurate with one's inputs), which are favored by liberals more than conservatives (Schlenker et al. Reference Schlenker, Chambers and Le2012). Similar concerns about measures of prejudice have been raised by others (e.g., MR/SR: Sniderman & Tetlock Reference Sniderman and Tetlock1986; Zuriff Reference Zuriff2002), but these critiques have largely been ignored and these measures continue to be widely utilized.
To illustrate the problem, we found that scores on these measures (e.g., RWA, MR/SR, SDO) were negatively related to evaluations of Left-leaning groups but positively related to evaluations of Right-leaning groups. In other words, these relationships again show prejudice on both sides, with conservatives (high scorers) and liberals (low scorers) each favoring groups who shared compatible values (Chambers et al. Reference Chambers, Schlenker and Collisson2013).
In two other studies, we manipulated both the race (Black or White) and ideological position (conservative or liberal) of the target and assessed participants' scores on MR and anti-black racism (Chambers et al. Reference Chambers, Schlenker and Collisson2013). If these measures assessed racial bigotry – as they are purported to do – they should predict negative attitudes towards Black targets and positive attitudes towards White targets, regardless of the target's ideological position. However, we found that they predicted attitudes based on the target's ideological position and not its race. Higher MR scores, for example, predicted negative attitudes (i.e., greater prejudice) towards both Black and White liberal targets, and positive attitudes (i.e., lower prejudice) towards both White and Black conservative targets. In other words, these measures seem to be tapping differences in core ideological beliefs and values. Ironically, they failed to predict the very thing they are supposed to predict – racial prejudice.
Although we focused on construct and external validity, internal and statistical conclusion validity can also be compromised. An example is the relationship between a measure of social inequality (Gini index) and happiness, which was used by Napier and Jost (Reference Napier and Jost2008) to conclude that, unlike conservatives, liberals “lack ideological rationalizations that would help them frame inequality in a positive . . . light” (p. 571). However, data re-analyses showed a major confound (between Gini index and time) and a failure to control for variables that were included in their other work (church attendance). When these were taken into account, the social inequality effect disappeared (Schlenker et al. Reference Schlenker, Chambers and Le2012).
Keep in mind that threats to validity can creep into any research, but whenever a particular ideological position dominates, corrective activities are more likely to be suppressed. The beauty of science is that corrective activities usually occur; the question is how long these might take.