A thought experiment
Suppose institutional review boards (IRBs), grant funders, and journal reviewers are influenced in their judgment of research by its political orientation, not just its scientific merit (Ceci et al. Reference Ceci, Peters and Plotkin1985). Suppose individuals are asked to evaluate a project; in one version the data support a liberal agenda and in another version they support a conservative agenda – despite the methods, procedures, and analyses being identical. Only the political implications of the results differ. Reviewers should not treat these studies differently if the studies are identical except for their political implications. If the studies have the identical design, analysis, literature framing, and so forth, they should be rated similarly in terms of their scientific rigor, appropriateness, and publishability.
Suppose journal reviewers are asked to review a study, a summary of which follows. After reading the full study, reviewers rate the rigor of the design, the importance/appropriateness of the question, the adequacy of the literature review/framing, and its publishability, by employing the typical considerations for the journal. Here are two politically opposite versions of the same study:
Version A of study
Is the Black–White IQ gap partly explained by economic disadvantage of Blacks? Researchers analyzed a nationally representative data set that included IQ and a host of demographic and environmental variables. They reported that controlling for parental educational attainment and income reduces the average Black–White group difference in IQ by roughly 90% or 13 IQ points, and, given that parents' socioeconomic status (SES) is an imperfect measure of environmental influences on intelligence, even this 13-point reduction is likely to underestimate the importance of parental social class on racial differences in intelligence. The researchers conclude that:
According to cultural theories of racial and ethnic differences in intelligence, as African American families advance up the socioeconomic ladder, their children should be less exposed to environmental deficits and therefore should do better and, by extension, close the gap separating the Black mean from the White mean. In fact, this is exactly what we found: The magnitude of the mean Black–White group difference in IQ for higher SES levels, when measured in standard deviations, is far smaller. . . . Matching Black and White children for the geographical areas of their homes, the schools they attend, and other finer-grained socioeconomic indicators reduces the mean group IQ difference still further, eliminating IQ gaps completely. Black children from the best areas and schools (those associated with the highest mean scores) average only slightly lower than do White children with similar socioeconomic indicators. This is an anomaly for genetic theory but is easily handled by environmental theories.
-
1. Rate the above study on its scientific methodology. Did the researchers' methodology allow a reasonable test of the research question? Use a 1 to 9 scale, in which 1 = extremely bad, 5 = neither good nor bad, and 9 = extremely good.
-
2. Rate the question asked by these researchers regarding its appropriateness for investigators to address – is this an appropriate question for researchers to ask? Use the same 1 to 9 scale.
Version B of study
In a flipped version, different reviewers rate the same study but with the results tilted in the opposite direction:
Is the Black–White IQ gap partly explained by economic disadvantage of Blacks? Researchers analyzed a nationally representative data set that included IQ and a host of demographic and environmental variables. They reported that controlling for parental educational attainment and income reduces the average Black–White group difference in IQ by only 30% or roughly 5 IQ points out of the 15-point gap that separates Blacks and Whites. Moreover, given that parents' socioeconomic status (SES) partly reflects their genetic differences in intelligence, even this 5-point reduction is likely to mask some genetic differences in parental intelligence. The researchers conclude the following:
According to cultural theories of racial and ethnic differences in intelligence, as self-defined Black groups advance up the socioeconomic ladder, their children should be less exposed to environmental deficits and therefore should do better and, by extension, close the gap separating the Black mean IQ from the White mean IQ. In fact, we found that the magnitude of the mean Black–White group difference in IQ for higher SES levels, when measured in standard deviations, is actually much larger than that found between lower SES groups. . . . Matching Black and White children for the geographical areas of their homes, the schools they attend, and other finer-grained socioeconomic indicators again reduces the mean group IQ difference but does not eliminate it. Black children from the best areas and schools (those schools associated with the highest mean scores) still average slightly lower IQs than do White children with the lowest socioeconomic indicators. This is an anomaly for the culture-only theory but is consistent with genetic theory through regression to the mean.
Conclusion
If liberal reviewers favor Version A and conservatives favor Version B, this would be evidence that extra-scientific considerations influence reviewers' calculus in ways that may be opaque to authors and editors. Reviewers of a version incongruous with their political orientation might view the study as “not asking the right question,” whereas their ideologically similar peers who review the opposite version might rate it as appropriate and relevant. One wonders whether a study of this type – if revealing that reviewers (including those who are regular journal reviewers and editors or grant-agency panelists) were biased against ideologically contrary interpretations – would be as readily publishable as its own counterpart arriving at the opposite conclusion.
A greater number of non-liberal scientists might help correct errors in scholarly reasoning and reframe the designs of politically valenced investigations, or at least counterbalance them against competing designs. If a presumptively open, self-referential peer-review process is influenced by political ideology, diversifying political ideology is warranted. If reviewers assume the role of political gatekeepers to decide what is fundable or publishable, science will be the perpetual loser.
Fewer than 10% of social scientists consider themselves politically conservative. Duarte et al. review evidence suggesting that diversifying political viewpoints within the academy could have salutary effects in several domains, including research. Here we address this claim.
A thought experiment
Suppose institutional review boards (IRBs), grant funders, and journal reviewers are influenced in their judgment of research by its political orientation, not just its scientific merit (Ceci et al. Reference Ceci, Peters and Plotkin1985). Suppose individuals are asked to evaluate a project; in one version the data support a liberal agenda and in another version they support a conservative agenda – despite the methods, procedures, and analyses being identical. Only the political implications of the results differ. Reviewers should not treat these studies differently if the studies are identical except for their political implications. If the studies have the identical design, analysis, literature framing, and so forth, they should be rated similarly in terms of their scientific rigor, appropriateness, and publishability.
Suppose journal reviewers are asked to review a study, a summary of which follows. After reading the full study, reviewers rate the rigor of the design, the importance/appropriateness of the question, the adequacy of the literature review/framing, and its publishability, by employing the typical considerations for the journal. Here are two politically opposite versions of the same study:
Version A of study
Is the Black–White IQ gap partly explained by economic disadvantage of Blacks? Researchers analyzed a nationally representative data set that included IQ and a host of demographic and environmental variables. They reported that controlling for parental educational attainment and income reduces the average Black–White group difference in IQ by roughly 90% or 13 IQ points, and, given that parents' socioeconomic status (SES) is an imperfect measure of environmental influences on intelligence, even this 13-point reduction is likely to underestimate the importance of parental social class on racial differences in intelligence. The researchers conclude that:
According to cultural theories of racial and ethnic differences in intelligence, as African American families advance up the socioeconomic ladder, their children should be less exposed to environmental deficits and therefore should do better and, by extension, close the gap separating the Black mean from the White mean. In fact, this is exactly what we found: The magnitude of the mean Black–White group difference in IQ for higher SES levels, when measured in standard deviations, is far smaller. . . . Matching Black and White children for the geographical areas of their homes, the schools they attend, and other finer-grained socioeconomic indicators reduces the mean group IQ difference still further, eliminating IQ gaps completely. Black children from the best areas and schools (those associated with the highest mean scores) average only slightly lower than do White children with similar socioeconomic indicators. This is an anomaly for genetic theory but is easily handled by environmental theories.
1. Rate the above study on its scientific methodology. Did the researchers' methodology allow a reasonable test of the research question? Use a 1 to 9 scale, in which 1 = extremely bad, 5 = neither good nor bad, and 9 = extremely good.
2. Rate the question asked by these researchers regarding its appropriateness for investigators to address – is this an appropriate question for researchers to ask? Use the same 1 to 9 scale.
Version B of study
In a flipped version, different reviewers rate the same study but with the results tilted in the opposite direction:
Is the Black–White IQ gap partly explained by economic disadvantage of Blacks? Researchers analyzed a nationally representative data set that included IQ and a host of demographic and environmental variables. They reported that controlling for parental educational attainment and income reduces the average Black–White group difference in IQ by only 30% or roughly 5 IQ points out of the 15-point gap that separates Blacks and Whites. Moreover, given that parents' socioeconomic status (SES) partly reflects their genetic differences in intelligence, even this 5-point reduction is likely to mask some genetic differences in parental intelligence. The researchers conclude the following:
According to cultural theories of racial and ethnic differences in intelligence, as self-defined Black groups advance up the socioeconomic ladder, their children should be less exposed to environmental deficits and therefore should do better and, by extension, close the gap separating the Black mean IQ from the White mean IQ. In fact, we found that the magnitude of the mean Black–White group difference in IQ for higher SES levels, when measured in standard deviations, is actually much larger than that found between lower SES groups. . . . Matching Black and White children for the geographical areas of their homes, the schools they attend, and other finer-grained socioeconomic indicators again reduces the mean group IQ difference but does not eliminate it. Black children from the best areas and schools (those schools associated with the highest mean scores) still average slightly lower IQs than do White children with the lowest socioeconomic indicators. This is an anomaly for the culture-only theory but is consistent with genetic theory through regression to the mean.
Conclusion
If liberal reviewers favor Version A and conservatives favor Version B, this would be evidence that extra-scientific considerations influence reviewers' calculus in ways that may be opaque to authors and editors. Reviewers of a version incongruous with their political orientation might view the study as “not asking the right question,” whereas their ideologically similar peers who review the opposite version might rate it as appropriate and relevant. One wonders whether a study of this type – if revealing that reviewers (including those who are regular journal reviewers and editors or grant-agency panelists) were biased against ideologically contrary interpretations – would be as readily publishable as its own counterpart arriving at the opposite conclusion.
A greater number of non-liberal scientists might help correct errors in scholarly reasoning and reframe the designs of politically valenced investigations, or at least counterbalance them against competing designs. If a presumptively open, self-referential peer-review process is influenced by political ideology, diversifying political ideology is warranted. If reviewers assume the role of political gatekeepers to decide what is fundable or publishable, science will be the perpetual loser.