A Cursory Examination of IER Detection

Organizational researchers often inquire about the extent to which respondents provide quality data, especially when they are recruited from online sources (Barger, Behrend, Sharek, & Sinar, Reference Barger, Behrend, Sharek and Sinar2011). Although some evidence suggests no differences in data quality between crowdsourcing and student samples (Behrend, Sharek, Meade, & Wiebe, Reference Behrend, Sharek, Meade and Wiebe2011), no studies have directly compared organizational samples with other sources on IER. Perhaps researchers perceive organizational samples as being of higher quality and free of IER. Consistent with this speculation, Liu, Bowling, Huang, and Kent's (Reference Liu, Bowling, Huang and Kent2013) survey of Society for Industrial and Organizational Psychology members showed that IER screening was adopted less frequently in organizational samples compared with student and paid online samples.

However, IER is not uncommon in the organizational literature among both organizational and student samples. In organizational samples, Green and Stutzman (Reference Green and Stutzman1986) found that 57% of 292 mental health workers reported spending time on tasks they could not have performed at work (i.e., infrequency tasks/items). Green and Veres (Reference Green and Veres1990) found that 18% of 101 clerical workers and 61% of 118 police corporals indicated they had performed infrequency tasks that were not part of their jobs. Stetz, Button, and Quist (Reference Stetz, Button and Quist2012) reported that endorsement of six infrequency items ranged from 3% to 15% among 72 police officers. Hough, Eaton, Dunnette, Kamp, and McCloy (Reference Hough, Eaton, Dunnette, Kamp and McCloy1990) chose a more stringent cutoff for carelessness (six out of eight endorsements on infrequency items) and identified 7.4% careless respondents in 9,152 military personnel. Meanwhile, in undergraduate student samples, Meade and Craig (Reference Meade and Craig2012) reported careless responding among 10%–12% of students in a lengthy survey. Chiaburu, Huang, Hutchins, and Gardner (Reference Chiaburu, Huang, Hutchins and Gardner2014) found that 15% of 133 undergraduate students devoted only 2 seconds per item in a low-stakes survey. Unfortunately, we could not find any published reports that directly address IER occurrence in crowdsourcing or other online samples.

Therefore, there is reason to suspect that respondents directly recruited from organizations may not always be attentive to survey measures. IER is likely present in datasets from a wide range of sample sources. Given the cursory nature of the review presented above, we attempted a more systematic examination of current IER screening practices in highly impactful journals in organizational research.

A Survey of IER Detection in Three Major Journals

Evidence-based strategies to detect and screen IER (e.g., infrequency, inconsistency, response pattern, response time, and multivariate outliers) have recently been advanced (Huang et al., Reference Huang, Curran, Keeney, Poposki and DeShon2012; Huang, Bowling, Liu, & Li, Reference Huang, Bowling, Liu and Li2015; Meade & Craig, Reference Meade and Craig2012). To compare quality control practices between sample sources, we examined data screening procedures pertaining to IER detection in recent articles published in the Journal of Applied Psychology, Journal of Management, and Academy of Management Journal. Our goals were to document common practices on quality control against IER and investigate whether more screening occurred for nonorganizational samples relative to organizational samples. We selected articles published between 2012 and 2014, following the release of two IER-related empirical articles in 2012 (Huang et al., Reference Huang, Curran, Keeney, Poposki and DeShon2012; Meade & Craig, Reference Meade and Craig2012). The search resulted in 252 samples from Journal of Applied Psychology, 104 samples from Journal of Management, and 107 samples from Academy of Management Journal.

Based on our analysis, reporting practices to prevent or screen IER are very rare in these top-tier journal articles. Most studies utilize completion of focal measures (i.e., missing data) as a proxy for data quality to determine inclusion of cases in analyses. However, we were surprised to find that only three studies from the targeted journals from 2012 to 2014 reported screening for IER. Johnson and Allen (Reference Johnson and Allen2013) and Pieterse, Van Knippenberg, and Van Dierendonck (Reference Pieterse, Van Knippenberg and Van Dierendonck2012) utilized multivariate outliers to remove individuals whose responses constituted extremely divergent combinations of scores. Bagger and Li (Reference Bagger and Li2014) examined the consistency of responses to demographic items to ensure data quality. These three studies utilized an online panel (Johnson & Allen, Reference Johnson and Allen2013), a student sample (Pieterse et al., Reference Pieterse, Van Knippenberg and Van Dierendonck2012), and a snowball sample of working adults (Bagger & Li, Reference Bagger and Li2014). No studies using organizational samples reported IER detection or screening. Overall, very few studies reported detecting or screening IER, regardless of sample source. An exception is experimental studies, where manipulation checks are usually applied to screen participants who did not attend to the experimental stimuli. In our survey of articles, experimental manipulation checks were reported as a proxy of IER screening with both organizational and student samples (King et al., Reference King, Botsford, Hebl, Kazama, Dawson and Perkins2012).

Two explanations may be offered for the limited application of IER detection and screening to the organizational literature. First, IER may not occur frequently. However, the existence of IER within organizational and student sample sources has been documented in the literature (as reviewed earlier) and highlighted in recent Society for Industrial and Organizational Psychology presentations (e.g., Nieminen, Kotrba, Denison, & Carter, Reference Nieminen, Kotrba, Denison and Carter2014). To capture the presence of IER across sample sources, we examined patterns of IER in our own datasets using samples of undergraduate students and Amazon Mechanical Turk (MTurk) workers. Responses to three extreme infrequency items (“I have never used a computer,” “I eat cement occasionally,” and “I can teleport across time and space”) were examined, where failure to disagree with these items indicated severe IER. Among three college student samples (Ns = 402, 1,046, and 1,250), IER frequencies for individual items ranged from 3.7% to 17.5%. Among four samples from MTurk (Ns = 325, 389, 415, and 428), item-level IER frequencies ranged from 2.5% to 11.2%. We conclude that most samples, regardless of source, are subject to the threat of IER. However, because different objectives, instructions, IER items, and survey measures were adopted across our datasets and in the publications reported earlier, we caution readers against making inferential IER comparisons between sample sources at this time.

This leads to a second possible explanation for the lack of IER detection and screening. Researchers, reviewers, and/or editors may overlook response effort as a critical determinant of validity or believe that it is a less important methodological factor compared with sample source, response rate, and other sample characteristics. Subsequently, insufficient attention to participant motivation may undermine the internal validity of results. It is important that researchers adopt an a priori IER screening method based on empirical recommendations (e.g., Huang et al., Reference Huang, Curran, Keeney, Poposki and DeShon2012; Meade & Craig, Reference Meade and Craig2012) and that reviewers and editors exert collective effort to create a standard for IER screening and reporting in the literature. This leads to our call for the advancement of evidence-based IER detection, screening, and prevention.

Call for Action

Organizational researchers have not paid sufficient attention to participant response effort nor consistently documented their treatment of IER. To better inform organizational researchers about IER, we call for more rigorous investigations to determine whether participant motivation and effort during surveys differ between convenience sampling sources (e.g., organizational, online, student samples). We provide three directions for future research on survey data quality assessment including the detection, screening, and prevention of IER. We recommend that these approaches be applied across sample sources.