Are Participants Who They Say They Are?

Landers and Behrend (Reference Landers and Behrend2015) describe how organizational samples are often seen as a “gold standard.” One probable reason for this is that in organizational samples, researchers (and journal editors and reviewers) can be reasonably certain that participants included in the study are actually working for that organization. Researchers may also be able to tell with some certainty what jobs participants hold, and in identified surveys, they may be able to verify which department or work group they are in, along with their demographics and other job-relevant information such as full-time status.Footnote ¹

Yet, with other data sampling sources—particularly crowdsourcing (including Mechanical Turk), snowball sampling, and online recruitment via social media, researchers (and journal editors and reviewers) must trust that the respondents are actually who they say they are (e.g., full-time workers). This is inherently important for industrial–organizational (I-O) psychologists, who are mainly interested in individuals who are currently working. Are we sure we are sampling individuals who are working for pay? And, if not, how meaningful are our findings regarding job satisfaction, work stress, and other work-related constructs?

There are also many cases when the validity of our findings rests on sampling specific populations. For instance, because laws around employment discrimination vary by country, researchers who study discrimination may desire to survey U.S. workers and not workers from other countries. As another example, a researcher interested in outcomes of job stress in older workers needs to be able to trust that workers in their samples are over a certain age. In these studies, note that researchers are not trying to generalize to the working population as a whole but to the specific population of interest, and therefore, attaining participants from that subpopulation is essential to the validity of the study.

By offering monetary incentives to respond to online studies, researchers may actually inadvertently motivate participants to identify themselves within a category of eligible participants when they are not. Some may argue that crowdsourcing sites such as Mechanical Turk (MTurk) pay so little that the money cannot possibly be incentivizing individuals to participate. There are two potential issues with this argument. First, for certain subsets of the population, small incentives may be attractive—particularly, people who are unemployed or people living in a nation where wages are much lower than they are in the United States. Indeed, anecdotal reports indicate that in some cases, MTurk workers who claim to be in the United States responded to questions about salary in rupees (Shoss, Reference Shoss2013). MTurk has taken recent steps to verify that individuals who say they are in the United States actually are, but there will arguably always be ways for individuals to get around location requirements online. Notably, working status, which is critical to many of our studies, is not verifiable. Second, low pay means that workers are likely motivated to complete as many tasks as possible in as little time as possible, leading to questions about data quality (which I expand on below).

This problem is not limited to MTurk and other crowdsourcing organizations; researchers may encounter similar problems when attempting to reach a specific population via snowball sampling and recruiting via social media when offering participation incentives. For example, in trying to reach full-time U.S. workers with chronic health conditions for a study on stigma and discrimination, my fellow researchers and I posted a link to an online survey on online sites specific to chronic health issues. Respondents were offered a small monetary gift card incentive. Shortly after posting, I discovered that the survey link had been shared on the site Reddit, and hundreds of anonymous responses had been entered from non-U.S. IP addresses within minutes.

There are some ways to avoid these potential pitfalls. First, and most obviously, researchers should be clear in their recruitment materials what the specific criteria are for participating and whether attaining an incentive for participating is contingent on meeting these criteria. Any researcher using online survey software should also consider using options within the software to help exclude individuals who don't qualify from participating. For instance, Qualtrics software has an option to screen people out based on geographical criteria using IP addresses. Researchers may also include a few screening questions on the first page of their surveys—if potential participants don't answer in a way that indicates they meet study criteria, they will be redirected to a message that states they are not qualified to complete the survey. Qualtrics also contains an option to only allow one survey response per IP address, which can help prevent individuals from completing a survey multiple times to obtain incentives.

Researchers using MTurk who want to study individuals with specific characteristics, including those who are currently working for pay, should consider deploying prescreening surveys to identify eligible individuals. Prescreening questions should be worded in such a way as to avoid revealing the characteristic the researcher is looking for. For example, a researcher wanting to survey older workers (e.g., workers over age 55) may first deploy a screening survey asking people to report their age in years. In MTurk, researchers can assign workers who are identified as having the desired characteristic a “qualification” based on their MTurk worker ID and set their full subsequent survey to only be able to be viewed by those qualified individuals.

When using snowball samples or social media, researchers may consider making their online surveys posted on social media identifiable based on e-mail address. Specifically, a researcher may post a recruitment advertisement for an incentivized study on social media but ask individuals to complete a brief online screening study for no pay first, in which they enter their e-mail addresses. Then, the researcher can create an individualized survey link and send it to just those participants who qualify. The researcher may choose options within their survey software to ensure that the individually e-mailed link will not work if e-mailed to another person or posted online and that the link will only allow one survey response. Further, researchers may consider alternative ways to attract potential participants besides monetary incentives. For instance, researchers might consider donating a specified amount of money to a charity for each survey completed or appeal to giving underexamined working populations a “voice.”

We agree with Landers and Behrend (Reference Landers and Behrend2015) that good data screening practices are essential (see also DeSimone, Harms, & DeSimone, Reference DeSimone, Harms and DeSimone2014). Researchers should look for red flags indicating possibly ineligible respondents, for example, job titles that aren't actually paying jobs (e.g., “full-time student”) when studying work-specific constructs. It would also be interesting and potentially instructive to include questions in an online survey asking participants about their motivation to participate (assuring them that receipt of their incentive will not be affected by the way they respond). For instance, are they completing the survey for the monetary incentive, to combat boredom at work, to do a favor for a friend, out of a genuine interest in the research topic, out of a desire to have their voice “heard,” and/or for other reasons? By more explicitly measuring participant motivations for responding, we may attempt to gain a better understanding of whether and how these motivations affect survey responses and our substantive findings.

Are Participants Paying Attention to the Study Instructions/Questions and Participating With Effort?

Beyond issues related to individuals’ motivation to decide to participate in a research study, researchers should pay attention to those regarding motivations to respond attentively and with effort. Recent evidence indicates that insufficient-effort responding (IER; Huang, Curran, Keeney, Poposki, & DeShon, Reference Huang, Curran, Keeney, Poposki and DeShon2012) results not simply in random error variance but in a systematic form of variance that may cause correlations to be inflated between variables for which average responses depart from the scale midpoints (Huang, Liu, & Bowling, Reference Huang, Liu and Bowling2014). This applies to a large number of variables of interest for I-O psychologists, including, for instance, absenteeism, workplace accidents/injuries, worker mistreatment, and discrimination. Therefore, IER should be of concern and steps should be taken to minimize its effects as noted below.

Research data obtained through MTurk have been found by some to be of good quality—for instance, Buhrmester, Kwang, and Gosling (Reference Buhrmester, Kwang and Gosling2011) found comparable levels of internal consistency (coefficient alpha) and test–retest reliability coefficients of their measures using MTurk when compared with data obtained using participants from more traditional sampling methodologies. MTurk workers are motivated to provide quality work in that they are held accountable for the quality of their work—their pay and MTurk reputation depend on the satisfaction of the requestors (those who post the study). This provides an incentive for workers to do a good job (not miss any response checks or skip an excessive number of questions). Despite this, I have found some evidence of IER in my own MTurk studies. For example in one survey of 990 MTurk workers who had been prescreened to be working full-time, 90 participants responded “yes” to the item “I have paid no attention to the questions in this survey.”

Many I-O psychology researchers also rely on student participants via their departments’ participant pools. In many departments, introductory psychology course students must complete a certain number of research credits as part of their course grade (or complete an alternative assignment, such as writing a paper). Students typically complete a screening study at the start of the semester, in which researchers may include questions in order to identify students with specific characteristics, making it easy for researchers to identify subpopulations for study recruitment. The main issue regarding motivation for participant pool students, therefore, is whether the students pay attention to the survey questions and put effort into their responses. It is not unreasonable to think that some students may be resentful of having to participate in studies to satisfy grade requirements and therefore may be unmotivated to respond with effort. It is also possible that some students choose online studies due to convenience (not having to go to a location to participate in-person) and out of a desire to complete the study as quickly as possible (particularly at the end of the semester). In my own research, I have found rates of IER in student samples that are consistently higher than those from MTurk—sometimes exceeding 30% of total respondents. Researchers may want to consider closing online surveys a few weeks prior to the end of the semester or at least carefully screening any data from individuals completing online studies at the very end of the semester.

We reiterate calls from Huang et al. (Reference Huang, Liu and Bowling2014) and others that survey researchers using any sample strategy should include survey measures to detect IER (e.g., instructed response or infrequency items) and remove participants who exhibit IER from data samples using predetermined cutoff criteria. Common method variance (CMV) potentially biasing relations between variables measured via single-source surveys is also a concern for researchers and can result when participants are not motivated to respond accurately (Podsakoff, MacKenzie, & Podsakoff, Reference Podsakoff, MacKenzie and Podsakoff2012). Researchers should consider the possible influence of motivation-related CMV on their findings before launching their surveys and include measures that may capture those sources. For instance, social desirability may be an issue for MTurk participants, who may be motivated to provide “good” answers to surveys in order to appease requesters and ensure receipt of payments. Social desirability and other measured and unmeasured sources of CMV can be addressed using latent variable techniques (see, for example, Williams & Anderson, Reference Williams and Anderson1994).

Of course, organizational samples are not immune to issues stemming from participant motivation. Individuals may feel coerced to participate in such studies, and/or may be wary of providing truthful responses out of fear the information will be used against them. There are also likely to be instances wherein participants intentionally misreport their demographics in an effort to disguise their identities—particularly in organizations if the culture is one of mistrust and they fear retaliation for negative responses. In addition to the practices already mentioned, it is important for researchers to assure anonymity or confidentiality as appropriate and attain buy-in from individuals through, for instance, involving employees in the survey design and educating them about the importance of accurate responding.

To be clear, I am not recommending that researchers abandon the use of any of the aforementioned sampling strategies. I am advocating that researchers consider issues related to participants’ motivations for engaging in research, attending to study instructions and questions, and putting effort into responding when considering various sampling strategies for their particular studies. By identifying potential motivational issues and the way they may affect their studies, appropriate measures can be taken to attempt to protect the validity of research findings.