Crowdlabor in Political Science Research

Crowdsourcing has become a buzz word. Surowiecki’s The Wisdom of Crowds, Tapscott and Williams’ Wikinomics, and Benkler’s The Wealth of Networks, among others, have glamorized crowdsourcing. Its promoters have high hopes for “the age of the crowd”: ^{Footnote 9} “A new economic democracy is emerging in which we all have a lead role,” Tapscott and Williams write. ^{Footnote 10} But what exactly is crowdsourcing? The trendy term conceals a variety of manifestly different types of labor networks. ^{Footnote 11} One form of crowdsourcing that is familiar to all of us is the likes of Wikipedia, an open-source, non-commercial platform. Wikipedia relies on volunteers whose labor ultimately creates a free resource for the commons. Much less lofty is the commercial crowdsourcing platform. In contrast to Wikipedia, crowdsourced labor is incentivized by pay rather than an aspiration to contribute to a greater cause. Unlike volunteers on Wikipedia, crowdworkers are required to hand over their intellectual property rights and consequently have no right to use their contributions. As will become apparent later, workers on commercial platforms are generally unaware of the final product of their labor and rarely have any use for it. The latter face of crowdsourcing labor markets, or crowdlabor markets for short, has shown itself in political science research.

Crowdlabor platforms are digital labor markets. ^{Footnote 12} Companies take a function hitherto performed by traditional employees, break it into small tasks, or “micro tasks,” and distribute these tasks to a large group of people online in the form of an open call. Jeff Howe coined the term in 2006:

The crowdlabor platform most often used for political science research is Amazon’s Mechanical Turk (AMT). ^{Footnote 14} It was established in 2005 as an online labor market. AMT serves as an intermediary between U.S.-based contractors (Requesters) and workers (Turkers) from across the globe. Crowdworkers on AMT complete “human intelligence tasks” by, for example, translating or manually coding text, audio, and visual materials.

Most studies in political science have used crowdlabor platforms, like AMT, to recruit participants in online experiments. ^{Footnote 15} AMT has also been used for the content analysis of text, where Turkers manually code blog posts, news stories, or party manifestos. ^{Footnote 16} Grimmer and Stewart recommend using crowdworkers to scrape text from websites when automated scraping is not an option. ^{Footnote 17} Most of these studies are in the subfield of political behavior and have been published in some of our leading journals, including World Politics, the American Political Science Review, the Journal of Politics, Political Analysis, and the American Journal of Political Science. AMT is becoming normalized as a research tool in our discipline. ^{Footnote 18}

Researchers have found crowdlabor platforms attractive both for their low cost and convenience. ^{Footnote 19} Crowdworkers code data manually for only a fraction of the cost of university research assistants, and there is no need to hire or supervise staff. Fielding survey experiments is less costly on crowdlabor platforms than it is through traditional survey companies, and one need not leave the premises of one’s office. Moreover, data collection is performed speedily with a global workforce around the clock. In a matter of hours, a researcher can administer a survey to thousands of individuals for a negligible cost. Requesters are able to reject unsatisfactory work, without having to pay for it, which is difficult to do in a traditional labor context. ^{Footnote 20}

Analyses of AMT as a research tool have also sprung up in our discipline, yet thus far they have been limited to technical issues. As far as research in political science goes, the great controversy surrounding the use of crowdlabor platforms revolves around sample representativeness, or the adequacy of using AMT samples to generalize findings to the broader U.S. population. Hence a number of studies have analyzed AMT demographics and have replicated existing studies with AMT samples. ^{Footnote 21} This stands in contrast to other disciplines, including economics, psychology, business, and language processing, where the ethical dilemmas and the scientific implications of relying on crowdlabor have received some attention. ^{Footnote 22} Researchers in these fields have brought to light concerns about ethical pay and the alienating nature of digital piece work, among others. Economists have considered the implications of moving toward an impersonal “bench science” characterized by limited contact with either colleagues or research participants. ^{Footnote 23} Political scientists in contrast have been silent on these and other ethical issues in using crowdsourcing labor markets in research.

The legal status of crowdlabor

The crowdlabor market currently falls outside the scope of federal and state laws. ^{Footnote 24} Crowdworkers are not covered by minimum wage, overtime laws, OSHA (health and safety), Title VIII (employment discrimination), FMLA (family leave), or the NLRA (union organizing). ^{Footnote 25} Taking advantage of the legal grey zone between statutory employees and independent contractors, crowdlabor platforms like AMT explicitly prevent permanent contracts between crowdworkers and requesters and prohibit collective bargaining, thus stripping workers of employment security and labor representation. AMT operates in “a state of legal exception.” ^{Footnote 26}

The thorny legal issue of whether crowdworkers ought to be considered independent contractors or statutory employees—and whether they ought to be paid minimum wage—has been taken up by several legal experts. ^{Footnote 27} Felstiner reviews seven conditions that qualify a worker as either employee or contractor under the FLSA. He weighs three of the seven factors in favor of employee status, one in favor of independent contractor, and the remaining three factors being conditional on the type of task performed. Cherry argues that FLSA should be amended or interpreted by the courts of appeal to require the payment of minimum wages to crowdworkers. ^{Footnote 28}

In 2013, crowdworkers filed a class-action law suit in California, alleging that they ought to have been paid the minimum wage under FLSA for work performed on the digital labor platform CrowdFlower. Otey v. CrowdFlower settled in Otey’s favor in 2014, and the law suit was dismissed. Although initially the case promised to give an answer to how the courts interpret crowdwork, the federal district court ultimately did not rule on questions pertaining to minimum wage or the contractor/employee status of crowdworkers. At the time of writing, crowdwork remains in a legal grey zone, and crowdworkers continue short of labor rights and protections.

Owing to a lack of legislation on digital labor markets, employer-employee relationships on crowdlabor platforms are profoundly asymmetrical in ways that favor employers. One aspect of this asymmetry is financial. While requesters do not pay taxes on employees, crowdworkers are required by law to report earned income and are taxed at the high rates set for independent contractors. ^{Footnote 29} Cherry points to the irony of this practice; while income earned on AMT is taxable under the law, the law does not in turn enforce a minimum wage for crowdworkers. ^{Footnote 30} Consider further that employers can reject work they deem unsatisfactory without having to pay for it. ^{Footnote 31} Not only do crowdworkers lack guarantee of payment for work done properly, but they have no recourse via official channels to redress employer wrongdoing. ^{Footnote 32} This makes crowdworkers especially vulnerable to employer wrongdoing. Finally, consider that crowdworkers are expressly prohibited from organizing collectively or from becoming contracted directly by requesters. This spells employment insecurity and no labor representation, while it simultaneously preempts any potential attempt to improve the lot of the crowdworker.

Some Economic and Political Ramifications of the Digital Labor Market

Crowdlabor may be added to the growing list of precarious forms of labor, along with temporary and part-time employment, unpaid internships, subcontracted employment, and false self-employment. ^{Footnote 33} Its susceptibility to labor arbitrage and its temporary, subcontracted nature links it unmistakably to extant forms of atypical labor. ^{Footnote 34} Like its siblings, crowdlabor at once relies on a pool of economically vulnerable individuals and reinforces their precarity. As Scholz writes, “These are new forms of labor but old forms of exploitation.” ^{Footnote 35} The cognitive, fragmented, and impersonal nature of crowdlabor, however, adds a distinct flavor to exploitation on digital labor markets. Anonymization, a gradual deskilling of the labor force and the continuous expansion of precarious forms of labor are some of its characteristic long-term effects. Scholars have taken note of the ongoing reconfiguration of labor from physical and factory-based onto “intellectual, immaterial, and communicative” forms of labor. ^{Footnote 36} I discuss in turn the economic and political implications of this reconfiguration.

One direct result of the digitalizing of labor is its anonymization. Foregoing the physical worker site, digital labor markets have made the employer-employee relationships anonymous and impersonal. There is normally no direct contact between employers and employees on crowdlabor platforms, and when there is, personal contact is limited. As a result, employers treat workers as something akin to “interchangeable replacements for computational processes.” ^{Footnote 37} The limited direct contact in the digital labor market and its anonymity facilitate paying crowdworkers less than minimum wage, or not paying at all, while brushing aside what consequences this might have on workers’ lives.

Another implication of digital work is the potential deskilling of its labor force in the long run. Under the crowdlabor model of work, a small group of individuals do the “big thinking,” break down work into micro tasks, and distribute the tasks to an assembly of crowdworkers. As a result of the division of labor, crowdworkers are nearly always removed from the final product of their work. They have insufficient information on which to base judgement of the cause to which their labor contributes. ^{Footnote 38} Far from offering opportunities for career advancement or training, digital labor platforms subject workers to dull and repetitive tasks. ^{Footnote 39} Two-thirds of AMT workers express no interest in the work they do as a result of micro tasks being redundant. ^{Footnote 40} Platforms for cognitive piecework do not aid individuals in reaching their full potential. ^{Footnote 41} Over the long run, the division of labor on digital labor could create structural labor inequalities where a select few manage a production line of a great many cognitive laborers.

Some observers have suggested that crowdworkers engage in digital piecework as a hobby, as “an expenditure of cognitive surplus.” ^{Footnote 42} This myth can be easily debunked by statistics on the AMT population. Most crowdlaborers report that they complete tasks for money. Roughly 20 percent of AMT workers use the platform as a primary source of income, and another 50 percent use it as a secondary source of income. More tellingly, 80 percent of the work is performed by the 20 percent of Turkers who spend more than fifteen hours per week on the platform. ^{Footnote 43} That is, most crowdworkers who take part in research studies will do so to make ends meet.

In their accounts, most crowdworkers report feeling that their labor is undervalued. One survey found that 64 percent of AMT workers felt exploited, yet 95 percent of those surveyed planned to continue working on the platform. ^{Footnote 44} Personal accounts indicate that crowdworkers keep returning to AMT due to the scarcity of better economic opportunities.

• I do it to pay my bills which includes rent and diapers for my kids until I find work again.

• I have a degree in accounting and cannot find a real job, so to keep myself off of the street I work 60 hours or more a week here on AMT just to make $150–$200. That is far below minimum wage, but it makes the difference between making my rent and living in a tent.

• I have been doing online surveys, freelance writing, and AMT to try to make the most money I can. I don’t make much but when you literally have no savings and no income you take what you can get.

• I realize I have a choice to work or not work on AMT, but that means I would also not need to make the choice to eat or not eat, pay bills or not pay bills, etc. ^{Footnote 45}

In great part crowdlabor platforms rely on a pool of individuals in precarious living conditions who lack decent income or the prospect thereof—and simultaneously perpetuate these workers’ vulnerable economic situation. The absence of legal regulation on digital labor and the accompanying employment insecurity effectively make crowdlabor markets synonymous with precarity. The demographic composition of AMT indicates further the platform’s disproportionate reliance on members of traditionally underprivileged groups. The number of workers based outside the United States has grown dramatically since the launching of AMT: Indian workers represented 10 percent of the labor force in 2008 and 50 percent in May, 2010. ^{Footnote 46} Women are also disproportionately represented. One study finds that women constitute 70 percent of AMT workers and 75 percent of the subject pool in some studies using AMT participants. ^{Footnote 47} As in other types of precarious employment, women are more vulnerable than men in unregulated labor markets. ^{Footnote 48} The precarious labor conditions on crowdlabor platforms place a further burden on underprivileged groups in society, thereby perpetuating existing socioeconomic inequalities. ^{Footnote 49}

Insecurity related to employment has a particularly demobilizing effect on workers as citizens. Temporary part-time employment, like that on AMT, has been linked to subjective job insecurity. ^{Footnote 50} When it comes to political engagement, experiencing employment insecurity seems to have an adverse effect, beyond that of sheer income levels. At least one study documents a demobilizing effect of income insecurity, noting that it is the most insecure citizens, not the poorest ones, who are most likely to abstain from active citizenship. ^{Footnote 51} It is interesting to note further that there exist parallels between disempowered workers and disempowered citizens. When it comes to working for a temp agency, for example, taking away decision-making powers from workers disempowers them as citizens. ^{Footnote 52} A parallel logic may well apply to crowdworkers who are at the receiving end of an assembly line. They have no say over how a project is divvied up into micro tasks and little knowledge of what the project may ultimately be.

While the jury is still out on the political ramifications of employment in the digital economy, precarious labor conditions more generally have been demonstrated to exert undesirable effects on political representation. Extant research suggests that the economic disadvantage of crowdworkers would likely translate into a political one. On one hand, workers in insecure employment are not represented by labor unions; and in the case of crowdworkers, they are expressly prohibited from organizing collectively. On the other hand, their interests are rarely reflected in the platforms of left-of-center parties; this is in part due to their high abstention rates in election and in part to their conflicting interests with the left’s core constituencies. ^{Footnote 53} Denied access to either labor unions or political parties, precarious workers have both channels to representation effective blocked.

Implications for Scientific Research Practice

When we as researchers field data collection on AMT, we assent and contribute to digital labor markets in which precarious forms of labor are the norm and which disempower workers as citizens. Even though most would agree that participating in a research experiment does not constitute actual labor, by fielding experiments through AMT we nevertheless support a legal and economic model based on employment insecurity, financial and legal asymmetry between employers and employees, and absence of labor representation. I therefore urge researchers, as well as institutional review boards, funding agencies, publishers, and scholarly associations, to acknowledge and address the unique set of challenges underpinning the use of crowdlabor markets in research.

Perhaps some readers may deduce from the discussion here that paying higher wages to crowdworkers is the solution out of this quandary. I wish to argue that this is not the case. Low wages—estimated to be somewhere in the range of $1.25 to $1.38 per hour on AMT ^{Footnote 54} —are only part of the problem on crowdlabor platforms. Even if academic contractors were to pay fair wages, the labor environment would continue unregulated, and crowdworkers would remain without job contracts, health benefits, or protection against employer wrongdoings. As I have argued, it is the legal and economic model that is at stake here. Paying nominally higher wages would not greatly alter the reality of precarious employment.

Over the long run, the extended use of crowdlabor platforms may well establish a below-minimum wage standard for remunerating research-related work. It may become increasingly difficult to obtain research funding for projects proposing fair wages and regular contracts to research assistants or what are deemed overly high compensation rates for study participants. ^{Footnote 55} If digital labor platforms become the norm for recruiting respondents for survey and experimental studies, researchers would be hard pressed to justify the contracting of professional survey companies.

As institutional review boards struggle to keep up with technological innovation, current human subjects’ regulation does not adequately address the unique labor issues raised by digital platforms. ^{Footnote 56} Some university officials have expressed concern that studies on AMT would likely not pass institutional review. ^{Footnote 57} At least one U.S. university waives its policy of $15-per-hour minimum rate for participation in research out of fear that such a high compensation—given the established low rates on AMT—would constitute subject coercion. ^{Footnote 58} Additional human subjects’ issues concern breached digital privacy on crowdlabor platforms. ^{Footnote 59} At the time of writing, there are no standard IRB guidelines for research conducted with human subjects on crowdlabor platforms.

Ethical alternatives to commercial platforms exist, and scholars in our field would do well to contribute to building and solidifying such initiatives. Nonproprietary crowdsourcing platforms, such as the Citizen Science Alliance, have emerged with the aim to generate crowdsourced data and foster scientific discovery and public engagement and understanding thereof. Not-for-profit crowdlabor platforms, like Samasource and MobileWorks, have also surfaced. Their goal is to empower underprivileged groups in rural and developing contexts and in the United States, respectively. CloudCrowd is an alternative commercial site that improves on many of AMT’s features but still falls short of offering legal rights and protection to crowdworkers. ^{Footnote 60} Prolific Academic is a crowdlabor platform launched recently by researchers at the University of Oxford and the University of Sheffield. The site sets a minimum wage of 7.50 USD, and workers are able to rate researchers. ^{Footnote 61} Each of these alternatives to commercial platforms begins to redress some of the deficiencies of commercial platforms like AMT.

These reflections ought to bring awareness to the unique challenges posed by data and services generated through crowdlabor. It is my hope that they open up a conversation in the field about the ways in which we can address these challenges. Building and contributing to ethical alternatives to commercial crowdsourcing platforms is one way to go forward. Relying on convenience samples for pretests or pulling resources to recruit nationally representative samples are other low-cost alternatives. ^{Footnote 62} Professional associations and IRBs can do a great deal to address these challenges by putting crowdlabor on their agendas and by devising a set of best practices for researchers to employ. AMT workers have devised their own guidelines for academic requesters, including on issues such as minimum wage, conditions for rejecting work, and communication with workers. ^{Footnote 63} We would do well to take these guidelines seriously and support AMT workers, and digital laborers more widely, in their struggle for fair labor conditions. As journal reviewers and editors, we can further raise questions about human subjects when considering research that relies on crowdlabor and thereby encourage authors to take these issues to heart. What we cannot do is trade cheap, convenient samples for fair labor standards and sweep the economic and political implications of this trade-off under the rug.