Skip to main content Accessibility help
×
Hostname: page-component-74656f9884-6grmk Total loading time: 0 Render date: 2026-11-05T12:18:45.012Z Has data issue: false hasContentIssue false

6 - Data for Machine Learning

A Human-Centered Perspective of Crowdsourcing

from Part II - Doing Human-Centered Machine Learning

Published online by Cambridge University Press:  11 September 2026

By
Edith Law  and
Ming Yin
Edited by
Rebecca Fiebrink ,
Marco Gillies  and
Gonzalo Ramos
Rebecca Fiebrink
Affiliation:
University of the Arts London
Marco Gillies
Affiliation:
Goldsmiths, University of London
Gonzalo Ramos
Affiliation:
CNRS, Sorbonne Université
Get access

Summary

In the past decade, human computation and crowdsourcing research has shown that, by tapping into human intelligence over the Web, one can collect massive amounts of labeled data for machine learning. At the same time, research on AI ethics has highlighted the “bias-in, bias-out” problem in machine learning and the importance of paying attention to the nature of the data on which models are trained. In this chapter, we shine a spotlight on the humans who help generate data, the various psychological processes – biases, motivation, and identity – that can affect the data generation process, and how such influences can be amplified or reduced through thoughtful design of crowdsourcing processes and work practices.

Keywords

human computationcrowdsourcingannotationhuman–machine partnership

Information

Type
Chapter
Information
Human-Centered Machine Learning , pp. 123 - 150
Publisher: Cambridge University Press
Print publication year: 2026

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

List of cognitive biases. https://en.wikipedia.org/wiki/List_of_cognitive_biases. Wikipedia. Accessed: 2022-02-23.Google Scholar
Alonso, O. and Marchionini, G.. The Practice of Crowdsourcing. Morgan & Claypool Publishers, 2019.CrossRefGoogle Scholar
Anik, A. I. and Bunt, A.. Data centric explanations: Explaining training data of machine learning systems to promote transparency. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–13, 2021.CrossRefGoogle Scholar
Antin, J. and Shaw, A.. Social desirability bias and self reports of motivation: A study of amazon mechanical turk in the us and india. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 2925–2934, 2012.CrossRefGoogle Scholar
Ashby, F. G. and Maddox, W. T.. Relations between prototype, exemplar, and decision bound models of categorization. Journal of Mathematical Psychology, 37(3):372–400, 1993.CrossRefGoogle Scholar
Ashby, F. G. and Maddox, W. T.. Human category learning. Annual Review of Psychology, 56(1):149–178, 2005.CrossRefGoogle ScholarPubMed
Bandura, A.. Self efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2):191–215, 1977.CrossRefGoogle Scholar
Barbosa, N. M. and Chen, M.. Rehumanized crowdsourcing: A labeling framework addressing bias and ethics in machine learning. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., and Kraut, R. E.. Using social psychology to motivate contributions to online communities. In Proceedings of the ACM SIGCHI Conference on Computer Supported Cooperative Work (CSCW’04), pages 212–221, 2004.CrossRefGoogle Scholar
Bennett, C. L., Gleason, C., Scheuerman, M. K., Bigham, J. P., Guo, A., and To, A.. “It’s complicated”: Negotiating accessibility and (mis)representation in image descriptions of race, gender, and disability. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–19, 2021.CrossRefGoogle Scholar
Berg, J.. Income security in the on demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law & Policy Journal, 37(3):1–27, 2016.Google Scholar
Bhattacharya, N., Li, Q., and Gurari, D.. Why does a visual question have different answers? In Proceedings of the International Conference on Computer Vision (ICCV’19), pages 4270–4279, 2019.CrossRefGoogle Scholar
Biswas, A., Kolczynska, M., Rantanen, S., and Rozenshtein, P.. The role of ingroup bias and balanced data: A comparison of human and machine recidivism risk predictions. In Proceedings of the ACM Conference on Computing and Sustainable Societies (SIGCAS’20), pages 97–104, 2020.CrossRefGoogle Scholar
Bless, H. and Fiedler, K.. Social Cognition: How Individuals Construct Social Reality. Psychology Press, 2014.CrossRefGoogle Scholar
Bowker, G. and Star, S.. Sorting Things Out: Classification and Its Consequences. MIT Press, 1999.CrossRefGoogle Scholar
Bruner, J., Goodnow, J., and Austin, G.. A Study of Thinking. Wiley, 1956.Google Scholar
Burns, K., Hendricks, L. A., Darrell, T., and Rohrbach, A.. Women also snowboard: Overcoming bias in captioning models. In Proceedings of the European Conference on Computer Vision (ECCV’18), pages 793–811, 2018.CrossRefGoogle Scholar
Burton, M. A., Brady, E., Brewer, R., Neylan, C., Bigham, J. P., and Hurst, A.. Crowdsourcing subjective fashion advice using vizwiz: Challenges and opportunities. In Proceedings of the International ACM Conference on Computers and Accessibility (SIGACCESS’12), pages 135–142, 2012.CrossRefGoogle Scholar
Callison-Burch, C.. Crowdworkers: Aggregating information across turkers to help them find higher paying work. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’14), pages 8–9, 2014.CrossRefGoogle Scholar
Calvo, R., Kane, S. K., and Hurst, A.. Evaluating the accessibility of crowdsourcing tasks on amazon’s mechanical turk. In Proceedings of the International ACM Conference on Computers & Accessibility (SIGACCESS’14), pages 257–258, 2014.CrossRefGoogle Scholar
Chandler, D. and Kapelner, A.. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization, 90:123–133, 2013.CrossRefGoogle Scholar
Chang, J. C., Amershi, S., and Kamar, E.. Revolt: Collaborative crowdsourcing for labeling machine learning datasets. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’17), pages 2334–2346, 2017.CrossRefGoogle Scholar
Cheng, J. and Bernstein, M. S.. Flock: Hybrid crowd machine learning classifiers. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15), pages 600–611, 2015.CrossRefGoogle Scholar
Chiang, C. W., Kasunic, A., and Savage, S.. Crowd coach: Peer coaching for crowd workers’ skill growth. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–17, 2018.CrossRefGoogle Scholar
Christoforou, E., Barlas, P., and Otterbacher, J.. It’s about time: A view of crowdsourced data before and during the pandemic. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), 2021.CrossRefGoogle Scholar
Chu, Z. and Ma, J.. Learning from crowds by modeling common confusions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’21), pages 1–9, 2021.Google Scholar
Coetzee, D., Lim, S., Fox, A., Hartmann, B., and Hearst, M. A.. Structuring interactions for large scale synchronous peer learning. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15), pages 1139–1152, 2015.CrossRefGoogle Scholar
Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., and Eccles, D.. Predicting protein structures with a multiplayer online game. Nature, 466:756–760, 2010.CrossRefGoogle ScholarPubMed
Crisan, A. and Correll, M.. User ex machina: Simulation as a design probe in human in the loop text analytics. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–16, 2021.CrossRefGoogle Scholar
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., and Allahbakhsh, M.. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys, 51(1):1–40, 2018.CrossRefGoogle Scholar
Das-Smaal, E. A.. Biases in categorization. In Advances in Psychology, volume 68, pages 349–386. North-Holland, 1990.Google Scholar
Díaz, M., Kivlichan, I., Rosen, R., Baker, D., Amironesei, R., Prabhakaran, V., and Denton, E.. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT’22), pages 2342–2351, 2022.CrossRefGoogle Scholar
Difallah, D. E., Catasta, M., Demartini, G., and Cudré-Mauroux, P.. Scaling-up the crowd: Microtask pricing schemes for worker retention and latency improvement. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’14), pages 1–9, 2014.CrossRefGoogle Scholar
Dizaji, K. G. and Huang, H.. Sentiment analysis via deep hybrid textual crowd learning model. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 1–8, 2018.Google Scholar
Dontcheva, M., Morris, R. R., Brandt, J. R., and Gerber, E. M.. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’14), pages 3379–3388, 2014.CrossRefGoogle Scholar
Doroudi, S., Kamar, E., Brunskill, E., and Horvitz, E.. Toward a learning science for complex crowdsourcing tasks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 2623–2634, 2016.CrossRefGoogle Scholar
Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B.. Shepherding the crowd yields better work. In Proceedings of the ACM SIGCHI Conference on Computer Supported Cooperative Work (CSCW’12), pages 1013–1022, 2012.CrossRefGoogle Scholar
Dweck, C.. Mindset: The New Psychology of Success. Random House, 2006.Google Scholar
Eickhoff, C.. Cognitive biases in crowdsourcing. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM’18), pages 162–170, 2018.CrossRefGoogle Scholar
Ellsberg, D.. Risk, ambiguity, and the savage axioms. The Quarterly Journal of Economics, 75(4):643–669, 1961.CrossRefGoogle Scholar
Fan, S., Barlas, P., Christoforou, E., Otterbacher, J., Sadiq, S., and Demartini, G.. Socioeconomic diversity in human annotations. In Proceedings of the ACM Web Science Conference (WebSci’22), pages 98–109, 2022.CrossRefGoogle Scholar
Fisher, K., Counts, S., and Kittur, A.. Distributed sensemaking: Improving sensemaking by leveraging the efforts of previous users. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 247–256, 2012.CrossRefGoogle Scholar
Gadiraju, U., Demartini, G., Kawase, R., and Dietze, S.. Human beyond the machine: Challenges and opportunities of microtask crowdsourcing. IEEE Intelligent Systems, 30(4):81–85, 2015.CrossRefGoogle Scholar
Gadiraju, U., Fetahu, B., and Kawase, R.. Training workers for improving performance in crowdsourcing microtasks. In Proceedings of the European Conference on Technology Enhanced Learning (ECTEL’15), pages 100–114, 2015.CrossRefGoogle Scholar
Gao, Y. and Parameswaran, A.. Finish them! Pricing algorithms for human computation. VLDB Endowment, 7(14):1965–1976, 2014.CrossRefGoogle Scholar
Gemalmaz, M. A. and Yin, M.. Accounting for confirmation bias in crowdsourced label aggregation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’21), pages 1729–1735, 2021.CrossRefGoogle Scholar
Gold, M., Wojatzki, T., Horsmann, D., and Zesch, T.. Do women perceive hate differently: Examining the relationship between hate speech, gender, and agreement judgments. In Proceedings of the 14th Conference on Natural Language Processing (KONVENS’18), pages 110–120, 2018.Google Scholar
Goldstone, R., Kersten, A., and Paulo, R.. Concepts and categorization. In Comprehensive Handbook of Psychology, Volume 4: Experimental Psychology, pages 599–621. John Wiley & Sons, 2003.Google Scholar
Goldstone, R. L. and Barsalou, L. W.. Reuniting perception and conception. Cognition, 65(2):231–262, 1998.CrossRefGoogle ScholarPubMed
Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., and Bernstein, M. S.. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’22), pages 1–19, 2022.CrossRefGoogle Scholar
Goyal, N., Kivlichan, I. D., Rosen, R., and Vasserman, L.. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–28, 2022.CrossRefGoogle Scholar
Gray, M. L. and Suri, S.. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Houghton Mifflin Harcourt, 2019.Google Scholar
Gray, M. L., Suri, S., Ali, S. S., and Kulkarni, D.. The crowd is a collaborative network. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’16), pages 134–147, 2016.CrossRefGoogle Scholar
Griffiths, T. L., Kemp, C., and Tenenbaum, J. B.. Bayesian models of cognition. In The Cambridge Handbook of Computational Psychology, pages 59–100. Cambridge University Press, 2008.Google Scholar
Guo, A., Kamar, E., Vaughan, J. W., Wallach, H., and Morris, M. R.. Toward fairness in ai for people with disabilities: A research roadmap. ACM SIGACCESS Accessibility and Computing, (125):1–9, 2020.Google Scholar
Gurari, D. and Grauman, K.. Crowdverge: Predicting if people will agree on the answer to a visual question. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’17), pages 3511–3522, 2017.CrossRefGoogle Scholar
Hanrahan, B. V., Willamowski, J. K., Swaminathan, S., and Martin, D. B.. Turkbench: Rendering the market for turkers. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’15), pages 1613–1616, 2015.CrossRefGoogle Scholar
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., and Bigham, J. P.. A data driven analysis of workers’ earnings on amazon mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’18), pages 1–14, 2018.CrossRefGoogle Scholar
Hara, K., Adams, A., Milland, K., Savage, S., Hanrahan, B. V., Bigham, J. P., and Callison-Burch, C.. Worker demographics and earnings on amazon mechanical turk: An exploratory analysis. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6, 2019.CrossRefGoogle Scholar
Heimerl, K., Gawalt, B., Chen, K., Parikh, T., and Hartmann, B.. Community sourcing: Engaging local crowds to perform expert work via physical kiosks. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 1539–1548, 2012.CrossRefGoogle Scholar
Ho, C. J., Slivkins, A., Suri, S., and Vaughan, J. W.. Incentivizing high quality crowdwork. In Proceedings of the International Conference on World Wide Web (WWW’15), pages 419–429, 2015.CrossRefGoogle Scholar
Ho, C. J. and Vaughan, J.. Online task assignment in crowdsourcing markets. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’12), pages 45–51, 2012.CrossRefGoogle Scholar
Hohman, F., Head, A., Caruana, R., DeLine, R., and Drucker, S. M.. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–13, 2019.CrossRefGoogle Scholar
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., and Wallach, H.. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–16, 2019.CrossRefGoogle Scholar
Huang, J., He, X., Ma, X., Ren, Y., Zhao, T., Zeng, X., Li, H., and Chen, Y.. Sequential biases on subjective judgments: Evidence from face attractiveness and ringtone agreeableness judgment. PLoS ONE, 13(6):1–23, 2018.Google ScholarPubMed
Huang, S. W. and Fu, W. T.. Don’t hide in the crowd! Increasing social transparency between peer workers improves crowdsourcing outcomes. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’13), pages 621–630, 2013.CrossRefGoogle Scholar
Hube, C., Fetahu, B., and Gadiraju, U.. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Humphreys, G. W. and Forde, E. M.. Hierarchies, similarity, and interactivity in object recognition: “category specific” neuropsychological deficits. Behavioral and Brain Sciences, 24(3):453–476, 2001.CrossRefGoogle ScholarPubMed
Ikeda, K. and Bernstein, M. S.. Pay it backward: Pertask payments on crowdsourcing platforms reduce productivity. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 4111–4121, 2016.CrossRefGoogle Scholar
Ikeda, K. and Hoashi, K.. Crowdsourcing go: Effect of worker situation on mobile crowdsourcing performance. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’17), pages 1142–1153, 2017.CrossRefGoogle Scholar
Irani, L. C. and Silberman, M. S.. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’13), pages 611–620, 2013.CrossRefGoogle Scholar
Jacques, J. T. and Kristensson, P. O.. Crowdworker economics in the gig economy. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–10, 2019.CrossRefGoogle Scholar
Johnston, A., Moran, N., Musgrove, A., Fink, D., and Baillie, S.. Estimating species distributions from spatially biased citizen science data. Ecological Modelling, 422:108927, 2020.CrossRefGoogle Scholar
Joseph, J. E.. Functional neuroimaging studies of category specificity in object recognition: A critical review and meta-analysis. Cognitive, Affective & Behavioral Neuroscience, 1(2):119–136, 2001.CrossRefGoogle ScholarPubMed
Kahneman, D., Sibony, O., and Sunstein, C.. Noise: A Flaw in Human Judgment. Little, Brown and Company, 2022.Google Scholar
Kajino, H., Tsuboi, Y., and Kashima, H.. A convex formulation for learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’12), pages 73–79, 2012.CrossRefGoogle Scholar
Kamar, E., Kapoor, A., and Horvitz, E.. Identifying and accounting for task dependent bias in crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’15), pages 1–10, 2015.CrossRefGoogle Scholar
Kamel Boulos, M., Resch, B., Crowley, D., Breslin, J., Sohn, G., Burtner, R., Pioke, W., Jezierski, E., and Chuang, K. Y.. Crowdsourcing, citizen sensing and sensor web technologies for public and environmental health surveillance and crisis management. International Journal of Health Geographics, 10:67, 2011.CrossRefGoogle ScholarPubMed
Kelling, S., Fink, D., Hochachka, W., Iliff, M., Sullivan, B., Wood, C., Munson, M., and Riedewald, M.. Obtaining new insights for biodiversity conservation from broadscale citizen science data. Nature Proceedings, pages 1–17, 2009.CrossRefGoogle Scholar
Khatib, F., DiMaio, F., and Foldit Contenders Group. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18:1175–1177, 2011.CrossRefGoogle ScholarPubMed
Khosla, A., Zhou, T., Malisiewicz, T., Efros, A. A., and Torralba, A.. Undoing the damage of dataset bias. In Proceedings of the European Conference on Computer Vision (ECCV’12), pages 158–171, 2012.CrossRefGoogle Scholar
Komatsu, L. K.. Recent views of conceptual structure. Psychological Bulletin, 112(3):500–526, 1992.CrossRefGoogle Scholar
Kost, D., Fieseler, C., and Wong, S. I.. Finding meaning in a hopeless place? The construction of meaningfulness in digital microwork. Computers in Human Behavior, 82:101–110, 2018.CrossRefGoogle Scholar
La Barbera, D., Roitero, K., Demartini, G., Mizzaro, S., and Spina, D.. Crowdsourcing truthfulness: The impact of judgment scale and assessor bias. In Proceedings of the European Conference on Information Retrieval (ECIR’20), pages 207–214, 2020.CrossRefGoogle Scholar
Larke, L., Brooke, S., Davies, H., Margaryan, A., and Lehdonvirta, V.. Skills formation and skills matching in online platform work. In Proceedings of the British Sociological Association Annual Conference, page 25, 2019.Google Scholar
Law, E. and von Ahn, L.. Input agreement: A new mechanism for collecting data using human computation games. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’09), pages 1197–1206, 2009.CrossRefGoogle Scholar
Law, E. and von Ahn, L.. Human Computation. Morgan & Claypool Publishers, 2011.CrossRefGoogle Scholar
Law, E., Yin, M., Goh, J., Chen, K., Terry, M. A., and Gajos, K. Z.. Curiosity killed the cat, but makes crowdwork better. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 4098–4110, 2016.CrossRefGoogle Scholar
Law, P. M., Das, S., and Basole, R. C.. Comparing apples and oranges: Taxonomy and design of pairwise comparisons within tabular data. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Lease, M.. On quality control and machine learning in crowdsourcing. In Proceedings of the AAAI Workshop on Human Computation (HCOMP’11), pages 97–102, 2011.Google Scholar
Lee, D. J. L., Lo, J., Kim, M., and Paulos, E.. Crowdclass: Designing classification based citizen science learning modules. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’16), 2016.CrossRefGoogle Scholar
Lee, H., Lee, S., Kim, N., and Seo, J.. Jigsaw map: Connecting the past to the future by mapping historical textual cadasters. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 463–472, 2012.CrossRefGoogle Scholar
Lekschas, F., Ampanavos, S., Siangliulue, P., Pfister, H., and Gajos, K. Z.. Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–12, 2021.CrossRefGoogle Scholar
Little, G., Chilton, L. B., Goldman, M., and Miller, R. C.. Turk it: Human computation algorithms on mechanical turk. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST’10), pages 57–66, 2010.CrossRefGoogle Scholar
Lucas, C. and Griffiths, T.. Learning the form of causal relationships using hierarchical bayesian models. Cognitive Science, 34:113–14747, Jan 2010.CrossRefGoogle ScholarPubMed
Mamykina, L., Smyth, T. N., Dimond, J. P., and Gajos, K. Z.. Learning from the crowd: Observational learning in crowdsourcing communities. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 2635–2644, 2016.CrossRefGoogle Scholar
Margaryan, A.. Understanding crowdworkers’ learning practices. In The Internet, Policy, and Politics, Oxford Internet Institute, pages 1–10, 2016.Google Scholar
Margaryan, A.. Selfregulated learning in the crowd workplace. In SIG 14: Learning and Professional Development, 2018.Google Scholar
Margaryan, A., Charlton, T., and Gadiraju, U.. Learning and Skill Development in Online Platform Work. Technical report, Copenhagen Business School, 2020.Google Scholar
Mason, W. and Watts, D. J.. Financial incentives and the “performance of crowds”. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP’09), pages 77–85, 2009.CrossRefGoogle Scholar
Mathur, N., Baldwin, T., and Cohn, T.. Sequence effects in crowdsourced annotations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’17), pages 2860–2865, 2017.CrossRefGoogle Scholar
McInnis, B., Cosley, D., Nam, C., and Leshed, G.. Taking a hit: Designing around rejection, mistrust, risk, and workers’ experiences in amazon mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 2271–2282, 2016.CrossRefGoogle Scholar
Medin, D. L.. Concepts and conceptual structure. American Psychologist, 44(12):1469–1481, 1989.CrossRefGoogle ScholarPubMed
Medin, D. L. and Schaffer, M. M.. Context theory of classification learning. Psychological Review, 85(3):207–238, 1978.CrossRefGoogle Scholar
Mervis, C. B. and Rosch, E.. Categorization of natural objects. Annual Review of Psychology, 32(1):89–115, 1981.CrossRefGoogle Scholar
Mitra, T., Hutto, C., and Gilbert, E.. Comparing person and process centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’15), pages 1345–1354, 2015.CrossRefGoogle Scholar
Morris, M. R., Bigham, J. P., Brewer, R., Bragg, J., Kulkarni, A., Li, J., and Savage, S.. Subcontracting microwork. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’17), pages 1867–1876, 2017.CrossRefGoogle Scholar
Muller, M., Wolf, C. T., Andres, J., Desmond, M., Joshi, N. N., Ashktorab, Z., Sharma, A., Brimijoin, K., Pan, Q., Duesterwald, E., and Dugan, C.. Designing ground truth and the social life of labels. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), 2021.CrossRefGoogle Scholar
Murphy, G. L.. The Big Book of Concepts. MIT Press, 2002.CrossRefGoogle Scholar
Newell, E. and Ruths, D.. How one microtask affects another. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 3155–3166, 2016.CrossRefGoogle Scholar
Nguyen, A. T., Kharosekar, A., Lease, M., and Wallace, B. C.. An interpretable joint graphical model for fact checking from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 1–8, 2018.CrossRefGoogle Scholar
Otterbacher, J.. Social cues, social biases: stereotypes in annotations on people images. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’18), pages 136–144, 2018.CrossRefGoogle Scholar
Pan, S., Larson, K., Bradshaw, J., and Law, E.. Dynamic task allocation algorithm for hiring workers that learn. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’16), pages 3825–3831, 2016.Google Scholar
Radanovic, G., Singla, A., Krause, A., and Faltings, B.. Information gathering with peers: Submodular optimization with peerprediction constraints. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 1603–1610, 2018.CrossRefGoogle Scholar
Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L.. Learning from crowds. Journal of Machine Learning Research, 11(4):1297–1322, 2010.Google Scholar
Rechkemmer, A. and Yin, M.. Motivating novice crowd workers through goal setting: An investigation into the effects on complex crowdsourcing task training. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’20), pages 122–131, 2020.CrossRefGoogle Scholar
Rechkemmer, A. and Yin, M.. Understanding the microtask crowdsourcing experience for workers with disabilities: A comparative view. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1–30, 2022.CrossRefGoogle Scholar
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., and Bernstein, M. S.. Expert crowdsourcing with flash teams. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST’14), pages 75–85, 2014.CrossRefGoogle Scholar
Richardson, B., Garcia-Gathright, J., Way, S. F., Thom, J., and Cramer, H.. Towards fairness in practice: A practitioner oriented rubric for evaluating fair ml toolkits. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–13, 2021.CrossRefGoogle Scholar
Rodrigues, F. and Pereira, F. C.. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 1611–1618, 2018.CrossRefGoogle Scholar
Rokicki, M., Chelaru, S., Zerr, S., and Siersdorfer, S.. Competitive game designs for improving the cost effectiveness of crowdsourcing. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM’14), pages 1469–1478, 2014.CrossRefGoogle Scholar
Rosch, E.. Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104(3):192–233, 1975.CrossRefGoogle Scholar
Rotman, D., Preece, J., Hammock, J., Procita, K., Hansen, D., Parr, C., Lewis, D., and Jacobs, D.. Dynamic changes in motivation in collaborative citizen science projects. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’12), pages 217–226, 2012.CrossRefGoogle Scholar
Roy, Q., Zhang, F., and Vogel, D.. Automation accuracy is good, but high controllability may be better. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–8, 2019.CrossRefGoogle Scholar
Sachdeva, P. S., Barreto, R., von Vacano, C., and Kennedy, C. J.. Assessing annotator identity sensitivity via item response theory: A case study in a hate speech corpus. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT’22), pages 1585–1603, 2022.CrossRefGoogle Scholar
Saha, M., Saugstad, M., Maddali, H. T., Zeng, A., Holland, R., Bower, S., Dash, A., Chen, S., Li, A., Hara, K., and Froehlich, J.. Project sidewalk: A web based crowdsourcing tool for collecting sidewalk accessibility data at scale. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–14, 2019.CrossRefGoogle Scholar
Saito, S., Chiang, C. W., Savage, S., Nakano, T., Kobayashi, T., and Bigham, J.. Predicting the working time of microtasks based on workers’ perception of prediction errors. Human Computation, 6:192–219, 2019.CrossRefGoogle Scholar
Saito, S., Chiang, C. W., Savage, S., Nakano, T., Kobayashi, T., and Bigham, J. P.. Turkscanner: Predicting the hourly wage of microtasks. In Proceedings of the World Wide Web Conference (WWW’19), pages 3187–3193, 2019.CrossRefGoogle Scholar
Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., and Milland, K.. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’15), pages 1621–1630, 2015.CrossRefGoogle Scholar
Salehi, N., McCabe, A., Valentine, M., and Bernstein, M.. Huddler: Convening stable and familiar crowd teams despite unpredictable availability. In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW’17), pages 1700–1713, 2017.CrossRefGoogle Scholar
Sannon, S. and Cosley, D.. Privacy, power, and invisible labor on amazon mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Sap, M., Swayamdipta, S., Vianna, L., Zhou, X., Choi, Y., and Smith, N. A.. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL’22), pages 5884–5906, 2022.Google Scholar
Schaekermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., and Law, E.. Understanding expert disagreement in medical data analysis through structured adjudication. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–23, 2019.CrossRefGoogle Scholar
Schaekermann, M., Goh, J., Larson, K., and Law, E.. Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–19, 2018.CrossRefGoogle Scholar
Sen, S., Giesel, M. E., Gold, R., Hillmann, B., Lesicko, M., Naden, S., Russell, J., Wang, Z. K., and Hecht, B.. Turkers, scholars, “arafat” and “peace”: Cultural communities and algorithmic gold standards. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15), pages 826–838, 2015.CrossRefGoogle Scholar
Singer, Y. and Mittal, M.. Pricing mechanisms for crowdsourcing markets. In Proceedings of the International Conference on World Wide Web (WWW’13), pages 1157–1166, 2013.CrossRefGoogle Scholar
Singh, A., Wehkamp, S., and Gadiraju, U.. Signupcrowd: Using sign language as an input modality for microtask crowdsourcing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’22), pages 184–194, 2022.CrossRefGoogle Scholar
Singla, A. and Krause, A.. Incentives for privacy tradeoff in community sensing. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’13), pages 165–173, 2013.CrossRefGoogle Scholar
Solymosi, R., Bowers, K. J., and Fujiyama, T.. Crowdsourcing subjective perceptions of neighbourhood disorder: Interpreting bias in open data. The British Journal of Criminology, 58(4):944–967, 2017.CrossRefGoogle Scholar
Stets, J. E. and Burke, P. J.. Identity theory and social identity theory. Social Psychology Quarterly, 63(3):224–237, 2000.CrossRefGoogle Scholar
Suresh, H., Gomez, S. R., Nam, K. K., and Satyanarayan, A.. Beyond expertise and roles: A framework to characterize the stakeholders of interpretable machine learning and their needs. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–16, 2021.CrossRefGoogle Scholar
Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., and Bernstein, M. S.. Atelier: Repurposing expert crowdsourcing tasks as micro internships. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’16), pages 2645–2656, 2016.CrossRefGoogle Scholar
Swaminathan, S., Hara, K., and Bigham, J. P.. The crowd work accessibility problem. In Proceedings of the International Web for All Conference (W4A’17), pages 1–4, 2017.CrossRefGoogle Scholar
Tang, B., Clark, J. S., and Gelfand, A. E.. Modeling spatially biased citizen science effort through the eBird database. Environmental and Ecological Statistics, 28(3):609–630, 2021.CrossRefGoogle Scholar
Tian, T., Zhou, Y., and Zhu, J.. Selective verification strategy for learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 4147–4154, 2018.CrossRefGoogle Scholar
Tommasi, T., Patricia, N., Caputo, B., and Tuytelaars, T.. A deeper look at dataset bias. In Domain Adaptation in Computer Vision Applications, pages 37–55. Springer International, 2017.CrossRefGoogle Scholar
Toxtli, C., Suri, S., and Savage, S.. Quantifying the invisible labor in crowd work. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–26, 2021.CrossRefGoogle Scholar
Uzor, S., Jacques, J. T., Dudley, J. J., and Kristensson, P. O.. Investigating the accessibility of crowd work tasks on mechanical turk. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–14, 2021.CrossRefGoogle Scholar
Van Kleek, M., Smith, D., Stranders, R., and schraefel, m.. Twiage: A game for finding good advice on twitter. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 889–898, 2012.CrossRefGoogle Scholar
von Ahn, L. and Dabbish, L.. Labeling images with a computer game. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’04), pages 319–326, 2004.CrossRefGoogle Scholar
von Ahn, L. and Dabbish, L.. Designing games with a purpose. Communications of the ACM, 51(8):58–67, 2008.CrossRefGoogle Scholar
von Ahn, L., Maurer, B., McMillen, C., Abraham, D., and Blum, M.. reCAPTCHA: Human-based character recognition via web security measures. Science, 321(5895):1465–1468, 2008.CrossRefGoogle ScholarPubMed
Wang, L., Qin, G., Yang, D., Han, X., and Ma, X.. Geographic differential privacy for mobile crowd coverage maximization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 200–207, 2018.CrossRefGoogle Scholar
Wang, Q., Ming, Y., Jin, Z., Shen, Q., Liu, D., Smith, M. J., Veeramachaneni, K., and Qu, H.. Atmseer: Increasing transparency and controllability in automated machine learning. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Wang, Y., Papangelis, K., Saker, M., Lykourentzou, I., Khan, V. J., Chamberlain, A., and Grudin, J.. An examination of the work practices of crowd farms. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–14, 2021.CrossRefGoogle Scholar
Waseem, Z.. Are you a racist or am I seeing things? Annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, 2016.CrossRefGoogle Scholar
Welinder, P., Branson, S., Perona, P., and Belongie, S.. The multidimensional wisdom of crowds. In Proceedings of the Advances in Neural Information Processing Systems (NIPS’10), pages 1–9, 2010.Google Scholar
Whitehill, J., Wu, T., Bergsma, J., Movellan, J., and Ruvolo, P.. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proceedings of the Advances in Neural Information Processing Systems (NIPS’09), pages 1–9, 2009.Google Scholar
Wigfield, A. and Eccles, J. S.. Expectancy value theory of achievement motivation. Contemporary Educational Psychology, 25:68–81, 2000.CrossRefGoogle ScholarPubMed
Willett, W., Heer, J., and Agrawala, M.. Strategies for crowdsourcing social data analysis. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 227–236, 2012.CrossRefGoogle Scholar
Williams, A. C., Mark, G., Milland, K., Lank, E., and Law, E.. The perpetual work life of crowdworkers: How tooling practices increase fragmentation in crowdwork. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 2019.CrossRefGoogle Scholar
Wu, A., Xie, L., Lee, B., Wang, Y., Cui, W., and Qu, H.. Learning to automate chart layout configurations using crowdsourced paired comparison. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’21), pages 1–13, 2021.CrossRefGoogle Scholar
Yang, C. J., Grauman, K., and Gurari, D.. Visual question answer diversity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP’18), pages 184–192, 2018.CrossRefGoogle Scholar
Yang, Y., Zhang, M., Chen, W., Zhang, W., Wang, H., and Zhang, M.. Adversarial Learning for Chinese NER from Crowd Annotations. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’18), pages 1627–1634, 2018.CrossRefGoogle Scholar
Yin, M. and Chen, Y.. Bonus or Not? Learn to Reward in Crowdsourcing. In Proceedings of the International Conference on Artificial Intelligence (IJCAI’15), pages 201–207, 2015.Google Scholar
Yin, M., Chen, Y., and Sun, Y. A.. The effects of performance contingent financial incentives in online labor markets. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’13), pages 1191–1197, 2013.CrossRefGoogle Scholar
Yin, M., Gray, M. L., Suri, S., and Vaughan, J. W.. The communication network within the crowd. In Proceedings of the International Conference on World Wide Web (WWW’16), pages 1293–1303, 2016.CrossRefGoogle Scholar
Yin, M., Suri, S., and Gray, M. L.. Running out of time: The impact and value of flexibility in ondemand crowdwork. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’18), pages 1–11, 2018.CrossRefGoogle Scholar
Yin, M., Wortman Vaughan, J., and Wallach, H.. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’19), pages 1–12, 2019.CrossRefGoogle Scholar
Yu, L., André, P., Kittur, A., and Kraut, R.. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’14), pages 967–978, 2014.CrossRefGoogle Scholar
Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., and Horvitz, E.. Human computation tasks with global constraints. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pages 217–226, 2012.CrossRefGoogle Scholar
Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K. W.. Men also like shopping: Reducing gender bias amplification using corpus level constraints. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’17), pages 2979–2989, 2017.CrossRefGoogle Scholar
Zheng, Y., Wang, J., Li, G., Cheng, R., and Feng, J.. QASCA: A quality aware task assignment system for crowdsourcing applications. In Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD’15), pages 1031–1046, 2015.CrossRefGoogle Scholar
Zhu, H., Dow, S. P., Kraut, R. E., and Kittur, A.. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the ACM conference on Computer Supported Cooperative Work & Social Computing (CSCW’14), pages 1445–1455, 2014.CrossRefGoogle Scholar
Zhuang, H., Parameswaran, A., Roth, D., and Han, J.. Debiasing crowdsourced batches. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’15), pages 1593–1602, 2015.CrossRefGoogle Scholar
Zyskowski, K., Morris, M. R., Bigham, J. P., Gray, M. L., and Kane, S. K.. Accessible crowdwork? understanding the value in and challenge of microtask employment for people with disabilities. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15), pages 1682–1693, 2015.CrossRefGoogle Scholar

Accessibility standard: WCAG 2.2 AAA

Why this information is here

This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.

Accessibility Information

The HTML of this chapter complies with version 2.2 of the Web Content Accessibility Guidelines (WCAG), offering more comprehensive accessibility measures for a broad range of users and attains the highest (AAA) level of WCAG compliance, optimising the user experience by meeting the most extensive accessibility guidelines.

Content Navigation

Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.
Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.

Reading Order & Textual Equivalents

Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.
Short alternative textual descriptions
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.
Full alternative textual descriptions
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.
Visualised data also available as non-graphical data
You can access graphs or charts in a text or tabular format, so you are not excluded if you cannot process visual displays.

Visual Accessibility

Use of high contrast between text and background colour
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×