Introduction
The “free labor” of generating data via digital platformsFootnote 1 is now well-known as a necessity for artificial intelligence (AI) production. Recent research has also shown the importance of “data work,” which cleans, labels, and verifies this data.Footnote 2 But in between these two essential moments in the AI production pipeline, there is another moment: scraping, or the process of downloading publicly available data from the web at scale such that it can be turned into training data. The essential contributions of web scraping to the production of machine learning (ML) training datasets have not been included in critical research on AI labor. This chapter shows that scraping is a distinct category of software labor and argues that understanding it is important for the political economy of AI. The economic utility of data is dependent on its transformation from web data into datasets, and this transformation is achieved by the labor of web scraping.
The chapter examines the labor involved in both the production and use of scraping software. We do this by providing a critical historical assessment of scraping dating back to the 1990s. We trace the development of scraping from a technical requirement for measuring and indexing the web, which underpinned the functioning of early communication software, to a commercialized practice and category of software for online advertising, which was subsequently adapted for contemporary use in the production of large language models (LLMs). This history is intended to highlight how the development of what we term the labor of web scraping is concomitant with the development of shifting regimes of data availability and the services and commodities desired to be produced from data. The specific qualities of the labor of web scraping evolved alongside that of the data ecosystem at large—from the volunteer and research-oriented non-profit early development of scraping software, to specialized scraping software production, on to the present moment of hyperscale scraping in view of the required data for the development of LLMs.
We argue that this is an important area of further study as the scraping of web data constitutes a preliminary step in the overall production of digital goods and further that it too should be considered as a form of data work.Footnote 3 We argue that the increased barriers to the practice (both legal and technical), resulting from the historical evolution of the web and different regimes of data collection, frame the current labor ecosystem of web scraping, which, despite being highly automated, continuously demands new kinds of human expertise—and labor—to successfully collect data.
We argue that the regimes of data utility, accessibility, and availability were party to specific historical circumstances which configure the possibilities and requirements for data scraping. These culminate in the current state of the web scraping ecosystem and frame the necessity for web scraping labor, defined as a perpetual cat-and-mouse game between scrapers and the shifting modes of data accessibility. These phenomena also complicate the discourse on purported automation of software in the specific case of data collection, as they show how automation itself is dependent on labor surrounding automated processes.
Our argument is based on the analysis of publicly available documents about scraping: industry-specific blogs, discussion boards, market reports, and conference presentations. We acknowledge at the outset that this selection of data sources is limited by the omission of direct engagement with web scraping workers and reliance on industry discourse. Our justification for this is the incremental nature of research. The first author is currently engaged in empirical research with the said scraping workers, and this paper presents an initial engagement with scraping as labor, in terms of its historical context. We offer this contribution as a step toward further empirical research. As it will become clear, scraping is mired in obscurity for political-economic reasons. But the current sources of information available at the time, consisting largely of market reports whose bias toward presenting the need for scraping should not be ignored, do shed light on the economic relevance of the data collection ecosystem and allow us to provide a macro-level political-economy snapshot of the conditions on which the labor of web scraping develops.
In view of the relative lack of authoritative and reliable information on the economic and labor aspects of web scraping, we find it worthwhile to expand on the history of web scraping.
In the following sections, first, we review the ML labor process and indicate where scraping fits in. Then, we consider relevant research on free labor and data work and show how neither adequately grapples with scraping. Next, we introduce the technical practices which constitute the scraping labor process in general and discuss the little that is known about the social composition of scraping labor. Next, we detail three historical eras of scraping (infrastructural, commercial, and AI) in terms of their social and technical composition and broader political-economic contexts. Finally, we discuss the distinctive aspects of scraping labor today.
The machine learning labor process
ML is an approach to AI where learning algorithms are used to assemble other algorithms, called models, from datasets, rather than by manually coding a program from the ground up.Footnote 4 We focus here on the subset of ML systems known as LLMs. LLMs are generative AI models which operate in the modality of text. They function by predicting “the likelihood of a token (character, word, or string) given either its preceding context or … its surrounding context”Footnote 5 and can be used to output textual results given a directing prompt. LLMs are poised, according to their advocates, to increase efficiency in a slew of different productive sectors. Thus far, LLMs appear to be having the greatest impact in software production.Footnote 6 Their future cannot be fully anticipated beyond asserting that it seems likely that, despite their flaws, LLMs and generative AI models will become an increasingly favored means of production for capital.Footnote 7
Like all ML models, LLMs are created from vast amounts of data, scraped mostly from the open web.Footnote 8 Such data are primarily a “recorded abstraction” of human actions.Footnote 9 In the LLM production, the web serves as a corpus of data with the “primary allure [of having the] ability to reflect the multifaceted nature of human language and communication as manifested online.”Footnote 10
The specific labor processes and working conditions required to produce ML models have been documented with regards to data preparation and processing, ranging from low-wage data work to high-status data science work.Footnote 11 Research on all such AI work is welcome. Less is, however, known about the labor processes by which data are collected in the first place, even though the importance of user-generated content to capital in the participatory web has been well-studied since the early 2000s.Footnote 12
To consider the ML labor process, we adopt the schematic model of Muldoon et al.: data collection, data curation, data annotation, model training, model evaluation, and data verification.Footnote 13 But within this model, data collection is often taken as a given in critical research on AI work; typically figured as an automated process performed by some sort of surveillance apparatus. However, as critical data study scholars have demonstrated, data do not come from nowhere, but have specific material, spatial, and temporal origins.Footnote 14 The critical literature on AI work thus evinces a gap between the generation of web data and its processing into training datasets. How does the former become the latter? Or posed differently: how are records of human activity collected from the internet and turned into datasets which can be used to train ML models? This is discussed frequently in technical literature.Footnote 15 Indeed, the collection of data is considered by data workers to be one of the (if not the) most challenging and time-consuming parts of data work.Footnote 16 However, it remains sparsely examined in sociological or critical research. We turn now to two important strands of critical research on AI work to demonstrate how scraping escapes their grasp.
Free labor and data
The free/digital labor debate, with its origins in the early 2000s,Footnote 17 broadly focused on how the digital economy relies on the unpaid production of data by users captured by platforms and used to sustain advertisement economies. The notion of free/digital labor places the onus of value production in the digital economy not only on “specific forms of production (web design, multimedia production, digital services, and so on), but is also about forms of labor we do not immediately recognize as such: chat, real-life stories, mailing lists, amateur newsletters, and so on.”Footnote 18 More recently, debates on the digital economy continue to argue for the importance of unpaid activities of users and how the data they produce is fundamental for the constitution of the advertisement spaces over which visibility is auctioned,Footnote 19 or how the data generated by users constitutes the raw material, or digital breadcrumbs required in platforms to produce digital goods.Footnote 20 This analysis remains salient insofar as some ostensibly “AI-first” companies such as Google generate the vast majority of their revenues from data-driven targeted internet ad sales.Footnote 21
Moreover, this unpaid production of data is also essential to LLM production. Contemporary models “require datasets featuring billions of words or hundreds of millions of images scraped from the internet”Footnote 22 with the written web serving as a vast repository of training data.Footnote 23 But much more than “free labour” is required for the production of AI, or of earlier digital artifacts.
Data work and data science work
The political economy of digital systems depends on many different types of labor, often hidden behind a perceived automatism. Muldoon et al.Footnote 24 note that “data work” constitutes the “precondition” for the multiple automated digital technologies which form the landscape of digital capitalism, despite the tendency “to forget that behind the seemingly automated processes of AI often lies the disguised labour of human workers forced to compensate for the limitations of technology.”Footnote 25
Although critical scholars recognize that the development of AI models is based on the use of publicly accessible web data, discussions on data work focus on labeling, annotation, and other forms of data processing and tend to neglect the question of where exactly data comes from and how it was collected. Gray and SuriFootnote 26 highlight how great amounts of labeling were necessary to make ImageNet a “gold standard” dataset, but find it sufficient to say that the first version of the pre-labelled dataset was obtained when “Li and her colleagues first wrote software to download millions of images from the World Wide Web.”Footnote 27 Similarly, Muldoon et al.Footnote 28 note that the “large” of LLMs comes from the size of their dataset and focus on the labor of labeling and annotating as part of their creation,Footnote 29 yet they do not directly address the anterior process required for the production of datasets, the collection of the data. Steinhoff recognizes that “[d]ata processing is, in reality, a second step after data collection” but does not provide detail about how collection occurs.Footnote 30 Srnicek notes that the “collection of data today is dependent on a vast infrastructure to sense, record, and analyse [it],”Footnote 31 but this gives an impression of a fully automated system, which, as we will discuss below, is not the case. One notable recognition of data collection as involving the practices of people comes from Slota et al., who argue that data science begins with “data prospecting” or “activities of discovering disordered or inaccessible data resources, thereafter to be ordered and rendered available for data scientific work” in order to make data science “fulsome with worldly material (e.g., data) from which value may be extracted.”Footnote 32 This theorization is valuable as it points out that collection is a practice, but it does not help us understand data collection as a concrete labor process.
Data collection work: (web) scraping
The different ways in which data can be collected from the web constitute a complex and wide topic, which for reasons of space cannot fully be broached here. However, it is important to note that web scraping is only one of different possible ways of collecting data. In many instances, the transfer of data between digital actors is fully automated, as is the case with application programming interfaces (APIs), which consist of protocols and standards which determine how two (or more) different pieces of software can interact,Footnote 33 allowing for external developers to access data from websites in a predetermined, authorized manner. Web scraping designates the phenomenon of the development of software products for data collection wherever these types of protocols are either lacking or too limiting.Footnote 34 Web scraping also refers to the process of collecting data even when they are purposefully formatted in ways to prevent their collection.
Web scrapingFootnote 35 is defined as the process by which unstructured data are automatically extracted from websites at scale and saved into databases for retrieval through the use of software toolsFootnote 36 whenever these data are not expressly available for collection. As data collection usually happens at scale, web scraping is a term generally used to refer to the actions of software programs, or bots, which run automatically through the web gathering data without the need for human intervention, that is, of manually visiting websites for data copying.
The concrete process of web scraping involves a transformation of any format of data from a website into a format that can be used for the further steps of the data work pipeline.
Web scraping is generally divided into a number of technical phases:
1. Defining scope—identification of what web page(s) are being scraped (which may involve crawling).
2. Fetching—request and receive data from websites.
3. Extracting and parsing—copying the intended data from the website in the form of HTML text and changing the data format by removing HTML tags.
4. Collecting—pasting the information in a database (structured or unstructured).
Concrete processes of web scraping are always conditioned by what specific data might be required and of the always particular conditions of accessibility of that data. As such, any data collection venture requires, first and foremost, the establishment of how to fetch the desired data. This involves, even before scraping, overviewing if there exist any website-specific APIs,Footnote 37 giving access to the desired website data.Footnote 38 If, however, this is not possible, proper web scraping is required, which involves the selection of the proper technical requirements on account of either static or differently complex web pages. Moreover, the necessity of labor in scraping is further required by its sometimes “gray” status. If data are purposefully protected from collection by external parties through varying anti-bot measures the data source might have, web scraping also involves the circumvention of these types of limitations.Footnote 39
To understand these two aspects which frame scraping–the technical complexity of websites and the technical and legal limitations to the practice, we explain briefly how the web works as a data transfer system involving both human authors and visitors as well as automated software before considering the history of different regimes of data accessibility alongside the evolution of scraping.
Scraping and automation
Although typically described as automated, the actual deployment of web scraping software is far from autonomous. As Valova et al. note in an overview of current web scraping solutions, their deployment requires “in-depth knowledge in the field of web scraping,” and users must “have high levels of programming” skills; scraping tools are meant for use by “trained specialists” or those with “the financial resources” to outsource their deployment.Footnote 40 For these reasons, while scraping is a highly automated technical process, it is simultaneously a peculiar labor process.
This derives from the fact that the internet has two faces. One is the front end which presents the human user with interfaces. The other is the back end where applications transmit information to one another and where data are present as collectable text.
The Web—as the collection of webpages linked through HTTP protocols—makes transfer of data possible through a series of request–response messages between client and server, allowing a user to request information from a server and receive it in the form of a webpage. Whenever a human visits a web page, they do no more than initiate this process of data retrieval in order to access the information they want in the form of a web document. At its most basic, scraping involves the automation of this process with the aim to collect the information accessed. In order to achieve this, a scraping tool (which may be called a bot or agent) interacts with web documents by accessing their underlying HTML and retrieves from it the desired data, which depending on the particular use-case could be names of products, comments, prices, etc. The process of web scraping thus entails the transformation of web data from an unstructured collection of HTML text into context-specific structured data. The process of scraping, although seemingly that of just collecting data, also constitutes a transformation of data.
The use of web data for the purposes of development of digital products is not possible without this extraction of specific data from the presented text in website interfaces, with themselves not primed for data extraction, but for human interaction. This is because textual data presented in the human interfaces of websites are not necessarily easily machine-readable,Footnote 41 as it is couched in layers of additional data within a website’s HTML code.
However, it is not the case that the web is exclusively made for human readership, since part of the actual technical infrastructure of the web is directed for machine intervention, as is the case of APIs or the purposeful production of text in view of machine readability for visibility, as is the case with SEO or search engine optimization. Yet, the practice of scraping is worthy of special attention because it is aimed at the automatic reading of web data regardless of whether it was designed for machine readability or not. For this reason, web scraping software mimics the way a human would navigate and read a given website,Footnote 42 in order to extract data in a form suitable for posterior processing and manipulation, even when the access to that data was not foreseen. Data extraction is something both sanctioned by websites and limited by them, and hence there is an aspect of machine-readability inherent in most web data. Web scraping goes beyond this aspect as it is intended to gather data beyond explicitly expressed website permission. Although much of web data are constructed in such a way as to have its data visible to some actors in the digital ecosystem, it is also constructed to not be fully readable by anyone, and it is precisely within this contradiction that scraping comes to the fore. It is this contradiction that also defies the purported automation of data collection, as the shifting position of data in terms of its accessibility does not allow for any one completely streamlined solution for data collection which does not require human labor in troubleshooting and maintenance.
Before expanding on this topic, it is necessary to locate where scraping takes place and to get a general overview of the scraping ecosystem.
The elusive social composition of web scrapers
There exists no authoritative account of the composition of scraping labor or under what conditions it operates. This is in part due to the fact that scraping often occurs in a gray zone and seeks to escape scrutiny. Web scraping is also underexplored as labor because it is not regularly a full-time job nor a profession; “web scraper” is not a commonly advertised job position. It often seems to constitute more of a specific task distributed across various categories of software work. Scraping is a necessary skill in the work of software engineers, data engineers, data analysts, product analytics, and more.Footnote 43 One industry survey of self-declared web scrapers notes that of 136 participants a majority of 58.8 percent were identified as software developers while 47.1 percent were identified specifically as web scrapers.Footnote 44 The pervasive ambiguity and constant change of tech work job titles is certainly a factor here too. At the time of writing, web scraping companies like Bright DataFootnote 45 and Zyte,Footnote 46 have open job postings for either developers or engineers specialized in the scaling of web scraping, while OpenAI also advertises positions for their data acquisition team.Footnote 47
However, the existence of lively, though relatively small, online fora attests to the existence of scraping as a distinct community of practice. This includes frequently updated professional blogs,Footnote 48 subreddits such as r/WebScraping,Footnote 49 and even a number of web scraping specific industry conferences,Footnote 50 which bring together scraping industry professionals to discuss the specificity of the task.
Scraping labor is performed by workers in a variety of organizational settings. According to one survey, 20.6 percent of web scrapers perform these tasks in the context of freelance work, and 3.7 percent as a hobby, while the “majority spend 5 hours or less per week on web scraping (52.9%),”Footnote 51 with web scraping being a prospect for monetizing coding skills for either the production of pre-built scraping software or scripts and of selling datasets on an independent basis.Footnote 52 Another industry report claims to have surveyed “hundreds of professionals” working in web scraping and concluded that “the majority of web scraping professionals work as freelancers (35.8%).”Footnote 53
But web scraping is also an avenue for professional careers.Footnote 54 The same reports conclude that only 10.3 percent of respondents work in web scraping full time,Footnote 55 and that “those working in startups represent 28.3% while those in small or medium-sized enterprises account for 20.8% […], while those working in large companies account for 11.7%.”Footnote 56 The same study provides some initial figures on the level of experience of web scraping workers, with 21.7 percent of surveyed workers having less than 1 year of experience but with 53.3 percent of respondents having “3+ years of experience.”Footnote 57
Accordingly, company reports suggest that many companies hire software developers with the explicit purpose of streamlining data collection, with Bright Data reporting that 82 percent of companies that use scraped data “need help overcoming challenges with data collection.”Footnote 58 In order to do so, even though they multiply the ways in which they acquire data by: (1) partnering with third-party data-gathering companies, (2) acquiring ready-made datasets, or (3) having dedicated scraping teams, with reports having found that the preference lies largely on the third option.Footnote 59
These demographic sources are necessarily limited on account of being produced by web scraping services themselves, but they suggest that the labor of web scraping is both a specialized job within the tech sector and a widespread informal and semi-formal task required for many data-intensive software applications. We highlight why this relative lack of information is itself a defining characteristic of web scraping labor and a consequence of its developmental history.
In the following sections, we will provide a brief history of both the technical development of web scraping software and of its applications in the digital economy to understand how the market for web scraping appeared and keeps evolving, with no single existing software being guaranteed to accurately reach the desired data, and how this came to define the labor process of web scrapers, and we show how the progressive automation of this data transformation operation has underpinned substantial changes in the digital economy and that this progressive automation has not—and arguably will not- elide a continuous need for human labor.
A brief labor history of scraping
The history of scraping can be broadly divided into three eras, as shown below. Namely, the first era of open-source developments, a second one marked by the complexification of dynamic websites and increased difficulty of scraping, leading to the commercialization of web scraping products, and lastly, the era of ML. Each era involves a particular configuration of scraping labor, technologies, and conflict between scrapers and those seeking to block scraping, as summarized in Table 1. The developments broached in this section frame the conditions of possibility of the labor of scraping, to which we return below.
Main eras of web scraping development
Table 1 Long description
Three eras are compared across four columns: Main eras, Scraping labour, Scraping technologies, and Scrape blocking. Infrastructural, 1960s to 1990s, lists labour as accumulated labour in existing automated software and infrastructure plus voluntary and enthusiast led software production, and technologies as background processes for organisation of information in early computers and communication networks and early bots. Infrastructural scrape blocking is described as non enforceable technical barriers for scraping on account of server overload, robots.txt, and the row also notes software intermediaries between computers and humans and attempts at automatically measuring and indexing the web. Commercial, 1990s to 2020s, lists specialised web scraping labour as a subset of software labour plus labour of automated software maintenance and troubleshooting, and technologies as open source web scraping development and production of datasets. Commercial scrape blocking lists technical barriers to scraping, IP blocks, captchas, and user login requirements, plus legal barriers to scraping, and the row also notes web scraping as a service and dataset commercialisation and scraping and search engines. AI, 2020s now, lists a shifting field of data transfer ecosystem, automation of scraping labour with continuous need for scraping labour, and AIs scraping bots. AI era scrape blocking is described as ongoing intensification of anti scraping measures, and the row also notes scraping for AI development and AI powered scraping.
Part 1—1990–2000: bots and scrapers as functional infrastructural open-source software
Taking a wide view, the phenomenon of automatically collecting and storing information arguably goes back to the 1940s and the notion of information retrieval.Footnote 60 From a more focused perspective, the history of automated information sorting software can be traced back to the 1960s when Fernando Corbato, a founding member of the MIT Laboratory for Computer Science, developed a software program which would automatically scan MIT’s computer network databank to check for what files had been modified since the last scan.Footnote 61 By the late 1980s, automated bots were important components of early digital communication systems, like BBSes and IRCs, serving as “intermediar[ies] between humans and computer”Footnote 62 which automated a series of background operations required to maintain these connected networks; doing “the boring stuff – keeping background processes running or chatrooms open,”Footnote 63 and guaranteeing the maintenance of usable interfaces. Such bots were created by committed users who freely contributed to the development of software with no commercial reward.Footnote 64
In the 1990s, with the flourishing of the public internet and the first File Transfer Protocols, the spread of files shared across computer networks similarly required some means of automated organization in order to be navigable. Early developments in this direction include the first proto-search engine, Archie, being developed by Alan Emtage, a university student at Montreal’s McGill University, in 1990,Footnote 65 information structuring software programs such as Gopher, created by Marc McCahill in 1991.Footnote 66 In 1993, Matthew Gray at MIT developed a program called WebWanderer which measured the web by automatically counting the number of new URLs,Footnote 67 and in the same year, Jonathon Fletcher developed JumpStation, a program which would index the web and allow for users to find pages through inputted search words,Footnote 68 which was quickly followed in 1995 by Brian Pinkerton’s development of WebCrawler. WebCrawler was a landmark as it was “the first web robot that was capable of indexing every word on a web page, while other bots were storing a URL, a title and at most 100 words,”Footnote 69 thus considered to be the first “full text” search engine.
The technical building blocks of scraping and the start of scraping limitations
Throughout this decade, the major developments in scraping software were led by individual enthusiasts who created increasingly sophisticated software for information organization and retrieval.Footnote 70 Broadly speaking, this first wave of web scraping was defined by researcher/hobbyist development of bots as experimentation in the interest of more functional programs for the operation of nascent communication networks. The early web was not user-friendly, since it required a level of technical skill to navigate. The technically-minded early adopters of networked computer technologies had both the knowledge and the skill to have a stake in discussing and developing the very technologies they were using. This production of scraping software was the purview of the free peer-to-peer labor of early web enthusiasts which formed the building blocks, or at the very least, the first significant prototypes, of scraping software. At this early stage, this software is the result of the unwaged work of programmers “willingly conceded in exchange for the pleasures of communication and exchange.”Footnote 71 We will see below how these early software developments survive in current scraping practices, as they still constitute background operations which frame the way data are maintained, read, and shared online, and how the accumulated capacities of the technical aspects of the web form one key technical component of the labor of web scraping.
Already in the 1990s, the spread of scrapers came to be seen as malicious because the volume of requests for large scraping operations could “[crash] websites by overloading them with automated traffic”Footnote 72 and this could also incur monetary costs for the website’s hosts. To solve this problem in 1994, Martijn KosterFootnote 73 created the Robots Exclusion Protocol through the robots.txt extension—a machine-readable text file which demarcates which parts of a website can and cannot be read by bots. Although it consists of no more than “an honor system that presumes good faith on the part of bot developers, who must actively decide to make each bot honor the convention and encode these values into the bot’s programming,”Footnote 74 this early mechanism built for the limitation of non-human agents visiting websites is still widely used today. This early episode displays a dynamic that will reappear throughout the history of web scraping. Labor is devoted to the development and deployment of scraping technologies which are quickly confronted by an opposing force: labor and technology for anti-scraping and bot-blocking.
Both the infrastructural importance of scraping bots for the organization of an ever-growing networked information space and the way in which these could be used to abuse the technical infrastructure of these networks, to the detriment of data holders, were to pave the history of web scraping development.
Part 2—the commercialization of scraping: scraping market and the labor of web scraping
The growing popularity of the web throughout the late 1990s and early 2000s significantly increased the number of websites, information, and data online, and the general complexification of the digital ecosystem.Footnote 75 The dot-com boom of the 1990s saw the proliferation of internet businesses such as Amazon, founded in 1994, and Google, founded in 1998. The subsequent bubble-pop of 2000 saw a slew of these new firms devastated, but some that survived, such as Amazon and Google, would recover and go on to achieve behemoth status in the following decade, changing the dynamics of the internet economy to one of an increasingly oligopolistic nature. The practice of web scraping evolved alongside these developments. Commercial scraping ventures led by large corporations appeared at this point, although non-commercial and open-source scraping remained as well, in the form of publicly available datasets and software.
Non-commercial web scraping
The growth in popularity of the web saw the continuation of volunteer and enthusiast led development of scraping technologies, which can broadly be divided into the production of open-source datasets and software development, required on account of the increased barriers to scraping explained above.
Open-source datasets
The expansion of the provision of open-source datasets was already initiated in the 90s, with the Internet Archive being created by Brewster Kahle in 1996. The project started with the intention of “building a digital library of Internet sites and other cultural artifacts in digital form.”Footnote 76 The Internet Archive is reliant on its crawler, Heritrix, developed by Gordon Mohr, which, instead of fetching data as plain HTML text, retrieved whole webpages so as to preserve their “original appearance and functionality.”Footnote 77
But these types of initiatives did not disappear with the growth of the web; on the contrary, by the mid-2000s, similar and more complex examples appeared. The case of CommonCrawl is worth highlighting, as it is still widely used in the training of popular ML models.Footnote 78 CommonCrawl started in 2007 with the expressed purpose of providing “academic samplings of the web”Footnote 79 by having its large-scale crawling bot CCbot run continuous sweeps of web data on a monthly basis and making them available for free.Footnote 80
These sorts of efforts, often stemming from the work of single programmers and small non-profit teams, followed in the lineage of providing accessible ways of collecting publicly accessible data and of having reliable copies of a growing corpus of web information.Footnote 81 These tools still mark the scraping ecosystem today, as they are still used to scrape most of the web or to have access to already scraped parts of the web, as we will elaborate on below.
Software
With the growth of the web, technical developments in scraping in the early 2000s placed emphasis on the creation of software which would streamline the possibility of scraping without the need for programming software from scratch. The possibility of having automated bots reading HTML pages was sought after, but at this time, much of the web was not structured in a standardized enough format for automated scraping. Although these formats were invisible for human readers, this lack of structure could stop scrapers from working.Footnote 82 For this reason, in 2004, Leonard Richardson launched BeautifulSoup, an automated scraping toolkit—still used today—which would make it so that “[v]aluable data that was once locked up in poorly-designed websites is now within your reach.”Footnote 83 Scrapy, another toolkit still used today, was developed to collect product information at scale in 2008. It was later developed as an open-source framework which further automatized the process of scraping. Compared to BeautifulSoup where “you had to write all the glue code yourself” with Scrapy “even in 2009, you would get an all in one tool.”Footnote 84
These scraping toolkits were developed in order to maintain the possibility of scraping for developers despite the lack of appropriate website structure for these projects. In the same vein, the complexification of website structures witnessed in the early 2000sFootnote 85 which included the addition of dynamic interfaces, further complicated automated web scraping. With the introduction of JavaScript (in addition to HTML) as part of the programming languages of websites, novel tools were required for the job.
Foremost among these are browsing automation technologies still widely used today. Selenium, which was developed as an open-source software, in 2004 by Jason Huggins.Footnote 86 Instead of running a script which merely fetches HTML pages from websites, as is the case with BeautifulSoup, Selenium would instead emulate the behavior of a user using a browser.Footnote 87
As a Scrapy developer notes in describing why scraping is a necessary concern, “In web scraping, you don’t get to choose when to change your code. Websites force your hand.”Footnote 88 These open-source initiatives, borne out of either volunteer projects or as non-profit arms of software companies, happened in tandem with the growth and complexification of the web. As websites became more dynamic, the automated technology required to scrape them too had to evolve. The dynamic of shifting technical standards in websites generating the need for specialized scraping software was to pave the way for the continued development of data collection, as it is a practice conditioned by the data ecosystem at large. But the major developments in the evolution of scraping came about not only through this imposed necessity, but through its subsequent commercialization.
Commercial web scraping—third-party services
The commercialization of web scraping stemmed from the increased importance of data for the functioning of the web, together with an increased difficulty of scraping data on account of the complexification of websites seen above and the imposition of novel technical barriers to data access. There are two main ways in which the initial non-commercial efforts of automatically reading and organizing the web were party to succeeding commercialization efforts: (1) the commercialization of information organization through the imposition of layers of technical mediation in the access of web information, as is the case with commercial search engines like Google and; (2) the commercialization of software specifically dedicated to the procurement of large amounts of web data concomitant with the development of other technical mechanisms to prevent it. We will analyze both in turn.
Search engine commercialization
To consider the commercial development of web scraping, we focus first on the case of Google Search. This focus makes sense because “search engines are essentially very large data mining applications.”Footnote 89 Google was first developed as a web crawler which, in addition to following through hyperlinks with a “Googlebot,” would rank the found URLs in order of calculated relevance (quantified as incoming links)—the famed PageRank algorithm.Footnote 90 Google’s underlying structure did not substantially differ from the open-source scraping projects previously mentioned, since it did not produce first-hand information and was instead built on giving access to an indexed copy of the “open web.”Footnote 91 Its particular innovation was the ability to commercialize this index through advertisement. Since, as a Google spokesperson stated, “neither Google nor any other search engine controls what pages are made public on the web,”Footnote 92 their business model is rather to build over existing information space in order to present it to users, while also generating a space of advertising which can be auctioned to companies. In this way, advertising was one of the main ways in which scraping became indelible for the development of the digital economy.
The economic importance of visibility generated by this ecosystem encapsulates the way in which websites would come to format their data in connection with the possibility of scraping. As Caliskan et al. note for Google, the contemporary internet is in part defined by a trade-off between visibility and access, in that websites allow for at least some of their data to be read by search engines in exchange for “a right to be seen” by them.Footnote 93 As such: “[t]he more linked and popular a page is, the higher it is ranked later. Therefore, web designers and SEO experts ensure that crawlers can access their sites and link their sites to many others.”Footnote 94
The economic importance of search engines is thus an important pole of the web scraping ecosystem as it shows how the readability and accessibility of the web are part and parcel of the digital economy. But search engine commodification was not the only way in which commercial scraping evolved. The commercialization of web scraping also stems from an increased perceived value of data together with an increased difficulty of scraping data.
Scraping commercialization
As early as 1958, IBM identified data procurement as a way of developing business intelligence and competitive advantage.Footnote 95 By the 1990s, data mining from open web sources was already seen as the new paradigm for this type of competitive intelligence,Footnote 96 with the appearance of scraping for business intelligence ventures like IntelliSeek, founded in 1997, whose “most successful early businesses was scraping message boards to find mentions of brand names for corporate clients.”Footnote 97 If the scraping of this information was relatively trivial in the 1990s, the increased market competition in the digital economy made it so that companies were trying to make their data less easily accessible. This was because of the changing ecosystem of data transfer online and the introduction at the general level of anti-bot measures alongside the popularization of APIs.
Anti-bot measures
In tandem with the increased difficulty of scraping due to the technical evolution of the web, there were also concerted efforts put in place in order to curtail the practice. As the ease with which the web could be scraped kept developing, there was unsurprisingly a marked increase in bots visiting websites. Although older metrics are not available, by 2013, the yearly Imperva Bad Bot Report found that automated bots constituted 61.5 percent of all website traffic,Footnote 98 with scrapers being considered one of the categories of bad bots as they were accused of “content theft and duplication.”Footnote 99 This is widely regarded to be detrimental to websites, as it can incur server overloads and costs in terms of downtime. For this reason, new ways of limiting bot access to websites were developed in the 2000s. These included both technologies to identify bots in the way they interact with website structures, like CAPTCHA systems and “I am not a robot” ticking boxes, as well as identifying the IP addresses of website clients in order to limit the amount of requests each can do—termed rate limiting—or even to block specific addresses associated with known irregular behaviour.Footnote 100 These impediments to streamlined scraping are moreover party to commercial ventures specifically dedicated to data protection, with companies such as Cloudflare, founded in 2009, offering anti-scraping software and now “manag[ing] and protect[ing] traffic for about 20% of the web.”Footnote 101
Web APIs
The 2000s also saw the appearance of new forms of mass data transfer which were not reliant on web scraping. APIs were introduced as an alternative way of commercializing data access. APIs serve as protocols providing the possibility of automatically accessing already structured data.Footnote 102 These were first developed commercially in 2000 by eBay and Salesforce and have since been widely adopted by all major internet companies, especially data-heavy social media companies,Footnote 103 as ways of determining the parameters by which their data can be accessed by third parties.Footnote 104
However, APIs can also serve as a limitation for the process of data collection since they constitute “a sanctioned means of gathering selected data from a website” in that the website that provides it “controls what information can access, as well as how much of it, by pre-defining limitations.”Footnote 105 Additionally, the use of APIs can come with monetary costs and the level of access permitted can change at a website’s discretion, thus not guaranteeing a reliable way of gathering data for long periods of time.Footnote 106
The concurrent development of APIs and the circumscription of data access through antibot measures constitute the material conditions against which web scraping developed as a commercial venture. Thus, the first decade of the 2000s was marked by the propagation of third-party scraping commodities purporting to easily extract data from complex websites without their authorization.
Specialized scraping companies
The appearance of these novel regimes of data transfer and of novel and increasingly complex scraping limitations in this way resulted in the generalized complexification of data accessibility. Scraping the web became less and less trivial but the concomitant importance of data for the development of digital products also made the need for data a widespread concern. For this reason, the practice of web scraping ceased to be only the purview of tech enthusiasts, but also a practical requirement for a plethora of initiatives. For this reason, while during the early web, scrapers could be found hacking away in university departments or independently creating and sharing scraping artifacts in specialized forums, since the mid-2010s, web scraping was also developed into an array of software commodities.
Some major actors in this field include Bright Data, started in 2014, which was initially developed as a commercial market for IP proxiesFootnote 107 and later made use of its proxy farm to become a commercial scraping software and dataset enterprise, as it could guarantee large-scale scraping through multiple IP addresses. In 2010, the initially open-source Scrapy also served as the basis for the development of commercial products as its creators rebranded it as Scrapinghub (and later Zyte): a suite of tools including crawlers, proxy services, and cloud-based solutions for avoiding scraping blocks.Footnote 108 A number of other companies like Apify (2015),Footnote 109 Oxylabs (2015),Footnote 110 Diffbot (2011),Footnote 111 ScraperAPI (2018),Footnote 112 and SerpAPI (2017)Footnote 113 were founded around the same time or subsequently.
While free and open-source technologies can still be used to scrape web data,Footnote 114 scraping companies make use of their scale and dedicated software in order to make scraping easy for non-technical actors. Effectively, they provide a way for outsourcing data ingestion needs at a premium through the provision of easier to navigate interfaces and actionable avenues for gathering data without the need of specialization in scraping, by selling either already produced datasets or software products specifically for data collection as ready-made automatic services. It is from this specialized need that the labor of web scraping, that is, the labor behind automated products and the labor of maintaining them, comes to the fore.
Web scraping labor in the commercial web
Whereas the early developments of web scraping resulted in the accumulated technical infrastructure of web data and were the purview of voluntary tech enthusiasts, the commercialization aspect we have just described and the consequent changes in requirements for accessing web data help explain additional characteristics of the labor of web scraping.
The commercial ecosystem of web scraping showcases how web scraping is of paramount importance for the development of digital products. A recent report indicates the economic importance of data collection: “98% of businesses face some negative impact when data collection is delayed. For 21%, the impact is severe” and if “interrupted, 95% of businesses face negative impacts within 24 hours.”Footnote 115 Scraping is now a significant bottleneck in the production of digital goods. To tackle the need for data, “34.8% of surveyed developers favor web scraping API [… to] address common hurdles like rate-limiting, JavaScript rendering, and data parsing” even if used in combination with open source alternatives,Footnote 116 but even if using these already built products, web scraping still requires specialized intervention, as we will expand on below.
For these reasons, whether as in-house scraping teams or through the utilization of specialized companies or freelancers, the period between the early 2000s and the present (2026) saw the appearance and evolution of web scraping as a distinct professional activity. While during the development of the early web the labor of scraping was visible mainly in the accumulated technical capacities of open-source software, the proliferation of companies specializing in industrial-scale scraping operations has made scraping tools into commodities.
As such, the contemporary scraping of the web is dependent on the fixed capital in already developed scraping solutions now available through software products, but also still on the continued existence of the practice. In this sense, the labor of web scraping is, on the one hand, the purview of accumulated capacities already existing in automated scraping software. On the other hand, the overcoming of these constantly updated barriers to scraping and the need for specialized continuous scraping labor, as the state of the web scraping market attests to, are indeed what constitutes the task proper of a web scraper, as we will expand on below. These dynamics have only intensified with the introduction of the AI economy, to which we turn next.
Part 3—Scraping data for AI
The new value of data and the increased difficulty of getting it
As we established above, the issue of data collection is of paramount importance in the development of generative AI products, and the race for data inaugurated by the appearance of companies such as OpenAI and Anthropic is changing the data collection market as the newfound value of data for the training of LLMs has prompted a series of important changes in the web scraping ecosystem at least since 2023.Footnote 117 Recent reports highlight this difficulty in ascertaining the state of the web scraping industry in 2026. For one, the estimates on the market size of web scraping vary greatly, with estimations ranging from a value of 363 million to 3.71 billion in 2023 and from 1.47 billion to 11.35 billion in the next decade.Footnote 118
The dependence of generative AI on scraped data is attested to, on the one hand, by the use made in their development of the free datasets broached above, with models like GPT-3 documented to have been trained using Common Crawl, from which it took 60 percent of its tokens (Brown et al., 2020) and other corpora of web text.Footnote 119 These datasets form “the background conditions—the environment—upon which ML research and development operates”Footnote 120 having become foundational for their production.Footnote 121 More recent public datasets have also been made accessible through other open-source projects. The most widely used is the open-source Hugging Face project,Footnote 122 which in 2021 released its “Datasets” library, containing more than 600 unique and freely available datasets of both big and small internet-scale corpora.Footnote 123 But the use of publicly accessible datasets does not exhaust the way in which generative AI is involved in scraping, since all major AI companies have developed their own scraping agents to gather “large portions of the web, mostly for their internal use.”Footnote 124 Googlebot,Footnote 125 GPTbot,Footnote 126 and ClaudebotFootnote 127 run uninterruptedly through millions of websites gathering data. At the same time, scraping-as-a-service companies, like Bright Data and Zyte, also pivoted their offers specifically for AI products.Footnote 128 These services are also used by major AI companies, with OpenAI, for example, using SerpAPI’s service to scrape Google without their consent.Footnote 129
The introduction of these new uses of data and of new actors in the web scraping ecosystem has resulted in an increased scale of scraping operations and has consequently brought the issue to the attention of data holders of all sizes. Like past scraping developments, this has led to the further development of data protection and bot-detection measures. One recent study has noted that restrictions to web scraping have increased significantly, with at least 28 percent of websites previously scraped by Common Crawl now explicitly restricting scraper bots.Footnote 130
This increased limitation of scraping results from the changing perceived usefulness of having website data scraped, in the face of AI companies which replace the utility of websites. To illustrate with the most salient example, Googlebot has been Google’s indexer since 1998, and its crawling of webpages was not seen as problematic for websites since websites got increased traffic from being “findable,” while Google was allowed to develop its search product with free data.Footnote 131 But since the introduction of Google’s AI summary feature, which presents information within Google’s search page, Googlebot’s scraping of the web no longer guarantees website traffic.Footnote 132 Cloudflare has framed these actions by Google as the gathering of website data without proper recompense and has created a Content Signals Policy which leverages the robots.txt protocol to not only block scrapers but to also specify what the data scraped can be used for.Footnote 133
But the contradictory stakes of data accessibility are not static, as the trend of increased blocking by websites, is also not itself perfectly consistent. While, as the cases above highlight, scrape blocking was the initial response to the increased presence of AI bots, since then, as the determinants of web visibility shifted from search engines to AI chatbots, as the main actors in internet traffic, the trend of GEO, or generative engine optimization, is also an observable trend. For this reason, websites are now allowing scrapers as a way of leveraging their data to be observable by LLM scrapers.Footnote 134
Yet anti-scraping sentiment still prevails, as it is also apparent in the multiplication of court cases surrounding the legal status of scraping data for commercial purposes. Several major LLM developers are currently under scrutiny and criticism on account of their use of publicly accessible data and of the measures they have put in place to access them. Both OpenAIFootnote 135 and AnthropicFootnote 136 have been accused of overloading websites and ignoring scraping restrictions.Footnote 137 Similar accusations have also been levied against third-party scraping companies, with cases being brought by Meta against Bright Data for the selling of Instagram user data datasets,Footnote 138 although the case was dropped in part because Meta itself had been a customer of Bright Data’s scraped datasets for years.Footnote 139
And if AI is on the one hand in constant need of more new data,Footnote 140 its main actors are also increasingly being kept away from getting it through blocking.Footnote 141 As an example, Reddit struck a data licensing of its existing corpus of data together with future data with Google to the tune of $204 million,Footnote 142 followed by a similar deal with OpenAI,Footnote 143 confirming that this data serves as the bedrock of how some of the more popular LLMs are trained. Although Reddit still allows data access to researchers and for non-commercial uses, it has explicitly required the establishment of contracts for commercial uses of its data,Footnote 144 whereas it was trivial to scrape until at least 2018.Footnote 145 Reddit has already cut major search engines from competing firms, namely Microsoft, from indexing its data on account of the risk of it being used non-contractually for the training of AI modelsFootnote 146 and has sued LLM developers with which it has no formal commercial agreement for having used their data for the training of AI models.Footnote 147 Moreover, Reddit’s CEO has recently hinted that Reddit’s data are so useful for the development of generative AI products that its licensing deals could be terminated in favor of a proprietary generative AI product of their own,Footnote 148 with internal use being favored over direct monetization as a commercial asset for third parties. At the same time, Reddit legally accuses data scraping companies of stealing their data.Footnote 149 These novel economic dynamics surrounding publicly accessible data pose a very real cost on how scraping has direct value implications.
What these shifting economic relationships highlight is that the lineage between the data production and data processing, which form the two poles of analysis of the political economy of digital products, is incomplete if it does not also specify the conditions of data collection. This is because transformation of web data into scraped data constitutes, in our view, a necessary step in the transformation of web data into data primed for machine-readability, and a bottleneck in the production of value for digital data companies. And although scraping does not exhaust the ways in which data are transferred between data companies, and the yet undetermined legal and technical conditions of data transfer across websites and data companies, whenever scraping is required, both the procurement and the transformation of data into modes appropriate for subsequent software development are marked by human scraping labor and expertise, which imparts value on otherwise useless—for production, at least—seas of data.
These ongoing tensions and evolution between different scrapers and websites showcase how the ecosystem for web scraping, in addition to being mired in technical difficulties, is continuously shifting in part by being curtailed through legal mechanisms, making scraping even harder and casting doubt on the legality and ethical appropriateness of the practice,Footnote 150 and in part by renewed strategies and regimes of data transfer coming about due to the necessity of having data accessible to economic actors.
Although the introduction of generative AI and its novel scraping agents is the reason for continued contradictions in the field of web scraping, it is still not possible to account for how this might impact the conditions of scraping in the long term. If on the one hand scraping seems to be increasingly difficult on account of renewed technical limitations to the practice, the need for data has never been more apparent, as is the continued need for human expertise for this to happen. We will see below how these dynamics impact the labor of web scraping.
Vicissitudes of scraping labor: cat-and-mouse game and secrecy
The preceding incursion into the historical trends surrounding data transfer regimes on the web is meant to highlight the complexity of actors, limitations, and practices which define the conditions of how data can be collected.
Until now, we have set aside how these conditions frame the concrete requirements of web scraping as a form of labor. The main reason for this omission is that although the introduction of AI companies into the field of web scraping ecosystem further exemplifies the growing scale of scraping, professional web scrapers are only rarely a distinct role in the tech industry job market. As an industry professional notes, the existence of big markets for both data scraping and proxy providers shows that there are people whose job is to develop specifically scraping technologies, even if they often seem invisible.Footnote 151 Focusing on the case of AI, they note that OpenAI, for example, has only 16 people whose job description mentions web scraping, yet it is centrally reliant on scraping as a means of gaining LLM training data.
Despite this liminal status, web scraping is in fact one of the most common behind-the-scenes operations for any internet-based technology and industry, although this “underlying data market that is off the maps […] is kept inefficient by the lack of a proper, stable interface for transferring data from those who have it (the websites) and those who want it (the scrapers).”Footnote 152
The reasons for this obscurity constitute the distinctive factors of web scraping labor. The main one concerns the cat-and-mouse nature of the scraping ecosystem as the maintenance of competitive advantage. The concurrent development of barriers to scraping and of scraping technology is thus best understood as a “bots arms race”Footnote 153 between entities who scrape data and entities which have data (which can many times be the same entity). The same phenomena illustrate the contradiction between sanctioned means of gathering data, through APIs and data licensing deals which effectively automate data transfer without the continued need of human intervention, but only between the largest actors in the data economy, namely LLM companies, whose own development is built on previously accumulated data scraping labor conducted in the past. For the web at large, the increased complexification of website structures and the increased commercial protection of data through bot-identification and blocking saw the rise of a market specifically dedicated to providing industrial scraping solutions together with web scraping through the use of freely developed and freely available software. Within this multiplication of both new modes of data transfer and of scraping offers (some which actually add value, and many which are just marketing), web scraping nonetheless remains a professionalized activity precisely because it provides “means of collecting any visible information from a website using a bot you have programmed to copy and save the information, whether or not the website owner permits you to do so.”Footnote 154 At the same time, the secrecy surrounding the practice keeps useful scraping solutions under wraps and gives companies the possibility to stay ahead of their competitors in how they acquire data.Footnote 155
The legality of web scraping also contributes to its secrecy, with the slew of court cases pertaining to the practice of data collection put into question many of the practices of web scraping and might hinder the profession. This is in fact a popular theme in web scraping conferences and it is noted that, due to the ongoing threat of continued legal action: “it’s important to remember that every action we [web scrapers] perform on the target server could cause monetary damage, so keeping our operations polite and lightweight is essential.”Footnote 156 Together with the general increase on automated traffic online and the perceived threat of scraping,Footnote 157 this necessary carefulness will likely continue to be required.
One recent report on the web scraping market found that blocking is considered one of the main challenges by 68 percent developers of scraping solutions inquired.Footnote 158 Similarly, another report surveying 506 software engineers and other professionals whose work involves web scraping, found that “[d]evelopers responsible for in-house web scraping operations face many challenges ranging from a lack of resources to the relentlessly improving anti-scraping measures,”Footnote 159 which were already specific challenges identifiable in the initial commercialization of the web and are of increased complexity in the age of AI.
These obstacles to web scraping, moreover, are noted by professionals to be consistently changing and becoming more difficult to circumvent.Footnote 160 Which means that whatever particular solution found for one website at one time might not be of use for different websites or for the same website at a different time.Footnote 161
In sum, as companies “beef up their security and their bot detection systems, bot builders must continue to step up their game to evade these new measures.”Footnote 162 One Oxylabs developer notes how this constitutes a major part of any web scraping project:
“I mean it’s good job security at least for me as a developer […] because every single day we have to unblock a new website and it seems you are […] playing a cat and mouse game. It’s pretty fun. It’s never like you do a task and then it’s done: you get two more [tasks]. I know it’s a sad topic for people paying for the products, but it’s a good topic for me”Footnote 163
In addition to the overcoming of scraping barriers and fetching data, the work of a scraper is not necessarily done. This is because simply fetching website data in its HTML format is likely not enough in order to have usable datasets. Data parsing, as “the process of transforming data from one format to another, typically, in order to change its structure and prepare for analysis,”Footnote 164 is also increasingly difficult and requires expert work.
In any given scraping project, the data fetched in HTML format is usually unstructured and with many elements that might not be germane to the project’s desired goal. Parsing designates the process of rendering this data into more appropriate formats. Considering textual data, this can include, isolating specific elements of information, the removal of whitespace, the removal of unnecessary metadata, etc.
This step of the process of web scraping is however complicated by the fact that, to put it bluntly, “websites are not primarily built to being scraped”Footnote 165 and hence desired data are usually not structured in a way to be easily identified and copied. Common barriers to parsing are “aspects such as pagination and dynamic content,”Footnote 166 but also, complex layouts with nested information, with the layout and information presented changing based on user interface interaction.Footnote 167 Data might even not be structured with unique HTML tag elements, making automated scraping difficult since a bot cannot locate it, and even if HTML tags are used, they can be used ambiguously or irregularly.Footnote 168 As one software developer puts it, there are cases in which: “the HTML source […] is not complete. The data is there in the source, but it’s in JavaScript objects and code which is in the same HTML.”Footnote 169 In such cases, “although the data is already there, there are no additional requests [… but] when you click on [an expandable bar] the JavaScript runs and it takes some piece of data and puts it into that HTML.”Footnote 170 The task of a software engineer maintaining a data scraping pipeline here becomes that of reverse engineering specific website structures, or “think about the logic that happens and think of what the front-end programmer intended to do and […] replicate that on the parser level” in order to get the desired data.Footnote 171
Unsurprisingly, the issue of parsing is also singled out by scraping professionals as one of the main challenges of the job.Footnote 172 This is not only because many websites now have these difficult to parse structures, but also because the establishment of long-term scraping solutions is made difficult by the constant “[d]eviations of the current web page structure with respect to the testbed structure could lead to failures in the extraction process.”Footnote 173 For these reasons, professionals have stated that they are required to change or fix their software as much as every day or “several times a week,” with 11–40 hours per week dedicated to troubleshooting parsing errors.Footnote 174
In both fetching and parsing it is clear that the difficulties in web scraping stem primarily from the fact that “implementing a web scraper comes along with several challenges and is considered an iterative and lengthy process.”Footnote 175
The market for web scraping appeared precisely because of these challenges. Developers, however, still note that this market for ready-made scraping solutions is complemented by free tools as seen above precisely on ensuring “scalability and flexibility” in the face of these constant changes,Footnote 176 but even then the technical requirements broached above mean that “the quality of data derived from web scraping technologies can vary significantly and requires validation and quality evaluation by expert interaction,”Footnote 177 on an ongoing basis with “the difficulties of extracting the data from complex web pages [making it so that] this process cannot be automatized and requires a lot of handcrafted work.”Footnote 178
Cloudflare, for example, explicitly states that their strategies for website security are to protect data not accessible through APIs, by rate limiting, modifying HTML markups at regular intervals, requiring proof of human challenges, and even embedding content inside of images instead of text.Footnote 179 All techniques which impede a streamlined scraping operation. The scraping ecosystem in general is marked by this perpetual cat and mouse game between companies and web scrapers (amateurs or professionals) with measures being put in place to curtail scraping and scrapers working to circumvent it.
The labor of a scraper thus involves not only the narrowly defined scraping labor process, but also the necessity of shifting through technical barriers, the procurement of website specific solutions which may involve, in addition to programming work, the choice of using third party applications, website specific APIs, free tools, or a combination of all.Footnote 180 The job of a web scraper thus includes a number of select tasks and decisions that are dictated in part (if not in full) by the changing ecosystem over which scraping happens. There is no one way in which the web can be scraped and the requirements here mentioned make it so that the actual task of web scraping is never straightforward and rather a continuous process of troubleshooting and maintenance.
These continuously shifting dynamics and the additional requirements they create for the extraction of data frame the aspects of web scraping labor which distinguish it from other forms of software labor. While in a sense, the labor of web scraping is that of producing software and party to specific challenges and qualities associated with it, as described above, the labor of scraping is party to a set of particularities resulting from its concrete dependence on extra conditions which frame it and are dependent on the larger dynamics of data holding and accessibility. As such, data scraping is in part distinct from other aspects of data labor as it is characterized specifically by dynamics that lay outside the working place.
If the above general aspects of web scraping labor continue to frame how data collection occurs in practice, the novelty of AI further reinforces the contradictory nature of web scraping labor.
On the one hand, the current development of web scraping is also implicated by how automated technologies have in theory made scraping solutions more streamlined, and thus with less need of human web scrapers. Web scraping is software and the development of automated solutions for software production also affects web scraping labor. As such the most popular industry-level web scraping solutions mentioned above have recently shifted their offers toward AI-powered web scrapers, under the aegis of “code less - scale your data pipeline with AI.”Footnote 181 Such indications would seem to suggest that scraping is more and more a type of process that does away with the necessity of human labor.
On the other hand, however, the observed dynamics in the web scraping ecosystem suggest otherwise, since, parallel to the development of the ever-improving automated web scraping solutions we showed above, the continuous limitations on scraping require bespoke solutions not yet achievable by AI. It is, in fact not rare to see scraping practitioners complaining about the uselessness of AI solutions as they fail at circumventing most anti-bot measures, since “AI tools just scrape toy websites with no bot protection[, t]he moment you throw it at a real, dynamic site with proper defenses [sic], it faceplants hard”Footnote 182 and with the lack of reliability being poised as one major issue with these solutions. In a dynamic that closely replicates that of the bots arm race between scrapers and anti-scraping technologies, automated scraping solutions using AI, are for now as liable to require constant maintenance and troubleshooting as those that preceded it, even if they might involve less direct code or script writing. More authoritative sources confirm this development as although there are increasing integration of AI in scraping processes, their effectiveness is still dependent on “custom implementations, fine-tuning, and integration with specialized systems,”Footnote 183 leading us to anticipate that the human labor of scraping is not necessarily disappearing.
In short, web scraping can be considered to form the basis of further data work, as it is only through web scraping that the primary material on which subsequent data manipulation is made available in the first place. And although scraping does not exhaust the ways in which data are transferred between data companies, whenever scraping is required, both the procurement and the transformation of data into modes appropriate for subsequent software development are marked by the continuous need for human scraping labor and expertise, which imparts value on otherwise useless—for production, at least—seas of data. The purportedly fully automated task of web scraping, in fact, requires continued human intervention, and it is only through the day-to-day labor of those software developers involved in scraping that the continued provision of data for commercial use is possible.
Conclusion
This incursion into the history of web scraping and the description of its current ecosystem is meant to highlight how the production of digital goods and specifically those of generative AI are preceded by the necessity of having vast amounts of data ready to be processed. Furthermore, it is not the case that the web data available online are ready-made to be included in the generative AI pipeline, instead, we mean to highlight that before data can be used, it needs to be gathered at scale.
In the preceding pages we have identified the necessary but specific labor of scraping required for data to be used in the production of digital products. Through an analysis of the history of web scraping, we have established how the labor of scraping is to be identified in the existing technical elements of languages, standards, protocols, and other infrastructure of the web, which were developed through open-source and free collaboration of software developers. These scraping conditions, although not necessarily recognized as labor now that these software products are easily available, are nonetheless the result of countless hours of individual and collaborative efforts of human workers. Additionally, web scraping, in line with much of the purported marketing of automated systems, is very much dependent on concrete continuous labor. This web scraping labor should thus be included in the general analysis of novel (or recently identified) forms of multiplying data work, in tandem with the later data pipeline production and maintenance recognized in data collection, data curation, data annotation, model training, model evaluation and data verification. As we have shown, this particular form of data work is marked by the constant mutation of the data transfer ecosystem, both made possible and limited by shifted technological and legal regimes, which frame the troubleshooting nature of web scraping software maintenance and necessity for handcraft human labor.
Taken together, the preceding indications allow us to suggest that scraping labor should be studied as a distinct component of data labor. As we have shown, the collection of data is dependent on conditions of data accessibility which exceed the aspect of technical difficulty which historically justified the need for scraping. Moreover, as these conditions do not remain static, the prospects of full automation in what concerns data collection remain uncertain, as the labor of scraping is defined by the constant need for maintenance of existing data pipelines which nonetheless are unable to accommodate the constant legal and technical changes to the web scraping ecosystem. It is on account of this that the labor of scraping is defined as a continuous cat-and-mouse game, which in addition to framing the concrete practices of web scrapers, contributes to its hidden nature, as it constitutes a form of gray work, whose legality is not always a given.
As these conditions keep changing and evolving, closer attention, focusing on this important yet relatively undertheorized moment of data collection, is warranted—a direction toward which this paper hopes to contribute.
Open access
