10.1 Introduction

This quote from the appeal case of Mr XX, an Iranian asylum seeker of Kurdish ethnic origin in the UK, addressed the use of social media, particularly Facebook, in asylum and immigration appeals, as well as the implications of surveillance by social media platforms and states. As asylum applications require extensive evidence, digital tools used by asylum seekers have also increasingly appeared in courts alongside identity documents, travel documents, medical certificates, biometric data and other evidence such as newspaper articles. In their guidelines on credibility, the UK’s Home Office does not list social media posts or other digital technologies among the evidence that can be considered by its caseworkers.Footnote ² Nevertheless, in the case of Mr XX, he was appealing the refusal of his asylum claim based on material critical of the Iranian government that he had posted on Facebook, and on the effects these digital traces might have in the case of his removal to Iran.Footnote ³

What was at stake in this case was not simply to evaluate the credibility of Mr XX’s Facebook postings. Rather, as Mr XX’s narrative of opposition to the Iranian government was deemed to lack credibility, the kinds of digital surveillance enabled by social media platforms were crucial to the appeal. Digital surveillance could expose Mr XX to the risk of harm if removed back to Iran. For asylum seekers, surveillance cannot be taken for granted but becomes a matter of contention in courts. What constitutes digital surveillance, who is targeted, by whom, and what are the probabilities of such surveillance? States, international, and humanitarian actors use digital devices to filter, hierarchise, and confine people on the move. Private and public actors are entangled in digital surveillance, with the role of big tech companies such as Palantir garnering public attention.Footnote ⁴ However, there has been less attention to the sociotechnical problematisations of data extraction and digital surveillance in asylum processes and how asylum seekers are required to connect their digital traces and surveillance.

The Upper Tribunal Immigration and Asylum Chamber (UTIAC), where Mr XX appealed a previous negative decision by the FtT, had to decide on ‘the issue of risk on return arising from a person’s social media use (in particular, Facebook)’.Footnote ⁵ What was required here was not certainty or even high probability, but the lower probability of a ‘reasonable degree of likelihood’.Footnote ⁶ The lower standard of proof used in asylum and immigration cases governs how data from digital platforms and related surveillance claims are evaluated in courts. Digital platforms appear in several guises: they are used by asylum seekers to collect data to substantiate their cases, they are data-extractive tools that commodify data, and they are also used by governments to track, surveil and monitor people on the move.

This chapter traces how asylum seekers are exposed to epistemic violence when using data from digital platforms to substantiate their claims for international protection. Epistemic violence has generally been understood as the violence that subjects experience as knowers, often through the erasure or silencing of their knowledge.Footnote ⁷ Border and migration scholars have formulated ‘epistemic borderwork’ as an instantiation of epistemic violence ‘used by state authorities to silence unwanted voices, undermine insurgent perspectives, and stifle the capacity of refugees to draw attention to their own mistreatment’.Footnote ⁸ Here, I show how new forms of epistemic violence emerge not just from the erasure and silencing of knowledge but also from the demand to know.

First, I argue that we need to distinguish between data extraction and surveillance to account for the difficulties that asylum seekers face in making their case, which I call here ‘data wrangling’. In Shoshana Zuboff’s acclaimed diagnosis of surveillance capitalism, data extraction is largely equivalent to surveillance.Footnote ⁹ While Zuboff focuses on big tech companies such as Facebook (now Meta) and Google, by equating data extraction and surveillance, surveillance extends to all actors involved in digital economies. However, asylum seekers have to evidence the likelihood of surveillance and, in the wake of the 2022 XX case, have been asked to bring ‘all the data’ to assuage suspicions of manipulation and instrumentalisation of social media. The demand to know has become an insatiable demand for more and more data.

Second, another form of epistemic violence emerges through the impossible demand to know the unknowable. I introduce the concept of ‘platform power’ to show that it is impossible to clarify which third parties can access data on digital platforms, whether data can be sold further, to whom, and how it can be recombined and reused for the purposes of surveillance. Platform power also raises questions about which concepts scholars and activists can mobilise to critique these forms of epistemic violence.

To develop these arguments, I discuss a selection of UTIAC decisions where the use of Facebook is problematised in relation to state and non-state digital surveillance. The UTIAC archive can be seen as integral to discourses of transparency and open government data, where data is made available to invisible publics to foster accountability. However, open government data enacts the ‘data subject as both auditor and entrepreneur’.Footnote ¹⁰ This is particularly the case where data that is made visible is not necessarily legible and intelligible.Footnote ¹¹ The UTIAC archive has not been prepared for archiving and has minimal metadata. The limited metadata that the archive contains distinguishes between ‘reported’ cases, which are citable and therefore relevant to future legal arguments, and ‘unreported’ cases, which constitute the majority of the archive.Footnote ¹² The archive is hosted on the Amazon Cloud and is searchable by keywords. In this chapter, I focus on a selection of reported and unreported cases where ‘Facebook’ is mentioned, given that it had the highest frequency over the years.Footnote ¹³ Facebook is also a paradigmatic case of surveillance capitalism.Footnote ¹⁴ Cases mentioning Facebook also continually increased in recent years. Up to the end of 2022, there had been 551 cases.Footnote ¹⁵

The chapter is structured in three sections. The first section situates digital surveillance in the context of the literature on the datafication of borders and migration. The second section analyses a selection of cases to show how discontinuities between data extraction and digital surveillance enable new forms of epistemic violence through the insatiable demand for ever more data. The third section introduces platform power to highlight another form of epistemic violence produced through the demand to know the unknowable. I conclude with reflections on how critical interventions need to simultaneously reject epistemic violence and defend asylum seekers’ rights.

10.2 Datafication, Data Making and Data Wrangling

The literature on digital technologies and migration has gradually shifted its attention from mobile devices as ‘lifelines’ for refugees, used to access information, track routes and keep in contact with family and friends,Footnote ¹⁶ to the datafication of migrant bodies.Footnote ¹⁷ Datafication is not simply the process of collecting data about people but also that of transforming bodies, actions, and things into data that can be processed by algorithms.Footnote ¹⁸ Biometric, digital, statistical, social media or transactional data have performative power, as mobile bodies under surveillance are classified according to racialised, gendered and class lines.Footnote ¹⁹

More recently, these critiques have been recast in light of the extensive processes of datafication and the use of machine learning algorithms. For Louise Amoore, the emergence of the ‘deep border’ powered through deep learning ‘explodes and scatters biometric data so that they are no longer primarily connected to characteristics as such, but rather gather together with a multiplicity of data features in a deep learning model that renders all data equivalent; all data as potential borders data’.Footnote ²⁰ However, border data includes many different forms of data. Migrants are also increasingly compelled to make data. They are turned into forced tech usersFootnote ²¹ and data makers.Footnote ²² Legal scholar Fleur Johns explains this transformation in the broader context of governance by data, whereby ‘subjects must now call themselves forth digitally to recognize themselves, and make themselves recognizable, within surrounding social and legal orders’.Footnote ²³

In this dual interpellation as tech users and data makers, migrants straddle the tensions of datafication – of using data and having their data continuously extracted by border and humanitarian actors, corporations, and increasingly by all. Media scholars Sandra Ponzanesi and Koen Leurs have outlined this duality of datafication, where data extraction and data making are not separate but entwined:

By using digital devices, platforms and apps, migrants generate more and more data required by states and valorised by platforms. This data comes in different forms, not just analogue and digital, but also biometric and biographical, statistical and sensor data, social media and location data, to name just a few.

The data is not only extracted by states – with the support of private and humanitarian actors – but also by big-tech commercial platforms. These platforms extract what we can call, following philosopher Bernard Harcourt, ‘expository’ data. Expository data is disclosed and volunteered on platforms such as Facebook, YouTube, or TikTok. This expository data refers to the content that migrants (and other users) post, whether as unstructured text data, images, or structured as ‘likes’. Harcourt outlines the insidious power of this expository data: ‘It is about every little desire, every preference, every want, and all the complexity of the self, social relations, political beliefs and ambitions, psychological well-being. It extends into every crevice and every dimension of everyday living of every single one of us in our individuality.’Footnote ²⁵ It is this granularity that makes expository data attractive for border authorities, as expository data can be used to challenge the credibility of an asylum seeker’s account of who they are, their connections, experiences and actions.

Expository data is supplemented by ‘surplus data’, which circulates on digital platforms, but it is not immediately visible.Footnote ²⁶ Surplus data is produced as migrants have to use digital platforms to connect with authorities, register their claims or bring evidence to substantiate their claims for international protection. Surplus data includes metadata and transactional data. While not directly used by states for border control, this data can be retrieved, for instance, through mobile phone extraction. Thus, alongside data extracted for ‘filtering’ mobile people at borders and assessing claims to protection and expository data extracted by platforms, surplus data circulates across multiple platforms that are ‘parasitic’ on the data that everyone – including migrants – produces.

As migrants are increasingly compelled to make and relinquish data, scholars have focused on processes of extraction by border and humanitarian actors, rather than on how asylum seekers have to work with data to lodge and substantiate their claims. For instance, they have explored the use of mobile data extraction and the challenges to it in asylum practices in Belgium, Germany, the Netherlands and Sweden.Footnote ²⁷ In the UK, the secret confiscation of mobile phones by the Home Office and subsequent data mining have been successfully challenged in a legal case led by Privacy International.Footnote ²⁸ Scholars and activists have also highlighted the effects of database interoperability on asylum seekers’ claims, particularly when applications are deemed inadmissible in one state and asylum seekers can be returned to another state deemed ‘safe’.Footnote ²⁹ Or they have shown how the humanitarian digital ecosystem intensifies colonial and paternalistic dynamics.Footnote ³⁰

Asylum seekers also undertake extensive data work to present digital and analogue information to caseworkers and judges for consideration as evidence.Footnote ³¹ They are compelled to work with data and become ‘data wranglers’ and not just ‘data makers’. While paperwork has not disappeared, digital data brings the promise of granularity, of minute details that could make a difference. Data wrangling is used in data science to refer to the processes of cleaning, combining and transforming datasets to make them analysable and to create new insights. Data scientists see three outcomes of data wrangling: making data usable, credible and useful.Footnote ³² I use data wrangling here to render the practices of making data usable, credible and useful for their claims for protection. Data wrangling draws attention to the struggles and difficulties that asylum seekers experience in transforming data into legal evidence of persecution and surveillance.

What does data wrangling mean for asylum seekers and their claims for protection? Asylum seekers have used expository data – their posts on digital platforms criticising governments in their country of origin, whether before arrival in the UK or sur place in the UK – to support their claims for international protection. To do so, they have needed to prove to a standard of ‘reasonable likelihood’ that they were under surveillance. How to connect data extraction and state and commercial surveillance has been a question that asylum seekers and their legal representatives have had to grapple with.

For asylum seekers, surveillance is uncertain, be it surveillance by social media companies or states. Zuboff’s acclaimed diagnosis of surveillance capitalism worldwide renders data grabbing as a new frontier of capitalism, producing profit from reality itself: ‘Machine intelligence processes behavioral surplus into prediction products designed to forecast what we will feel, think, and do: now, soon, and later’.Footnote ³³ Monitoring and manipulation are at the heart of surveillance capitalism, where advertising and marketing have taken a central role. Through surveillance capitalism, data extraction folds onto surveillance and vice versa. This also enables the extension of commercial surveillance into state surveillance through ‘the growing interdependencies between the intelligence agencies, resentful of constitutional constraints on their prerogatives, and the Silicon Valley firms’.Footnote ³⁴

While data extraction fuels and is fuelled by surveillance, scholars have also started problematising the equation of data extraction with surveillance. For example, drawing on Michel Foucault’s conceptualisation of power, Harcourt questions the dominance of surveillance to understand our relation to digital technologies. ‘We are not being surveilled today so much’, he argues, ‘as we are exposing ourselves knowingly, for many of us with all our love, for others anxiously and hesitantly’.Footnote ³⁵ Harcourt’s argument goes beyond work that has nuanced or specified surveillance understood through the architectural model of the Panopticon via variations on ‘opticon’ such as synopticon, oligopticon, polyopticon or banopticon.Footnote ³⁶ Media scholar Taina Bucher has also challenged the imaginary of a Facebook Panopticon, arguing that it is not the threat of monitoring that is at stake, but ‘the possibility of constantly disappearing, of not being considered important enough’.Footnote ³⁷ AI researcher Stephanie Sherman has similarly gestured towards the limits of equating datafication with surveillance:

Digital surveillance does not account for the challenges that asylum seekers face connecting their digital traces and the likelihood of surveillance by state and non-state actors. It also does not account for their data-making, the political acts they engage in, and the forms of digital citizenship they enact.Footnote ³⁹ If we start from the asylum seekers’ data-wrangling rather than the certainty and ubiquity of surveillance, we can understand how they are exposed to new forms of epistemic violence. To do so, the following section discusses a selection of UTIAC decisions in which the relation between data extraction and surveillance was disputed.

10.3 Data Extraction and the Limits of Digital Surveillance

It was only at the beginning of 2022 that the XX case began to change judicial understandings of the roles of surveillance by state and social media platforms. Mr XX claimed asylum ‘based on his fear of persecution because of his political beliefs, specifically his support for the Kurdistan Free Life Party (Partiya Jiyana Azad a Kurdistanê, (“PJAK”)’. He adduced evidence of being persecuted in Iran and of sur place activities in the UK, where he ‘had attended demonstrations and set up a Facebook account, which included material critical of the Iranian government’.Footnote ⁴⁰ While his account of events was deemed to lack credibility by the FtT, and this finding was preserved by the UTIAC, the question remained whether he would be at risk of persecution if removed to Iran, given his social media activities and potential surveillance by the Iranian government.

The case raised issues of government surveillance through data extraction from digital platforms, but the Tribunal went one step further than previous cases that relied on social media activities. It asked Facebook questions of clarification on its own surveillance and data-sharing practices. After an initial lack of response from Facebook to Home Office’s questions, Facebook Ireland responded to a Tribunal request. The Tribunal wanted to understand how Facebook content could be accessed by third parties – including governments – and what kind of surveillance could still be in place based on digital traces after an individual’s account had been deleted. Two experts called by the appellant gave oral and written evidence on social media platforms and the Iranian surveillance capabilities, respectively.

The Tribunal assessed the different types of state surveillance that are ‘reasonably likely’ based on social media activities. Mass surveillance by the Iranian state was found unlikely, both because data is assumed not to be easily ‘scraped’ and given an assumed discrepancy between the capabilities of the Iranian authorities and their discourse.Footnote ⁴¹ The Tribunal deemed Iran to claim to do more than it was effectively able to do in practice. If mass surveillance was improbable, the only reasonable likelihood remaining was that of targeted surveillance. Yet, the probability of targeted surveillance was also speculatively diminished as Facebook did not answer questions concerning activities by third parties:

…

The Tribunal grappled with different forms of surveillance – surveillance on social media platforms, surveillance by third parties, mass surveillance and targeted surveillance. There was no continuous, certain or ubiquitous surveillance. The connections between social media posts and different forms of surveillance were a matter of contestation and weighing of probabilities.

Given the focus of the case on whether the removal of Mr XX to Iran would pose a risk of harm, the key question was how the asylum seekers’ expository data might entail surveillance by third parties, in this case by the Iranian government. Surveillance studies scholars have shown that ‘Iran has a complex assemblage of overlapping governmental and non-governmental institutions that are mandated with the regulation of internet content and access’.Footnote ⁴³ However, the UTIAC found that the difficulties of scraping data from Facebook would make mass surveillance by Iran unlikely. If mass surveillance by third parties was unlikely, then the question became one of targeted surveillance. Targeted surveillance presupposes a hierarchy of significance. What qualification would then turn a person into a target of surveillance? Only high-profile individuals whose Facebook accounts had attracted a lot of attention on social media were deemed to be likely targets of surveillance: ‘Whose Facebook accounts will be targeted, before they are deleted, will depend on a person’s existing profile and where they fit onto a “social graph;” and the extent to which they or their social network may have their Facebook material accessed’.Footnote ⁴⁴ As third-party surveillance, including state surveillance, relies on the visible digital inscriptions that users produce on Facebook, it was not seen to work at scale or to operate on the ‘mass’ of the population.

Nonetheless, for Mr XX, it was the materiality of digital traces, rather than the content of expository data, that enabled the positive result of his appeal. As the UTIAC put it, ‘the nature of his activities puts him at real risk of persecution’.Footnote ⁴⁵ Mr XX’s appeal was accepted given that he had not chosen any of the privacy settings on Facebook, had an extensive social graph with thousands of friends, and continually posted content that attracted ‘likes’. Although Mr XX’s narrative of having taken part in demonstrations in the UK was deemed to be ‘contrived’, the perdurance of digital traces of his activities meant that his forced removal to Iran would incur the risk of persecution. The Tribunal concluded that, compounded with the fact that none of his Facebook settings had been set to private, ‘[h]is carefully curated (albeit contrived) social graph was, in this particular case, just sufficient in our judgment to establish a risk that he has been subject to surveillance in the past that would have resulted in the downloading and storing of material held against his name’.Footnote ⁴⁶

It was exactly because Mr XX did not make his settings private, because data might have been monitored before he deleted his account and because he was actively associating with large numbers of Facebook friends that he could not be removed back to Iran. The public materiality of his digital traces was compounded by the materiality of his networked friends such that his profile gained sufficient network value on social media to be deemed a likely target of surveillance. By generating expository data, Mr XX was afforded legal protection and ‘safety’ from the risk of state violence. But to do so and avoid the risk of harm and persecution, he also needed to make himself demonstrably surveillable.Footnote ⁴⁷

Before this case, in appeals where UK asylum applicants used social media extracts to document their political activities and resistance – both in countries of origin and sur place – the use of social media accounts, particularly Facebook, was often part of the reason the appeal was dismissed.Footnote ⁴⁸ This was mainly because digital traces were seen as effaceable. For instance, the UTIAC adopted such a judicial assessment in the case of a Kurdish asylum seeker from Iran:

In these cases, judges often assumed that all that was required to nullify the risk of surveillance in the countries of origin was for asylum seekers to delete their social media accounts. Indeed, in the case of Mr XX, the FtT decided that the asylum applicant could simply delete their Facebook account before being forcibly removed from the UK to Iran, as his claim was rejected. These requests to delete social media accounts have been common in cases where rejected asylum seekers challenged decisions to remove them from the UK.Footnote ⁵⁰

In having to connect data extraction and digital surveillance, asylum seekers are confronted with new forms of epistemic violence. This is not just the epistemic violence of erasure and deletion, but the epistemic violence of the demand to adduce more and more data and to know how their digital inscriptions are used. This demand to know is undergirded by a suspicion of manipulation and faking. Another UTIAC case of an Iranian citizen of Kurdish ethnicity highlighted this shift from demonstrations and organising (e.g., strike organising) as a form of political activism to the use of digital platforms. Mr BR ‘recorded anti-Government raps on his mobile phone while in Iran’. Subsequently, ‘His phone was lost or stolen, and he became aware that his anti-Government raps had been shared on various online platforms (Facebook, YouTube and Telegram)’.Footnote ⁵¹ Mr BR left Iran after his home was raided by the police, made his way to Germany, where his asylum claim was rejected and then reached the UK via France (presumably via Calais). Mr BR’s appeal against the Home Office’s refusal of his asylum claim was rejected at the FtT, where the judge questioned the credibility of Facebook posts as eminently falsifiable: ‘posts can be taken down and altered, settings can be changed between public and private, and screenshots only show the status shown on a page for the moment of the photograph and may only show a temporary change to a page made for the purpose of the screenshot itself.’Footnote ⁵²

However, at the Upper Tribunal, the judge did not accept this view of social media inauthenticity and highlighted the materiality of social media data, concluding that it was not ‘reasonable to assume that the anti-Government Facebook posts are not accessible to the Iranian authorities, even if they have been taken down’.Footnote ⁵³ Here, the materiality of data extraction on digital platforms meant that state surveillance was a probability that could not be discounted any more than it could be assumed.

Despite the positive decision in the XX case, subsequent decisions in cases in which asylum seekers used Facebook to express criticism, opposition, or resistance minimised or denied the likelihood of asylum seekers’ surveillance by states or state proxies. In the wake of this reported case, courts deemed surveillance ‘reasonably likely’ only for high-profile activists or party members.Footnote ⁵⁴ The likelihood of surveillance was assessed based on extended temporalities of posting and networking, which would have allowed governments to become aware of these anti-government activities. The production of networks (‘social graphs’) could turn certain users into ‘nodes’ and, therefore, potential targets for state surveillance.

Asylum seekers’ data making on digital platforms also enabled intensified data extraction, as judges demanded that more and more data be submitted or deemed the Facebook posts in the appellants’ bundle insufficient. Following discussions about ‘invisible’ metadata, such as location or time of postings in the XX case, the UTIAC made requests for surplus data to supplement the submission of expository data. For instance, in another decision citing the XX case, the UTIAC judge declared printouts of Facebook posts of limited value as ‘we have not been provided with the appellant’s full Facebook activity profile by way of the Download Your Information (“DYI”) function in Facebook. The importance of full disclosure was highlighted by the Upper Tribunal at [96] of XX (PJAK – sur place activities – Facebook) Iran CG [2022] UKUT 23 (IAC).’Footnote ⁵⁵

‘Download Your Information’ was meant to provide the Tribunal with the data about the location, time or frequency of social media activities. Download Your Information also contains sensitive personal data, family connections, networks and intimate histories.Footnote ⁵⁶ To make themselves demonstrably surveillable, asylum seekers had to make themselves entirely and absolutely knowable through all the data.

In the XX case, the UTIAC used the extraction of metadata on digital platforms to argue that ‘a wealth of wider information, including a person’s locations of access to Facebook and full timeline of social media activities, readily available on the “Download Your Information” function of Facebook in a matter of moments’ should be submitted to the court.Footnote ⁵⁷ In the absence of such metadata, printouts were deemed manipulable and therefore lacking in credibility. Moreover, the applicant in this case, Mr HA, was found not to have attracted the attention of Bangladeshi authorities due to his low-profile activities sur place, including participation in demonstrations and Facebook posts. Alongside the lack of metadata, the judge contrasted governmental practices in Bangladesh to those in Iran so that the risk of surveillance appeared unlikely:

Yet, Iranian asylum seekers themselves could still have their appeals rejected in the wake of the XX case on similar grounds of lack of surveillance by the state and assumed state inability to access their digital traces.

In another Tribunal decision following the XX case, Facebook posts submitted by an Iraqi citizen of Kurdish ethnicity were not accepted as evidence based on the entangled materialities of networks, privacy settings, language and metadata. The evidentiary value of Facebook posts was discounted given the absence of translations and exhaustive metadata, as now expected through ‘Download your data’:

Alongside the responsibility of asylum seekers to know the intricacies of social media platforms, their interfaces and privacy settings, the legal cases analysed above continued to reinforce their responsibility to ‘collect, assemble and order data’ without attention to asylum seekers’ capabilities and resources to do so.Footnote ⁶⁰

The UTIAC requested translations of Facebook posts and extensive data disclosure across several cases citing the XX case, without any consideration of the resources and labour required to make data and translations available. While in some cases Google translations seemed to be acceptable to the UTIAC, in other cases the question of certified translations was brought up against the applicants:

Because all this data was potentially available from Facebook, the UTIAC reasoned, it should be disclosed and submitted to the courts. The demand to know was severed from the material infrastructure and resources that enable knowledge production.

Following the XX case, asylum seekers in the UK have faced intensified demands for data making and wrangling. As we have seen, expository data has become increasingly insufficient as evidence, and it needs to be supplemented by the metadata extracted by digital platforms. Echoing Tasniem Answar’s and Beste İşleyen’s observations in the context of terrorist financing trials, here too, ‘the burden to know’ was placed on ordinary people, who were expected to know while simultaneously having their knowledge contested, discarded or dismissed.Footnote ⁶² The next section introduces the concept of platform power to show how the demand for ever more data is supplemented by an impossible demand to know the unknowable.

10.4 Platform Power and the Violence of Knowing the Unknowable

Asylum seekers wrangling with data to substantiate their cases in court have to connect data extraction and digital surveillance. For them, surveillance cannot be taken for granted as the aim and purpose of data extraction, whether by the state or companies. However, the demand that asylum seekers connect data extraction and surveillance, even to a lower standard of proof, is an impossible demand. This is not, I argue, due to Facebook’s secrecy about its data sharing and commodification practices but due to how platforms operate in the digital economy.

As media theorists have explained, online platforms are a ‘programmable digital architecture designed to organize interactions between users – not just end users but also corporate entities and public bodies … geared toward the systematic collection, algorithmic processing, circulation, and monetization of user data’.Footnote ⁶³ Platforms are characterised by two types of relations: on the one hand, data extraction from users and the incitement of users to make more and more data and on the other, service provision through relations with ‘complementors’. On digital platforms, the more data is made, the more valuable the platform is.

The expansion of platforms has given rise to ‘worries about the concentration of power and an oligopolistic market structure’ particularly given the ownership of large platforms by the US Big Five tech companies.Footnote ⁶⁴ This assumption of centralised legal-political power underpins the UTIAC questions to Facebook. However, as Tobias Blanke and I have argued, platform power also needs to be understood through the opposite move: of breaking up digital components into ‘services’ and ‘microservices’ offered to an indefinite array of third parties.Footnote ⁶⁵ In that sense, ‘platform power emerges through the dual move of decomposing into small components and recomposing these across the web’.Footnote ⁶⁶

More importantly, in doing so, platforms blur their own boundaries: they can extract data by connecting to other platforms and share it with third parties. Platforms have become ‘an opaque and complex ecosystem in which connections are invisible to the public eye and hence largely beyond societal control’.Footnote ⁶⁷ Blanke and Jennifer Pybus have also drawn attention to ‘the underlying and integrated network of technical actors that supports the capture of user data well outside the discrete boundaries of the platform’.Footnote ⁶⁸ This ecosystem connects advertisers and data brokers who can sell the data to an even more extensive array of distant parties, including governmental actors. The result of this move is that ‘the Internet user is permanently connected to platforms – often without realizing it’.Footnote ⁶⁹ Moreover, the decomposition of platforms into a vast array of services and microservices means that ‘[p]latforms are so dispersed and heterogeneous that only machine learning can bring them together again’.Footnote ⁷⁰

Platform power works through data extraction and microservices dispersed across the globe. These platform operations mean that it is difficult, if not impossible, to say when or if data really disappears from the internet. It is also impossible to clarify which third parties can access data on digital platforms, whether data can be sold further, to whom, and how it can be recombined and reused for the purposes of surveillance. This impossibility is reflected in hesitations and ambiguities across Facebook’s policies. Facebook is not only ambiguous about what data remains after an account is deleted, but also about all forms of attempted disconnection that could lead to data loss. For instance, if a post is deleted while the user still has an account, Facebook explains that ‘Facebook may keep service-related information about your account, like IP address logins or email changes on your account, to protect your security, prevent abuse, and improve our services’.Footnote ⁷¹ Moreover, any data shared by friends about an account will not be deleted. Facebook also keeps log data but argues that it is depersonalised and, therefore, cannot be traced back to an individual.Footnote ⁷²

There are many ways in which data perdures, not just through Facebook’s reluctance to delete all the data, either upon account closure or after other deletions. Although in the XX case, expert evidence referred to an investigation by the New York Times, which found that Facebook had shared personal data with 150 companies, the UTIAC discounted any likelihood of data sharing with the Iranian government.Footnote ⁷³ An analysis of platform power indicates that such data sharing is not improbable, even if it is impossible to identify the exact third parties with whom data is shared. Data brokers further sell data that they have acquired from digital platforms. Given the ambiguous boundaries of platforms and the multiple temporalities of data, it becomes impossible to ascertain whether some data will continue to exist on the web and whether governments can access it with or without the mediation of other parties. The data is never sufficient, not just because social media platforms produce many opacities about the data they collect and share but because their operations are dispersed and can only be held together through machine learning algorithms.

Facebook transforms its boundaries through apps and services that allow it to expand into new markets and industries. Anne Helmond and Fernando van der Vlist have shown that ‘Facebook has been steadily growing by accommodating various strategic stakeholder groups through its architectural design and programmability’.Footnote ⁷⁴ Platform power blurs these binaries by being neither fully opaque nor fully transparent, neither private nor public, neither open nor secret, but by being both. In written evidence before the US Congress, technology researcher Caitlin Chin pointed out that intelligence services in the US can gain access to Facebook data exactly through third parties, without having to ‘break’ Facebook’s security arrangements or demand the disclosure of personal data:

States can request user data from social media companies as part of legal investigations or court cases. As a New York Times article notes, ‘which data law enforcement can get depends on the sort of request they make’.Footnote ⁷⁶ In their Transparency Report for January–June 2022, Facebook reported 237,000 government requests for 412,285 user accounts.Footnote ⁷⁷ Most requests come from the US, India, Germany, France and the UK. Yet, countries of origin for refugees are also listed with data requests: Pakistan has 1,373 requests, Nepal has 602, while Iran and Iraq have only 2 and 6 requests, respectively. However, the transparency reports do not tell us anything about data brokers and other non-state third parties.

The operations of platform power illuminate the epistemic violence of the demand to know the unknowable. As we have seen, asylum seekers do not only face the epistemic violence of knowledge erasure and disbelief. While these forms of epistemic violence also infuse the politics of asylum, I have focused here on epistemic violence enacted through the demand to know. Asylum seekers are confronted with the violence of the demand to know by making and wrangling with more and more data. Moreover, they are also confronted with the demand to know the unknowable.

10.5 Conclusion: An Impossible Critique?

This chapter has argued that critical diagnoses of generalised surveillance and surveillance capitalism do not account for the asylum seekers’ experiences of data wrangling. These disconnections between data extraction and surveillance come into stark relief when asylum seekers mobilise their expository data to support their claims for protection and to evidence their actions of protest and resistance. Drawing on appeal cases before the UTIAC in the UK, I have shown how asylum seekers have been required to submit more and more data in order to prove that they might be likely targets of surveillance in their countries of origin. Even as the materialities of data extraction on social media platforms like Facebook have supported claims of future risks of harm and persecution, these materialities have also given rise to intensified demands by courts to access ‘all the data’, to make metadata and not just expository data available in court, and present it in many different forms – both as original data through ‘Download Your Data’ on Facebook and through translations of posts, which turns data making into a difficult and expensive undertaking for asylum seekers.

Through an analysis of court decisions, I have drawn attention to how asylum seekers are exposed to new forms of epistemic violence. Following the social media guidance issued by the UTIAC in the XX case, subsequent decisions showed that the Tribunal required more and more data from asylum seekers, including metadata from digital platforms. Having all the data is not only a question of privacy and the exposure of personal and often sensitive information to courts. The request for all the data intensifies the asymmetries of power and confronts asylum seekers with new forms of epistemic violence. Asylum seekers have to make themselves surveillable in ways that harbour future risks of harm. They are forced to engage in more and more data making and wrangling without the money, support or resources that data work entails. Finally, by introducing the concept of platform power, I have shown that an intractable demand to know the unknowable is foisted upon asylum seekers and their legal representatives. The concept of platform power sheds light on the impossibility of connecting data extraction and the likelihood of surveillance by state or non-state actors. As platform power operates by taking the inside out and bringing the outside in, it is impossible to clarify which third parties can access data, whether data can be sold further, to whom, and how it can be recombined and reused. Platform power undoes the binaries of privacy/publicity, transparency/opacity, openness/secrecy, and monitoring/indifference. It also obfuscates the attempts to ascertain the likelihood of digital surveillance.

What can critical interventions do to challenge the insatiable demand for data and the impossible demand to know the unknowable? I suggest that critical interventions – by researchers, activists, journalists or lawyers – need to refuse the epistemic violence of knowing the unknowable in order to resist the dismantling of asylum seekers’ rights. As we have seen, more data wrangling does not bring better outcomes, as ultimately data extraction on social media platforms can always be found insufficient or otherwise deficient. The materiality of digital traces means that this data rarely disappears. A UTIAC decision preceding the XX case shows that this reasoning is possible. Here, the UTIAC judge situated the materiality of digital traces in the context of the likelihood of harm for the asylum seeker: ‘The appellant faces a real risk of persecution because the authorities will impute an anti-regime political opinion to him as a result of a combination of his Kurdish ethnic origin, illegal departure and sur place activities (whether undertaken in good faith or not and whether his Facebook account is deleted or not).’Footnote ⁷⁸ The judge avoided the blackmail of ‘either … or’ by working with the blurring of boundaries and materialities of digital platforms.

Could such a refusal of binaries inform the assessment of the likelihood of surveillance? While the boundaries between data extraction and surveillance remain unknowable, despite activist and academic attempts to make some of these circuits and entanglements visible, surveillance is neither certain nor uncertain. Its reasonable likelihood can be rarely, if at all, proven, but the operations of platform power also mean that one can reasonably hold that surveillance is not improbable. The double negation of ‘not improbable’ can be counterposed to the demand to know the reasonable likelihood of surveillance.

Introduction

On 22 April 2023, Adam D’Angelo – Quora founder, former CTO at Facebook, and board member at OpenAI – tweeted: ‘AI is going to enable the creation and enforcement of laws 1000x as complex as we have today. This is mostly a good thing’.Footnote ¹ Such claims are not new. Nick Bostrom warns against the superintelligence, while Benjamin Alarie propagates for ‘legal singularity’, which ‘contemplates complete law’, finally making ‘legal uncertainty obsolete’.Footnote ² Similar polarizations populate the blockchain space. Some excitingly claim that blockchain will establish a ‘truly international law’, ‘done programmatically’ and ‘by global technologists rather than country-specific lawyers’.Footnote ³ Others value blockchain’s capacity of creating new governance models potentially allowing for ‘more democratic and participatory decision-making’.Footnote ⁴ What is rarely disputed, however, is that new digital technologies – artificial intelligence (AI), machine learning (ML), and blockchain technology – are powerful and strongly influence modes of global and non-global governance in times of indeterminacy, political unrest, growing inequality, hyper-financialization, economic and social instability, and global warming, to mention just a few challenges.Footnote ⁵ Even as scholarship and practice agrees on challenges and risks of these developments, few have taken on the question of how fascisms fare in and under these conditions. Drawing on Gilles Deleuze and Félix Guattari’s work on fascism, this is what our chapter does.Footnote ⁶

Considering Guattari’s caution that ‘the new totalitarian regime … cannot be identified purely and simply with national fascisms of the Hitler or Mussolini type’, we step back from – and dive into – the question of technologically mediated governance and ask, in a Spinozian manner: What can fascism do in the twenty-first century, where does analysis begin, and how to make non-fascist modes of resistance, alliance, and community-building sense-able, thinkable, and do-able?

What we offer in this chapter is a transdisciplinary conversation on law, politics, and computational science, which probes questions and topics for rethinking contemporary global law and governance on a conceptual (yet real) level.Footnote ⁷ Our writing ‘in conversation’ is presented in three individually authored entries. Methodologically, we draw from critical collective writing as a ‘writing device’ of trans-subjective and transgressive openness.Footnote ⁸ Our individual contributions maintain ‘openness to dissensus, to the co-existence of multiple positions and perspectives’, while defining the critical alliance between the three of us as the basis for the force of our argument.Footnote ⁹ We do not collapse our stances into a common argument or suggestion. Even as we share a fundamental understanding of the problems embedded in AI, we do not propose a single ‘ready-to-go’ solution. Instead, we commit to an ethos of non-fascist living where a multiplicity of approaches and praxes are necessary and helpful – ultimately engaging in a multi-voiced ongoing questioning of the workings of fascism, more precisely of micro-fascisms, in our time.Footnote ¹⁰

Given the focus of this volume, we considered it appropriate to ask ChatGPT – a large language model developed by OpenAI using the Generative Pre-trained Transformer architecture – to enter the conversation by answering the following question: ‘If writing a chapter on fascism in an academic book titled “Global Governance by Data: Infrastructures of Algorithmic Rule”, what should be included?’ This is ChatGPT’s advice:

Regarding the possible impacts of new digital technologies and fascism on law, ChatGPT replied that although ‘these technologies can potentially enhance legal systems, they must be used in an ethical and responsible manner to avoid contributing to the erosion of civil liberties and human rights’.

Certainly, these suggestions are not wrong, but what could such a chapter do? The AI-generated blueprint is ripe with assumptions: what contemporary fascist regimes and movements are, that their use of data and technology give us insight into both their workings and our resistance strategies, and that democracy, human rights, and individual freedoms are fascism’s absolute opposite. Clearly, scholarship must do better, even as online scholarship provides massive amounts of training data for the Open AI model by which ChatGPT operates. Programmed to ‘think’ in ways reminiscent of the political as molecular decisionism (to borrow from Deleuze and Guattari), ChatGPT mirrors normative patterns of thought embedded in academia. As with ChatGPT’s mechanical recombination of tokens, the narcissism of the thinking subject remains attracted to what it already knows – what it has already established as true and real.

This presents us with a conundrum tangled with what we want this chapter to do: To make sense-able desires, dynamisms, and operations preceding the generation of patterns (of thought, perception, action, data) and to render thinkable alternatives grounded in an ethics of non-binary and equivocal relationality – whether as blockchain-informed modes of commoning, people’s councils, or posthuman alliances – we, like ChatGPT, must begin with a given prompt. We rely on mechanisms already having determined ‘which words or events are predicted and not, or which things are classified as “this” and not “that”’ (11.2). We do not start from nowhere: In the history of thought alternatives are present, each one building on a network of encounters, relations, and alliances. Thus, a philosophy of difference, as well as experiences from praxis (some mentioned, others tacitly informing) guide our approaches towards fascisms and fascistic desires. Although three different entry points, some concepts recur throughout the chapter, allowing us to push analysis beyond common-sensical analysis and thinkability: indeterminacy, plane of immanence, desire, de-individuation and dividualization, micro-fascisms, and decisionism.

Our contribution is less a stance against something (‘anti-’) – it encourages ‘the art of not being governed like that and at this price’.Footnote ¹¹ Thus we remain critical in the Foucauldian sense and affirmatively creative as we argue that forms of tech governance, global and across different scales, demand more than negation and oppositionality.Footnote ¹²

In what follows, Daniela sets out some fundamentals about fascisms and fascist desire – foundational for all three entries – building on Deleuze and Guattari. She exposes the complicities between fascist desire and dominant sense-making practices within and outside academia as well as in relation to new digital technologies. Dan focuses on how concrete operations of AI resonate with fascistic tendencies at the molecular and molar levels of social and technical structures. He proposes a non-dualistic approach to technopolitics and a non-fascist restructuring via people’s councils and cybernetic systems. Building on central aspects of fascistic desires and micro-fascism (set out by Daniela in 11.1) and a convivial technopolitics of commoning (set out by Dan in 11.2), Matilda turns to posthuman feminism in 11.3 to consider how such alliances foster politics, law and global governance towards a transversive human-non-human resistance of fascist desires. In conclusion, we return to our main points and the questions that are on all of us – also our readers – to continuously pursue.

11.1 In the Beginning(s): Speaking Up Against, About, and Through Fascism (Daniela Gandorfer)

A poststructuralist trope is to begin – a talk, essay, or chapter – by questioning the possibility of speaking in, with, and through language. Jacques Derrida performs this gesture in Force of Law. In it, Derrida deconstructs the concept of law and equates deconstruction with justice, depicting difference as ‘difference of force’ and ‘force as différance’.Footnote ¹³ This deferral of meaning as a construction of semantic undecidability/indeterminacy – taking place through the representational layer of language – serves not only as rhetorical force, but as pre-emptive limitation of accountability. What lies beyond representation? How to navigate force-fields undermining conditions of represent-ability? How to become response-able – that is, able to respond – to forces performing existential exclusion without representational traces?Footnote ¹⁴ Such exclusions and segmentations precede texts and materialize in the (molecular) unconscious and belong to the realm of physics,Footnote ¹⁵ where intensities ‘are not metaphors, but matter itself’.Footnote ¹⁶ That is where fascistic desire is real. Recognizability fails as neither form/content nor signifier/signified have yet taken hold. The question(ing) does not concern being addressed, but coming to matter (being excluded from mattering).Footnote ¹⁷

Before Force of Law, Derrida had already taken on ‘a natural force that might actually deconstruct atoms’Footnote ¹⁸ in ‘No Apocalypse’.Footnote ¹⁹ Unaware of the ongoing nuclear war, he declared that ‘the stakes of the nuclear question are those of humanity, of the humanities’.Footnote ²⁰ While the equation of humanity with humanities already registers as peculiar, it is another conclusive claim that reveals how power hides even within modes of thought operating in the name of justice. For it is not the annihilation of bodies (of all kinds) that are threatened by nuclear war and the unimaginable (nuclear) force, but literature as ‘the body of texts whose existence, possibility, and significance [that] are the most radically threatened, for the first and last time, by the nuclear catastrophe’. What is at stake, Derrida argues, is ‘the destruction of the symbolic order and a particular condition of representation’ that grounds power and agency of the humanities scholar.Footnote ²¹ Despite such limitations, deconstruction, as well as the work of many other poststructuralist (and especially feminist) thinkers, including Judith Butler, Luce Irigaray, and Julia Kristeva, were important strategies to counter semantic determinations, essentialized meanings, and binary oppositions. Language, after all, is a powerful technology that orders, determines, represents, and excludes. The linguistic turn – especially its post–World War II variations – demonstrated that language not simply represents or reflects reality, but actively constructs it.Footnote ²² The horrors of twentieth-century fascism posed the question of how discourse and language could play such a major role in atrocities whose degree and abjectness resides (and hide) in ineffability. Derrida’s concern with the possibility of literature emerged from language assisting the unspeakable crime of genocide. Hannah Arendt, Victor Klemperer, Theodor Adorno, and Lynn Tirrell all show how fascism hijacks language, mobilizing it for the material-discursive inscription of absolute essentialized friend/enemy distinctions into narratives of non/being. Adorno famously declared that ‘poetry after Auschwitz is barbaric’,Footnote ²³ drawing attention to the complicity of language and its capacity of enabling utmost violence.

There is much more to say about lethal discourses and the complicity of technologies in the proliferation of fascisms in the twentieth century. Yet, I open with the question of academic modes of thought, complicity, and speakability, returning to the question of beginning differently. Allow me thus to begin again from the middle, exploring the relationship between fascistic desire and sense-making practices to demonstrate how new modes of governance easily become attracted to a ‘desire that desires its own repression’Footnote ²⁴ and how new digital technologies might assist in this perversion. I have three entry points: the risk of speaking up, the tendency of speaking about, and the power of speaking through.

11.2 Machinic Micro-Fascisms and the Possibility of an Alternative Apparatus (Dan McQuillan)

11.3 Alternatives: More-Than-Human Posthuman Feminist Alliances (Matilda Arvidsson)

Let us begin (again) with Foucualt’s preface to Anti-Oedipus where non-fascist living means resisting ‘not only historical fascism, the fascism of Hitler and Mussolini – which was able to mobilize and use the desire of the masses so effectively – but also the fascism that is in us all, in our heads and in our everyday behaviour, the fascism that causes us to love power, to desire the very thing that dominates and exploits us’.Footnote ⁹⁴

How to do such living while embodying fascist desire for ‘one’s own annihilation, or desiring the power to annihilate’?Footnote ⁹⁵ ‘It is’, as Deleuze and Guattari say, ‘too easy to be antifascist on the molar level, and not even see the fascist inside you, the fascist you yourself sustain and nourish and cherish with molecules both personal and collective’.Footnote ⁹⁶ It is too easy to conflate being charged with fascism, with fascism having anything to do with what one does (be it AI/blockchain/algorithmic global/governance by data/law, or something else).

Historical fascism has immunized us somewhat from desiring ‘strongmen’ to ‘make “us” great again’ (willing our own annihilation through birthing the great ‘us’). And it has become commonplace to conceptualize fascism through the concepts of its own devising as historical events, personified in political figure-heads, forgetting how fascism feeds from our forgetfulness to resist our love of power, from our desire for the things that dominate and exploit us – to interrogate fascisms in their molar and molecular forms. Why applaud desires to governance through abstract optimizations by AI amplifying reactionary transformations of our societies, eager to let AI work through ‘black boxes’ and deep learning (Dan’s entry) – the black holes through which micro-fascisms connect in larger networks? Why this forgetfulness? Or, is it intentional?

The mode of resistance I envision – the ‘I prefer not to’ without succumbing to negativity, that Daniela has outlined – involves posthuman feminist alliances, generative towards present-future societies, sense-ative to detect and resist fascisms. Drawing on Deleuze and Guattari’s understanding of fascisms and desire (Daniela’s entry) and turning to posthuman feminism for analysis and an alternative politics,Footnote ⁹⁷ the question is how to refuse.Footnote ⁹⁸ But, what is the more-than-human posthuman feminist ‘we’ –‘We-Are-All-In-This-Together-But-We-Are-Not-One-And-The-Same’Footnote ⁹⁹ – promising a non-fascist alternative to the ever-present micro-fascisms nurtured by the decontextualization and decisionism of AI and its industry?

11.4 Where to Go from Here: Enter Alliances, Resist

Having approached the relations between digital technologies, law, and fascism from three distinct starting points, recognizing the generative differences between them, the concerns raised by our chapter are converging on a shared set of problems and possibilities. We have posed the questions of what fascism can do in the twenty-first century, where analysis begins, and how to make non-fascist resistance, alliances, and community-building sense-able, thinkable, and do-able. Is there non-fascist living with digital technologies and law? And, if so, what is needed in (or instead of) global law and governance in order for such a life to materialize?

Responding as a transversal alliance, writing not as one-and-the-same but as we-are-in-this-together is a method of resistance – by which we invite others to enter into alliance – but not an answer. Yet, it is not answers, let alone responses, but response-ability we are committed to. Our loyalty and labour is owed not to rhetorical strategy or transcendent principles but to ongoing questionings as creative and affirmatively critical practices. Such questioning must ‘go all the way down’, to the molecular unconscious, to quanta of fascist energy, the mass of movements, to attractions, and charges of all kinds, to fusions and fissions with the gravity of World Wars and to the most intimate cut, determining who or what matters and is excluded from mattering.

Academic analysis, method, thought – even if not always acknowledged – are legal political actions, and as such complicit: Our chapter has suggested that there is no escape from fascisms or no safe zone. No-one, however well-meaning, clever, and or conceptually clear, is immune. Concepts are not safe from fascist desire and capture, nor is language, or any other (digital or non-digital) technology of global governance. Indeed, it is not immunity, but creative refusal that we call for: ‘I prefer not to’, people’s councils, modes of commoning, feminist transversal alliance, the art of not being governed at that price, a collective commitment to non-fascist living.

12.1 Introduction

The Greek island of Samos, September 2021: Greek and European authorities inaugurated a fenced and high-security camp for refugees of Zevrou, equipped with ‘turnstiles, magnetic gates, X-rays, two-factor access control system (identity and fingerprint) which secure[d] and controlled entry and exit from the access center to whoever is entitled to it’.Footnote ¹ The camp, which was presented by the EU as a blueprint for others, has been the object of harsh criticisms by journalists, NGOs and activists. Since the so-called ‘refugee crisis’ in 2015, Greece has been considered one of the key ‘technological testing grounds’Footnote ² where the EU and Greek authorities implement advanced digital technologies to surveil migrants. Large datasets to predict migration flows, algorithms that score how suspicious a vessel is at the sea or biometric systems to enter refugees camps are examples of technological testing projects funded by the EU. Yet, the constant announcement of Artificial Intelligence (AI) projects at the border focuses attention on the use of this technology, more than on the existence of infrastructures which enhance structural violence. Indeed, as Rodgers and O’Neill contend ‘infrastructure is not just a material embodiment of violence … insofar as the material organization and form of a landscape not only reflect but also reinforce social orders, thereby becoming a contributing factor to reoccurring forms of harm’.Footnote ³

The criticisms raised about Zevrou camp in Samos constitute a case in point about how technology-based surveillance, that is, a normative approach to surveillance that criti-hypes the use of technology at the border,Footnote ⁴ has gained central stage in debates on borders and migration. Indeed, it is noteworthy that what unleashed indignation among scholars and journalists was the technological surveillance, more than the conditions of twofold carcerality that refugees experience – by being confined on an island and inside a closed camp. Notwithstanding a myopic focus on surveillance and AI ethics in many circles, the implementation of algorithmic surveillance systems at the border has also triggered broader contestations and mobilised campaigns about the harmful consequences they generate both on marginalised populations and on those who are labelled and governed as ‘migrants’. This chapter contributes to and amplifies debate surrounding these broader concerns.

This chapter examines how algorithmic power has transformed the ways in which the mobility and presence of people labelled as ‘migrants’ is controlled and hampered. Claudia Aradau and Tobias Blanke argued in Algorithmic Reason that algorithmic power stems from the use of algorithms as governance tools.Footnote ⁵ This power ‘works by producing truths – both as outcomes or outputs of systems and as part of the discursive reinforcement of particular norms, approaches and modes of reasoning’.Footnote ⁶ Despite computer scientists having neglected the political feature of algorithms by advocating for its neutrality, we build on critical data studies to confront this perspective. For instance, Luke Munn claimed that algorithmic power ‘is political, it calibrates subjectivities towards certain ends, it prioritises space in specific ways, and it privileges particular practices whilst suppressing others’.Footnote ⁷

However, in this piece we contribute to this concept of algorithmic power at the border by focusing on the production of racialised digital subjectivity through algorithmic systems. While data has being historically instrumentalised to identify people on the move,Footnote ⁸ we argue that the use of algorithmic systems in migration governmentality invisibilises, fosters and recrafts racialised bordering mechanisms. By speaking of racialised bordering mechanisms we draw on E. Tendayi Achiume’s definition of the term: ‘border regimes that variously allocate and curtail mobility and migration on a racial basis, largely relying upon facially race neutral mechanisms’.Footnote ⁹ Such bordering mechanisms are not confined to national frontiers: in fact, we refer to heterogeneous racialised bordering mechanisms, both visible and invisible (digital borders), through which hierarchies among people and their nationality status are established and reinforced.Footnote ¹⁰ Considering that ‘the border is a site of identity production’,Footnote ¹¹ we ask: Which racialised bordering mechanisms are enacted and to what extent existing ones are strengthened and invisibilised through use of algorithmic power? Which (racialised) subjectivities are recrafted and produced? In so doing, we shift attention from questions on surveillance, ethics and AI, which imply debating over the threshold of acceptability of algorithmic systems, towards an interrogation on the entanglements between emerging and sedimented modes of violence.

In order to address these questions, we join other scholarsFootnote ¹² in contending that it is essential to move beyond what Michel Foucault has designated as a ‘repressive hypothesis’ on power which underpins debates about the ethical risks of using algorithmic systems at the border, to instead analyse power’s constitutive effects, as Foucault and others have.Footnote ¹³ First, we contend that the use of algorithmic systems in migration governmentality invisibilises, fosters and recrafts racialised bordering mechanisms. Indeed, algorithms do not only target, discriminate and affect individuals in differential ways: they also shape digital subjectivities at the border finding patterns and connections on digital evidence. Investigating the productive subject-making dimension of algorithmic power is essential, we suggest, for shifting the focus from technologies as such and their uses and functioning towards how both collective and individual subjectivities are re-made and adapted. In fact, the production of racialised subjectivities is to a large extent coded by the algorithms; yet, in part they exceed the intentionality as well as the original purpose for which these AI-driven systems have been implemented, due to the frictions, resistance and appropriations enacted by the individuals.

Algorithmic power relies on the computational ability of finding patterns and connections on large datasets. For instance, biometric systems are able to find whether a biometric sample is similar to millions of fingerprints stored in a biometric database or which nationality is more likely to be scrutinise by border authorities. Throughout data, probabilities, mathematical functions and algorithms are put to work by nation-states to automatise the racialised mechanism of borders. Second, we suggest that in order not to see technology and migration like a state involves not taking for granted surveillance, bias and privacy as the main concerns, interrogating instead the differential harms it produces on migrants.

The chapter starts by engaging with the ‘repressive hypothesis’ on power and argues that a critical analysis of the use of algorithmic systems at the border requires challenging this and the juridical understanding of power that sustains it. The chapter moves on by focusing on two case studies of algorithmic power in operation in the governance of migration. First, it deals with biometric technologies enforced at the border, showing how through biometrics existing racialised hierarchies are replicated and new ones emerge. Second, it focuses on the algorithm-based visa system in the UK (or, the Streaming Tool), to illustrate how this has invisibilised the nationality-based criteria through which visa applications are assessed. The chapter concludes by suggesting that a critical analysis of ‘AI at the border’ should be coupled with a critique of racialised bordering infrastructures.

12.2 Critical AI Theories and Surveillance Paradigm at the Border

Building on publicly available EU and state documents, this chapter intervenes in debates about AI and racialised borders by foregrounding the productive, racialising and racialised effects of algorithmic bordering mechanisms. This contributes to critical theory on AI and raceFootnote ¹⁴ to migration scholarship on the use of technologies in migration and refugee governmentalityFootnote ¹⁵ and to legal literatures on racialised digital bordering practices in ways that this section outlines.Footnote ¹⁶ Migration literature has explored how in the aftermath of 9/11, digital technology has been increasingly used to enhance racialised surveillance.Footnote ¹⁷ Digital labour scholars have shown how AI has multiplied and mutated processes of racialisation that affect in particular those who are classified and controlled as ‘migrants’.Footnote ¹⁸ A growing body of scholarship across law and the social sciences has drawn on longstanding, critical traditions in those fields to challenge the disproportionate focus on digital surveillance encouraged by some work in surveillance and migration studies, drawing attention instead to the political ecology and the political economy of data extraction. For instance, speaking of data as oil Sy Taffel advocates shifting ‘the focus of digital exploitation away from a critique of surveillance capitalism … foregrounding unequally distributed social and environmental harms associated with the material dimensions of digital extractivism’.Footnote ¹⁹ This chapter makes a similar gesture – shifting away from a surveillance-centred analysis, and the repressive and juridical understanding of power with which such analysis is associated, by investigating how algorithmic systems are instrumentalised to invisibilise and foster racialised borders, and generate new ones.

Academic debates on the use of AI and biometrics at the border and in refugee camps mostly focus on the risk of enhanced surveillance or on the bias and discrimination which are invisibly enacted through AI-driven systems to assess asylum claims and to identify migrants.Footnote ²⁰ We argue that too great a focus on surveillance and discrimination overshadows the extent to which algorithm-based systems are embedded in and reinforce existing exclusionary borders and how they recraft racialised borders.

Critical works on AI and race have pointed to the risks of automated discrimination associated with the widespread use of socio-technical systems and shown how algorithms reinforce historical oppressions and embed forms of racism and white supremacy.Footnote ²¹ At the border, Louise Amoore has examined the impact of AI.Footnote ²² Yet, this chapter argues that there is a need to move beyond the surveillance narrative and further examine the impact of algorithms at the border by avoiding techno-hype narratives.Footnote ²³ For instance, Simone Browne has called for rethinking surveillance ‘through the multiplicity of blackness’:Footnote ²⁴ ‘rather than seeing surveillance as something inaugurated by new technologies, such as automated facial recognition or unmanned autonomous vehicles’, she advocates analysing digital technology as a ‘factor in how racism and antiblackness undergird and sustain the intersecting surveillances of our present order’.Footnote ²⁵ Our chapter contributes to this literature by exposing how AI is nowadays integrated as a tool within the racialised mechanism of border infrastructures in the European context. But rather than claiming that this technology is opening a new paradigm in the surveillance context, adding to the critical literature on racism and AI, we suggest that critical inquiry should problematise the concept of racialised border infrastructures that produces racialised.

Critical works on socio-technical systems are mainly centred on the modes of surveillance and tracking enforced through these latter and on the discrimination induced through automation.Footnote ²⁶ By questioning such an analytical perspective, we do not deny that repressive mechanisms are enforced through AI. Rather, we highlight that prohibition, discrimination and repression are situated in a broader economy of power relations, that enact subjectivities, automatically connects digital evidence and multiply racialised borders. That is, we are interested in looking at the productive dimension of power relations which have been deeply reconfigured by the systematic use of AI and algorithmic systems and exploring these productive relations and effects as sites of critique. In line with Isin and Ruppert’s account of ‘an analysis of the forms of power that generate, assemble and organise data’,Footnote ²⁷ we are interested not simply in studying which new technologies are deployed at the border but also in investigating ‘how new ways of life are brought into being’.Footnote ²⁸

12.3 Making Up Migrants Beyond Surveillance and Repression

Oppression, bias and discrimination are terms which are widely used for critiquing algorithmic systems. And algorithmic-based bordering and biometrics technologies are often represented in critical literature as repressive tools which work by enforcing bias and remote surveillance. However, in this section we argue that in order to develop a critique of the use of algorithms at the border it is key to undoing a repressive and juridical understanding of power and to question analyses that reiterate the image of panopticon-based surveillance. These two approaches are problematic because they narrow attention to the ways algorithmic mechanisms strengthen repression and bias and enhance surveillance. Such an understanding of the power exercised through AI also circumscribes contestation and forms of refusal of algorithmic and biometric technologies, including how they are conceptualised and enacted. To move beyond these limitations, we draw from Foucauldian scholarship on the productive dimensions of power and literature highlighting the material heterogeneity of digital surveillance practices. In this way, we gesture towards an analysis that investigates how some people are turned into migrants – what we call ‘making up migrants’Footnote ²⁹ – through algorithmic-based systems and processes. Our claim is that focusing on algorithmic infrastructures for ‘making up migrants’ opens up more productive avenues for contesting digitally racialised bordering regimes.

Resistance to biometrics and algorithmic systems is mainly framed as refusal, obfuscation and partial or total withdrawal.Footnote ³⁰ Indeed, if power – in this case exercised through AI – is deemed to be essentially repressive, enacting resistance would consist in practices that block or limit such repression. Questioning the repressive hypothesis which underpins many analyses of AI attunes us to a conceptualisation of resistance that Nancy Ettlinger has defined as ‘productive’: ‘some strategies of resistance make use of, rather than reject or disrupt and obfuscate, the digital environment. Productive approaches to resistance … target and subvert a particular strategy of subjection to serve digital subjects’.Footnote ³¹ An example of productive resistance, Ettlinger explains, consists in digital subjects who ‘may act critically on their freedom to challenge digital norms by making use of the affordances of the digital environment to produce new elements that serve their needs’.Footnote ³² Building on this analysis, we argue that it is essential to ‘cut off the King’s head’;Footnote ³³ that is, to move beyond a juridical understanding of power and attend to the ways algorithmic-based systems are being used to recraft racialised bordering mechanisms. This entails not taking for granted the modes of subjection enforced through AI and questioning whether enhanced surveillance and tracking are the key harms caused by using algorithmic systems at the border.

According to Foucault, a juridical understanding of power relations posits that power works by prohibiting and repressing and, relatedly, that resistance consists in refusal. In The Will to Knowledge, Foucault claimed that ‘we need to cut off the King’s head: in political theory that has still to be done’.Footnote ³⁴ In so doing, Foucault questioned what he called the ‘repressive hypothesis’, according to which power is conceived in terms of prohibitions and interdictions. Against this, Foucault shifted attention to the productivity of power relations, focusing on the regimes of knowledge and truth that are at play and how subjectivities are shaped by power relations.Footnote ³⁵ For Foucault, the notion of repression is inadequate for capturing these productive effects of power. It obfuscates that ‘what makes power hold good, what makes it accepted, is simply the fact that it doesn’t only weigh on us as a force that says no, but that it traverses and produces things, it induces pleasure, forms knowledge, produces discourse’.Footnote ³⁶

This repressive hypothesis about power relations, which is grounded in a juridical understanding of power, is, however, surreptitiously replicated in the ongoing debates on algorithmic and biometric technology, where AI is deemed to be an instrument through which repression and discrimination are enforced.Footnote ³⁷ Although it is certainly important to reiterate the general argument that AI-based bordering systems enable and reinforce discrimination, we suggest that this focus obscures other productive effects. For this reason, we gesture towards an analysis that investigates how some people are turned into migrants – what we call making up migrantsFootnote ³⁸ – through algorithmic-based systems and processes. As we show below, algorithmic bordering mechanisms are used for enforcing existing laws to restrict and obstruct migrants’ movements – such as the Dublin Regulation in Europe – to consolidate racialised hierarchies of mobility, and to multiply exclusionary classifications. As we illustrate in this chapter, EURODAC is a biometric database that reinforces the Dublin RegulationFootnote ³⁹ by storing fingerprints of migrants that are attached to a categorical classification based on their status in the territory (asylum seeker, ‘irregular’ entrant or ‘illegal’ stayer).

Alongside juridical conceptualisations of power, a growing scholarship refers to the panopticon as a surveillance model to explain the functioning of AI in migration governmentality and the. In fact, while it is crucial to understand how surveillance is enforced and strengthened through the systematic use of algorithms, our point is that a disproportionate focus on surveillance reiterates the image of the panopticon as a paradigm of power control.Footnote ⁴⁰ For instance, the above mentioned high-security refugee camp in Samos, Zovrou, has been defined a ‘panopticon for refugees’: ‘those who want to enter have to show a chip and have their biometric data checked on a fingerprint scanner’ and the cameras surrounding the camp detect ‘suspicious behaviours with the help on artificial intelligence’.Footnote ⁴¹ The designation of the refugee camp in Samos as a ‘surveillance panopticon’Footnote ⁴² on the one hand obscures the porosity of borders as well as migrants’ practices of resistances and tactics of flight;Footnote ⁴³ and on the other it narrows down the analysis to (capillary) surveillance while marginalise investigations about the carceral economies which are enforced and transformed through technology.

Notably, the panopticon is an architectural dispositive of surveillance which is characterised by a central tower from where all surrounding cells are monitored. The disciplinary function of the panopticon relies on an asymmetric visibility, in which the person in the tower can see without being seen, while prisoners in the cells do not know when and whether they are monitored.Footnote ⁴⁴ The trope of the panopticon is often mobilised in critical literature on technology for highlighting the pervasive and capillary modes of surveillance in a datafied society.Footnote ⁴⁵ While capturing important and alarming features of how technology is used in key sites, like workplaces, for monitoring workers or at the external frontiers of Europe for detecting migrants,Footnote ⁴⁶ as critical surveillance studies scholars have stressed, the trope of the panopticon reiterates an exclusive attention to (centralised) surveillance and, at once, assumes an overwhelming disciplinary gaze.Footnote ⁴⁷ Indeed, following Jef Huysman, it is worth noticing that many contemporary surveillance practices do ‘not work in such bounded institutional spaces and their hierarchical organisation of visibility’.Footnote ⁴⁸ According to Benoit Dupont, in order to fully grasp the possibility to resist surveillance mechanisms, it is key to develop a ‘more grounded knowledge of the actual interactions between those who watch, the machines and infrastructure they design and use to carry out their surveillance’.Footnote ⁴⁹ For instance, Aila Spathopoulou and Anna Carastathis have studied techniques of resistance among migrants on the Aegean islands of Lesvos and Samos whereby they have contested classifications and distinctions applied to them in their interactions with border police and infrastructure.Footnote ⁵⁰

By conceiving control in terms of (panopticon) surveillance scholars who subscribe to this view end up seeing migration and technology ‘like a State’,Footnote ⁵¹ corroborating states’ narratives about the possibility to exercise full control in real time. More precisely, the panopticon model eclipses the patchy visibility and the uneven temporality that underpin many mechanisms of surveillance.Footnote ⁵² For instance, in some contexts state authorities are not interested in constantly monitoring and detecting migrants; in other cases, even if they do, it is not possible to keep a constant and overwhelming visibility. In a nutshell, the analytics of the panopticon overshadow the heterogeneous modes of subjugation that are enforced also through AI and that in part have been changed and re-shuffled because of the incursion of digital technology. Instead, interrogating which racialised bordering mechanisms are put in place through the use of AI allows rethinking and widening the range of resistance, tactical appropriation and refusal.

One illustration of the tendency to analyse technological governance in ways that overstate the coherence and capacity of the State is the scholarly and popular debate around facial recognition technology. In many jurisdictions, the implementation of facial recognition technologies has triggered criticisms about privacy violations and discrimination, such as the civil society initiative against facial recognition, entitled Reclaim your Face, which calls for a ban on biometric mass surveillance practices. Proposals of this kind are usually grounded in concerns about surveillance and tracking activities and tend to emphasise the violation of privacy that the unregulated use of facial recognition might lead to. Both the public debate on facial recognition and initiatives set for resisting biometrics have been crucial in bringing to the fore the ethical-political risks connected to the use of facial recognition. These initiatives have built a critical mass towards the use of biometric systems by law enforcement authorities. However, these anti-surveillance campaigns do not move forward by questioning the violence of the State as such.

While biometric systems, such as facial recognition, are being implemented to gather digital evidence for police investigation and law enforcement, there is a need to question the violence of borders beyond these measures. By framing the critique of facial recognition technology around a concern about surveillance, scholars neglect to question racialised bordering mechanisms and the legitimacy of borders as such. If we start from the consideration that power, and more specifically power through AI, works in heterogeneous ways, it becomes possible to see that even resistance might take different forms.Footnote ⁵³ Refusal of and withdrawal from technologies of control illustrate how subjects engage in practices of resistance, both individually and collectively. The heterogeneity of resistance echoes the heterogeneity of modes of subjugation, given that the former is never fully external to the latter.Footnote ⁵⁴ Resistances are characterised by ‘tactical immanence’Footnote ⁵⁵ to power relations, while at the same time they also introduce an asymmetrical element with respect to power. In the field of algorithmic power, this means drawing attention to practices of resistance that enact a tactical twisting of AI, as Joler and Pasquinelli note: ‘rather than studying only how technology works, critical inquiry studies also how it breaks, how subjects rebel against its normative control and workers sabotage its gears … and to illuminate the invisible labour of knowledge that makes machine intelligence appear ideologically alive’.Footnote ⁵⁶

Accounting for the multiplicity and diversity of practices of resistance is key to highlight how the boundaries between consensus and coercion are often blurred in contexts where migrants are pushed to use technology for their own good. In other contexts, such as when authorities identify migrants upon arrival, they are not even asked to consent to the use of biometric technologies: they are just fingerprinted. And yet, at times they find ways to temporarily avoid mandatory fingerprinting through acts of self-harm – that is, by burning their fingertips. The effects of such practices of resistance depend on the socio-legal context in which they take place. For instance, citizens’ resistance to police facial recognition will have different consequences than migrants’ resistance to being fingerprinted at the border. From conditions of obstructed agency, such as those experienced by migrants before the border, binary oppositions between coercion and consent, freedom and obligation, tacit withdrawal and overt refusal as well as sheer opposition and tactical appropriation are deeply unsettled.

12.4 Algorithmic Power at the Border: Making People Illegal Through Algorithmic Systems

As the previous section illustrates, a radical questioning of the repressive hypothesis means investigating the effects of subjectivation that are produced by enforcing biometric technology at the border.Footnote ⁵⁷ The border is a context where the unequal impact and the racialised functioning of biometrics explicitly emerges. Indeed, biometrics and algorithmic systems are not simply deployed at the border:Footnote ⁵⁸ their implementation changes the ways in which migrants’ subjectivities are shaped (and what becomes an object of control and data extraction).Footnote ⁵⁹ The lexicon of surveillance is inadequate for fully grasping the modes of control, exclusion and racialised partitions enforced on migrants through technology.

Databases and algorithmic systems work to enhance and invisibilise exclusionary bordering mechanisms in the field of migration and refugee governance. At the same time, they also help invisibilise and recraft social injustice along new lines. These productive dimensions of bordering infrastructures are illustrated in the two empirical case studies we analyse below – the use of biometric systems by EU states for migrant ‘identity management’ and the use of algorithmic screening tools by the UK Home Office to filter ‘authorised’ movements and scrutinise visa applications.

12.4.3 EURODAC and Dodging Statistics

The development of deep learning approaches for biometrics has enhanced the performance of fingerprint systems that are systematically used by state authorities as a tool for border control.Footnote ⁸⁰ On the basis of the Dublin Regulation, EU Member States are obliged to fingerprint migrants who enter the territory without authorisation or who lodge an asylum application, and to send this biometric data into the EURODAC. In this database, Member States’ law enforcement authorities are obliged to categorise subjects in four different groups:

• Category 1 (CAT 1): Applicants for international protection.

• Category 2 (CAT 2): Irregular border crossing or unauthorised entry.

• Category 3 (CAT 3): Unauthorised stay of third-country nationals.

• Category 4-5 (CAT 4-5): Searches by states and Europol for the ‘detection or investigation of terrorist offences or of other serious criminal offences’.Footnote ⁸¹

Figure 12.1

Distribution of Categories 1, 2 and 3 of EURODAC from 2015 to 2019 in different countries (UK,Footnote ⁸² Greece, Spain and Italy). The difference on distributions shows how states make-up migrants and create different migration status inconsistently. Source: EURODAC statistics reports 2015–2019

(eu-LISA 2015, eu-LISA 2020).

Through these categories, states decide if migrants should be classified as deportable (CAT 2) or temporarily authorised on the territory (CAT 1). These statistics shows how EURODAC, a biometric database, fosters the distinction between legal and illegal migrants. Legal migrants are classified in CAT 1 while illegal migrants are classified in CAT 2 or CAT 3. Those in CAT 1 are forced to stay in the country where they have been fingerprinted. Figure 12.1 shows the distribution of categories in 2015 and 2019 on four different countries: UK, Greece, Spain and Italy. However, this categorisation of migrants at the EU border is not systematically fostered by Member States. For instance, while Greece classified most migrants in CAT 2 (which made them deportable), the distribution drastically changed in 2019 with a more equal number of migrants in CAT 1 and CAT 2. While Spain merely fingerprinted migrants in 2015 during the so-called ‘refugee crisis’, the country reported a large number of migrants in 2019. Italy also changed the distribution of categories: increasing CAT 3 and decreasing CAT 2.

EURODAC illustrates how biometric databases foster racialised bodies by assigning fingerprints to categories that assign migration status, that are in turn solely decided by border authorities. Moreover, significant differences observed in the distribution of these categories in only four years (see Figure 12.1) show how biometric systems are also instrumentalised by states to recraft racialised borders. We observe how the status of migrants being fingerprinted that can then be matched throughout biometric systems in EURODAC is recrafted due to policy or regulation shifts. For instance, states were allowed in 2015 to duplicate fingerprints if those entered ‘irregularly’ (CAT 2) claimed asylum (CAT 1) by copying instances from CAT 2 to CAT 1. However, in 2019 this practice was forbidden: if a person changes category, it should be moved but not copied to avoid duplicates in the statistics. In fact, EURODAC regulation has evolved over the years, which might have also impacted on the statistical distribution of categories. As the European Parliament states: ‘[T]he purpose and the scope of application of the EURODAC Regulation have been continually broadened since its creation’.Footnote ⁸³

The EURODAC Regulation will allow authorities to verify the identity or nationality of migrants through algorithmic systems that find patterns in fingerprints which will add new grounds for detention.Footnote ⁸⁴ The continuous upgrade of biometric database regulations will be instrumentalised to foster and create racialised borders. In the case of EURODAC, the new regulation approved moves beyond the scope of the first regulation of this database, allowing more data collection such as names and ID numbers for unforeseen purposes. The categorisation embedded in EURODAC creates and reproduces hierarchies of legal and non-legal status which are linked to fingerprints. Law enforcement authorities are able to know whether a person is an asylum-seeker, an ‘irregular’ border crosser or an authorised stayer by fitting their fingerprint into a biometric system within EURODAC. Depending on this biometric match, states can decide when they should be detained or not. Thus, a non-juridical understanding of power enables capturing how biometrics is used both for enacting migrants and crafting and multiplying legal and digital subjectivities.

12.5 The Streaming Tool: An Algorithmic System to Automate Visa Applications in the UK

To what extent are racialised borders invisibilised, fostered and recrafted through other algorithmic-based solutions? Not all AI-based solutions designed for border control rely on biometric systems. In 2015, the UK Home Office implemented an algorithm, named the ‘Streaming Tool’, to automatically assess visa applications. The system was designed to calculate a risk score on key features of the applicant to further scrutinise the visa application. In order to design this algorithm, the Home Office designed a tree-based structure with a series of yes/no questions that reflected past ‘immigration abuses’.Footnote ⁸⁵ Part of this application was made accessible after a freedom of information (FOI) request. As the UK Independent Chief Inspector of Borders and Immigration has argued, this tool was developed to ‘enhance the security of decision-making and support more efficient processing’ of decision makers.Footnote ⁸⁶ Moreover, ‘[t]he streaming tool is regularly updated with data of known immigration abuses (for example, data relating to breaches of visa conditions after entry)’ and ‘[t]he streaming tool is updated regularly using feedback from decision makers and intelligence’.Footnote ⁸⁷ In other words, this tool proposes a tree-bank structure of questions that are supposedly key to automatically distinguish between successful and non-successful applications. Among these factors, nationality was considered a relevant one.

One of the main limitations of using this type of algorithm is the feedback loop. This concept was coined by Lum and Isaac in their work ‘To Predict and Serve?’Footnote ⁸⁸ In their research on predictive policing, Lum and Isaac argued that the over-policing of areas which usually correspond to the most deprived neighbourhoods is more likely to be predicted as high risk by the algorithm. Therefore, if police patrol those areas more than others following the algorithmic recommendation and they then feed the algorithm again with this new activity, the probability that in the future the system will choose these areas as high risk increases: ‘[T]his creates a feedback loop where the model becomes increasingly confident that the locations most likely to experience further criminal activity are exactly the locations they had previously believed to be high in crime: selection bias meets confirmation bias’.Footnote ⁸⁹

In other words, the problem is not the algorithm per se, but the police bias in the data. The system is not predicting nor analysing crime, but policing activity patterns. This is also the case with the UK Home Office Streaming Tool. In the case of the visa system, the algorithm was giving an output based on the several questions proposed by visa officers and intelligence services. This algorithmic system identified certain patterns in past cases, such as some nationalities were more likely to obtain a higher scrutiny score than others, the reason why this factor was considered a relevant one to distinguish between high and low risk applications. However, the streaming tool was entering into a feedback loop within the UK visa processing system. Given that decisions were made taking into account previous data, the system was predicting visa officers’ patterns and reinforcing their past decisions. In other words, citizens from Bangladesh, Algeria or Ghana have significantly higher refusal rates for UK work visas when compared to other countries. If we fit this data into an algorithm, the system will find that nationality is a key element to classify whether a visa is high risk or not. It will automatically assess that nationals from Bangladesh, AlgeriaFootnote ⁹⁰ or Ghana are more likely to be rejected because in the past their rejected rate was higher than other nationalities.Footnote ⁹¹ Therefore, if your visa application satisfies the requirements to enter the UK but you belong to one of these nationalities, the algorithmic system will score you with a high-risk score. In this context, algorithmic systems automatise racialised borders by quantifying racialisation mechanisms through data, probabilities and mathematical functions.

As a result, in August 2020, the UK Home Office decided to scrap this socio-technical system after being legally challenged by the Joint Council for the Welfare of Immigrants (JCWI) and the legal non-profit organisation Foxglove.Footnote ⁹² While the Home Office did not provide any details about this system, as required by the claimants, they decided to abandon this system rather than opening a legal battle to prove that their system was not discriminatory. In this lawsuit, JCWI and Foxglove argued that some nationalities visas were more likely to be score as high risk: ‘[A]pplications made by applicants holding “suspect” nationalities are, other things being equal, more likely to receive a higher risk rating. Applications with higher risk ratings receive greater and more intensive scrutiny by Home Office officials, appear to be approached with considerably more skepticism, take longer to determine, and are much more likely to be refused.’Footnote ⁹³

Therefore, the Streaming Tool was alleged to be unlawful on the grounds of the Equality Act 2010 which ‘legally protects people from discrimination in the workplace and in wider society’.Footnote ⁹⁴ In an interview, the Independent Chief Inspector of Borders and Immigration claimed that: ‘Immigration Enforcement’s intelligence about illegal working mostly consisted of low-level allegations made by members of the public, which were lacking in detail and the reliability of which was difficult to assess. … Bangladeshis, Indians, Pakistanis and Chinese made up almost two-thirds (63%) of all illegal working arrests’.Footnote ⁹⁵ If experience of intelligence in the UK was used to design the Streaming Tool, this fact about illegal arrests and nationalities could be embedded within the tool.

The Streaming Tool invisibilised sedimented racialised classifications at the UK borders, fostering the digital Hostile Environment.Footnote ⁹⁶ The users of this system, that is, visa applicants, found themselves totally uninformed against a visa refusal caused in part by the automatic system. However, the harm that algorithms, such as the Streaming Tool, are causing nowadays can be confronted by an exhaustive audit of the source code. In fact, the algorithmic evidence that the tool was discriminating towards certain nationality could be challenged in courts pleading that the Home Office was discriminating towards these nationality previous to the Streaming Tool.Footnote ⁹⁷ Rather than challenging the algorithmic black-box, there is a need to challenge the institutional opacity which underpins migration policies.Footnote ⁹⁸ In fact, the opacity of the Streaming Tool being used by the Home Office was another factor claimed in this lawsuit:

The Streaming Tool has made visible racialised bordering mechanisms. Through a quantification mechanism of visa applications, the system reveals that some nationalities are more likely to be rejected by the Home Office. In fact, there were some nationalities more likely to be targeted by this algorithm, which made visible the racial bias of the migration system. Such evidence could have been used to challenge the UK visa process in the past. More broadly, this case study suggests that critical debate on AI should not focus exclusively on the racial bias of algorithms. In the words of the scholar and activist J. Khadijah Abdurahman: ‘[I]t is not just that classification systems are inaccurate or biased, it is who has the power to classify, to determine the repercussions or policies associated thereof and their relation to historical and accumulated injustice?’Footnote ¹⁰⁰ Systems such as the Streaming Tool illustrate from a quantitative perspective of racialised border mechanisms: why are some nationalities more likely to be scrutinised than others? Why are nation-states like the UK rejecting more visa applications from one nationality than another one? As Narita eloquently claims: ‘However, rather than simply pointing to the flawed outcomes of technology, an infrastructural lens provides useful critiques of the future being built by bordering technology through revealing the logics and rationalities being “built in”’.Footnote ¹⁰¹

While evidence found by Foxglove and JCWI clearly exposed that the Streaming Tool was systematically discriminating against African nationalities, the case was closed without further scrutiny which could make the racialisation of borders even more evident. First, the Home Office has not provided the algorithmic code, so the decisions made by this tool are still invisible for public scrutiny. Second, we still do not know whether nationality is a key element for the scrutiny and denial of visas to the UK. The Home Office scrapped the tool after the lawsuit launched by JCWI and Foxglove, but the decision-making of visa assessment caseworkers remains largely invisible. Moreover, the Streaming Tool fostered and recrafted the racialisation of borders from a quantitative perspective by statistically distinguishing between suspect and non-suspect nationalities. Such algorithms are based on statistical mechanisms of data and probabilities that quantify human decisions in the past. In the case of the Streaming Tool, the system transformed applicant’s personal history and caseworkers’ decision-making into visa risk scores which shows how algorithms are reshaping the racialised mechanism of borders.

However, the Streaming Tool could have been used to expose discriminatory mechanisms within the Home Office’s visa decision-making processes. Algorithms like the Streaming Tool could have been employed by migrant and legal advocacy organisations to highlight the racialisation of border control mechanisms. Since algorithms rely on historical data for training, the detection of bias in an algorithmic tool implies that the training data itself was biased, indicating past instances of discriminatory practices by border authorities. Thus, algorithmic bias could be used to expose discrimination patterns before the algorithm was implemented and be used in lawsuits. For instance, the Streaming Tool statistically concluded that nationality was a key element to distinguish between high-risk and low-risk visas, proving that border authorities could potentially made unlawful decisions – based on the Equality Act 2010 – in the past, before the algorithm was implemented. Therefore, this algorithmic evidence could be useful to visibilise oppressions and quantify racialised borders in the future, rather than just challenging algorithmic bias, inaccuracy and opacity.

12.6 Conclusion

Through algorithmic systems existing racialised borders are invisibilised, enhanced and re-structured. A critical investigation of the use of algorithmic systems in migration governmentality entails scrutinising not only how these are deployed at the border but also how their implementation has transformed the ways in which algorithmic power is exercised. This requires forging a new political lexicon that does not take for granted oppression, discrimination and surveillance as the main ways in which racialised borders are enacted through algorithmic systems, investigating instead which specific harms and modes of exclusion are enforced. Importantly, racialised borders are not fixed once for all, and actually it is key to register ‘the mutability of race’:Footnote ¹⁰² that is, who is racialised and classified as ‘migrant’, as well as the degrees of racialisation change over time. The use of data-based systems and algorithms to enact exclusionary laws and policies contributes to invisibilise consolidated hierarchies of mobility as well as the ongoing transformations of racialised borders.

Algorithmic systems do much more than monitoring and tracking migrants. Algorithms are used to invisibilise, enhance and recraft the ways in which the unequal and racialised access to mobility is enforced, grounding on discriminatory classifications among individuals which are mainly predicated on nationality. By questioning the juridical understanding of power and the repressive hypothesis that underpin debates and literature on algorithmic surveillance, this chapter has opened avenues for interrogating how the critique of biometric systems and AI in migration governmentality should be framed. Indeed, algorithmic governmentality induces ways of ‘un-thinking’ the future, which in the context of borders and migration entails narrowing down the critique of racialised borders to a matter of bias and discriminatory functioning.Footnote ¹⁰³ Critiques of AI at the borders mainly focus on the discriminatory effects produced through algorithmic-based systems and on the risk of enhanced mass surveillance. In so doing, they partly overshadow the (un)acceptability of borders. Indeed, to truly engage with a critical perspective of AI at the border, we should start from the consideration that the border is the problem – as long as through borders racialised hierarchies of lives are established and fuelled – instead of articulating a critique that hinges on the discriminatory functioning of technologies at the border.

By retracing the colonial legacies of biometrics, Simone Browne has argued that ‘it is at the border – territorial, epidermal, and digital – a site where certain bodies are cast out and made out of place, that a critical biometric consciousness’ could be developed.Footnote ¹⁰⁴ In fact, the border is not simply one of the many sites for investigating biometrics. Rather, it constitutes a case in point, this chapter has shown, for excavating racialised exclusionary mechanisms without ‘seeing like a state’Footnote ¹⁰⁵ nor like a citizen. If on the one hand, nowadays AI and more specifically biometric systems affect everyone, on the other it is key to assess the differential impact they have on citizens and on those who are racialised as ‘migrants’ or whose mobility is ‘illegalised’. Considering the making of racialised subjectivities as a standpoint, we suggest, will enable to re-think the critique of algorithmic power at the border by interrogating how does it look like to see AI at the border like a migrant: indeed, from such a perspective, claims around consent and privacy would be decentred in favour of claims for undoing racialised borders.

This commentary begins at the end. For all chapters in this section conclude with a reflection on violence and power. From a discussion of platform power,Footnote ¹ the violence that results from state surveillance and deportation to an analysis of racialised biometric borders,Footnote ² and an honest reflection on being complicit in and resisting fascist politics,Footnote ³ all authors agree that there is no escape from the racist, sexist and classist structures that shape how populations are governed using digital technologies. A critical engagement with notions of being, belonging and citizenship are central in all chapters, illustrating empirically how digital technologies are at the heart of exclusionary practices. Conceptually, all three chapters speak to a growing debate at the intersection of Science-and-Technology Studies, Feminist Theory and Critical Surveillance Studies. While the chapters differ greatly, in terms of empirical focus, methodology and style, this commitment to a critical analysis of digital technology and violence brings out a few common themes that deserve more attention. In this commentary, I focus on three of these themes and engage with the chapters, bringing out their similarities and differences. Furthermore, following the themes of this Part, I attempt to further the insights of the chapters by connecting to Critical Race Theory. In doing so, I took up Ardvidsson et al’sFootnote ⁴ invitation to think: what can a chapter do? There is no need to overestimate what this commentary can do, yet one can have a modest ambition to foster a generous debate, in solidarity, with care.Footnote ⁵

Truth

An important element of algorithmic violence that we observe in the different chapters, is how digital politics and algorithmics impact our understanding of ‘truth’, and how truth-telling via data can effect epistemic violence. Epistemic violence refers to the violent ways of producing or erasing knowledge in the production of the Other.Footnote ⁶ In Tazziolli and Valdivia’s chapter ‘truth’ is unpacked in the context of biometric surveillance of migration and the use of algorithmic calculation at the border. In these algorithmic calculations the ‘truth’ is shaped by what is considered the ‘ground truth’. This ‘ground truth’ is a process where the model is shaped through an initial labelled dataset that should mirror reality. Tazzioli and Valdivia show how algorithms are trained to find matches based on biometric traits, connecting images to actual faces. To avoid mismatches in this identification process, it is necessary to train the algorithm with the ground truth improve its robustness: the ability to carry on with calculations and prediction on a new dataset or data that contains outliers. In making these mathematical calculations, algorithms are inscribed with a form of neutrality, pretending their decisions are not political but the outcome of a neutral mathematical process. Tazzioli and Valdivia contest this assumption of neutrality and show how biometric databases are a historical continuation of colonial governance practices using biometrics to sort people into different categories.Footnote ⁷ The (bad) quality of the images, or the discrimination embedded in the training data, however, have disproportionally impacted racialised populations in facial recognition models. In other words, the ‘ground truth’ with which many algorithms at the border are trained is already filled with racial hierarchies and histories of violence. Considering this history, ‘biometric methods and methodologies can only produce dehumanizing mathematical results’.Footnote ⁸ In other words, algorithmic truth entrenches historical asymmetry, administering epistemic violence. Truth, just as neutrality, fits into a sanitised and quantified view by system designers on improving accuracy of the socio-technical systems at work, without attention to the broader socio-political contexts in which these systems work.Footnote ⁹

Aradau’s chapter shifts our focus from algorithmic to legal truths, that is: the establishment of facts through legal decision-making. Studying how digital data is presented in asylum cases, Aradau shows how courts decide what kind of legal truths can be attributed to asylum applicants based on their social media activities. Social media platforms, however, exercise a form of platform power, which obfuscates the distinction between surveillance and data extraction. Therefore, courts’ assumptions that more data will lead to a better assessment of what really happened ignores how platforms access, extract, monetise and curate data in multiple ways. Nevertheless, this assumption and associated demands by courts for more and more data place ‘a burden to know’ on ordinary citizens to know exactly what happens with their data and what consequences this data might have.Footnote ¹⁰ The argument in this chapter is that platform power, like biometric data collected at the border, obfuscates the boundaries of public/private if one makes statements on social media, and even of what is true/false. Despite this complexity, courts need to come to a verdict: a legal truth. Asylum or deportation. Violence or care.

Arvidsson, Gandorfer and McQuillan’s chapter helps us to understand the broader implications of the way algorithmic calculations impact what we understand as the ‘truth’. Algorithmic calculation often confirms what is already known and accepted as “true”. This concurs with Tazzioli and Valdivia’s description of ‘ground truth’ which takes a particular representation of the world as the truth to begin with. Yet, as Ardvisson et al write: ‘nothing starts without a history, or with a clean slate. Beginning can thus only ever mean speaking from the middle’.Footnote ¹¹ Indeed, ‘truth’ in algorithmic governance, as evidenced in these chapters, is a particular way of pretending that there is truth in the way algorithms fixate the past as (datafied) evidence. Through algorithmic practices, the meaning of some (datafied) evidence is sedimented, rather than a product of interpretation or potential change. While scholars have paid attention to the fluid nature of socio-technical systems,Footnote ¹² it seems that racial hierarchies are fixed rather than evolving. Fixation, as BenjaminFootnote ¹³ describes, or an ‘image that does not change’ lies at the core of stereotypes and at the heart of racist practices. Fixation, stereotyping, othering: a beginning of violence. In sum, what the chapters therefore illustrate is how algorithms create a distinction between ‘true’ and ‘false’ representations of reality. The chapters show us that such binary understandings produce algorithmic violence as they close off other ways of interpreting or knowing the world.

Visibility

The making of truth, violence and visibility are intimately connected in (algorithmic) surveillance. The three chapters in this Part engage with current debates in (algorithmic) surveillance studies and illustrate what legal or political becomes visible, obfuscated, foreclosed or illuminated, and how these optics and apertures can open onto violence.Footnote ¹⁴ Looking closely at the term ‘algorithmic surveillance’, it indeed means nothing more than having a set of rules that determines what to see or watch over.Footnote ¹⁵ Critical scholarshipFootnote ¹⁶ reminds us that surveillance practices are not neutral processes of ‘watching over’, but a way of producing racialised, gendered and classist accounts of living bodies. Browne argues that surveillance technologies work through the assumption that they can reveal ‘the truth’ on the racial identities of bodies. Identifying bodies through fingerprinting is an example that finds its origin in the colonial context where race-making and scientific method worked together to ‘identify untrustworthy colonial subjects in India and Egypt’Footnote ¹⁷ and justified racial violence.Footnote ¹⁸

To date, academic conversations on algorithmic surveillance have focused on questions of (in)visibility and privacy to protect individuals from violence. There is a shared concern, however, across the chapters to move beyond issues of privacy and attune to (epistemic) violence. In the chapter on biometric surveillance by Tazzioli and Valdivia, we learn how the debates around ethical use of technologies have rendered existing racial borders invisible. The authors argue ‘[t]he use of data-based systems and algorithms to enact exclusionary laws and policies contributes to invisibilise consolidated hierarchies of mobility as well as the ongoing transformations of racialised borders’.Footnote ¹⁹ In other words, discussions on ‘ethical use of data’ might focus on what data is extracted from migrants at the borders, but they obfuscate the racist structures that govern migration in the first place. Similarly, Aradau presents data wrangling as a conceptual tool that is used to describe the practices of cleaning data and making it presentable. The concept ‘carries the meaning of dispute, of struggle and contestation over what data migrants can produce, how they can then collect it, and how they need to transform it into legal evidence’.Footnote ²⁰ Aradau observes that courts demand a disclosure of data by applicants with a disregard of potential violent outcomes of this making-public. She cautions that issues of privacy are not sufficient to understand the full effects of platform power, as ‘[h]aving all the data is not only a question of privacy and the exposure of personal and often sensitive information to courts. The request for all the data intensifies the asymmetries of power and exposes asylum seekers to a specific form of epistemic violence’.Footnote ²¹ What we observe here is again a dual enactment of (in)visibility: while privacy and transparency might be understood in the eyes of the court as binaries, to understand the interactions between platforms, citizens and data, we need to move beyond (in)visibility as an issue of privacy alone.

Together, the chapters push current debates on privacy/transparency, by unpacking the power structures that are embedded in the gaze of algorithmic surveillance, and their complicity in violence. As such, they commence a conversation towards future change and critique. Arvidsson et al propose that visibility, like the ‘truth’, is not disembodied nor objective; there is no seeing everything from nowhere. Algorithmic vision from nowhere is just another ‘eye that fucks the world’.Footnote ²² Tazzioli and Valdivia suggest indeed that: ‘Further research should interrogate how does it look like to see AI at the border like a migrant: indeed, from such a perspective, claims around consent and privacy would be decentred in favour of claims for undoing racialised borders’.Footnote ²³ Algorithmic surveillance, if we follow the critical approach that is present in all the chapters, becomes more than only the set of rules that determines who or what to see. Rather, we gain an important insight that writing from a situated place rather than an objective one is a starting point to resist the racist gaze that continues to dehumanise. It is through this gaze that racialised bodies (or other marginalised bodies) become visible, while pre-existing racial hierarchies remain invisible as ‘blind spots’ of the algorithm.Footnote ²⁴ Prior to algorithmic (in)visibility, however, Fanon already discussed how dehumanisation is the result of a white gaze, where whiteness is the prototypical way of seeing. Fanon writes that: ‘already I am being dissected under White eyes, the only real eyes. I am fixed. Having adjusted their microtomes, they objectively cut away slices of my reality’.Footnote ²⁵ As Fanon shows such dehumanising gazes are not just conceptual tools, they are lived experiences of becoming fixated epistemologically. Again: fixation, stereotyping, othering: a beginning of violence.

Fixation

Thinking through the different chapters, I propose that we should stay (just a little bit longer) with the trouble of fixation. Recall Fanon’s statement: ‘I am fixed’, signalling both a racial fixation of the body and an epistemic closure of being. Connecting to the chapters and the aforementioned themes, epistemic closure and (racial) fixation is also at the heart of algorithmic violence. Ruha Benjamin writes about this as ‘a racial fix is itself a social technology that creates vertical realities – freedom, security and happiness above and bondage, precarity and misery below – all the while masking and distorting the lives of the latter from those who experience the former’.Footnote ²⁶

To describe algorithmic calculations as fixations might seem counterintuitive. After all, the three chapters illustrate a continuous desire of companies and governments to collect and share more data. In this desire, we see the opposite of fixation. Data should flow. Data should improve our knowledge. Data should enable innovative policies. Whether this is in the case of asylum proceedings in courtFootnote ²⁷ or at the borderFootnote ²⁸ or in realising a fascist desire.Footnote ²⁹ However, all authors resist the idea that more data is better or more revealing or enabling. Amidst the fluid and ever-growing ways that data can be shared, collected and stored, the extraction of data from vulnerable and marginalised populations continues to fixate persons and things into specific categories that justify political action. More data essentially reproduces the same epistemic closure: I am fixed.

As the moments of injustice and violence that are addressed in these chapters have shown us, there is no quick-fix for these issues. Furthermore, the chapters agree that a technical fix cannot address the social and political context that is (re-)produced through algorithmic calculations. We are therefore not in need of better apps or more accurate data to form ground truths. Perhaps we are in need of unfixing global data infrastructures.

Conclusion?

In the previous three sections I have examined common themes that help us understand how algorithms produce violence and complicate accountability. The chapters in this section have critiqued the assumption that algorithmic decision-making is disembodied and neutral, not troubled by a racist and sexist history or the social realities in which it operates. Such assumptions amplify a binary vision of the world and its inhabitants: visible/invisible, guilty/innocent, truth/false. Nevertheless, what possibilities are left for those who want to resist or push back against algorithmic violence and want to move beyond the binaries? Arvidsson et al suggest that we should turn to post-human feminist alliances: ‘posthuman feminist transversal alliances are de-territorial, moves along lines of flights, are Deleuze-Guattarian “packs-swarms”. Yet, these are not necessarily “bro-packs”, rather they are feminist, sense-active, de-de-selectionary and becoming’.Footnote ³⁰ Yet, what does that mean concretely? What practical or conceptual tools do such statements give us?

Tazzioli and Valdivia propose a deeper interrogation of racist borders as a site where the differential treatment of migrants becomes apparent. Their suggestion for further research is to see AI at the border like a migrant to fully understand what an alternative future could look like. Epistemic openness is furthered by shifting our perspective from the powerful to the marginalised. Similarly, Aradau’s chapter critiques the way a ‘burden to know’ is placed on ordinary citizens, while powerful institutions have limited understanding of the way data extraction and surveillance functions. Arvidsson et al similarly suggest ‘speaking is speaking up against, from below’.Footnote ³¹ Further research is needed to formulate better protection against algorithmic violence to those who are marginalised and fixed. As such, research can move from a ‘quick-fix’ to an ‘epistemological unfixing’. Such unfixing requires opening up research to Fanon’s lived experiences of racialised and marginalised communities, unsettling interpretations of the past, and making space for ‘a mode of being constantly ongoing (becoming), always shifting its relations’.Footnote ³²

McKittrick writes that ‘the sharing of ideas (no beginnings, no ends) enables a terrain of struggle, through which different futures are imagined’.Footnote ³³ As the chapters in this section show, we are in dire need of imagining and realising a different algorithmic future. Until now, algorithmic tools and governance have provided a repetition of the past. Past knowledge, hierarchies and politics re-appear through novel configurations. Much of the violence remains the same. We are fixed. To move out of this epistemic closure, perhaps sharing of critical ideas and novel suggestions of accountability are the way to envision a new future. As I find myself struggling with words to conclude, perhaps what this commentary can do, is to share and bring together ideas (no beginnings, no ends).

When I was appointed United Nations Special Rapporteur on Contemporary Forms of Racism, Racial Discrimination, Xenophobia and Related Intolerance in 2017, it had never crossed my mind that of all the global challenges confronting racially marginalized groups, emerging digital technologiesFootnote ¹ would present themselves as among the most personally terrifying fronts. I am a scholar of international migration, borders, and racial governance, but I was an international human rights lawyer first. Some of my most professionally formative experiences have involved bearing witness to soul-crushing systems and regimes of oppression and the testimonies of those subject to them. I think, for example, about someone I interviewed to prepare for an asylum hearing who had been forced to give birth alone in a forest as she fled ethnic cleansing, or someone else who was part of an incarcerated group I worked with at a maximum-security prison in California, destined to live the rest of her life in a tiny box.

The horror of spectacular violence, and increasingly that of institutionalized violence, is within easy reach for most people, even though, as the ongoing, widely broadcast Israeli genocide of Palestinians shows with each passing day, it takes more than widely held revulsion to bring an end to such violence. Whatever we might say about emerging digital technologies in 2024, while concerns with their societal ramifications are more widespread than they were in 2017, it seems a stretch to say these technologies are widely perceived as warranting a sense of exceptional unease. In this brief epilogue, I reflect on the nonetheless persisting sense of unease, verging on terror that I developed in my UN work on emerging digital technologies, and how the contributions in this edited volume have helped me make sense of these emotions, and the circumstances that provoked them.

I am neither a technology nor a data expert of any sort. Within the human rights framework I have devoted most attention to equality and nondiscrimination rights, rights that typically have not been at the fore of international legal debates on the global governance of emerging digital technologies, and that arguably remain a side show, at best. Within the field of human rights, concerns with emerging digital technologies have traditionally been framed as implicating the right to privacy and to a lesser extent the right to freedom of expression and others. Historically, the inclusion of equality and nondiscrimination concerns has been on narrow terms that have resulted in impoverished mobilization of these rights. From the very beginning of my time in my UN mandate I received invitations to participate in global policymaking and related processes at the intersection of technology and human rights. Yet almost invariably, these invitations asked me to speak specifically to the role of equality and non-discrimination rights in the adjudication of hate speech online, often rehearsing broader debates within the human rights system that have seen states – with the United States at the helm – pit speech and expression rights against equality and nondiscrimination rights.Footnote ²

The tech and human rights debate within and outside the UN system has shifted in important ways, but at the time – even with limited human rights exposure to the tech world – I found the narrowing of the relevance of equality and nondiscrimination norms to the realm of hate speech concerns unsettling, if familiar. Unsettling because the equality and non-discrimination effects of any phenomena that quite literally transform and remake societal relations on the scale of emerging digital technologies could hardly only or mostly be confined to online hate speech. As a Special Rapporteur the sheer (supra-)sovereign force of some of the big Silicon Valley tech companies was palpable. Whereas bilateral meetings coordinated by the Office of the High Commissioner for Human Rights for me in my UN capacity had solely been with diplomatic representatives of nation-states, I recall being stunned one year when such a meeting had been arranged for me with the “diplomatic corps” of one such company. More was always obviously at stake than content moderation.

Yet the reductive framing of the stakes, as I mentioned, was familiar. The tendency of the field of international human rights and the broader global governance terrain of multilateral institutions has been to marginalize racial justice and equality and to adopt a reductive approach to their stakes.Footnote ³ Where they have received the most attention, these causes have typically been reduced to issues of combating the explicitly racist conduct of individuals while completely disregarding (but reifying) structural and systemic racism operationalized through liberal law and policy, including at the international level.Footnote ⁴ Global governance within the United Nations, even through initiatives such as the Sustainable Development Goals that are intended to address global inequality adopt what Critical Race Theorists have termed a “color-blind” approach.Footnote ⁵ A purported even-handedness that entails ignoring racially disparate conditions in favor of treating all individuals and groups the same, even if these individuals and groups are differently situated, including because of historical structures of intentional discrimination such as European colonialism. It is thus not surprising that the human rights preoccupations of the primary global governance forum – the United Nations – would effectively narrow the relevance of racial equality and nondiscrimination norms mostly to influencing social media content moderation policies, and even then, doing so on terms seemingly set by social media platforms.Footnote ⁶ Meta’s politically and economically motivated recent abandonment of its independent fact-checking policy illustrates the pitfalls. On the one hand, content moderation matters for racial justice and for human rights more broadly. On the other hand, without approaches that target the worldmaking capabilities of social media companies, these companies remain unconstrained. The problem of global corporate predation is, of course, a much deeper one than can be addressed within the tech and human rights frame, although it implicates the voluntary business and human rights regime that so poorly mitigates this predation. But the problem might be materially easier to tackle within the context of a global discourse that begins from recognition and repudiation of the structural and systemic account of this problem, as opposed to accounts that render the problem two dimensional.

It was against this backdrop that I produced two thematic reports focusing on racial equality and emerging digital technologies, one of which dealt specifically with the context of immigration and border enforcement. My first report attempted to bring racial equality and nondiscrimination principles to bear on the structural and institutional impacts of emerging digital technologies, relying on the research of critical scholars of technology and on the experiences of racially marginalized communities subject to these impacts in the realms of employment, education, health care, and criminal justice among others.Footnote ⁷ At the heart of the report was an examination of how emerging digital technologies exacerbate and compound existing inequities, many of which exist along racial, ethnic, and national origin grounds. The examples highlighted in the report raised concerns about different forms of racial and intersectional discrimination in the design and use of emerging digital technologies. In some cases, this discrimination is direct, and explicitly motivated by intolerance or prejudice. In other cases, discrimination results from disparate impacts on groups according to their race, ethnicity, or national origin, even when an explicit intent to discriminate is absent. And in yet other cases, direct and indirect forms of discrimination exist in combination, and can have such a significant holistic or systemic effect as to subject groups to racially discriminatory structures that pervade access to and enjoyment of human rights in all areas of their lives. The second report detailed the ways that “digital borders”Footnote ⁸ – international borders increasingly reliant on machine learning, big data, automated algorithmic decision-making systems, predictive analytics, and so on – enhance the scope and precision of already racially discriminatory borders.

The modest ambition of these two reports was to move the human rights debate on racial discrimination and tech beyond the narrow confines of content moderation concerns. In some sense, I aimed to help broaden the scope of the global governance debate to encourage policymaking attention to the structural reification of racial hierarchies across and within societies, and to the means of that reification. By the time the reports were done, I had the nagging sense that though worthwhile, an analysis that treated global governance and those governed as independent variables, itself reproduced a myopia not unrelated to fixating on online hate speech as the racial equality concern of emerging digital technologies. This edited volume, and the nuance and brilliance of its framing by the editors confirm that intuition. Global Governance by Data illustrates how we might ask the right questions across disciplinary silos to make sense of our world as it, and we are iteratively remade by emerging digital technologies. Some of the contributors, for example, push us to think more deeply about how global governance by data reconstitutes racial meaning (Phan and Wark) even though, it seems we cannot quite say yet whether this entails a thoroughgoing reconstitution of the nature and targets of racial injustice.

Phan and Wark argue that AI and machine learning calculate and construct populations premised “not on discrete, shared characteristics – on ‘identity’ – but on measures of alignment determined using complex, automated machine learning techniques: on what’s referred to as the ‘proximity’ between individuals” possibly resulting in “racialisation and racial discrimination premised on logics of proxification.” They offer a compelling account in the shifting techniques of classification and construction and this will surely impact racialization, and this is something we can know based on our experiences with what prior technological innovations have done. Consider, for example, the work that international borders do to classify and construct, and how as their technologies for doing so have shifted over time, so too, have the forms and outputs of their racializing power. The category “non-Western migrant” for example is a twenty-first-century construction made possible by border institutions, including innovations such as the EU’s Schengen visa regime.Footnote ⁹ What remains to be seen, and what I am not yet convinced of (perhaps due to a failure of imagination) is that proxy logics can fundamentally, or even significantly dislodge the structural, material, and especially colonial grooves that are constitutive of race as a global, historical structure of injustice. Although new, the category non-Western migrant enacts racialized injustice that is most pronounced for immigrants and their descendants who are not White and Christian. Can global governance by data, without a whole lot more, dislodge White supremacy as a global institution?Footnote ¹⁰ Phan and Wark are surely right that the “politics of proxies” will demand different political techniques of resistance, in particular I think they increase the urgency for building solidarity across different groups of racially or otherwise marginalized people. Even in the absence of “associative” identities,Footnote ¹¹ solidarity across communities subjected to the same systems of domination but on different terms remains difficult work. Unless AI and machine learning can fundamentally remake the colonial, capitalist DNA of the contemporary world order, the core of our racial paradigm, and the nature of racial injustice within it will remain largely intact.

The provocation of the title – global governance by data – to my mind, calls attention to, the de facto and by design shared consciousnesses that are shaped by the ascendance of data. Data, the digital technologies of their manipulation and enlivening, and the corporate entities firmly in the driver’s seat, all sit at the very heart of our imagination and our connections, two vital foundations for any real hope of emancipatory futures. For example, climate disorder – a catastrophe on a planetary scale – has become a growing site of techno-chauvinistFootnote ¹² fever dreams actualized in the name of justice, in some cases in ways that compound the racial injustices that have structured and enabled the extraction, industrialization, and consumption driving the disorder.Footnote ¹³ The editors of this volume seek to train our attention on “what data does as an active force in global governance: the ways it recomposes and shapes new subjectivities, including legal subjectivities, forms of material agency and sociality; draws people, places and things together into novel relations to be valorised, known and ruled through calculative and representational techniques, and the ways it reconfigures institutional routines, legal and regulatory practices.”Footnote ¹⁴ What data does is not merely a matter of scope, degree, or kind with respect to what is governed globally. This volume makes the compelling case that what is at stake in novel or at least remarkable ways is the terms on which we make sense of our internal and external worlds, who we are, and what governance is.

Inquiries along the lines pursued in this volume are critical. They probe the “logics” of big data analytics – the promises, assumptions, beliefs, and rationalities at their heart – and how they are put into effect. As one contribution highlights the ontological condition of complicity for life under fascism (Arvidsson, Gandorfer, and McQuillan citing Massumi), so, too, I think, the entire volume serves to remind that there may no longer be an “outside” of global governance by data.

The volume leans into the often-neglected reality that “frames” – the paradigms we mobilize to make sense of the world – in effect constitute our multiple worlds. Where borders specifically are concerned, the volume includes contributions exploring how global governance by data in the realm of border infrastructure is reconstituting state agencies (Raso). Raso offers an account I found haunting of how digital border infrastructure contributes to the intensification of state power even as it diffuses state agencies in ways that push accountability further from reach. As a legal scholar, I was unnerved anew by Raso’s implicit reminder that liberal proceduralism will not save us – not that it ever could. She writes, “conventional remedies cannot prevent infrastructural elements from focusing authority into the future, allowing traces of riskiness to remain in circulation potentially indefinitely.” In my own work I have observed that so-called digital or virtual borders are digital racial borders. The contributions to this volume, and the work of its editors offer an even deeper and more urgent cut: the “digital” may be remaking the nature of the “racial.” As mentioned above, the extent of this shift remains to be seen. But it seems different, data-associated logics are firmly in the game as shifting not only the nature of the “racial” but also of the “border.” As one of the editors noted elsewhere, the “virtual border” can be a “a system of discrimination and division where the standards of hierarchy or inclusion are always kept in play.”Footnote ¹⁵

To return to the sentiment of unease/borderline terror which I described at the start, it has never been anchored in an understanding of emerging digital technologies as inherently or singularly “bad” or “dangerous.” Rather, I have come to think after reading this volume that what I experienced and continue to experience as deep unease is the most tractable of a range of other as yet intractable emotions. A suite of emotions gradually triggered by a growing awareness of how central these technologies and their related infrastructures have become (not exclusively of course) to the constitution of human subjectivities globally, and how for now, the power to dictate the trajectory of these technologies, at least as a matter of political economy, is concentrated in too few incredibly wealthy individuals. Private corporations wield monumental influence in the design and use of emerging digital technologies. Among digital platforms, at the time I published my report seven “super platforms” – Microsoft, Apple, Amazon, Google, Facebook, Tencent, and Alibaba – accounted for two thirds of the total market value of the world’s seventy largest platforms.Footnote ¹⁶ In 2019, Google has 90 percent of the global market for Internet searches; occupying two thirds of the global social media market, Facebook was the top social media platform in more than 90 percent of the world’s economies, and Amazon had an almost 40 percent share of the world’s online retail activity. These corporations are not only quasi-sovereign, but they are also world-makers. Contributions in this volume by Yeung and Cohen offer a granular account of how commercial logics, data logics, and governance logics interact in sometimes unexpected ways. Yeung’s work, for example, explores the transposition of tools tailored to low-stakes commercial consumer decision-making to high-stakes public governance decision-making contexts. She calls attention to the specific trajectories of injustice that are triggered with this shift, which is difficult to do without the more granular political economic study she conducts.

The tendency in the forums where the global is governed is to keep our analyses tethered to the dominant and preferred categories and modes of analysis. These are the funded, platformed conversations within the United Nations and other multilateral governance forums, but they are also the academic mainstream in fields adjacent to policy. How can we make AI fairer? How do we eliminate bias? And so on. The opportunity to engage more critically, more open-endedly, and in the registers exemplified by the contributors to this volume are essentially nonexistent. I cannot help but wonder what it would mean or do if global policymaking could encounter (not incorporate) the sort of transdisciplinary exchange exemplified by the volume, perhaps most pointedly by Arvidsson, Gandorfer and McQuillan?

Whereas it is the form of the UN report to conclude with tidy recommendations or “best practices,” the heart of the editors’ project – it seems to me – includes an invitation to dive headfirst into the unknown (and even the unknowable) of emerging digital technologies; to develop new ways of seeing and knowing and connecting; and to begin from the premise that nothing has changed, and everything has changed.