12.1 Introduction

The Greek island of Samos, September 2021: Greek and European authorities inaugurated a fenced and high-security camp for refugees of Zevrou, equipped with ‘turnstiles, magnetic gates, X-rays, two-factor access control system (identity and fingerprint) which secure[d] and controlled entry and exit from the access center to whoever is entitled to it’.Footnote ¹ The camp, which was presented by the EU as a blueprint for others, has been the object of harsh criticisms by journalists, NGOs and activists. Since the so-called ‘refugee crisis’ in 2015, Greece has been considered one of the key ‘technological testing grounds’Footnote ² where the EU and Greek authorities implement advanced digital technologies to surveil migrants. Large datasets to predict migration flows, algorithms that score how suspicious a vessel is at the sea or biometric systems to enter refugees camps are examples of technological testing projects funded by the EU. Yet, the constant announcement of Artificial Intelligence (AI) projects at the border focuses attention on the use of this technology, more than on the existence of infrastructures which enhance structural violence. Indeed, as Rodgers and O’Neill contend ‘infrastructure is not just a material embodiment of violence … insofar as the material organization and form of a landscape not only reflect but also reinforce social orders, thereby becoming a contributing factor to reoccurring forms of harm’.Footnote ³

The criticisms raised about Zevrou camp in Samos constitute a case in point about how technology-based surveillance, that is, a normative approach to surveillance that criti-hypes the use of technology at the border,Footnote ⁴ has gained central stage in debates on borders and migration. Indeed, it is noteworthy that what unleashed indignation among scholars and journalists was the technological surveillance, more than the conditions of twofold carcerality that refugees experience – by being confined on an island and inside a closed camp. Notwithstanding a myopic focus on surveillance and AI ethics in many circles, the implementation of algorithmic surveillance systems at the border has also triggered broader contestations and mobilised campaigns about the harmful consequences they generate both on marginalised populations and on those who are labelled and governed as ‘migrants’. This chapter contributes to and amplifies debate surrounding these broader concerns.

This chapter examines how algorithmic power has transformed the ways in which the mobility and presence of people labelled as ‘migrants’ is controlled and hampered. Claudia Aradau and Tobias Blanke argued in Algorithmic Reason that algorithmic power stems from the use of algorithms as governance tools.Footnote ⁵ This power ‘works by producing truths – both as outcomes or outputs of systems and as part of the discursive reinforcement of particular norms, approaches and modes of reasoning’.Footnote ⁶ Despite computer scientists having neglected the political feature of algorithms by advocating for its neutrality, we build on critical data studies to confront this perspective. For instance, Luke Munn claimed that algorithmic power ‘is political, it calibrates subjectivities towards certain ends, it prioritises space in specific ways, and it privileges particular practices whilst suppressing others’.Footnote ⁷

However, in this piece we contribute to this concept of algorithmic power at the border by focusing on the production of racialised digital subjectivity through algorithmic systems. While data has being historically instrumentalised to identify people on the move,Footnote ⁸ we argue that the use of algorithmic systems in migration governmentality invisibilises, fosters and recrafts racialised bordering mechanisms. By speaking of racialised bordering mechanisms we draw on E. Tendayi Achiume’s definition of the term: ‘border regimes that variously allocate and curtail mobility and migration on a racial basis, largely relying upon facially race neutral mechanisms’.Footnote ⁹ Such bordering mechanisms are not confined to national frontiers: in fact, we refer to heterogeneous racialised bordering mechanisms, both visible and invisible (digital borders), through which hierarchies among people and their nationality status are established and reinforced.Footnote ¹⁰ Considering that ‘the border is a site of identity production’,Footnote ¹¹ we ask: Which racialised bordering mechanisms are enacted and to what extent existing ones are strengthened and invisibilised through use of algorithmic power? Which (racialised) subjectivities are recrafted and produced? In so doing, we shift attention from questions on surveillance, ethics and AI, which imply debating over the threshold of acceptability of algorithmic systems, towards an interrogation on the entanglements between emerging and sedimented modes of violence.

In order to address these questions, we join other scholarsFootnote ¹² in contending that it is essential to move beyond what Michel Foucault has designated as a ‘repressive hypothesis’ on power which underpins debates about the ethical risks of using algorithmic systems at the border, to instead analyse power’s constitutive effects, as Foucault and others have.Footnote ¹³ First, we contend that the use of algorithmic systems in migration governmentality invisibilises, fosters and recrafts racialised bordering mechanisms. Indeed, algorithms do not only target, discriminate and affect individuals in differential ways: they also shape digital subjectivities at the border finding patterns and connections on digital evidence. Investigating the productive subject-making dimension of algorithmic power is essential, we suggest, for shifting the focus from technologies as such and their uses and functioning towards how both collective and individual subjectivities are re-made and adapted. In fact, the production of racialised subjectivities is to a large extent coded by the algorithms; yet, in part they exceed the intentionality as well as the original purpose for which these AI-driven systems have been implemented, due to the frictions, resistance and appropriations enacted by the individuals.

Algorithmic power relies on the computational ability of finding patterns and connections on large datasets. For instance, biometric systems are able to find whether a biometric sample is similar to millions of fingerprints stored in a biometric database or which nationality is more likely to be scrutinise by border authorities. Throughout data, probabilities, mathematical functions and algorithms are put to work by nation-states to automatise the racialised mechanism of borders. Second, we suggest that in order not to see technology and migration like a state involves not taking for granted surveillance, bias and privacy as the main concerns, interrogating instead the differential harms it produces on migrants.

The chapter starts by engaging with the ‘repressive hypothesis’ on power and argues that a critical analysis of the use of algorithmic systems at the border requires challenging this and the juridical understanding of power that sustains it. The chapter moves on by focusing on two case studies of algorithmic power in operation in the governance of migration. First, it deals with biometric technologies enforced at the border, showing how through biometrics existing racialised hierarchies are replicated and new ones emerge. Second, it focuses on the algorithm-based visa system in the UK (or, the Streaming Tool), to illustrate how this has invisibilised the nationality-based criteria through which visa applications are assessed. The chapter concludes by suggesting that a critical analysis of ‘AI at the border’ should be coupled with a critique of racialised bordering infrastructures.

12.2 Critical AI Theories and Surveillance Paradigm at the Border

Building on publicly available EU and state documents, this chapter intervenes in debates about AI and racialised borders by foregrounding the productive, racialising and racialised effects of algorithmic bordering mechanisms. This contributes to critical theory on AI and raceFootnote ¹⁴ to migration scholarship on the use of technologies in migration and refugee governmentalityFootnote ¹⁵ and to legal literatures on racialised digital bordering practices in ways that this section outlines.Footnote ¹⁶ Migration literature has explored how in the aftermath of 9/11, digital technology has been increasingly used to enhance racialised surveillance.Footnote ¹⁷ Digital labour scholars have shown how AI has multiplied and mutated processes of racialisation that affect in particular those who are classified and controlled as ‘migrants’.Footnote ¹⁸ A growing body of scholarship across law and the social sciences has drawn on longstanding, critical traditions in those fields to challenge the disproportionate focus on digital surveillance encouraged by some work in surveillance and migration studies, drawing attention instead to the political ecology and the political economy of data extraction. For instance, speaking of data as oil Sy Taffel advocates shifting ‘the focus of digital exploitation away from a critique of surveillance capitalism … foregrounding unequally distributed social and environmental harms associated with the material dimensions of digital extractivism’.Footnote ¹⁹ This chapter makes a similar gesture – shifting away from a surveillance-centred analysis, and the repressive and juridical understanding of power with which such analysis is associated, by investigating how algorithmic systems are instrumentalised to invisibilise and foster racialised borders, and generate new ones.

Academic debates on the use of AI and biometrics at the border and in refugee camps mostly focus on the risk of enhanced surveillance or on the bias and discrimination which are invisibly enacted through AI-driven systems to assess asylum claims and to identify migrants.Footnote ²⁰ We argue that too great a focus on surveillance and discrimination overshadows the extent to which algorithm-based systems are embedded in and reinforce existing exclusionary borders and how they recraft racialised borders.

Critical works on AI and race have pointed to the risks of automated discrimination associated with the widespread use of socio-technical systems and shown how algorithms reinforce historical oppressions and embed forms of racism and white supremacy.Footnote ²¹ At the border, Louise Amoore has examined the impact of AI.Footnote ²² Yet, this chapter argues that there is a need to move beyond the surveillance narrative and further examine the impact of algorithms at the border by avoiding techno-hype narratives.Footnote ²³ For instance, Simone Browne has called for rethinking surveillance ‘through the multiplicity of blackness’:Footnote ²⁴ ‘rather than seeing surveillance as something inaugurated by new technologies, such as automated facial recognition or unmanned autonomous vehicles’, she advocates analysing digital technology as a ‘factor in how racism and antiblackness undergird and sustain the intersecting surveillances of our present order’.Footnote ²⁵ Our chapter contributes to this literature by exposing how AI is nowadays integrated as a tool within the racialised mechanism of border infrastructures in the European context. But rather than claiming that this technology is opening a new paradigm in the surveillance context, adding to the critical literature on racism and AI, we suggest that critical inquiry should problematise the concept of racialised border infrastructures that produces racialised.

Critical works on socio-technical systems are mainly centred on the modes of surveillance and tracking enforced through these latter and on the discrimination induced through automation.Footnote ²⁶ By questioning such an analytical perspective, we do not deny that repressive mechanisms are enforced through AI. Rather, we highlight that prohibition, discrimination and repression are situated in a broader economy of power relations, that enact subjectivities, automatically connects digital evidence and multiply racialised borders. That is, we are interested in looking at the productive dimension of power relations which have been deeply reconfigured by the systematic use of AI and algorithmic systems and exploring these productive relations and effects as sites of critique. In line with Isin and Ruppert’s account of ‘an analysis of the forms of power that generate, assemble and organise data’,Footnote ²⁷ we are interested not simply in studying which new technologies are deployed at the border but also in investigating ‘how new ways of life are brought into being’.Footnote ²⁸

12.3 Making Up Migrants Beyond Surveillance and Repression

Oppression, bias and discrimination are terms which are widely used for critiquing algorithmic systems. And algorithmic-based bordering and biometrics technologies are often represented in critical literature as repressive tools which work by enforcing bias and remote surveillance. However, in this section we argue that in order to develop a critique of the use of algorithms at the border it is key to undoing a repressive and juridical understanding of power and to question analyses that reiterate the image of panopticon-based surveillance. These two approaches are problematic because they narrow attention to the ways algorithmic mechanisms strengthen repression and bias and enhance surveillance. Such an understanding of the power exercised through AI also circumscribes contestation and forms of refusal of algorithmic and biometric technologies, including how they are conceptualised and enacted. To move beyond these limitations, we draw from Foucauldian scholarship on the productive dimensions of power and literature highlighting the material heterogeneity of digital surveillance practices. In this way, we gesture towards an analysis that investigates how some people are turned into migrants – what we call ‘making up migrants’Footnote ²⁹ – through algorithmic-based systems and processes. Our claim is that focusing on algorithmic infrastructures for ‘making up migrants’ opens up more productive avenues for contesting digitally racialised bordering regimes.

Resistance to biometrics and algorithmic systems is mainly framed as refusal, obfuscation and partial or total withdrawal.Footnote ³⁰ Indeed, if power – in this case exercised through AI – is deemed to be essentially repressive, enacting resistance would consist in practices that block or limit such repression. Questioning the repressive hypothesis which underpins many analyses of AI attunes us to a conceptualisation of resistance that Nancy Ettlinger has defined as ‘productive’: ‘some strategies of resistance make use of, rather than reject or disrupt and obfuscate, the digital environment. Productive approaches to resistance … target and subvert a particular strategy of subjection to serve digital subjects’.Footnote ³¹ An example of productive resistance, Ettlinger explains, consists in digital subjects who ‘may act critically on their freedom to challenge digital norms by making use of the affordances of the digital environment to produce new elements that serve their needs’.Footnote ³² Building on this analysis, we argue that it is essential to ‘cut off the King’s head’;Footnote ³³ that is, to move beyond a juridical understanding of power and attend to the ways algorithmic-based systems are being used to recraft racialised bordering mechanisms. This entails not taking for granted the modes of subjection enforced through AI and questioning whether enhanced surveillance and tracking are the key harms caused by using algorithmic systems at the border.

According to Foucault, a juridical understanding of power relations posits that power works by prohibiting and repressing and, relatedly, that resistance consists in refusal. In The Will to Knowledge, Foucault claimed that ‘we need to cut off the King’s head: in political theory that has still to be done’.Footnote ³⁴ In so doing, Foucault questioned what he called the ‘repressive hypothesis’, according to which power is conceived in terms of prohibitions and interdictions. Against this, Foucault shifted attention to the productivity of power relations, focusing on the regimes of knowledge and truth that are at play and how subjectivities are shaped by power relations.Footnote ³⁵ For Foucault, the notion of repression is inadequate for capturing these productive effects of power. It obfuscates that ‘what makes power hold good, what makes it accepted, is simply the fact that it doesn’t only weigh on us as a force that says no, but that it traverses and produces things, it induces pleasure, forms knowledge, produces discourse’.Footnote ³⁶

This repressive hypothesis about power relations, which is grounded in a juridical understanding of power, is, however, surreptitiously replicated in the ongoing debates on algorithmic and biometric technology, where AI is deemed to be an instrument through which repression and discrimination are enforced.Footnote ³⁷ Although it is certainly important to reiterate the general argument that AI-based bordering systems enable and reinforce discrimination, we suggest that this focus obscures other productive effects. For this reason, we gesture towards an analysis that investigates how some people are turned into migrants – what we call making up migrantsFootnote ³⁸ – through algorithmic-based systems and processes. As we show below, algorithmic bordering mechanisms are used for enforcing existing laws to restrict and obstruct migrants’ movements – such as the Dublin Regulation in Europe – to consolidate racialised hierarchies of mobility, and to multiply exclusionary classifications. As we illustrate in this chapter, EURODAC is a biometric database that reinforces the Dublin RegulationFootnote ³⁹ by storing fingerprints of migrants that are attached to a categorical classification based on their status in the territory (asylum seeker, ‘irregular’ entrant or ‘illegal’ stayer).

Alongside juridical conceptualisations of power, a growing scholarship refers to the panopticon as a surveillance model to explain the functioning of AI in migration governmentality and the. In fact, while it is crucial to understand how surveillance is enforced and strengthened through the systematic use of algorithms, our point is that a disproportionate focus on surveillance reiterates the image of the panopticon as a paradigm of power control.Footnote ⁴⁰ For instance, the above mentioned high-security refugee camp in Samos, Zovrou, has been defined a ‘panopticon for refugees’: ‘those who want to enter have to show a chip and have their biometric data checked on a fingerprint scanner’ and the cameras surrounding the camp detect ‘suspicious behaviours with the help on artificial intelligence’.Footnote ⁴¹ The designation of the refugee camp in Samos as a ‘surveillance panopticon’Footnote ⁴² on the one hand obscures the porosity of borders as well as migrants’ practices of resistances and tactics of flight;Footnote ⁴³ and on the other it narrows down the analysis to (capillary) surveillance while marginalise investigations about the carceral economies which are enforced and transformed through technology.

Notably, the panopticon is an architectural dispositive of surveillance which is characterised by a central tower from where all surrounding cells are monitored. The disciplinary function of the panopticon relies on an asymmetric visibility, in which the person in the tower can see without being seen, while prisoners in the cells do not know when and whether they are monitored.Footnote ⁴⁴ The trope of the panopticon is often mobilised in critical literature on technology for highlighting the pervasive and capillary modes of surveillance in a datafied society.Footnote ⁴⁵ While capturing important and alarming features of how technology is used in key sites, like workplaces, for monitoring workers or at the external frontiers of Europe for detecting migrants,Footnote ⁴⁶ as critical surveillance studies scholars have stressed, the trope of the panopticon reiterates an exclusive attention to (centralised) surveillance and, at once, assumes an overwhelming disciplinary gaze.Footnote ⁴⁷ Indeed, following Jef Huysman, it is worth noticing that many contemporary surveillance practices do ‘not work in such bounded institutional spaces and their hierarchical organisation of visibility’.Footnote ⁴⁸ According to Benoit Dupont, in order to fully grasp the possibility to resist surveillance mechanisms, it is key to develop a ‘more grounded knowledge of the actual interactions between those who watch, the machines and infrastructure they design and use to carry out their surveillance’.Footnote ⁴⁹ For instance, Aila Spathopoulou and Anna Carastathis have studied techniques of resistance among migrants on the Aegean islands of Lesvos and Samos whereby they have contested classifications and distinctions applied to them in their interactions with border police and infrastructure.Footnote ⁵⁰

By conceiving control in terms of (panopticon) surveillance scholars who subscribe to this view end up seeing migration and technology ‘like a State’,Footnote ⁵¹ corroborating states’ narratives about the possibility to exercise full control in real time. More precisely, the panopticon model eclipses the patchy visibility and the uneven temporality that underpin many mechanisms of surveillance.Footnote ⁵² For instance, in some contexts state authorities are not interested in constantly monitoring and detecting migrants; in other cases, even if they do, it is not possible to keep a constant and overwhelming visibility. In a nutshell, the analytics of the panopticon overshadow the heterogeneous modes of subjugation that are enforced also through AI and that in part have been changed and re-shuffled because of the incursion of digital technology. Instead, interrogating which racialised bordering mechanisms are put in place through the use of AI allows rethinking and widening the range of resistance, tactical appropriation and refusal.

One illustration of the tendency to analyse technological governance in ways that overstate the coherence and capacity of the State is the scholarly and popular debate around facial recognition technology. In many jurisdictions, the implementation of facial recognition technologies has triggered criticisms about privacy violations and discrimination, such as the civil society initiative against facial recognition, entitled Reclaim your Face, which calls for a ban on biometric mass surveillance practices. Proposals of this kind are usually grounded in concerns about surveillance and tracking activities and tend to emphasise the violation of privacy that the unregulated use of facial recognition might lead to. Both the public debate on facial recognition and initiatives set for resisting biometrics have been crucial in bringing to the fore the ethical-political risks connected to the use of facial recognition. These initiatives have built a critical mass towards the use of biometric systems by law enforcement authorities. However, these anti-surveillance campaigns do not move forward by questioning the violence of the State as such.

While biometric systems, such as facial recognition, are being implemented to gather digital evidence for police investigation and law enforcement, there is a need to question the violence of borders beyond these measures. By framing the critique of facial recognition technology around a concern about surveillance, scholars neglect to question racialised bordering mechanisms and the legitimacy of borders as such. If we start from the consideration that power, and more specifically power through AI, works in heterogeneous ways, it becomes possible to see that even resistance might take different forms.Footnote ⁵³ Refusal of and withdrawal from technologies of control illustrate how subjects engage in practices of resistance, both individually and collectively. The heterogeneity of resistance echoes the heterogeneity of modes of subjugation, given that the former is never fully external to the latter.Footnote ⁵⁴ Resistances are characterised by ‘tactical immanence’Footnote ⁵⁵ to power relations, while at the same time they also introduce an asymmetrical element with respect to power. In the field of algorithmic power, this means drawing attention to practices of resistance that enact a tactical twisting of AI, as Joler and Pasquinelli note: ‘rather than studying only how technology works, critical inquiry studies also how it breaks, how subjects rebel against its normative control and workers sabotage its gears … and to illuminate the invisible labour of knowledge that makes machine intelligence appear ideologically alive’.Footnote ⁵⁶

Accounting for the multiplicity and diversity of practices of resistance is key to highlight how the boundaries between consensus and coercion are often blurred in contexts where migrants are pushed to use technology for their own good. In other contexts, such as when authorities identify migrants upon arrival, they are not even asked to consent to the use of biometric technologies: they are just fingerprinted. And yet, at times they find ways to temporarily avoid mandatory fingerprinting through acts of self-harm – that is, by burning their fingertips. The effects of such practices of resistance depend on the socio-legal context in which they take place. For instance, citizens’ resistance to police facial recognition will have different consequences than migrants’ resistance to being fingerprinted at the border. From conditions of obstructed agency, such as those experienced by migrants before the border, binary oppositions between coercion and consent, freedom and obligation, tacit withdrawal and overt refusal as well as sheer opposition and tactical appropriation are deeply unsettled.

12.4 Algorithmic Power at the Border: Making People Illegal Through Algorithmic Systems

As the previous section illustrates, a radical questioning of the repressive hypothesis means investigating the effects of subjectivation that are produced by enforcing biometric technology at the border.Footnote ⁵⁷ The border is a context where the unequal impact and the racialised functioning of biometrics explicitly emerges. Indeed, biometrics and algorithmic systems are not simply deployed at the border:Footnote ⁵⁸ their implementation changes the ways in which migrants’ subjectivities are shaped (and what becomes an object of control and data extraction).Footnote ⁵⁹ The lexicon of surveillance is inadequate for fully grasping the modes of control, exclusion and racialised partitions enforced on migrants through technology.

Databases and algorithmic systems work to enhance and invisibilise exclusionary bordering mechanisms in the field of migration and refugee governance. At the same time, they also help invisibilise and recraft social injustice along new lines. These productive dimensions of bordering infrastructures are illustrated in the two empirical case studies we analyse below – the use of biometric systems by EU states for migrant ‘identity management’ and the use of algorithmic screening tools by the UK Home Office to filter ‘authorised’ movements and scrutinise visa applications.

12.4.3 EURODAC and Dodging Statistics

The development of deep learning approaches for biometrics has enhanced the performance of fingerprint systems that are systematically used by state authorities as a tool for border control.Footnote ⁸⁰ On the basis of the Dublin Regulation, EU Member States are obliged to fingerprint migrants who enter the territory without authorisation or who lodge an asylum application, and to send this biometric data into the EURODAC. In this database, Member States’ law enforcement authorities are obliged to categorise subjects in four different groups:

• Category 1 (CAT 1): Applicants for international protection.

• Category 2 (CAT 2): Irregular border crossing or unauthorised entry.

• Category 3 (CAT 3): Unauthorised stay of third-country nationals.

• Category 4-5 (CAT 4-5): Searches by states and Europol for the ‘detection or investigation of terrorist offences or of other serious criminal offences’.Footnote ⁸¹

Figure 12.1

Distribution of Categories 1, 2 and 3 of EURODAC from 2015 to 2019 in different countries (UK,Footnote ⁸² Greece, Spain and Italy). The difference on distributions shows how states make-up migrants and create different migration status inconsistently. Source: EURODAC statistics reports 2015–2019

(eu-LISA 2015, eu-LISA 2020).

Through these categories, states decide if migrants should be classified as deportable (CAT 2) or temporarily authorised on the territory (CAT 1). These statistics shows how EURODAC, a biometric database, fosters the distinction between legal and illegal migrants. Legal migrants are classified in CAT 1 while illegal migrants are classified in CAT 2 or CAT 3. Those in CAT 1 are forced to stay in the country where they have been fingerprinted. Figure 12.1 shows the distribution of categories in 2015 and 2019 on four different countries: UK, Greece, Spain and Italy. However, this categorisation of migrants at the EU border is not systematically fostered by Member States. For instance, while Greece classified most migrants in CAT 2 (which made them deportable), the distribution drastically changed in 2019 with a more equal number of migrants in CAT 1 and CAT 2. While Spain merely fingerprinted migrants in 2015 during the so-called ‘refugee crisis’, the country reported a large number of migrants in 2019. Italy also changed the distribution of categories: increasing CAT 3 and decreasing CAT 2.

EURODAC illustrates how biometric databases foster racialised bodies by assigning fingerprints to categories that assign migration status, that are in turn solely decided by border authorities. Moreover, significant differences observed in the distribution of these categories in only four years (see Figure 12.1) show how biometric systems are also instrumentalised by states to recraft racialised borders. We observe how the status of migrants being fingerprinted that can then be matched throughout biometric systems in EURODAC is recrafted due to policy or regulation shifts. For instance, states were allowed in 2015 to duplicate fingerprints if those entered ‘irregularly’ (CAT 2) claimed asylum (CAT 1) by copying instances from CAT 2 to CAT 1. However, in 2019 this practice was forbidden: if a person changes category, it should be moved but not copied to avoid duplicates in the statistics. In fact, EURODAC regulation has evolved over the years, which might have also impacted on the statistical distribution of categories. As the European Parliament states: ‘[T]he purpose and the scope of application of the EURODAC Regulation have been continually broadened since its creation’.Footnote ⁸³

The EURODAC Regulation will allow authorities to verify the identity or nationality of migrants through algorithmic systems that find patterns in fingerprints which will add new grounds for detention.Footnote ⁸⁴ The continuous upgrade of biometric database regulations will be instrumentalised to foster and create racialised borders. In the case of EURODAC, the new regulation approved moves beyond the scope of the first regulation of this database, allowing more data collection such as names and ID numbers for unforeseen purposes. The categorisation embedded in EURODAC creates and reproduces hierarchies of legal and non-legal status which are linked to fingerprints. Law enforcement authorities are able to know whether a person is an asylum-seeker, an ‘irregular’ border crosser or an authorised stayer by fitting their fingerprint into a biometric system within EURODAC. Depending on this biometric match, states can decide when they should be detained or not. Thus, a non-juridical understanding of power enables capturing how biometrics is used both for enacting migrants and crafting and multiplying legal and digital subjectivities.

12.5 The Streaming Tool: An Algorithmic System to Automate Visa Applications in the UK

To what extent are racialised borders invisibilised, fostered and recrafted through other algorithmic-based solutions? Not all AI-based solutions designed for border control rely on biometric systems. In 2015, the UK Home Office implemented an algorithm, named the ‘Streaming Tool’, to automatically assess visa applications. The system was designed to calculate a risk score on key features of the applicant to further scrutinise the visa application. In order to design this algorithm, the Home Office designed a tree-based structure with a series of yes/no questions that reflected past ‘immigration abuses’.Footnote ⁸⁵ Part of this application was made accessible after a freedom of information (FOI) request. As the UK Independent Chief Inspector of Borders and Immigration has argued, this tool was developed to ‘enhance the security of decision-making and support more efficient processing’ of decision makers.Footnote ⁸⁶ Moreover, ‘[t]he streaming tool is regularly updated with data of known immigration abuses (for example, data relating to breaches of visa conditions after entry)’ and ‘[t]he streaming tool is updated regularly using feedback from decision makers and intelligence’.Footnote ⁸⁷ In other words, this tool proposes a tree-bank structure of questions that are supposedly key to automatically distinguish between successful and non-successful applications. Among these factors, nationality was considered a relevant one.

One of the main limitations of using this type of algorithm is the feedback loop. This concept was coined by Lum and Isaac in their work ‘To Predict and Serve?’Footnote ⁸⁸ In their research on predictive policing, Lum and Isaac argued that the over-policing of areas which usually correspond to the most deprived neighbourhoods is more likely to be predicted as high risk by the algorithm. Therefore, if police patrol those areas more than others following the algorithmic recommendation and they then feed the algorithm again with this new activity, the probability that in the future the system will choose these areas as high risk increases: ‘[T]his creates a feedback loop where the model becomes increasingly confident that the locations most likely to experience further criminal activity are exactly the locations they had previously believed to be high in crime: selection bias meets confirmation bias’.Footnote ⁸⁹

In other words, the problem is not the algorithm per se, but the police bias in the data. The system is not predicting nor analysing crime, but policing activity patterns. This is also the case with the UK Home Office Streaming Tool. In the case of the visa system, the algorithm was giving an output based on the several questions proposed by visa officers and intelligence services. This algorithmic system identified certain patterns in past cases, such as some nationalities were more likely to obtain a higher scrutiny score than others, the reason why this factor was considered a relevant one to distinguish between high and low risk applications. However, the streaming tool was entering into a feedback loop within the UK visa processing system. Given that decisions were made taking into account previous data, the system was predicting visa officers’ patterns and reinforcing their past decisions. In other words, citizens from Bangladesh, Algeria or Ghana have significantly higher refusal rates for UK work visas when compared to other countries. If we fit this data into an algorithm, the system will find that nationality is a key element to classify whether a visa is high risk or not. It will automatically assess that nationals from Bangladesh, AlgeriaFootnote ⁹⁰ or Ghana are more likely to be rejected because in the past their rejected rate was higher than other nationalities.Footnote ⁹¹ Therefore, if your visa application satisfies the requirements to enter the UK but you belong to one of these nationalities, the algorithmic system will score you with a high-risk score. In this context, algorithmic systems automatise racialised borders by quantifying racialisation mechanisms through data, probabilities and mathematical functions.

As a result, in August 2020, the UK Home Office decided to scrap this socio-technical system after being legally challenged by the Joint Council for the Welfare of Immigrants (JCWI) and the legal non-profit organisation Foxglove.Footnote ⁹² While the Home Office did not provide any details about this system, as required by the claimants, they decided to abandon this system rather than opening a legal battle to prove that their system was not discriminatory. In this lawsuit, JCWI and Foxglove argued that some nationalities visas were more likely to be score as high risk: ‘[A]pplications made by applicants holding “suspect” nationalities are, other things being equal, more likely to receive a higher risk rating. Applications with higher risk ratings receive greater and more intensive scrutiny by Home Office officials, appear to be approached with considerably more skepticism, take longer to determine, and are much more likely to be refused.’Footnote ⁹³

Therefore, the Streaming Tool was alleged to be unlawful on the grounds of the Equality Act 2010 which ‘legally protects people from discrimination in the workplace and in wider society’.Footnote ⁹⁴ In an interview, the Independent Chief Inspector of Borders and Immigration claimed that: ‘Immigration Enforcement’s intelligence about illegal working mostly consisted of low-level allegations made by members of the public, which were lacking in detail and the reliability of which was difficult to assess. … Bangladeshis, Indians, Pakistanis and Chinese made up almost two-thirds (63%) of all illegal working arrests’.Footnote ⁹⁵ If experience of intelligence in the UK was used to design the Streaming Tool, this fact about illegal arrests and nationalities could be embedded within the tool.

The Streaming Tool invisibilised sedimented racialised classifications at the UK borders, fostering the digital Hostile Environment.Footnote ⁹⁶ The users of this system, that is, visa applicants, found themselves totally uninformed against a visa refusal caused in part by the automatic system. However, the harm that algorithms, such as the Streaming Tool, are causing nowadays can be confronted by an exhaustive audit of the source code. In fact, the algorithmic evidence that the tool was discriminating towards certain nationality could be challenged in courts pleading that the Home Office was discriminating towards these nationality previous to the Streaming Tool.Footnote ⁹⁷ Rather than challenging the algorithmic black-box, there is a need to challenge the institutional opacity which underpins migration policies.Footnote ⁹⁸ In fact, the opacity of the Streaming Tool being used by the Home Office was another factor claimed in this lawsuit:

The Streaming Tool has made visible racialised bordering mechanisms. Through a quantification mechanism of visa applications, the system reveals that some nationalities are more likely to be rejected by the Home Office. In fact, there were some nationalities more likely to be targeted by this algorithm, which made visible the racial bias of the migration system. Such evidence could have been used to challenge the UK visa process in the past. More broadly, this case study suggests that critical debate on AI should not focus exclusively on the racial bias of algorithms. In the words of the scholar and activist J. Khadijah Abdurahman: ‘[I]t is not just that classification systems are inaccurate or biased, it is who has the power to classify, to determine the repercussions or policies associated thereof and their relation to historical and accumulated injustice?’Footnote ¹⁰⁰ Systems such as the Streaming Tool illustrate from a quantitative perspective of racialised border mechanisms: why are some nationalities more likely to be scrutinised than others? Why are nation-states like the UK rejecting more visa applications from one nationality than another one? As Narita eloquently claims: ‘However, rather than simply pointing to the flawed outcomes of technology, an infrastructural lens provides useful critiques of the future being built by bordering technology through revealing the logics and rationalities being “built in”’.Footnote ¹⁰¹

While evidence found by Foxglove and JCWI clearly exposed that the Streaming Tool was systematically discriminating against African nationalities, the case was closed without further scrutiny which could make the racialisation of borders even more evident. First, the Home Office has not provided the algorithmic code, so the decisions made by this tool are still invisible for public scrutiny. Second, we still do not know whether nationality is a key element for the scrutiny and denial of visas to the UK. The Home Office scrapped the tool after the lawsuit launched by JCWI and Foxglove, but the decision-making of visa assessment caseworkers remains largely invisible. Moreover, the Streaming Tool fostered and recrafted the racialisation of borders from a quantitative perspective by statistically distinguishing between suspect and non-suspect nationalities. Such algorithms are based on statistical mechanisms of data and probabilities that quantify human decisions in the past. In the case of the Streaming Tool, the system transformed applicant’s personal history and caseworkers’ decision-making into visa risk scores which shows how algorithms are reshaping the racialised mechanism of borders.

However, the Streaming Tool could have been used to expose discriminatory mechanisms within the Home Office’s visa decision-making processes. Algorithms like the Streaming Tool could have been employed by migrant and legal advocacy organisations to highlight the racialisation of border control mechanisms. Since algorithms rely on historical data for training, the detection of bias in an algorithmic tool implies that the training data itself was biased, indicating past instances of discriminatory practices by border authorities. Thus, algorithmic bias could be used to expose discrimination patterns before the algorithm was implemented and be used in lawsuits. For instance, the Streaming Tool statistically concluded that nationality was a key element to distinguish between high-risk and low-risk visas, proving that border authorities could potentially made unlawful decisions – based on the Equality Act 2010 – in the past, before the algorithm was implemented. Therefore, this algorithmic evidence could be useful to visibilise oppressions and quantify racialised borders in the future, rather than just challenging algorithmic bias, inaccuracy and opacity.

12.6 Conclusion

Through algorithmic systems existing racialised borders are invisibilised, enhanced and re-structured. A critical investigation of the use of algorithmic systems in migration governmentality entails scrutinising not only how these are deployed at the border but also how their implementation has transformed the ways in which algorithmic power is exercised. This requires forging a new political lexicon that does not take for granted oppression, discrimination and surveillance as the main ways in which racialised borders are enacted through algorithmic systems, investigating instead which specific harms and modes of exclusion are enforced. Importantly, racialised borders are not fixed once for all, and actually it is key to register ‘the mutability of race’:Footnote ¹⁰² that is, who is racialised and classified as ‘migrant’, as well as the degrees of racialisation change over time. The use of data-based systems and algorithms to enact exclusionary laws and policies contributes to invisibilise consolidated hierarchies of mobility as well as the ongoing transformations of racialised borders.

Algorithmic systems do much more than monitoring and tracking migrants. Algorithms are used to invisibilise, enhance and recraft the ways in which the unequal and racialised access to mobility is enforced, grounding on discriminatory classifications among individuals which are mainly predicated on nationality. By questioning the juridical understanding of power and the repressive hypothesis that underpin debates and literature on algorithmic surveillance, this chapter has opened avenues for interrogating how the critique of biometric systems and AI in migration governmentality should be framed. Indeed, algorithmic governmentality induces ways of ‘un-thinking’ the future, which in the context of borders and migration entails narrowing down the critique of racialised borders to a matter of bias and discriminatory functioning.Footnote ¹⁰³ Critiques of AI at the borders mainly focus on the discriminatory effects produced through algorithmic-based systems and on the risk of enhanced mass surveillance. In so doing, they partly overshadow the (un)acceptability of borders. Indeed, to truly engage with a critical perspective of AI at the border, we should start from the consideration that the border is the problem – as long as through borders racialised hierarchies of lives are established and fuelled – instead of articulating a critique that hinges on the discriminatory functioning of technologies at the border.

By retracing the colonial legacies of biometrics, Simone Browne has argued that ‘it is at the border – territorial, epidermal, and digital – a site where certain bodies are cast out and made out of place, that a critical biometric consciousness’ could be developed.Footnote ¹⁰⁴ In fact, the border is not simply one of the many sites for investigating biometrics. Rather, it constitutes a case in point, this chapter has shown, for excavating racialised exclusionary mechanisms without ‘seeing like a state’Footnote ¹⁰⁵ nor like a citizen. If on the one hand, nowadays AI and more specifically biometric systems affect everyone, on the other it is key to assess the differential impact they have on citizens and on those who are racialised as ‘migrants’ or whose mobility is ‘illegalised’. Considering the making of racialised subjectivities as a standpoint, we suggest, will enable to re-think the critique of algorithmic power at the border by interrogating how does it look like to see AI at the border like a migrant: indeed, from such a perspective, claims around consent and privacy would be decentred in favour of claims for undoing racialised borders.