EXAMPLES

Consider for a moment these examples of the application of machine learning that may alter the balance of power in relationships between private individuals and landlords, and criminal defendants and the state.

‘Tenant Assured’ enables landlords to vet potential tenants in advance by scraping information from various social networks, build a picture of their personality and financial situation, and provide an overall score.Footnote ¹ The privacy implications are obvious, but these are not the most interesting issues. The profile for a prospective tenant is assembled by algorithms drawing conclusions from data from a variety of sources. The Tenant Assured website claims ‘If an applicant is using a lot of negative words and regularly argumentative online, then he/she will likely have a negative online reputation. And vice versa.’ Leaving aside the question of whether this is a legitimate processing of personal data, a valid conclusion, or a relevant consideration in deciding whether to rent a property, consider the number of subjective choices implicit in that statement, and how these must be converted to some quantitative measure. What is a negative word? What are a lot of them? What is regular? Programmers will have made decisions about all of these factors, which will be embedded in difficult-to-change software code, which is not available to the public to challenge or correct. Instead individuals are told that they can't rent a particular apartment because their online score is not high enough. This happens for reasons that cannot be adequately explained to them because the code is secret and probably difficult to understand, especially by a non-programmer.

Secondly, several jurisdictions in the USA are relying on computer systems to predict outcomes for criminal suspects and defendants.Footnote ² These are used to identify individuals who are likely to commit crime and locations where they are likely to do so – by judges to assist in determining sentences and by prison officials in deciding how to manage particular prisoners. Eric L Loomis has challenged the use of the Compas algorithm developed by Northpointe Inc, as part of his sentencing.Footnote ³ The Wisconsin Supreme Court has allowed continued use of this system but has highlighted that it is only one factor to be considered when passing sentence.Footnote ⁴ On a more positive note, a pilot project in New York City using big data and machine learning proved more accurate than judges at predicting which defendants were likely to offend if released on bail, while not considering their race, a development which is likely to lead to a re-balancing of the prison population.Footnote ⁵

Thirdly, Joshua Browder, a British Stanford University student, has developed a chatbot-based expert system, DoNotPay.co.uk, which assists with challenges to parking tickets. Tens of thousands of tickets have been overturned with its help, and he has expanded the system to assist with applications for refugee status.Footnote ⁶

ADOPTION

Digital information and communications technology, particularly big data, machine learning tools, and networked devices, have been enthusiastically adopted by individuals and businesses, altering the texture of commercial and social relationships in profound ways. It is clear that these new tools are becoming a very important part of modern society and a significant factor to consider when developing political or business strategies, developing new markets, or trying to solve problems. All of these changes have raised many legal questions, and issues around data protection, deregulation and automation of public transport, and copyright infringement have become headline news. What has received less attention until quite recently is the use of these technologies by the state.

This was not so obviously relevant to the individual or socially significant. There seems to be little attention paid to how this use may impact on fundamental legal values. The Snowden revelations changed that with regard to mass surveillance. However, many other aspects of government use of information technology go unexplored. Large public databases, obscure software systems, and creaky user interfaces are not interesting to the average citizen or the journalists whose role it is to inform them of what matters.

There is a great deal more to the wide-scale adoption of digital technology than meets the eye, particularly in the hidden corners of the modern state. This is slowly becoming better understood by professionals and the general public. In very recent years, stories have emerged that highlight the key role that algorithms (which are, at their heart, processes of selection) play in shaping the world around us. Google Image Search has displayed racial and gender biases – labeling pictures of black people as ‘gorillas’,Footnote ⁷ selecting mug shots when asked to show ‘three black teenagers’ but selecting happier images when asked to show ‘three white teenagers’,Footnote ⁸ and under-representing women in searches for career-related keywords.Footnote ⁹ Facebook has been accused of manipulating the news stories which are highlighted in individuals’ feeds.Footnote ¹⁰ Volkswagen has admitted installing software in its diesel cars that cheat on emissions tests, something which has led to prosecutions for executives and fines of $4.3 billion.Footnote ¹¹ Other car manufacturers are accused of using similar ‘defeat devices’.Footnote ¹²

PUBLIC SECTOR USE AND DIFFICULTIES

If this is occurring in the private sector, what biases, errors and assumptions are causing difficulties for individual citizens from the use of algorithms in the public sector?

We still know very little about how government uses these technologies and what may go wrong. Software is now a key component in a great deal of the machinery of government – for example, social welfare, the tax system, environmental management and regulation – and is increasingly playing a role in policing and warfare through profiling of criminal and terrorist suspect, the use of drones, and the deployment of autonomous weapons systems. The issues that are buried in these systems should be a matter of serious concern for law and lawyers as we try to protect fundamental values in the 21st century, and we try to properly represent our clients' interests against systems errors that we cannot easily identify.

How are individuals selected for tax audit? Why are they denied social welfare benefits? Why are they investigated by the police? Often, this will be because of outputs from software programs, sometimes machine learning systems that sift through masses of big data. As Danielle Keats Citron has discussed in her writings,Footnote ¹³ this development – the taking of decisions about individual rights and entitlements based on unknown, inscrutable, and often unchallengeable systems – raises serious challenges to the rule of law in the modern state. Similarly, as private sector systems become the conduits for our daily lives, how many individuals are side-lined because of unfair factors? (For example, Google advertising seems to have a gender bias; it generally will not show executive jobs to women.Footnote ¹⁴ )

The impact of error in these processes can often arise from mundane causes and affect a small number of people, rather than arising from the large-scale application of the current buzzwords of ‘big data’ and ‘algorithms’, but the impact on individual lives can nonetheless be significant. For example, easy-to-anticipate problems in transitioning from one computerised case management system to another in Alameda County, California led to dozens being arrested or jailed in error, with others being forced to register as sex offenders when they should not have done so.Footnote ¹⁵ Big data can produce problems on a much larger scale – the Australian Centrelink social welfare compliance system has required many thousands of citizens to needlessly prove that they were properly in receipt of benefits because it generated far too many false positives.Footnote ¹⁶ Overall, it is clear that the use of databases, algorithms, and big data by government and bureaucracy has the potential to go seriously awry, with significant negative consequences for individuals and populations.