There is nothing like a good story. This amazing book is full of them—stories about poor men, women, children, abled and disabled; stories about encounters between users and providers of social services, all mediated by online onboarding forms, numbers, formulas, and, most of all, life-altering verdicts rendered by computers. It is a phenomenally and phenomenologically rich book, which mercifully attaches actual people’s names and flesh and sweat and tears to the possibly driest subject of all: algorithmic decision-making.
Automating Inequality opens, actually, with a story about the author herself, struggling with an investigation for healthcare fraud by her insurance company. She suspects an algorithm red-flagged her family for submitting expensive medical claims shortly after she started a new job. Her long-term partner had been the victim of a random street aggression that sent him off to the hospital. This may have been an unlikely development from a statistical point of view, but unlikely things happen to real people. It took a year-long administrative, financial and emotional rollercoaster to clear the mistake and restore their benefits. And yet, the author remarks, she was fortunate. She had the cultural capital, material resources, social supports, and grit to question the Byzantine apparatus that had improperly classified her. But, as automated systems increasingly make their way into people’s lives, most of those targeted find themselves in a far more vulnerable position. Unlike Eubanks, people at the “bottom of society”—the mass of working and non-working poor in America—lack her familiarity with technology, her self-confidence about what they are entitled to, and perhaps her flexibility to devote time to righting a wrong. And while her encounter was chilling and upsetting, it was a one-off event. In contrast, poor people are on the receiving end of such glitches day in and day out. Those who rely on public services for survival have little choice but to put their personas under a microscope, and to see their lives disaggregated into thousands of data points that are then recombined for efficient sorting and processing by anonymous machines in obscure places.
Automating Inequality, then, is a book about automation, about what is happening when we relinquish important—vital—decisions to computers. It is a book about the growing black boxing of administrative practice, the disappearance of human mediation in the interactions between citizens, particularly poor citizens, and the state. We’ve come a long way from the time when social theorists hailed the impartiality of the well-trained, rule-bound government agent as a guarantee of procedural fairness. The ideal-typical bureaucrat, Max Weber wrote, was rational, objective, acting only by the rule and “without regard for persons.” Yesterday’s algorithm was embodied in an actual person (who, as a side note, was likely to be male), trained to exert his judgment in a highly codified manner. But he was not perfect, far from it: sometimes corrupt, often prejudiced, frequently incompetent, and looking out for his own survival above everything else. So, it would seem that his replacement by a computer should deliver us from his failings—that the cold, mechanical objectivity of mathematics should be a cause for celebration! Indeed this is what the companies that peddle these tools want you to believe.
And yet to a scrupulous observer like Eubanks, this shift of authority from persons to machines is delivering something else—something that Max Weber also foresaw: “an unreal realm of artificial abstractions, which with their bony hands seek to grasp the blood-and-the-sap of true life without ever catching up with it.” In Virginia Eubanks’ world of social services for the poor, the frontline agent (who, as a side note, is now likely to be female) is disappearing. She is losing her discretionary power over individual situations, bound as she is by automated rules, by thresholds that start investigations or by red flags that deny benefits. That agent now second-guesses herself when the computer renders a verdict different from her own analysis of a case. She has been turned into a pure executant, Taylorized into discrete tasks, and stripped of the ability to override the algorithm. She lives in fear of being downsized, demoted and cheapened. Decision by decision, she is forced to detach herself emotionally from the people she is supposed to serve. The rational state is, indeed, an unfeeling state.
Eubanks is no fool, though. As the title of her last substantive chapter (“the digital poorhouse”) intimates, the rule of machines exacerbates forms of moral regulation that have long been institutionalized, going back all the way to the original poorhouse. But this analogy prompts a question: is something bigger at stake? In other words, is automation the real culprit, or should we rather incriminate the way we—as a society—treat the poor? In Automating Inequality Eubanks draws all of her examples from the United States, in spite of similar pushes occurring everywhere else around the world. This is a felicitous choice: the social consequences of automated decision-making are especially visible—and unnerving—in the United States for three reasons. First, since the 1970s the country has made virtually no progress on the poverty front. Second, this abysmal performance in relative terms was in part the result of the dismantling of those public infrastructures that were tasked with mitigating poverty. Third, whatever is left to help is a hopelessly incoherent patchwork of programs whose benefits and criteria vary across states, counties and cities, all with their own systems and requirements. It is striking (and of course Eubanks designed her book that way) that the three core empirical chapters are about, in short order: the state of Indiana, the City of Los Angeles, and Allegheny County in Western Pennsylvania. Three locales, three systems, three levels of government.
It is useful to step back and ask ourselves: would Eubanks or cerebral palsy-stricken Sophie Stipes have experienced a denial of their healthcare benefits (whatever its cause) in a country where healthcare is simply framed as a basic human right accessible to all? Would so many children have been taken away from their parents for “failure to provide” in a country that offers decent financial support to all families? Algorithms failed the populations they were meant to help not only because they were poorly implemented, but first and foremost because they were deemed useful to sort, to manage at the margins—that is, through a default assumption of exclusion rather than inclusion, difference rather than similarity, responsibility rather than solidarity—all of it deeply steeped in the particularly tortured history of the American welfare state.
In other words, the problem is very much a political problem. It is thinking about poverty through the lens of deservingness and morality rather than through the lens of human dignity, universal rights and solidarity. The best argument for unconditional social benefits and services is that they don’t demand us to choose who the recipients should be––only who should be taxed, and how, so they can be delivered. If there is no one to sort, because everyone is included, then algorithms lose their visibility and stigmatizing power. Eubanks’ cautious endorsement of Universal Basic Income in her forceful last chapter emerges from that logic (though she is silent on the necessary tax counterpart).Footnote 1
The economic story is even more subdued. The book suggests, but does not fully interrogate the many ways in which the new data allows for a rethinking of social provision itself, making the digital poorhouse a fundamentally different political animal from the old one. We are often left wondering: what is the deal? What is the state really gaining from this? How do its powers change through it? Also: what happens to the data collected by public agencies, or by vendors on behalf of public agencies? Actually, do they even collect this data themselves, or is the process of collection being externalized? What happened to poor and working-class Hoosiers’Footnote 2 data when the state cancelled its contract with IBM for determining welfare eligibility? Was that part of the settlement?
To the extent that data is “treated” and put to algorithmic work by private corporations, who has a right to use it, and how? Whose data has to be given away, along with the expensive contract, and for whom are special precautions or special prices the norm? How do public officials behave around poor people’s data, children’s data, or prisoners’ data—three vulnerable populations under the care of the state, but eliciting different kinds of moral feelings? In the realm of education, for instance, online platforms often offer their hardware and software for free (or at steeply reduced prices) to resource-starved school districts. Data, of course, is the real currency here, which is often bargained away without democratic oversight or proper safeguards.(Good deals both remove the true value of the counterpart (the data) from explicit monetization, and protect against public scrutiny.) As a result, corporations and non-profit organizations have been quietly appropriating much more data about students than is educationally necessary, storing it indefinitely, for future uses that even they cannot yet fathom. It is through the implementation of these data-heavy technologies, however, that a nuclear cloud of somewhat ad hoc numbers about real people materializes and starts to dynamically circulate in the digital ether with very little accountability, threatening to re-entangle itself in people’s lives every time they come into contact with it.
In other words, the state and its functions have been reconstructed by private economic actors as a profitable ground for generating the new oil of the new economy: data. In this very particular economic circuit, populations under state control (including, perhaps state employees themselves) are being “traded”, bit by bit, against the promise of institutional innovation, fairness, efficiency, red-tape-cutting, roll-cutting, and staff-cutting. Eubanks’ vital contribution is to uncover the tyrannies and personal injustices that takes place at the intersection between infrastructure and biography. But it is just as urgent that we start questioning the whole political economy upon which these infrastructures have thrived. We must follow the money—that is, often, the data itself—, wherever it may go.
