Assessing the Ethics of Algorithms

The first epistemic concern holds that the connection between the production of evidence and the resulting conclusion might be inscrutable.^{Footnote 24} Inscrutability or opacity can derive from intentional hiding (e.g., because of competition on the market), from people’s inability to technically understand programming and algorithms, or from the nature of algorithms and their high dimensionality and complexity. Algorithms are often judged as black boxes because, especially in machine-learning algorithms, the working mechanisms are opaque, even to experts.

Second, the evidence generated by algorithms can be inconclusive. There is an inevitable amount of uncertainty attached to the conclusion drawn from the data an algorithm uses, because, in essence, algorithms typically detect correlations between variables, but rarely causation.^{Footnote 25} Imagine the case of correlating one’s place of living and one’s financial situations: credit rating algorithms take this correlation (which is not a causal relation) as an indicator of one’s potential to pay back. However, these kinds of correlations might be spurious, as many authors maintain.^{Footnote 26}

Third, epistemic worries arise because the quality of an algorithm’s output depends on the data that serve as input (“misguided evidence”). If the data are misguided, so is the resulting conclusion, it seems. The term “misguided” in this context refers to biases on the part of those who design and implement these algorithms. Biases can enter an algorithm as the result of a designer’s or user’s decision (e.g., when automated surveillance incorporates the values and biases of law enforcement authorities), but they can also be technologically driven (e.g., when flaws in the data reoccur in the resulting conclusions), or represent an emergent phenomenon linked to new contexts of use (e.g., when clinical decision-support systems [CDSSs] can only recommend treatments that are within its database).

Where Are the Problems?

There a number of problems related to these issues. Opacity or lack of transparency in algorithms’ data mining or decisionmaking, for example, decreases trust in algorithms and data processors. Because users of algorithms; such as banks, insurance companies, or courts, depend on their customers’ and citizens’ trust, making use of inscrutable algorithms whose decision-logic defies human comprehension has negative implications for cooperation.

Furthermore, opacity makes oversight, comprehension, interpretation, and control quite impossible. If an algorithm and its decisionmaking logic cannot be adequately interpreted and understood, the resulting decisions cannot be controlled, tested, and investigated. Both the concrete logic behind a decision and the wider implications of using a certain model in the algorithm potentially remain opaque and are therefore hidden from control and oversight.

Algorithmic opacity, hampering accessibility and comprehensibility, also stands in the way of informational privacy.^{Footnote 27,Footnote 28} If the basic elements on which a decision against a person is based are inscrutable and defy accessibility and control, the person loses the capacity to control what is known and who knows it. Moreover, people can no longer control what is done with the information available if they does not know what information exists in the first place. This has implications for autonomy as well. People who do not have knowledge of, or control over. their data, namely its acquisition and processing, might be restricted, or even restrict themselves preemptively, in their choices, and, therefore, suffer a decrease of autonomy.

Inscrutability, but also inconclusiveness, are often related to questions of discrimination, because not knowing how an algorithm works makes it harder to detect or locate biases and, consequently, to find solutions to address these possible shortcomings. Finally, through algorithmic discrimination, existing inequalities might be reinforced, leading to less autonomy and participation on the part of the discriminated groups and persons. An example of the reinforcement of inequalities might be a case in which homeless people did not receive assistance because an algorithm made the assumption that someone who had been in prison previously cannot be homeless.^{Footnote 29}

Solutions

Several scholars have nonetheless pointed out a number of possible solutions to these problems. First, there is the relatively intuitive idea of creating transparency as a means to overcome opacity and other shortcomings. However the usefulness of transparency in itself can also be questioned, with the argument that transparency is often neither necessary nor sufficient for accountability in algorithmic decisionmaking.^{Footnote 30} Transparency itself may not even be the most desirable goal. Additional technological instruments are suggested to provide accountability instead. Transparency can be created either by making explicit input–output relations such that the algorithm only produces results that are expected beforehand. This, however, would amount actually to not using machine-learning algorithms altogether because, almost by definition, machine learning defies the inclusion of fixed decisionmaking criteria.^{Footnote 31} The problem is that if algorithms are programmed in this manner, their effectiveness might be reduced.

Alternatively, algorithms can also be tested and licensed by third parties, also known as auditing.^{Footnote 32,Footnote 33} The auditing approach might be useful in the case of effective machine- learning algorithms that remain opaque, sometimes even to experts.

Moreover, implementing an explanation module that exposes the decisionmaking of the algorithm “post-hoc” in humanly understandable terms might be another way to address opacity in machine-learning algorithms.^{Footnote 34,Footnote 35} This solution has the problem of potentially leading to false reassurance or at least to an incomplete understanding if supposedly humanly comprehensible explanations are given. Public education in programming, code writing, and computational thinking in general represents a further path toward enhancing people’s awareness and sensitivity to algorithms, and therefore toward creating oversight and control. The use of open source software might also contribute to this end.^{Footnote 36}

With the aim of avoiding discrimination, further technological solutions are discussed; for example, programming without discriminating factors.^{Footnote 37} Finally, other innovative approaches have been suggested to resolve the responsibility gap. These approaches tend to presuppose that not only humans can act morally or be responsible, and thus suggest a framework for programming moral competence in robots or of devising a means to implement moral reasoning in machines.^{Footnote 38,Footnote 39,Footnote 40,Footnote 41}

The ethics of algorithms debate has identified a number of epistemic and resulting ethical concerns and has provided some possible solutions for these concerns. In the following, we will address the question of if and how these concerns and solutions can be used to shed light on the ethics of BCIs.

Algorithms and Ethical Concerns Regarding BCIs

The ethics of algorithms debate shows how algorithmic opacity creates problems in BCIs as well. The evidence created by BCI algorithms can be inscrutable, because their working mechanisms are somewhat opaque. Moreover, the evidence can also be inconclusive. In the case of BCIs, the “evidence” in question is the identification of relevant brain signals (filtering out all artefacts) and their classification by the algorithm; for example, as a certain intention or affective state. Thus, algorithms create evidence for the conclusion that a certain state of mind in the user has occurred. In most cases, this evidence is directly transmitted to the output device; for example, an exoskeleton, a wheelchair, or a communication device. This kind of evidence is opaque to the extent that the particular decision-logic defies understanding on part of the user, and it is inconclusive to the extent that there is a degree of uncertainty attached to the identification and classification of brain states.

It can be argued, however, that opacity in BCIs is not as relevant a question as it is in other algorithms. In BCIs, algorithms only use a relatively narrow set of data. The purpose of their use is, as stated previously, to identify relevant brain signals, classify them, and correlate them with a certain output. Compared with the huge amount of data acquired by, for example, a credit rating algorithm, the volume of BCI data is rather low. Thus, the range of possible outcomes that need to be understood is limited.

Moreover, contrary to algorithms in general, in BCI use, the user (i.e., the one who uses an algorithm to make decisions), and the entity mostly dealing with the effects of the algorithm, is one and the same person. In other contexts considered in the debate around algorithmic decisions, algorithms are used by one party (such as a bank, an employer, a court) to make decisions about another party (e.g., a person wishing to get credit, an applicant for a job, or an offender in a trial). If a person or passer-by sees a problem emerging with the algorithm in these cases, that person might complain directly to the respective user and ask for an explanation or, ultimately, the two parties might mutually agree that not using the algorithm is the preferred option in certain contexts. The consequences of algorithmic opacity seem thus to differ qualitatively in BCI use. A quadriplegic person who uses a BCI to direct a wheelchair is both the user of an algorithm and the recipient of its use, because it is that person who experiences the algorithm’s outcomes. Moreover, in the case of BCIs, the algorithms are more closely linked to the action that the user wishes to perform, compared with other contexts of algorithm use (as in the case of a credit-rating algorithm). The problem of opacity seems, therefore, to be less far-reaching in BCIs. A BCI user cannot simply complain to another party because of the lack of transparency of an algorithm and ask or demand that the algorithm not be utilized in certain contexts. This picture and its related ethical consequences change again, if third parties are also involved through the BCI use, as might be the case if a wheelchair controlled by a BCI hits another person who happens to stand aside.

None of these thought experiments deny the opaque and inconclusive nature of the algorithms in BCIs as such. The algorithm’s output, that is, a particular movement of the BCI-directed exoskeleton or wheelchair, might still be “surprising” or opaque for the user, precisely because the formation of that movement must be attributed to the algorithm whose decisionmaking process remains opaque. It is true that some amount of training is needed to enable a user to control an external device via the BCI, but presumably there remains a degree of uncertainty at least in some cases. This dynamic demonstrates that the inscrutable and inconclusive nature of algorithmic decisions in BCIs can be reframed as a safety problem. The BCI literature has addressed this problem as the trade-off between speed and accuracy: the longer an algorithm can do its work, the better the results of identification and classification.^{Footnote 42} Increasing BCI accuracy through training, however, is an often frustrating endeavor and can lead to a lack of trust in BCIs.^{Footnote 43} This would have serious implications for further studies in improving BCIs, with serious repercussions for those who stand to benefit from these technological advancements.

Algorithmic opacity is at the core of other problems that have been identified with BCIs, with one among these being privacy, for example. At first sight, the issue of privacy as a right to control access to one’s data,^{Footnote 44} is touched only to the extent that data are shared with third parties, because BCI algorithms are tailored to the individual brain. However, one might consider the case of a passive BCI that detects brain states unknown to the individual, and imagine that the BCI acts accordingly. For example, a BCI might detect a negative affective state during an interaction with another person. This might indeed become a privacy-related issue because it questions the individual person’s capacity to control access to that person’s core personality. In a sense, it is no longer the person who has privileged access, but the BCI. The BCI user shares effective control and access with the algorithms.

Such questions raised by the use of the algorithmically fixed BCIs have direct implications for the notion of autonomy. It rapidly becomes a questionable matter whether or not people are autonomous if they do not have privileged knowledge about the mental states that drive their motivations and actions. If algorithms are used to detect brain states and to inform an external device to perform a certain task, this might be seen as enhancing autonomy for those who cannot perform that task otherwise. But autonomy can also be reduced in BCI use if people have no insight into and control over the resulting effects of their activities with a BCI. Furthermore, if there is a constant mismatch between what users believe about their mental state, and what the BCI detects, users’ self-concept might be confused to an extent that their ability for self-determination is ultimately diminished. Somewhat paradoxically, individually tailored algorithms could be designed to enhance autonomy by opening a path toward real self-knowledge, enabling people to find their “true” preferences, or deepest emotions. However, if users are not in the epistemic position to discern their real selves when the BCI gives outputs that are not clearly understood or expected, then autonomy may be significantly reduced.

A further aspect for ethical consideration is that the evidence generated by a BCI algorithm is often incomplete. Imagine that a person’s BCI system detects that the person feels uneasy and therefore stops the person’s wheelchair automatically. Whether the uneasiness was meant to be the action-guiding mental state is an open question. It seems that the question of which mental state should be considered as “action guiding,” needs to be well defined prior to using a BCI, if a person wants to stay autonomous.

Because algorithmic opacity is said to contribute to biases and discrimination, the question arises whether opacity is observable in BCIs as well. Discrimination in BCIs appears to be not as clear-cut as in other cases. What could emerge as “discrimination” is a bias in the ability to use BCIs. As Wilhelm Bauer and Mathias Vukelic discuss,^{Footnote 45} elderly people exhibit differences in their assessment of negative emotions compared with younger people. This has an impact on older people’s neural correlates of affectivity, which makes it more difficult to detect affective brain states that then are informing a passive BCI. If these results are confirmed in future studies, the problem of opacity for elderly BCI users is increased, and, more fundamentally, using a BCI will not be open to everybody on an equal level. To state that this is a problem requires evidence to the extent that people have a right to use such a technology, or have a vital interest in it. One could argue that should BCIs be used widely in society one day, the unequal ability to use a BCI constitutes a disadvantage. As Guglielmo Tamburrini^{Footnote 46} notes, unequal access to BCIs because of personal variations in BCI proficiency might raise questions of healthcare justice. The workplace context is another area in which the age-based tendency to be unable to use BCIs could pose serious problems of discrimination. Note, however, that in this case, the ethical problem is not one that arises out of a working algorithm’s inherent bias (wherever it comes from), but one of an algorithm being unusable for a given population. This adds a particular aspect to BCI ethics that goes beyond what the ethics of algorithms has offered so far.

Potential Solutions

Algorithmic transparency has been put forward as a solution to decrease the level of opacity; to increase trust, understanding and oversight, privacy, and autonomy; and to reduce bias and discrimination. One particular means to achieve transparency would be to refrain from using machine-learning algorithms altogether. This, however, would most probably lead to less effective BCIs.

A more promising approach would likely involve making the algorithm’s model interpretable post-hoc; namely, by explaining the decision logic and working mechanism after a particular decision has been made. One can imagine that BCI users gradually learn about the algorithm and its working mechanism through behavioral, but also explanatory feedback, and can subsequently adapt to it. However, whether this is the case depends on the individual’s ability to understand computational issues, and it makes room for potential discrimination because it limits the range of people who could understand and subsequently use BCIs with more insight. Post-hoc explanation might even lead to a false assurance and reliance on algorithms, potentially causing harm at a later point in time.^{Footnote 47} If a BCI repeatedly explains its decision, the user might increasingly trust the technology and lose the awareness that the technology could still be prone to making errors.

As a further option, auditing has been proposed as a solution to the problem of inscrutable evidence and opacity. Auditing does appear to be highly useful for BCIs. Trusted auditors such as researchers or external regulators could (be required to) analyze algorithms and either make them explainable or certify their acceptability. These auditors could examine cases of BCI uses and see how reliably the algorithms work (e.g., ex-post audit studies). They could run simulations or use algorithm internal reporting mechanisms. All these audits leave the black box model intact; that is, they do not attempt to understand the algorithm, but rather prove its reliability and verify correct functioning. However, more research is needed to find practical ways of conducting audits of BCIs, particularly because some of the scenarios envisaged here do not even exist yet.

Education in programming and coding has been put forward as yet another means to counter inscrutable evidence, and is also applicable in the case of BCIs. However, knowledge about programming is a rather generic solution that does not guarantee that a particular algorithm and its particular decisions are understood. And if some machine-learning algorithms are said to remain opaque even to experts, by extension this will hold true for educated BCI users. Using open source codes and software will not improve BCI transparency, because it is typically not the BCI user who elaborates on the BCI’s algorithm.

Given the opacity of a BCI algorithm and the impact of this question of opacity on safety, privacy, and autonomy, it might be useful to implement measures that locate the final say as to what action the technological device performs with the user. A veto right to intervene in the BCI-mediated action can be justified on these grounds, through implementing an emergency stop. This is, at root, a technical question that affects the design of the algorithm. The case appears to be that algorithms need to be created that do not only detect and classify brain signals, but also leave control over the action with the user. This distinguishes BCI algorithms from other, non-BCI instances of using algorithms. Here, users have control over the algorithms because they are the ones to decide whether or not to use an algorithm for a certain task; for example, to decide whether people who apply for credit can be granted their wish. There is no need to talk of a special veto right here, because the user would have the veto right in the first place (to use or not use an algorithm). One part of the explanation is that the action of interest here does not rely on algorithms. To decide whether an applicant should receive credit does not need to make use of algorithms. The decision can be made using various other means. In BCIs, however, the algorithm directly connects with the intention-generating and, therefore, action-initiating processes within a person’s brain. A BCI is constitutive of an action and not merely a replaceable instrument. Therefore, a veto right appears to be necessary in BCIs, which complements what some authors have termed the “right to explanation” in the case of other algorithms.

With regard to the discriminatory elements in BCIs, available remedies do not directly target the algorithm and its design. Rather, these elements appeal to the way that those who put the BCI algorithm to use in the real world have to compensate for the resulting harms. It could include mandatory provisions of alternative means that those with less or no BCI proficiency can use. The workplace context is one area in which age-based tendency toward being unable to use BCIs could pose serious problems of discrimination. Technological solutions could be provided by taking into account the age-bias and program algorithms accordingly. Political instruments might be applicable here, too; for example, that using BCIs as automation tools in factories could require paying an extra tax.

On a more general level, the debate about the ethics of algorithms has also revealed various reinterpretations and reallocations of responsibility as a solution to the responsibility gap. One way is to maintain that there remains some portion of responsibility with designers and developers of algorithms to test their algorithms and to care for epistemic justification as well as nondiscrimination. It might be interesting and fruitful to approach this question from the perspective of corporate social responsibility^{Footnote 48} (CSR) or even political corporate social responsibility^{Footnote 49} (PSCR) so as to include a broader range of potential bearers of responsibility. This approach would introduce a debate that has a long-standing experience with ethically assessing market behavior and could shed new light on designing and selling medical devices.

As a final note, an entirely novel way to evaluate user responsibility might be necessary. This is reflected by attempts to conceptualize the notion of responsibility toward BCI-mediated actions as being analogous to the scenario of interaction with animals. To explain, similarly to the case of the owner of a domestic animal, the BCI user does not always know how the algorithm will eventually “act” or behave. Nonetheless, the owner has responsibility for the results of the interaction of the animal. Whether harmful or unwanted results occur depends on the knowledge, the virtues, or the morality of the user, or on other factors at least partially and sufficiently under the control of the user.