Accuracy

It may be considered that sufficient accuracy is achieved when covert speech is predictably detected, recorded, and used as a trigger for the device. This sense of accuracy might best be thought of in terms of the technical functioning of the device. We call this technical accuracy. In another sense, we can think of covert speech in terms of propositional content. In this sense, the accuracy of operation of a neuroprosthesis depends on how well it discloses the content or meaning the user intends to communicate. We call this semantic accuracy.

Accuracy in the technical sense indicates the good functioning of the neuroprosthetic system, from the recording of neural activity through processing, to the production of an acoustic output. In this narrow sense, ‘accuracy’ means that there is a close correspondence between neural activity, the recording made by the neuroprosthetic device, and the acoustic output. This can be summarized as follows:

The neuroprosthesis can be called ‘accurate’ if it:

1. 1) detects and records neural activity associated with an instance of covert speech,

2. 2) maps this to articulatory-motor properties,

3. 3) derives acoustic properties from the articulatory-motor properties, and

4. 4) produces a synthetic verbal output matching those acoustic properties

Even if 1–4 are met, the accurate neural recording, decoding, and synthesized verbal output may yet not appropriately correspond to what a speaker wishes to say. This might be because the act of covert speech triggers the device properly, but in the process of being externalized, some fidelity is lost—perhaps the algorithmic processing of the neural activity produces something not quite intended, or perhaps the user has not generated the neural activity that would produce the desired output. The system here is still ‘accurate’ in terms of the covert speech acting as a trigger for the device, and the reconstruction algorithm working properly, but it externalizes something unintended by the user. Whether or not the device is technically accurate is not affected by the user’s intended meaning—it is a question of mechanism.

The different parts of the system seem able to detach from one another in ways not familiar in everyday communication. The technical accuracy of the prosthetic in representing neural activity is one thing, but the assessment of how this ought to relate to the ultimate synthesized speech involves thinking about the propositional content of the user’s covert speech activity.

In this context it ought to be asked: in relation to what, exactly, is synthesized speech being deemed accurate? Speech perception activates phoneme-specific (or phonetic feature-specific) responses in the superior temporal cortex, similar to the articulator motor-specific activation patterns in the inferior frontal cortex during speech production.^{Footnote 29,Footnote 30} Accuracy of neural recording in motor areas may reflect not just what is intended by the speaker, but also what they hear—something not in their control. There is the suggestion at least of overlapping or coextensive properties between perceiving and producing speech that could well ground analyses of speech processing.^{Footnote 31} These could feasibly bear upon how to understand the speech produced or intended to be produced—the pertinent question might concern how we know what exactly we are decoding. Once we have a clear picture of this, we can still ask to what standards ought we to assess the speech the neuroprosthesis produces.

‘Accuracy’ in a broader sense is not just a question of correctly mapping electrical field activity in articulatory-motor areas of the brain, via algorithmic reconstruction, to acoustic output. That is, we ought not to think of a neuroprosthetic device as ‘accurate’ in its output simply if it registers perfectly the electrical field activity of specified neural areas.

We ought to distinguish technical accuracy, the accurate functioning of the device, from semantic accuracy, the accurate representation of what the user wanted to say. As argued, the former is a technical question. Even if difficult to know for sure, there will be an objective fact of the matter about the degree of match-up between neural activity and machine representation. The latter is more difficult to operationalize, yet clearly ethically significant. Some argue there may be no equivalent ‘fact of the matter.’^{Footnote 32} Natural speech often does not quite match what a speaker attempts to say, and speakers sometimes seem to find out what they ‘think’ through speaking.^{Footnote 33}

In the next section, we demonstrate how semantic accuracy in particular relates to the user’s control of the device. In order for the user to be satisfied with the output of the device, she not only needs to be able to exert voluntary causal control, triggering the device when and only when she intends, but she also must be satisfied with the semantic accuracy of the output such that she identifies with the words and phrases produced. We will argue below that descriptive semantic accuracy is important given the value we place on our utterances authentically representing us. In combination, these bear on the right sorts of relations between sets V and E from Figure 1. Further pragmatic dimensions will affect E, but will be discussed later.

Reflection on both senses of accuracy and how we set the bar for adequacy in these domains prompts broader discussion about the nature of synthetic speech and whether we should hold it to the same standards as natural speech. Is it that neuroprosthesis-aided speech is to restore a speech capacity that has been lost, or is this case more like the instigation of a new communicative practice that will replace that which has been lost? If the former, the way we assess the adequacy of synthesized speech will be in terms of everyday speech, which will constitute a high bar for synthetic speech. If the latter, we must consider whether a replacement for standard speech can be considered adequate if it is not as linguistically rich and fluent as a hypothetical ‘restored’ speech. This has some bearing on the pragmatic dimensions of speech explored below.

As argued, semantic accuracy requires us to assess factors beyond the technical function of the device, and to examine the wider interaction between the device and the user. Conceptual questions of accuracy shade into ethical issues when we examine their relevance for the user’s control of her speech neuroprosthesis.

Control

Control of the device in terms of merely triggering it via covert speech is parallel to the technical functioning sense of accuracy just mentioned. But user control ought to also be thought of in terms of the user’s ability to moderate the activity of the prosthesis, to ‘steer’ it, and curate its speech outcomes. This is most pertinent because control concerns freedom of action which, Nita Farahany writes, “…means the freedom of intending an action, being able to bring it about without obstacles or impediments, and identifying with the action that results.”^{Footnote 34}

Both the voluntariness of the production of an output and the authenticity of that output will be important to assessing whether the user has sufficient control over their device and can be satisfied with the output. We therefore draw a distinction between the voluntariness of the speech and the authenticity of the speech, both of which are of ethical significance. Together, these dimensions of control affect how we assess instances of synthesized speech as intended, both in the sense of volition (whether the user intended to produce speech), and of meaning (whether the output is an authentic representation of what the user intended). We first examine features of the device that will influence the voluntariness of the speech.

Given the general nature of the recording and decoding approach so far described, it is conceivable that a neuroprosthesis could be too sensitive, or too coarse, in decoding neural activity. This effect could result in the involuntary externalization of much more covert speech than a user might desire. Any activity sufficiently covert language-like (in neuroanatomical terms) to be decoded as such by the neuroprosthetic speech system could be externalized. This presents at least four areas for consideration. An instance of covert speech may not be:

1. 1) intended to be externalized at all (e.g., private thought),

2. 2) ready to be externalized (e.g., reflection, ‘practice’ or ‘preparatory’ speech),

3. 3) externalized as was intended (e.g., malapropism), or

4. 4) intended at all (e.g., mind-wandering)

The externalization of these kinds of covert speech might amount to a failure of control of the neuroprosthetic device. As such, they might be mitigated via training and practice. This ought not to be surprising—the systems envisioned here permit the externalization of covert speech via mechanisms controlled through brain activity. This requires discipline, and some practice.^{Footnote 35,Footnote 36} Nonetheless, a private thought, or mind-wandering moment, may plausibly be manifested as an instance of covert speech. The neuroprosthetic device could record, decode, and ultimately make overt this instance. However, it might go against the speaker’s volition.

Further, as a ‘mere’ thought, it may not reflect the user’s desires, values, or intentions. Depending on one’s account of authentic speech and thoughts, the content of fleeting thoughts may not be judged to be ‘authentic’ unless reflectively endorsed.^{Footnote 37} ‘Reflective endorsement’ would mean that the content of a thought would be validated by the agent, given further opportunities to evaluate it. The user would identify with it. This is a complex area drawing in discussions of autonomy among others, which requires more analysis than the present context can provide, but which can be seen in the discussions of De David Grazia,^{Footnote 38} Alexandre Erler and Tony Hope,^{Footnote 39} and Johnny Pugh, Hannah Maslen and Julian Savulescu.^{Footnote 40}

For the purposes of this paper, the idea of reflective endorsement demonstrates that involuntariness may sometimes produce inauthentic outputs. Crucially, this would be a case where there may be semantic accuracy but inauthentic speech. We might not want to preclude the possibility of voluntary inauthentic speech (part of natural speech), but would want to avoid all instances of involuntary speech (even where semantically accurate), and cases where semantically inaccurate yet authentic speech is externalized, i.e., the speech externalized is not ‘as intended’ yet the prosthesis user identifies with it and reflectively endorses it.

Semantic accuracy alone is not sufficient for authentic speech. Involuntarily externalized ‘random’ thoughts might exemplify this. Nor is it necessary for authentic speech: a prosthesis user might not plan in detail what she’s going to say, or there is significant shaping by processing of the device, yet she identifies with and endorses the output. The question of whether the speech is authentic requires us to assess the broader relationship between the utterance and the agent.

In short, the speech produced by the prosthesis could be involuntary but authentic (as in the case of reflectively endorsed private thought externalized by mistake). Further, voluntary speech could also be ultimately inauthentic if, for example, too much prediction or ‘smoothing out’ by processing were to occur.

We remain neutral on what precisely counts as authentic speech. However, we now argue that, regardless of one’s account, the way the device functions may generate new challenges for assessing authenticity, since the device itself will contribute to and shape the output. In cases where the covert speech instigation of the neuroprosthetic system and final synthesized speech output diverge—i.e., where there is semantic inaccuracy—or where covert speech is externalized involuntarily, the recording, processing, word prediction, and decoding functions of the prosthesis could be playing a significant role. This is ethically significant, as individuals often learn about or refine what they value and believe through engaging in communication with others—i.e., they take their own utterances to represent something about their beliefs, desires and values. Whilst potentially improving efficiency, the more the output is shaped by the processing, the greater the potential for some degree of disconnect between the user and the output.

In worst-case scenarios, the user could be like a ship’s captain without a working tiller, knowing where to go, but without means to steer.^{Footnote 41} Or, where there is a significant contribution to the output made during processing, the user could be like a ship’s captain who has the impression he is steering and picking his course, when in fact his hands are nudged more or less often by autopilot. More coarse-grained control could (at least) allow the user to omit to say anything at all, to halt a course begun in error, or taken up unexpectedly by the system, or any other such sequence of events that diverges from that intended by the user. Some examples are summarized in Table 1.

Table 1.

Summary of possible types of speech and concerns.

For example, a feedback loop, ‘first listen’ approach would enable a user to evaluate and to release only those externalizations fitting their evaluative standards. The idea of a feedback loop is a promising candidate to buttress control. However, where the ambition is to create a device that can ultimately provide a communicative ability akin to unimpaired natural speech, this would represent an obstacle—introducing a delay by design would inhibit the free-flowing nature of speech expected in everyday communication. Given its value in providing assurances over the proper functioning of the device, however, this might still have value as a part of the research and training process.

In terms of technology readiness levels (TRLs)^{Footnote 42} we can think of the role for feedback as an early stage heuristic in the development of the technology. Its usefulness is particular to the development of the early stages of the prosthetic technology, as it is at this stage that researchers and early users must come to an understanding of the nature and scope of applications of the devices under development. This is specifically for TRLs of 1 and 2 (basic research, proof of concept), where development is underway. By TRLs 5–9 (validation in context, to final product), the device would be expected to operate more comparably with natural speech and so the feedback mechanism would represent an obstacle.

The focus upon control is not a straightforward solution in itself. For instance, a feedback mechanism between prosthetic and user might yet be considered in terms of ‘verbal overshadowing,’ wherein sincere reports made by a user are subsumed by ‘objective’ descriptions given to them by the device.^{Footnote 43} Ben Alderson-Day et al. note that this phenomenon came to the fore “…following evidence that verbal description of the perpetrator of a crime was associated with a 25% reduction in recognition of the perpetrator’s face.”^{Footnote 44} Specifically in terms of auditory perception, Timothy J. Perfect et. al write that,

Might the speech produced by a prosthesis, once played through a speaker and heard by the prosthesis-user, risk producing a verbal overshadowing effect in that user? The superior temporal gyrus is sensitive to the phonetic features of heard speech, which could be problematic were the effects of the prosthetic’s playback to interfere with its recording activity through stimulating the neuronal activity it is supposed to be monitoring.^{Footnote 46} The speech system, or the users themselves, could become confused over what is being intended as speech, and heard as playback, especially in cases where medication or cognitive issues may play a role. Not least, this would seem highly salient where complicated issues about informed consent are at stake.

It is extremely difficult to say for any speaker if or how their eventual utterance relates to their initial intention and actual verbalization.^{Footnote 47} It might be part of everyday discourse that one is more pleased with an utterance than expected. Situating speech within its context, with careful critical attention, can be central to discerning the meaning of that speech as it relates to a speaker and a particular conversation. These are potential constraints upon the set ‘E’ from Figure 1.

Pragmatic considerations

There is a pragmatic dimension to this that requires attention: What role does an interlocutor have in assessing accuracy and inferring degree of control, and what are her interpretive duties? Departing from the first-person perspective, ’accuracy’ may in part be dependent on the listener performing her role as participant in a communicative exchange. What an audience knows of a speaker’s character, who they are, and of the context’s influence on speech activity will be seen to be important in ascribing meaning to communicatively employed language. In the novel case of neuroprosthetically mediated communication, norms of communicative action require scrutiny.

Pragmatic dimensions of speech require some focus given potential failures in control that may be relevant to judgements of accuracy, especially in terms of authenticity. How an interlocutor notices a mistake, or something amiss, in a discussion can relate to how well a speaker appears to reflect their own perspective in their verbal behaviour. The remarks made here require further elaboration, likely in another paper focussing upon informed consent. Nevertheless, this dimension deserves some exploration in the context of the discussion so far.

Pragmatics draws upon interaction with other people, which in the case of neuroprosthetic devices presupposes user control of their device. Thus, where pragmatic considerations arise in technologically-mediated speech, volition and control are central issues. Apparent divergence between what a speaker likely means and what they actually say can result from involuntariness or failure of control. In either case, this can be addressed in pragmatic terms.

Paul Grice’s pragmatic analyses are particularly relevant here, as they centre upon the relations among speakers and their utterances, and among speakers and their audiences, in order to account fully for linguistic meaning. This draws attention to the pragmatic dimension of meaning, highlighting the responsibilities upon audiences in understanding utterances.

Grice characterizes his ‘cooperative principle’ as follows:

User control can play a role in terms of ensuring speech externalized by the prosthesis is a voluntary, authentic reflection of the user’s covert speech. Grice’s principle serves to highlight the remaining interpretive task of the listener. Where clarity of meaning is at stake, there is an onus upon the listener to actively seek to recognize that meaning. Therefore, in addition to technical solutions, ‘social’ measures ought also to be developed in order to prepare the context for speech realized via neuroprosthetic devices.

Careful phrasing of questions put to the users of prosthetics; the construction of multiple phrasings in order to verify the apparent contents of speech produced; a suitable degree of interpretive charity with which to contextualize potential misunderstandings;^{Footnote 49} as well as other such measures on behalf of the audience can serve well to mitigate issues here. Primarily, the recognition of this kind of communication as potentially different from that routinely encountered, and so deserving of care in reception, is what counts.