In the section of his paper dedicated to What should we (not) do, Seth writes that “The development of real artificial consciousness for its own sake should not be an explicit goal.” This touches upon the fundamental question why to pursue artificial consciousness in the first place. What are the expected benefits deriving from a conscious AI, and do they outweigh the possible risks?
Developing conscious AI could advance our understanding of biological consciousness, consistently with the idea that it is easier to understand what we ourselves build. It might also exponentially accelerate the entire field (LeCun, Reference LeCun2022), maximising its benefits for society. For instance, a conscious AI would be able to develop models of the world and to intentionally use them in order to anticipate the impact of its own functioning and of other agents’ actions. On this basis, on an optimistic reading, it could avoid negative consequences and promote positive ones, including a better alignment with human values (Khamassi, Nahon, & Chatila, Reference Khamassi, Nahon and Chatila2024).
Seth acknowledges these arguments in support of conscious AI, but he also (rightly, in our opinion) add that it should not be taken for granted that conscious AI is a holy grail.
We agree, not only for the reason that it may not actually solve the present technical shortcomings of AI, but more fundamentally because a conscious AI may eventually replicate and even exacerbate the shortfalls of human consciousness-cum-intelligence which is either explicitly or implicitly assumed as a reference model in the effort to build a conscious AI (Farisco et al., Reference Farisco, Evers and Changeux2024). The combination of intelligence and consciousness does not entail benevolence. To the contrary, the combination has in human beings, as a result of cultural as well as biological evolution, typically been coupled with aggression against “out-group” individuals, reserving sympathy and aid for those who belong to the group (Evers, Reference Evers, Metzinger and Windt2015). Moreover, humans create a very large number of group distinctions exhibiting profound diversities in terms of the values they endorse. Therefore, the question of how to ascertain that intelligent and conscious artificial systems (“they”) created by humans do not react with malevolence but with benevolence towards (some) humans (“us”) is important. Artificial systems are trained and programmed with and have access to vast amounts of human-generated data, where examples of, for example, xenophobic, misogynistic, and violent behaviour abound and universal benevolence shines by its absence. For this reason, the confidence that a conscious machine created by humans would be engaged in universal human well-being appears at best unfounded.
Why should ”they” (machines) care about ”us” (humans)? And if they were to engage, why should they feel benevolence instead of malevolence towards out-group individuals (that would be far more in line with the mind of their programmer, their human “creator”)? In view of how selectively and xenophobically conscious intelligence operates in humans, taking machine benevolence towards humans for granted appears dangerously naïve.
We may also reverse the perspective: hypothetical artificial consciousness, specifically sentience, introduces the issue of machine welfare. Seeing how humans treat other human and non-human animals, there is ample reason to doubt that machines would face a happy destiny if we, whether intentionally or inadvertently, were to construct machines capable of subjective experience. And, as in the case of, for example, vivisected animals, or enslaved and intellectually stunted humans, their suffering might long go unacknowledged and even undetected.
We think that the above suffices to provide grounds for scepticism concerning the desirability of artificial consciousness, in addition to the grounds for scepticism concerning its feasibility developed in detail by Seth in his paper.
There are also epistemic reasons for scepticism concerning the feasibility of artificial consciousness. Presently, consciousness is only known to exist in living things. That is a fact about our knowledge that does not logically exclude artificial consciousness. Conscious AI is assumed to be theoretically possible within certain theoretical frameworks (Evers et al., Reference Evers, Farisco, Chatila, Earp, Freire, Hamker, Nemeth, Verschure and Khamassi2025), but for now, no external empirical evidence or validation is available. In that situation, we can neither logically exclude nor affirm the possible existence of artificial consciousness in the real world. Empirical validation is essential to turn logical possibility into actuality.
Moreover, artificial consciousness would likely be qualitatively very different from human and animal consciousness due to deep differences in substrate, architecture, and functions (Evers, Reference Evers2024; Farisco, Evers, & Changeux, Reference Farisco, Evers and Changeux2024). Here, the epistemic and psychological challenges introduced by the gaming problem are central (Birch & Andrews, Reference Birch and Andrews2024). When artificial systems use human-generated training data to mimic human behaviours, they may, if successful, psychologically persuade human users of their sentience, as a form of anthropomorphism; yet logically, mimicking human behaviours in artificial systems has no evidential value whatsoever. This poses a formidable obstacle to detecting sentience or consciousness in artificial systems trained with human-generated data and deeply influences our risk–benefit assessments in this domain. To avoid the gaming problem, we should arguably focus less exclusively on similarities and pay more attention to differences, since AI is materially, functionally and architecturally very different from a human brain. But here a different problem arises: the question of commensurability. A challenge in detecting differences is that we cannot think entirely beyond our own perspective, since we are imprisoned by the limits imposed by our bodies, so if the differences are sufficiently deep, we cannot detect them. There must also be some similarities to justify the application of the same concept to such distinct phenomena.
Our conclusion is that engineering conscious AI is not desirable because the potential risks far outweigh the possible benefits. In view of how human nature has expressed itself throughout history, conscious AI created by humans would likely be monumentally dangerous. It could also be very different from human consciousness and thus (a) undetectable, (b) incomprehensible, and (c) incommunicable. This could increase the risks and cut both ways if the conscious AI has valenced experience.
You have
Access
In the section of his paper dedicated to What should we (not) do, Seth writes that “The development of real artificial consciousness for its own sake should not be an explicit goal.” This touches upon the fundamental question why to pursue artificial consciousness in the first place. What are the expected benefits deriving from a conscious AI, and do they outweigh the possible risks?
Developing conscious AI could advance our understanding of biological consciousness, consistently with the idea that it is easier to understand what we ourselves build. It might also exponentially accelerate the entire field (LeCun, Reference LeCun2022), maximising its benefits for society. For instance, a conscious AI would be able to develop models of the world and to intentionally use them in order to anticipate the impact of its own functioning and of other agents’ actions. On this basis, on an optimistic reading, it could avoid negative consequences and promote positive ones, including a better alignment with human values (Khamassi, Nahon, & Chatila, Reference Khamassi, Nahon and Chatila2024).
Seth acknowledges these arguments in support of conscious AI, but he also (rightly, in our opinion) add that it should not be taken for granted that conscious AI is a holy grail.
We agree, not only for the reason that it may not actually solve the present technical shortcomings of AI, but more fundamentally because a conscious AI may eventually replicate and even exacerbate the shortfalls of human consciousness-cum-intelligence which is either explicitly or implicitly assumed as a reference model in the effort to build a conscious AI (Farisco et al., Reference Farisco, Evers and Changeux2024). The combination of intelligence and consciousness does not entail benevolence. To the contrary, the combination has in human beings, as a result of cultural as well as biological evolution, typically been coupled with aggression against “out-group” individuals, reserving sympathy and aid for those who belong to the group (Evers, Reference Evers, Metzinger and Windt2015). Moreover, humans create a very large number of group distinctions exhibiting profound diversities in terms of the values they endorse. Therefore, the question of how to ascertain that intelligent and conscious artificial systems (“they”) created by humans do not react with malevolence but with benevolence towards (some) humans (“us”) is important. Artificial systems are trained and programmed with and have access to vast amounts of human-generated data, where examples of, for example, xenophobic, misogynistic, and violent behaviour abound and universal benevolence shines by its absence. For this reason, the confidence that a conscious machine created by humans would be engaged in universal human well-being appears at best unfounded.
Why should ”they” (machines) care about ”us” (humans)? And if they were to engage, why should they feel benevolence instead of malevolence towards out-group individuals (that would be far more in line with the mind of their programmer, their human “creator”)? In view of how selectively and xenophobically conscious intelligence operates in humans, taking machine benevolence towards humans for granted appears dangerously naïve.
We may also reverse the perspective: hypothetical artificial consciousness, specifically sentience, introduces the issue of machine welfare. Seeing how humans treat other human and non-human animals, there is ample reason to doubt that machines would face a happy destiny if we, whether intentionally or inadvertently, were to construct machines capable of subjective experience. And, as in the case of, for example, vivisected animals, or enslaved and intellectually stunted humans, their suffering might long go unacknowledged and even undetected.
We think that the above suffices to provide grounds for scepticism concerning the desirability of artificial consciousness, in addition to the grounds for scepticism concerning its feasibility developed in detail by Seth in his paper.
There are also epistemic reasons for scepticism concerning the feasibility of artificial consciousness. Presently, consciousness is only known to exist in living things. That is a fact about our knowledge that does not logically exclude artificial consciousness. Conscious AI is assumed to be theoretically possible within certain theoretical frameworks (Evers et al., Reference Evers, Farisco, Chatila, Earp, Freire, Hamker, Nemeth, Verschure and Khamassi2025), but for now, no external empirical evidence or validation is available. In that situation, we can neither logically exclude nor affirm the possible existence of artificial consciousness in the real world. Empirical validation is essential to turn logical possibility into actuality.
Moreover, artificial consciousness would likely be qualitatively very different from human and animal consciousness due to deep differences in substrate, architecture, and functions (Evers, Reference Evers2024; Farisco, Evers, & Changeux, Reference Farisco, Evers and Changeux2024). Here, the epistemic and psychological challenges introduced by the gaming problem are central (Birch & Andrews, Reference Birch and Andrews2024). When artificial systems use human-generated training data to mimic human behaviours, they may, if successful, psychologically persuade human users of their sentience, as a form of anthropomorphism; yet logically, mimicking human behaviours in artificial systems has no evidential value whatsoever. This poses a formidable obstacle to detecting sentience or consciousness in artificial systems trained with human-generated data and deeply influences our risk–benefit assessments in this domain. To avoid the gaming problem, we should arguably focus less exclusively on similarities and pay more attention to differences, since AI is materially, functionally and architecturally very different from a human brain. But here a different problem arises: the question of commensurability. A challenge in detecting differences is that we cannot think entirely beyond our own perspective, since we are imprisoned by the limits imposed by our bodies, so if the differences are sufficiently deep, we cannot detect them. There must also be some similarities to justify the application of the same concept to such distinct phenomena.
Our conclusion is that engineering conscious AI is not desirable because the potential risks far outweigh the possible benefits. In view of how human nature has expressed itself throughout history, conscious AI created by humans would likely be monumentally dangerous. It could also be very different from human consciousness and thus (a) undetectable, (b) incomprehensible, and (c) incommunicable. This could increase the risks and cut both ways if the conscious AI has valenced experience.
Financial support
This research has received funding from the project Counterfactual Assessment and Valuation for Awareness Architecture – CAVAA (European Commission, EIC 101071178).
Competing interests
None.