2. Artificial Intelligence as Goal-Optimization

To see why evaluating human intelligence by the normative standards we use to evaluate AI may be detrimental, we need to begin by getting a handle on what I shall mean by AI for the present discussion. As a starting point, it is worth noting that AI researchers and engineers themselves do not have any settled theoretical account of intelligence. The label “artificial intelligence” covers a heterogeneous family of techniques and systems (Boden, Reference Boden2016; Legg & Hutter, Reference Legg and Hutter2007). It would therefore be misleading to attribute a single conception of intelligence to the field. There is, however, an influential operational tradition in which systems are designed and evaluated through successful performance on specified tasks. That is, at an operational level, intelligence is approached pragmatically: it is as it does.

Historically, this tradition has clear roots in Turing’s pathbreaking decision to replace the question “Can machines think?” with an imitation game organized around observable performance (Turing, Reference Turing1950). It is also visible in Legg and Hutter’s definition of intelligence in terms of an agent’s ability to achieve goals across environments (Legg & Hutter, Reference Legg and Hutter2007). A similar orientation appears in the rational-agent framework developed by Russell and Norvig, and in reinforcement learning, where a system is trained to maximize expected cumulative reward (Russell & Norvig, Reference Russell and Norvig2021; Sutton & Barto, Reference Sutton and Barto2018). Stuart Russell, a distinguished leader in the field, calls the resulting approach the “standard model” of AI, which he takes to be widespread across the range of commercial AI being developed today (Russell, Reference Russell2022).Footnote ⁶ In practice, engineers specify goals, reward signals, loss functions, benchmarks, or feedback criteria and build systems that perform increasingly well relative to them. A reinforcement learning system maximizes expected cumulative reward (Sutton & Barto, Reference Sutton and Barto2018). Large language models are pretrained on statistical prediction tasks (Brown et al., Reference Brown, Mann and Ryder2020); some are subsequently fine-tuned using demonstrations and human preferences to improve instruction-following (Ouyang et al., Reference Ouyang, Wu and Jiang2022). The systems differ enormously, but they share an operational family resemblance: their success is assessed relative to standards supplied through their design and use.

In sum, while at the theoretical level intelligence remains contested, at the operational level the field has largely treated intelligence as successful performance relative to externally supplied goals. However thin, this model has driven many of the most spectacular developments in contemporary AI, from reinforcement learning to large language models capable of self-supervised, statistically driven learning.

The danger, as I shall argue in the next section, is that the success and potency of these models in the context of their increasing ubiquity puts further pressure on societies to value human intelligence as though it were essentially no different from its machine-style counterpart. That is, the worry is that the success and ubiquity of AI will accelerate and broaden the existing technocentric tendency to evaluate human intelligence predominantly or exclusively in terms of measurable outputs. In what follows, my aim is to explore both how AI may serve to accelerate this trend and why that would be undesirable.

3. From Engineering Abstraction to Institutional Norm

The attractions of an output-relative model of intelligence are clear. As an engineering abstraction, it replaces difficult metaphysical questions with observable performance, thereby facilitating comparison, benchmarking, and the measurement of improvement. Problems begin when three distinct moves are collapsed. The first is methodological. Certain machines are designed and evaluated by their success relative to specified goals. The second is conceptual. In the context of the growing ubiquity and cultural dominance of AIs, successful performance relative to externally supplied goals—or optimization—is treated as a sufficient account of intelligence as such. The third move is institutional. Intelligent human practices are further reorganized on the assumption that their relevant value consists in producing the same kinds of outputs delivered by AIs. The first move may be entirely legitimate while the second and third remain ethically troubling, or so I shall argue.

Russell makes the second move—from engineering abstraction to human anthropology—unusually explicit:

This passage does not establish that this narrowly instrumental characterization of human intelligence is culturally or institutionally dominant. It does, however, show a highly influential theorist generalizing an operational model from the design of AI to the interpretation of human intelligence.Footnote ⁷ The sort of technocentrism I am interested in names precisely this: the evaluative tendency involved when such a model is tacitly treated as sufficient for understanding and judging human cognitive activities. When treated in this way, human cognitive activities such as perceiving, thinking, learning, and inventing are valued and regarded as intelligent to the extent that they contribute to measurable outputs.

This conception of human intelligence is not peculiar to Russell. Newell and Simon’s influential information-processing account of human problem-solving models such activity as heuristic search through a problem space, in which operators transform states in search of a specified goal state (Newell & Simon, Reference Newell and Simon1972). More recently, Tegmark defines intelligence as the ability to accomplish complex goals and applies this performance-relative concept to biological and artificial systems alike (Tegmark, Reference Tegmark2017). These accounts differ in important respects, but they share a goal-relative structure: cognitive performance is characterized in terms of successfully reaching or accomplishing an objective. What such models do not represent is the formative value that the activity through which the objective is reached may have for the person undertaking it.

Of course, the output-oriented valuation of human intelligence did not originate with AI. On the contrary, this tendency has been an increasingly marked feature of modern societies for at least several centuries. Standardized testing, scientific management, bureaucratic administration, market valuation, and productivity metrics have long encouraged modern institutions to evaluate human activity in instrumental terms and, often, to translate such activity into measurable results.Footnote ⁸

Famously, the Frankfurt School made the social consequences of this tendency an object of critique. In Eclipse of Reason, Horkheimer contrasts subjective reason, concerned with selecting adequate means to purposes taken largely for granted, with objective reason, under which purposes and forms of life themselves remain open to rational assessment. When subjective reason becomes dominant, “rational” comes to mean useful, adaptive or efficient relative to antecedently assumed purposes. Thus, persons and institutions come to be judged by how well they function within an existing order (Horkheimer, Reference Horkheimer1947). Horkheimer and Adorno’s Dialectic of Enlightenment radicalizes this diagnosis. A rationality organized around calculation, abstraction and control reduces heterogeneous things to commensurable units and treats nature and human beings as mere objects to be controlled and administered (Horkheimer & Adorno, Reference Horkheimer and Adorno1947 [2002]).

These diagnoses show that concern with the instrumentalizing tendency central to technocentrism significantly predates AI. Its increasing power and ubiquity, however, may be expected to give this older tendency powerful new reach. When machines can reproduce the outputs of an increasing number of cognitive practices, institutions acquire both new means and new incentives to identify those practices with their outputs alone. The next question is how this normative-conceptual pressure may be converted into an institutional norm.

I suggest this may happen through a four-step process. First, much contemporary AI development relies on benchmarking. That is, the use of standardized tasks, datasets, metrics and testing protocols to compare systems by assigning them scores on selected forms of performance. Benchmarks are treated as indispensable engineering devices insofar as they make training progress and system comparison tractable. But they are also selective operationalizations. They must specify in advance which outputs will stand for the capacity being measured. Second, when an institution considers procuring or deploying an AI system to perform a human activity, it faces corresponding pressure to interpret or “see” the activity in question as a set of observable outputs tied to corresponding success criteria. From this perspective, writing can appear as text production, reasoning as answer production, teaching as information transmission, judgment as decision-output, and so on. These redescriptions need not be imposed by any explicit theory; they can arise from the practical requirement that an activity be made measurable before a system can be trained, purchased or compared. Third, once the activity is described principally through its output, machine substitution acquires a powerful new institutional rationale. If the relevant good is the text, answer, decision or prediction, then a faster, cheaper or more scalable route to it appears, other things equal, preferable. Fourth, machine substitution of the human activity may serve to reinforce the original redescription. As institutions remove occasions for human performance, the formative goods realized through those activities become less visible, and further machine substitution may correspondingly appear more reasonable and desirable. In this way, I suggest, AI threatens to help turn an engineering abstraction into a self-confirming, self-perpetuating institutional norm that entrenches technocentric attitudes toward human cognitive activity.

Of course, AI can be designed and used to complement human capacities.Footnote ⁹ It can reduce drudgery, widen access to expertise and advance knowledge in ways that were previously impossible. The above analysis does not deny any of that. Nor does every human activity at risk of AI automation possess significant formative value worth protecting. My concern is more specific. Where institutions are already disposed to value human activity by outputs, and where AI makes a greater range of those outputs reproducible without human performance, the operational standard can become the evaluative standard for human cognitive activity by default. To determine when, where, and why that may be detrimental, we need to dig deeper into the formative value of human intelligent activity itself.

4. Learning and the Value of Process

In this section, I deepen the claim that the institutional tendency described above is normatively objectionable insofar as it neglects goods realized through or constitutive of the human performance of cognitive activity. I use Kant’s writings on enlightenment and education to help specify one such good: the formation of character through the exercise of rational capacities. More specifically, I draw on Kant to support three claims. First, that rational capacities become one’s own through their use; second, that the practices through which they are used or exercised involve public answerability; and third, that repeated participation in these practices can establish relatively settled dispositions of thought, belief, judgment, perception and conduct—in other words, character. Considered together, these claims explain how the development of cognitive capacities helps to shape the person who exercises them.

Most people would agree that some human cognitive activities possess value that is not contained in or exhausted by their outputs. Learning provides the clearest case. By formative value, I mean the value an activity has insofar as performing it develops or sustains capacities, dispositions and traits we value in the person who performs it. No one denies, of course, that learning has instrumental value. It gives the learner information, skills, qualifications and opportunities. It can produce knowledge and discoveries of broad or narrow social benefit. But we also value learning because of what it makes of the learner. It can deepen understanding, discipline attention and strengthen the capacity to identify and correct error. It can make the learner more patient with difficulty, more humble before evidence and more ready to revise her judgment in conversation with others. Perhaps most importantly, it can enlarge the learner’s ability to see from perspectives other than her own and induct her into a shared culture of inquiry, in which claims are supported by reasons that can be both challenged and shared. These are not merely discrete competencies, but instead mutually reinforcing elements of character.Footnote ¹⁰

Consider two students who arrive at the same correct answer while only one can explain why it is correct, respond to an objection or recognize the limits of the argument that supports it. The difference is absent from the output but present in the learner. Similarly, a student who works through a philosophical argument learns to follow a line of thought, distinguish premises from conclusions, detect ambiguity and make her judgment answerable to reasons that others can share and assess. Likewise, a writer who labors over a sentence may discover what she means and what she can responsibly say. Something similar is true of the researcher. She meets resistance in her material and thereby learns, through practice, what her methods can and cannot establish. In each case, the activity forms capacities and dispositions that are not contained in the resulting output, be it answer, text or discovery. The process is not an indifferent route to a separably valuable product. It is partly constitutive of the intellectual or moral good at stake in the person who undertakes it.

The formative importance of cognitive activity is, of course, widely recognized in the discourse and theory of education and has been for centuries.Footnote ¹¹ Recent critical work on technology and AI is often animated by this recognition, too. Work on technological deskilling examines how technologies may erode capacities and skills required for practical wisdom and virtuous character, while also allowing that they can create possibilities for reskilling or upskilling (Vallor, Reference Vallor2015). The distinction between automation and augmentation likewise directs attention to whether AI replaces human performance or complements and extends it (Brynjolfsson, Reference Brynjolfsson2022). A recent debate on generative AI in education examines whether and how delegating the writing of humanities essays may deprive students of formative activities through which their rational capacities are cultivated (Aylsworth & Castro, Reference Aylsworth and Castro2024, Reference Aylsworth and Castro2026; Gorrieri, Reference Gorrieri2025; Sticker, Reference Sticker2026). Related work applies the problem of deskilling to capacities required for public reason and sustained through practices of democratic deliberation (Ferdman, Reference Ferdman2025).

What needs further explanation, I suggest, is how exactly the exercise of cognitive capacities contributes to character. While the philosophy of education has much to say on this issue, Kant can also help connect the dots in a perspicuous way. On his account, one learns to judge by exercising one’s judgment, but such exercise can only meaningfully take place in the context of practices where one’s reasons can be questioned by others. To participate repeatedly in these practices is, on my reading, to learn how to attend to a difficulty, ask for and offer reasons, acknowledge error and persist in pursuing difficult questions. The same activity both develops rational capacities and shapes the broader dispositions with which they are exercised.

Kant’s discussion of immaturity in his essay “An Answer to the Question: What Is Enlightenment?” sheds further light on this. In that piece, he defines immaturity as the inability to use one’s own understanding without the guidance of another and takes “Have courage to make use of your own understanding!” as the motto of enlightenment (WA 8:35). Kant’s examples—a book that thinks for me, a spiritual adviser who serves as my conscience and a doctor who determines my diet—are meant to distinguish uncritically accepting others’ judgments from acquiring the capacity to judge for oneself. The problem Kant is concerned with, however, is not reliance on others as such. He would agree that human beings learn through testimony, instruction, criticism, and example.Footnote ¹² The problem arises when the uncritical acceptance of another’s judgment replaces the exercise of one’s own in a way that leaves one’s rational powers unused and underdeveloped.

Kant’s appeal to courage is especially illuminating. Making use of one’s understanding involves attempting judgments whose correctness cannot be guaranteed in advance. It involves exposing oneself to error and to the possibility of correction by others. It involves being open to revising one’s beliefs. Courage thus concerns the willingness to undertake this risky enterprise, to tolerate the vulnerability it implies, and to assume responsibility for the results rather than rely passively on others’ assurances. In this way, an uncritically accepted answer supplied from elsewhere spares the learner the difficulty of exercising courage and reaching an answer herself. Possessing the answer and acquiring the capacity to judge are, in this respect, very different achievements.

This capacity for independent judgment cannot, however, develop in isolation. Kant’s account helps us see why. His defense of the public use of reason places the exercise of one’s understanding in a setting of shared inquiry and criticism (WA 8:36–37). Thinking for oneself requires neither dispensing with others’ views nor treating one’s private judgment as final.Footnote ¹³ Instead, it involves making one’s judgments answerable to others by offering reasons, considering objections and revising when warranted. Independence of judgment is thus acquired within a collective, critical discipline, with shared concepts, inherited questions and common standards of evidence. I use “public answerability” as a blanket term for this collective discipline, through which the exercise of cognitive capacities gives rise to socially learned dispositions. Learning to judge, on this account, includes acquiring the courage to expose one’s judgments to criticism and the responsiveness to objections required by shared practices of reason-giving.

The further claim that Kant’s view helps us appreciate is that sustained participation in such practices forms character. Kant understands character as a settled readiness to orient one’s thought and conduct by shareable maxims arising from one’s own judgment or understanding (LP 9:480–81).Footnote ¹⁴ His Lectures on Pedagogy emphasize that the formation of character is a temporal process. Human beings do not arrive with their rational powers fully formed; they require education, in which one generation participates in the formation of the next (LP 9:441–45). The value of education, on this account, cannot be reduced to information transmitted, skills acquired or intellectual competence displayed on particular tasks. Instead, it is expressed in relatively stable, reason-responsive dispositions that guide thought, perception, attention and conduct. Character formation is not, of course, guaranteed by the competent performance of cognitive tasks, but nor can it be separated from the processes and activities through which these competences are acquired.

The institutional norms discussed in the previous section become especially worrying in light of the social-temporal dimension of character-formation I have been describing. Since each generation participates in educating the next, opportunities to acquire and exercise one’s own judgment depend on settled practices that outlast particular learners. Institutions can provide settings in which their members learn to make use of their own understanding (together with others) or organize those settings in ways that foreclose that possibility, entrenching intellectual dependence—or immaturity, to use Kant’s word. The formative contribution of cognitive activities and the institutions in which they occur cannot therefore be assessed in reference to results alone. It depends on whether and how they sustain the practices through which new participants become capable of independent judgment and answerability, the hallmarks of character on the view I am sketching.

Applied to the problem of AI substitution, my Kant-inspired account helps us articulate what meaningful normative evaluation of AI-use should include. Where human performance of cognitive tasks contributes to character formation, a superior machine output should not by itself be taken to establish that substitution is normatively preferable. We need to know what exactly is formative in the activity, who continues to undertake the work in question, whether participants can answer for the results, and how the whole arrangement affects the relevant human capacities over time. These concerns point to four diagnostic questions, which I develop in the section below.

5. Augmentation, Substitution, and the Social Danger of Technocentrism

In Section 3, I described the technocentric tendency to identify cognitive activities with their outputs alone. In Section 4, I supplied reasons to reject that identification where the activity in question contributes to the formation of human character. The normative upshot for present purposes is limited. The fact that an AI system produces an equivalent or superior result does not settle whether replacing human performance is desirable, since it leaves the activity’s contribution to the formation of its participants normatively unassessed. How, in light of the above considerations, should people and institutions make that assessment? How, concretely, should they distinguish desirable augmentation from potentially objectionable substitution?

The following four diagnostic questions may be helpful. First, what, if anything, is formative in the human performance of the activity in question? Which capacities and dispositions does it exercise or help develop, and how central are they to the good of the practice and to character-formation more generally? These questions draw attention to important values that a comparison of outputs alone leaves out, without assuming that every component of the activity in question has the same formative importance, if any at all. These questions do not therefore suggest a general normative presumption against automation. They leave open the possibility that some cognitive tasks are tedious, dangerous, exhausting or of little formative importance, and that certain forms of automation may free human time and attention for more valuable formative activity.

The second question I propose follows from the dependence of capacity-formation on repeated exercise in the context of answerability. How, exactly, does the AI system distribute or reconfigure the relevant cognitive work? Does it prompt users to interpret, compare, articulate reasons, test claims, and revise their judgments? Or does it simply supply a result that leaves the user with the task of selection or approval? Once again, these questions leave open the possibility that AI assistance can be extensive while yet preserving the human work that gives the activity in question its formative value. Substitution becomes potentially objectionable when it bypasses or erodes that work.

The third diagnostic question I propose concerns the public answerability through which judgment is exercised. To what extent is the AI user in a position to answer for the result obtained? Can she identify its grounds and detect relevant error? Can she respond to criticism? A person may be formally charged with approving an output while lacking the understanding required to assess it fully and assume meaningful responsibility for it. In such cases, retaining a human at the final stage of an automated process may preserve responsibility in a nominal sense while yet gutting its genuine basis. One important consideration is whether and to what extent the user can exercise independent judgment about the result obtained and revise it when challenged.

The fourth question I propose concerns the conditions under which capacities are acquired, and human character is shaped over time. What are the likely effects of repeated, institutionalized AI use on those who participate in the affected practices? Does the arrangement in question preserve opportunities for novices to learn to judge, as well as opportunities for experienced participants to continue doing so? Does it sustain and support shared practices of criticism and reason-giving? These questions follow from the tight connection Kant draws between repeatedly making use of one’s own understanding and character formation. A pattern of AI use, I suggest, should be assessed in relation to the development of character it supports, thwarts or erodes over time.

These questions do not provide an algorithm for distinguishing desirable from undesirable uses of AI. Instead, they help identify what needs to be established before an appeal to output equivalence or superiority should be made to support AI substitution. Meaningful answers to these questions will depend on a host of context-dependent variables. Answers must be informed by the nature of the activities in question, the design of the AI systems in play, as well as the nature, experience, and vulnerability of their users. Answers must draw from empirical evidence about how such systems are used and the impacts of their use.

The fourth question about the development of human character over time may be particularly difficult to answer because the goods at issue are social and diachronic. Select acts of delegation may have no appreciable effect on a person’s capacities now or in the near term, particularly when those capacities are already well developed and exercised elsewhere. Institution-wide patterns of delegation, however, pose greater risks. Education, research, professional judgment, writing and public deliberation are among the practices through which participants acquire habits of judgment and answerability. If these practices are reorganized around the routine procurement and approval of machine outputs, institutions may continue to depend on human rational capacities and character traits they no longer reliably cultivate or even actively undermine.

Academic writing provides an obvious case in point. A generative AI tool may augment a student’s reasoning by presenting objections to a claim she has formulated. But it may also supply the argument before she has learned to do that for herself. If the latter use becomes normalized across educational institutions, polished texts and arguments may continue to be produced even as students receive fewer occasions to acquire the capacities the production of those texts and arguments would ordinarily foster. This worry is not limited to education, of course. Similar dynamics can also arise in professional settings. Decision-support may sharpen a practitioner’s judgment by bringing new evidence and alternatives into view, and yet a workflow organized around accepting AI-generated recommendations may gradually reduce the opportunities through which human judgment is fostered and formed.

Similar risks are explored in recent work on “capacity-hostile environments” created by AI mediation and on AI-related threats to epistemic agency (Coeckelbergh, Reference Coeckelbergh2023; Ferdman, Reference Ferdman2026). The Kantian account I have developed here deepens our appreciation of these risks by tying them to the problem of character formation in the context of public answerability. The practices through which people exercise cognitive capacities are the very practices through which they learn how to attend, perceive, judge, and respond to others in stable, reason-sensitive ways. Their reorganization may therefore affect who participants have opportunities to become, even before any damage to specific skills or outputs is apparent.

A Kantian humanist approach to AI therefore asks what forms of human participation its design and institutional deployment support, preserve, bypass or displace. The relevant unit of evaluation is often not the isolated user-tool interaction, but the institutions that distribute cognitive work, authority and opportunities for human character development. We must ask who is still enabled to exercise judgment, who is learning to do so, who can answer for and challenge the results produced, and which shared practices of criticism and reason-giving remain in place over time. My central normative claim is that successful machine performance cannot by itself establish that the corresponding human activity has become dispensable. Whether substitution is normatively desirable depends also on what is formative in that activity, what cognitive work remains in human hands, whether those involved retain meaningful answerability for its results, and how the resulting practices shape the capacities and character of those who participate in them. Technocentrism obscures precisely these questions when it treats the continuing availability of satisfactory or superior outputs as sufficient evidence that nothing of importance has been lost.