This chapter charts the author’s personal and professional journey from an evangelist for AI to an AI critic. It begins by exploring the binary visions of AI’s future: the utopian promise of solving global challenges (curing cancer, reversing global warming) against the existential danger of narrowly optimized superintelligence. The narrative details the author’s tenure at the Allen Institute for Artificial Intelligence, where initial excitement over a commitment to “AI for the common good” and ethical development turned to disillusionment as core ethical initiatives were deprioritized, licenses were changed, and the organization’s focus seemed to shift, and his time as a lecturer on AI, law, and ethics at the University of Texas at Austin. This experience propelled the author into a deep dive into the burgeoning legal and ethical challenges of generative AI (GenAI). The chapter concludes by arguing that GenAI is the decade’s most consequential technology, yet its rapid, legally ambiguous rise is largely outside public consciousness. It makes a forceful call for greater AI literacy and public involvement to ensure that society, rather than a handful of companies, shapes how this technology transforms our world.

This chapter offers a sobering assessment of artificial general intelligence (AGI), which remains frustratingly difficult to define. The debate is polarized between optimistic “effective accelerationists,” who foresee boundless abundance, and “doomers,” who warn of existential risk from misaligned systems. The chapter discusses my skepticism of imminent AGI and explores indicators suggesting that current capabilities are being significantly oversold. It concludes that the AGI narrative primarily serves to attract investment and deflect accountability. Therefore, law and policy should regulate AI based on its actual capabilities and current harms, rather than speculative future potential.

The widely accepted “AI Race” narrative, which pits the US against rivals like China, is a flawed metaphor that actively harms thoughtful AI development and policy. This chapter deconstructs the “race” by showing it lacks a clear finish line or coherent definition of “winning,” unlike historical contests such as the space race or the Manhattan Project. The narrative is based on three shaky assumptions: that innovation inherently equals progress, that generative AI is already capable of near-term, large-scale usefulness, and that the “winner” will control global AI values. Critically, these flawed premises trigger predictable, problematic policy reactions, including unlimited corporate and governmental spending, indefinite delays on necessary regulation, and hasty, thoughtless deployment. By prioritizing speed and raw capability over genuine utility, the “AI race” risks squandering resources on unreliable systems and undermining public trust. The chapter concludes that the real challenge lies in human coordination, not computation, and calls for abandoning the race framework in favor of a wise, goal-oriented approach that serves human needs over corporate interests.

Whereas today’s “narrow” AI machines can be easily controlled, the probable advent over the coming decades of machines endowed with Artificial General Intelligence (AGI) will pose a far more difficult control challenge – for such machines will function as agents endowed with many human-like capabilities. The AI expert Stuart Russell has proposed a radical new design for the motivational architecture of AI machines, which he calls “humble AI.” Rather than programming AI machines to pursue specific goals, he argues, it would be much safer to give them a single broad goal, namely, maximizing the realization of human preferences. The key innovation here lies in programming the machines so that they can never be 100 percent certain about what those human preferences are – a gambit that restores ultimate control over the machines’ behavior to their human overseers. Russell argues persuasively that fundamental safety research in the field of AI needs to be pursued as a much higher priority than it currently receives today.