This chapter critically examines the prevalent practice of measuring generative AI (GenAI) capabilities through performance benchmarks, which major AI companies frequently cite to assert the superiority of their models. It begins by defining benchmarks as standardized tests used to compare models and confer “state-of-the-art” (SOTA) status, despite the tenuous link between high scores and true general ability. The discussion then details the three primary evaluation methods: automated benchmarking, humans as judges, and models as judges.

The chapter then pivots to the numerous problems that undermine the utility of these benchmarks. These limitations include data contamination (training models on the test data), overfitting, inherent dataset and measurement biases, the narrow focus on incrementalism over fundamental research, and the misleading nature of aggregated leaderboards. Ultimately, the chapter argues that benchmarks suffer from a lack of construct validity, testing only what is easily measurable rather than the messy, ill-defined problems of the real world, leading to an over-acceptance of GenAI in critical life domains.