This chapter combines the methods from the previous two chapters, namely, function approximation and simulation. It introduces policy-centric methods that are based on approximating the action-choice probability distribution of a randomized policy. Value function approximation using Monte Carlo and temporal differencing methods is described. Optimization algorithms classified as value-based algorithms (Q-learning and Q-policy iteration) and policy-based algorithms (policy gradient and actor-critic) are presented. The chapter concludes with brief remarks on additional topics, including simultaneous learning and control, experience replay, deep learning, importance sampling, Monte Carlo tree search, and partially observable models.

This chapter describes simulation-based methods to evaluate value functions and state-action value functions, focusing on problems that are sufficiently small so that policies can be represented in tabular form, that is, as a look-up table. Three broad classes of techniques are considered: Monte Carlo methods, temporal differencing, and Q-learning. Numerous variants of these methods are presented. Important trade-offs are considered for each method, including offline vs. online implementation, synchronous vs. asynchronous updating, exploration vs. exploitation, on-policy vs. off-policy recursions, and variance vs. bias. Policy evaluation and optimization using simulation-based methods are presented for episodic models, discounted models, and average reward models. Policy iteration type algorithms are introduced. The chapter concludes with a case study of the queuing service rate control application, comparing simulation-based solutions with those obtained using classical methods from Chapters 5–7.