Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Acknowledgements
- Chapter One Introduction
- Part I Foundations
- Chapter Two Markov Decision Process Fundamentals
- Chapter Three Examples and Applications
- Part II Classical Markov Decision Process Models
- Chapter Four Finite Horizon Models
- Chapter Five Infinite Horizon Models: Expected Discounted Reward
- Chapter Six Infinite Horizon Models: Expected Total Reward
- Chapter Seven Infinite Horizon Models: Long-Run Average Reward
- Chapter Eight Partially Observable Markov Decision Processes
- Part III Reinforcement Learning
- Chapter Nine Value Function Approximation
- Chapter Ten Simulation in Tabular Models
- Chapter Eleven Simulation with Function Approximation
- Appendices
- Bibliography
- Index
Chapter Four - Finite Horizon Models
Published online by Cambridge University Press: aN Invalid Date NaN
Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Acknowledgements
- Chapter One Introduction
- Part I Foundations
- Chapter Two Markov Decision Process Fundamentals
- Chapter Three Examples and Applications
- Part II Classical Markov Decision Process Models
- Chapter Four Finite Horizon Models
- Chapter Five Infinite Horizon Models: Expected Discounted Reward
- Chapter Six Infinite Horizon Models: Expected Total Reward
- Chapter Seven Infinite Horizon Models: Long-Run Average Reward
- Chapter Eight Partially Observable Markov Decision Processes
- Part III Reinforcement Learning
- Chapter Nine Value Function Approximation
- Chapter Ten Simulation in Tabular Models
- Chapter Eleven Simulation with Function Approximation
- Appendices
- Bibliography
- Index
Summary
This chapter formulates a finite horizon model, in which the MDP ends at a fixed future stage that is known at the start of the planning horizon. It introduces the concepts of the value of a policy (policy value function), an optimal policy, the optimal value function, the state-action value function, the principle of optimality, the optimality (Bellman) equation, and backward induction. These concepts are formally defined and illustrated through numerous examples. Interpretability and optimality of structured policies are also discussed. The intuition developed in this chapter is critical to understanding infinite horizon MDPs.
Keywords
Information
- Type
- Chapter
- Information
- Markov Decision Processes and Reinforcement Learning , pp. 103 - 149Publisher: Cambridge University PressPrint publication year: 2026
Accessibility standard: Unknown
Why this information is here
This section outlines the accessibility features of this content - including support for screen readers, full keyboard navigation and high-contrast display options. This may not be relevant for you.Accessibility Information
Accessibility compliance for the PDF of this chapter is currently unknown
and may be updated in the future.