This chapter addresses the challenge of dealing with large MDP models by approximating the value function or state-action value function with linear and nonlinear lower-dimensional basis functions (also known as features). It shows how to use least squares methods to fit the basis functions to the value function and evaluate a fixed policy. An alternative approach using linear programming is also presented. Least squares value iteration, least squares policy iteration, least squares modified policy iteration, and linear programming methods are developed to identify good policies based on the approximations. Applications in advance appointment scheduling and golf are provided at the end of the chapter.