Introduction to Machine Learning is designed as a textbook for senior undergraduate machine learning courses and aims to provide a general overview of various machine learning techniques that could be applied in statistics, data mining, artificial intelligence, pattern recognition, and other related areas. It targets a wide audience and provides insights into popular machine learning techniques. Students with an appropriate background on math and statistics, covering the concepts of random variables, probability distributions, etc., can easily follow the book. One unique feature of the book is that it provides an informative summary at the beginning of each chapter on what that chapter covers. Furthermore, important terms and concepts are displayed on the side margins to highlight these to the reader. This is not an unknown technique in textbooks but one that I have found especially useful when searching for a previously seen keyword to refresh my recollection.
This book has nineteen chapters and includes a preface and an appendix on probability. Chapter 1 introduces the concept of machine learning and provides examples of machine learning applications. It starts with examples of supervised learning methods, such as classification and regression, and then continues to discuss unsupervised learning methods, focusing on density estimation and clustering, two of the major unsupervised learning problems. The chapter then introduces reinforcement learning and ends with pointing to some important resources in machine learning: journals and conferences, dataset repositories, and publicly available benchmark test collections. Thoughout the chapter, we see discussions of applications of various machine learning methods in a wide variety of areas such as bioinformatics, natural language processing, robotics, etc.
Chapter 2 covers supervised learning techniques: It introduces learning a binary classification model and then continues with fundamental and important topics such as Vapnik Chervonenkis dimension and Probably Approximately Correct (PAC) Learning. PAC learning bounds the error a hypothesis is allowed to make and hence gives the designer control over the effectiveness of the algorithm. The chapter further introduces other important and general concepts, such as learning with noise, overfitting, inductive bias, etc., that appear often in applications.
Chapter 3 provides an overview of probability theory covering concepts such as likelihood, prior and posterior probabilities, expected loss minimization and expected utility maximization, etc. Although this chapter could help unprepared readers to refresh their probability background, it would probably have been better if the information here was spread out throughout the book as needed, perhaps with some of it moving to an appendix. For instance, Chapter 14 could have been a good place to incorporate the discussions on Bayes rule, Bayes classifier, and posterior estimation.
After introducing making decisions under uncertainty using probabilities, Chapter 4 discusses how to capture such probabilities with model parameters and learn them from training data, focusing on parametric methods for classification and regression. For classification, this chapter covers likelihood estimation, bias-variance trade-off. The chapter then covers Bayesian estimation from both classification and regression perspectives, pointing to approximation techniques, such as Laplace approximation, and then discusses Gaussian processes, another classification approach that estimates the distribution of class labels for each input example when the weight vector is drawn from a Gaussian distribution. This chapter also goes into the evaluation of models based on the bias and variance of its parameters, since unbiased models with consistency are always desirable. The chapter does a good job of bringing to bear the connection between bias-variance trade-off and overfitting, and ends with a discussion on model selection procedures with a focus on regularization, minimum description length, and structural risk minimization. Chapter 5 takes the concepts of Chapter 4 up in the multivariate coxtext, where mean and variance in the univariate case are now represented as the mean vector and covariance matrix.
Chapter 6 discusses dimensionality reduction in the context of real-world problems with a large set of features. It presents techniques such as principal component analysis, factor analysis, multidimensional scaling, and linear discriminant analysis while doing a good job of motivating the need for dimensionality reduction, It explains each reduction technique in detail so that a reader can apply them in real-life problems.
Chapter 7 discusses clustering, a well-studied machine learning problem that aims to learn the data distribution to represent the data as mixtures, covering popular clustering techniques such as k-means clustering, and the expectation maximization algorithm widely used not only in clustering but also in solving many problems involving latent variables. The chapter also briefly touches on how the data can be mapped into a new space defined by clustering and how supervised learning such as regression could be applied in this new space, pointing out the similarities and differences of clustering and dimensionality reduction methods.
Chapter 8 introduces the nonparametric perspective and covers nonparametric density estimation, classification, and regression. It also links nonparametric regression to smoothing, as nonparametric methods prefer that noise in the data be smoothed out to have better generalization performance. The visualizations prove particularly useful to better understand the concepts.
Chapter 9 discusses decision trees, one of the earliest nonparametric machine learning methods, covering classification and regression problems and explains pruning and its importance. It also covers rule learning and multivariate trees. This chapter should be taken as a brief introduction to decision trees, which can be followed by some of the material pointed out by the author, e.g. a survey by Rokach and Maimon (2005).
Chapter 10 discusses linear discrimination with copious help from visual aids for the geometry of linear discriminant. It focuses on logistic regression, one of the most widely used linear classification algorithms. Logistic regression, like many other discrimination techniques, relies on gradient descent, an iterative optimization procedure that updates the estimate based on the gradient and the step size, and is mostly applied in situations when no analytical solution is available. It would have been better if this chapter had provided more on gradient descent-based techniques such as Newton's method, conjugate gradients, simulated annealing, etc.
Chapter 11 covers artificial neural networks, starting with a discussion on how the brain works and then introduces single-layer and multi-layer perceptrons. It discusses how one chooses the structure of the network and tune its size, especially the number of hidden layer units, and ends with a complexity analysis of training neural networks. Chapter 12 continues with neural networks, focusing on learning locally specific models. It covers normalized and competitive basis functions as the main building blocks for local models, and concludes with a discussion on the connection between neural networks, radial basis function networks, and mixture of experts. This chapter, in my opinion, provides the right insights in order for a reader to appreciate the similarities between different methods in machine learning and the underlying ideas behind them.
Chapter 13 discusses kernel machines, one of the most important and influential techniques in machine learning. Kernel machines rely on the concept of maximum margin. When two hypotheses lead to the same training error, the maximum margin principle favors the one that places the closest instances from different classes the furthest apart. Thus, maximum margin methods care only about the instances close to the boundary and ignore the rest. Such points are known as support vectors and define the decision boundary. This chapter gives a very informative overview of these concepts and then elaborates, linking kernel machines to regression problems.
Chapter 15 discusses Hidden Markov Models (HMMs), one of the oldest, well-known, and widely used techniques in machine learning. HMMs are widely used speech and language processing problems, among others. This chapter presents the main problems of training and employing HMMs: learning HMM parameters from a training set, and decoding, the process of finding the most likely state sequence for a given sequence of observations. The presentation is detailed and extends into a brief discussion on the learning with continuous variables. The next chapter, Chapter 16, extends HMMs to the graphical models, first describing different types of associations and interaction between different variables of a model, and how one represents these relations with a graphical model. It then provides examples of graphical models, and continues with a discussion of belief propagation, how one estimates parameters over observations and propagates these estimates and observations throughout the graph. The chapter also covers different types of tree-structured graphical models and undirected models.
Chapters 17 and 18 are devoted to two additional topics in machine learning: ensemble methods and reinforcement learning. Ensemble methods are not stand-alone techniques; they are instead combined with other machine learning methods as base learners. The idea of an ensemble is to create a diverse committee of learners which will complement each other and hence generalize better with higher accuracy. Obviously, the question of how to combine the base learners is a central one. This chapter covers different combination schemes, including error correcting output codes, bagging, boosting, stacked learners, etc., and does a good job of giving a clear overview of the major ensemble techniques. On the other hand, the chapter on reinforcement learning, one of the earlier techniques in machine learning, originally developed for control systems, robot navigation, etc., could have been put under a separate final chapter called “Additional Topics,” perhaps together with other more specialized topics such as the Bayesian Decision Theory.
Chapter 19 combines various topics in a unified setting and gives guidelines on how to conduct more reliable experiments in practice, either as a machine learning researcher or practitioner. Most of the topics such as hypothesis testing, cross-validation, confidence interval estimation, etc. could be found in other books, but will most likely not come under the same section or chapter. Most people who use off-the-shelf machine learning algorithms may have problems when designing the experiments, analyzing the results, and drawing conclusions. This chapter, without doubt, is very helpful for a beginner in machine learning, to understand the important points to consider when conducting such experiments. These skills come with experience in most cases, but for students this chapter provides a very relevant background.
In summary, as an introductory book, Introduction to Machine Learning generally maintains a good balance between the breadth and depth. It provides exposure to various machine learning topics, some of which can be found in a more detailed form in other books. However, the book could also have covered more recent topics such as active learning and semi-supervised learning. These are quite important topics in current machine learning research and applications which have not been adequately covered in earlier textbooks.
