Deep neural networks have become a cornerstone of modern machine learning, demonstrating remarkable performance across a wide range of applications. Despite this empirical success, the theoretical understanding of deep neural networks, particularly in the context of generalisation, remains relatively underdeveloped. The rectified linear unit (ReLU) is one of the most widely used activation functions in deep learning, valued for its simplicity and effectiveness in training deep networks. In this thesis, we focus on advancing the theoretical analysis of deep ReLU networks from two perspectives: pairwise learning tasks and gradient descent methods.
For pairwise learning tasks, we are concerned with the generalisation performance of nonparametric estimation. Most of the existing work requires the hypothesis space to be convex or a VC class, and the loss to be convex. However, these restrictive assumptions limit the applicability of the results in studying many popular methods, especially kernel methods and neural networks. We significantly relax these restrictive assumptions and establish a sharp oracle inequality of the empirical minimiser with a general hypothesis space for the Lipschitz continuous pairwise losses. As an application, we apply our general results to study pairwise least squares regression and derive an excess generalisation bound that matches the minimax lower bound for pointwise least squares regression up to a logarithmic term. The key novelty here is to construct a structured deep ReLU neural network as an approximation of the true predictor and design the target hypothesis space consisting of the structured networks with controllable complexity. This successful application demonstrates that the general results obtained indeed help us to explore the generalisation performance on a variety of problems that cannot be handled by existing approaches.
We study the generalisation performance of metric and similarity learning by leveraging the specific structure of the true metric (the target function). Specifically, by deriving the explicit form of the true metric for metric and similarity learning with the hinge loss, we construct a structured deep ReLU neural network as an approximation of the true metric, whose approximation ability relies on the network complexity. Here, the network complexity corresponds to the depth, the number of nonzero weights and the computation units of the network. Considering the hypothesis space which consists of the structured deep ReLU networks, we develop the excess generalisation error bounds for a metric and similarity learning problem by estimating the approximation error and the estimation error carefully. An optimal excess risk rate is derived by choosing the proper capacity of the constructed hypothesis space. To the best of our knowledge, this is the first ever known generalisation analysis providing the excess generalisation error for metric and similarity learning. In addition, we investigate the properties of the true metric of metric and similarity learning with general losses.
For gradient descent methods, we focus on studying the statistical generalisation performance of gradient descent (GD) and stochastic gradient descent (SGD) for overparameterised neural networks within the neural tangent kernel regime. Most of the existing work on regression problems is limited to shallow network architectures, leaving a notable gap in the theory for deep neural networks. We address this gap by presenting a comprehensive generalisation analysis for deep ReLU networks trained using GD and SGD. Specifically, we establish the first known minimax-optimal rates of excess population risk for both GD and SGD with deep ReLU networks, under the assumption that the network width scales polynomially with respect to the network depths and training sample size. Our results demonstrate that with sufficient widths, gradient descent methods for deep ReLU networks can achieve the optimal rates of generalisation on a par with kernel methods.
Some of these research results have appeared in [Reference Zhou, Huang, Feng, Wang and Zhou1].
