Generalised additive models (GAMs) offer a parsimonious, flexible and interpretable framework for regression, particularly when handling a large number of candidate predictors. This thesis addresses the GAM variable selection problem: categorising each candidate predictor’s effect on the mean response as linear, nonlinear or zero. We use Bayesian model selection paradigms and group least absolute shrinkage and selection operator (LASSO) priors. Two types of priors are explored for the sparse fits. The first, Laplace-zero and grouped LASSO-zero priors, is applied to Gaussian and binary responses. The second, (grouped) horseshoe priors, is used for Gaussian, Poisson and negative-binomial responses. For both prior types, tailored auxiliary variable representations enable practical implementation of Markov chain Monte Carlo (MCMC) sampling. Specifically, the sampling procedure reduces to Gibbs sampling for Gaussian and binary responses, and to slice sampling for count responses. To improve computational scalability, particularly for large datasets, we also derive the mean field variational Bayes (MFVB) algorithms under the Laplace-zero and grouped LASSO-zero priors. These MCMC and MFVB methods for Gaussian and binary GAM selection are implemented in the publicly available R package gamselBayes. Although less accurate than MCMC, this variational approach offers substantial gains in speed. The GAM selection framework is further extended to generalised additive mixed models (GAMMs) with random intercepts for Gaussian and binary responses using both MCMC and MFVB. Finally, the properties of the grouped horseshoe distribution and its use in Bayesian GAM selection are investigated. While many characteristics of the univariate horseshoe distribution are carried over, some distinctions arise in the grouped case.

Some of these results have appeared in [Reference He and Wand1, Reference He and Wand2].