Complex bioconvection patterns are observed when a suspension of the oxytactic bacterium Bacillus subtilis is placed in a chamber with its upper surface open to the atmosphere. The patterns form because the bacteria are denser than water and swim upwards (up an oxygen gradient) on average. This results in an unstable density distribution and an overturning instability. The pattern formation is dependent on depth and experiments in a tilted chamber have shown that as the depth increases the first patterns formed are hexagons in which the fluid flows down in the centre.

The linear stability of this system was analysed by Hillesdon & Pedley (1996) who found that the system is unstable if the Rayleigh number Γ exceeds a critical value, which depends on the wavenumber k of the disturbance as well as on the values of other parameters. Hillesdon & Pedley found that the critical wavenumber kc could be either zero or non-zero, depending on the parameter values.

In this paper we carry out a weakly nonlinear analysis to determine the relative stability of hexagon and roll patterns formed at the onset of bioconvection. The analysis is different in the two cases kc≠0 and kc=0. For the kc≠0 case (which appears to be more relevant experimentally) the model does predict down hexagons, but only for a certain range of parameter values. Hence the analysis allows us to refine previous parameter estimates. For the kc=0 case we carry out a two-dimensional analysis and derive an equation describing the evolution of the horizontal planform function.