Choosing the outcome distribution

GLMMs offer a general framework for modeling binary, continuous, and count outcomes.Reference Stroup¹¹ This flexibility comes from the ability to choose different probability distributions depending on the outcome, for example, infection counts or individual patient outcomes. Available distributions for GLMMs include the binomial distribution for binary data, the normal and gamma distributions for continuous data, and the Poisson distribution for count data. Examples of each distribution for common infection control outcomes are provided in Table 2.

Different outcome distributions will determine how changes due to the intervention and other factors are modeled. For example, data collected on number of healthcare-associated infections before and after a hand hygiene intervention can be modeled using a Poisson distribution where the intervention is associated with changes in infection rates.Reference Barnett, Page and Campbell¹² For studies involving continuous outcomes, such as patient length of stay, choosing a normal distribution will model changes in average length of stay.Reference Graves, Weinhold and Tong¹³

Our cleaning data were modeled using a binomial GLMM in which the outcome was the number of cleaned FTPs per the total number audited (Figure 3). Subsequent modeling, therefore, focused on the impact of the cleaning bundle on the probability of cleaned FTPs.

Fig. 3.

GLMM Specification for Case Study

Defining model effects

GLMMs use different types of effects for describing relationships between study variables and the outcome of interest, and for accounting for correlation among related observations. These effects are specified in the same way as a regression equation (Eq. 1):

(1) $$g\left( {\{ expected{\rm{ }}outcome\} } \right) = \left\{ {fixed{\rm{ }}effect\left( s \right)} \right\} + \left\{ {random{\rm{ }}effect\left( s \right)} \right\} + \left\{ {offset} \right\}$$

In a fixed effects model, the impact of the intervention and other study variables (eg, season, ward versus ICU) on the expected outcome is modeled. These models can also be used to account for outcome trends unrelated to the intervention. This is common in ‘interrupted time series’ designs in which time is included as a fixed effect, for example, in antimicrobial use before and after introducing additional antibiotic stewardship measures.Reference Taggart, Leung, Muller, Matukas and Daneman¹⁴

GLMMs differ from regression models using random effects to account for how data are collected under a study design. The role of random effects in multisite studies is 2-fold. First, for the designs in Figure 2, including a site-specific random effect effectively groups observations belonging to the same site. In doing so, we account for expected correlation between these observations (eg, see Fig. 1). Second, random effects account for between-site differences that are unrelated to intervention effectiveness, for example, baseline differences in cleaning thoroughness prior to the REACH bundle. By removing these differences, the estimated intervention effect represents the expected within-site change in the outcome of interest. This is important if modeling results are to be generalized, that is, if study sites are representative of the wider population. Examples of random effects for explaining other sources of variation are provided in the Supplementary Material online.

For count data, differences in site population sizes should be accounted for by specifying a model offset. For example, when modeling infection counts, differences in hospital size will inevitably create differences in infection counts between hospitals. Including hospital bed days as an offset will ensure that rates are modeled on a common scale (eg, infection rate per 10,000 bed days).Reference Mitchell, Hall and White^6, Reference Barnett, Page and Campbell¹²

We tested 3 different intervention effects for describing the impact of the REACH bundle on cleaning thoroughness (Figure 3). Models that included temporal trends used audit times in weeks since the start of each hospital’s intervention phase. Changes over time were scaled by a factor of 10 to transform the trend to a more practically meaningful change (per 10 weeks).

Link function

Linear models assume that study variables share linear relationships with the outcome of interest and that residuals are normally distributed. These assumptions are inappropriate for outcomes modeled by nonnormal distributions. In some cases outcomes can be transformed to satisfy these assumptions; however, modeling results are difficult to interpret.

GLMMs resolve this issue by specifying a link function that connects the expected outcome with model effects. The form of the link function, g(·) in Equation (1), depends on the chosen outcome distribution. For infection counts modeled using the Poisson distribution, the log link function is used; fixed effects therefore summarize changes in each variable on the log relative risk of infection. For our binomial GLMM, using a logit link function means that effect of the bundle on cleaned FTPs is modeled on the log odds scale.

Software

GLMMs are available in many statistical software packages, including Stata (meglm), SAS (proc glimmix), and SPSS (GENLINMIXED). In R, the lme4 package (version 1.1-18-1)Reference Bates, Mächler, Bolker and Walker¹⁵ implements several likelihood-based methods for GLMM estimation and is compatible with other R packages for model assessment. Data and R code used in this article are available at https://github.com/nicolemwhite/ICHE_analysis.

Bayesian GLMMs are available in a number of software packages, including WinBUGS,Reference Sturtz, Ligges and Gelman¹⁶ JAGS,Reference Depaoli, Clifton and Cobb¹⁷ Greta,Reference Golding¹⁸ and R packages including MCMCglmmReference Hadfield¹⁹ and INLA.Reference Martins, Simpson, Lindgren and Rue²⁰ Bayesian approaches differ from traditional likelihood methods by modeling uncertainty in both the data and unknown parameters, the latter requiring additional specification of prior distributions for each parameter in the model. In our experience, likelihood and Bayesian approaches for GLMMs yield similar results, except in cases where prior information may aide model estimation (eg, small sample sizes). When using Bayesian approaches, we recommend trying different prior distributions as part of model assessment because different choices may influence estimated model effects.Reference Lunn, Jackson, Best, Spiegelhalter and Thomas²¹ A review of Bayesian approaches to GLMMs is available elsewhere.Reference Fong, Rue and Wakefield²²

Model selection

Information criteria are recommended as a parsimonious approach for choosing between GLMM specifications. Unlike model selection procedures based on hypothesis testing, information criteria combine goodness of fit with a complexity term to penalize models with more parameters. Commonly used information criteria include Akaike information criterion (AIC) and the Bayesian information criterion (BIC).Reference Lavergne, Martinez and Trottier²³ Lower values indicate improved model fit. When comparing models, criterion differences >10 indicate strong support the better fitting model.Reference Burnham and Anderson^24, Reference Raftery²⁵

Information criteria can also be used to combine inferences from multiple models using model averaging.Reference Hoeting, Madigan, Raftery and Volinsky^26, Reference Wagenmakers and Farrell²⁷ This method is particularly useful when there is uncertainty about a key feature of the model, for example, the timing of the intervention effect.Reference Kürüm, Warren and Schuck-Paim²⁸

We used AIC to compare the 3 models fitted in our case study. The results indicated that a step change in cleaning performance after introducing the bundle coupled with a linear trend over the intervention phase was the best fit to the data (AIC = 3,039 for model 1; 2,923 for model 2; 2,658 for model 3).

Model assessment

The selected model should be assessed for discrepancies between the observed data and model-based predictions because these may indicate problems with the data and/or model fit. Discrepancies can be visualized by plotting observed outcomes against fitted values; however, this is not always useful depending on the type of outcome being modeled (eg, binary data versus estimated probabilities).

Simulation-based model checking is a useful technique for model assessment; it compares summaries of the observed data with the same summaries produced by replicate data that are simulated from the model. By repeating this simulation many times, uncertainty in predicted outcomes can also be assessed.

Residual analysis should be conducted to check for unusual observations and correlation that is unexplained by the model. For data collected over time, unexplained temporal correlation can be examined visually or formally via the autocorrelation function applied to model residuals for each site (eg, the ‘acf’ function in R). If observations with large residuals are identified, these should be checked as their inclusion may influence model estimates.

In multisite studies, model results may be strongly influenced by individual sites, to the extent that excluding a site from the dataset affects study findings. The contribution of each site should therefore be examined to determine model sensitivity to changes in the data and, if so, implications for study conclusions should be reported. Diagnostics, including the Cook distance and the DFBETA statistic, have been adapted for GLMMs.Reference Ouwens, Tan and Berger^29, Reference Pinho, Nobre and Singer³⁰ These methods apply a leave-one-out approach that refits the model excluding data from 1 site each time then compares changes in parameter estimates. Selected diagnostics are available in the ‘influence.me’ package in R.Reference Nieuwenhuis, Grotenhuis and Pelzer³¹

The model assessment for our selected model (model 3) is summarized in Figure 4. Simulation results showed a high level of overall agreement between observed versus predicted cleaning performance (Fig. 4a). Residual plots over time (Fig. 4b) did not indicate unexplained temporal correlation; however, residual plots by site (Fig. 4c) identified large negative residuals for 2 sites (sites 6 and 9). Follow-up analysis revealed unexpected poor performance recorded for 1 audit in the control phase (site 6) and 1 audit in the late intervention phase (site 9). Refitting the model without these observations did not impact parameter estimates, and therefore, these data were retained in the dataset.

Fig. 4.

Assessment of model 3 (binary change + linear intervention trend): (a) comparison of observed cleaning performance and model-based simulations by audit (black = observed sample means; grey = outlying simulations); (b) residuals versus time since the start of the control period; (c) boxplots of residuals by site.

Using the Cook distance, we identified 2 sites (sites 3 and 9) as potentially influencing model results (Table 3). These 2 sites had the largest impact on the estimated binary effect, reflected in the percentage changes using DFBETA. However, the absolute effect of excluding each of these sites was small, with parameter estimates varying between 0.61 and 0.78 for the binary change and between 0.27 and 0.32 for the linear trend.

Table 3.

Influence Diagnostics for Assessing the Contribution of Each Hospital to the Estimated Intervention Effects

Note. SE, standard error.

^a Sites are ordered by decreasing Cook distance. Estimated intervention effects following the exclusion of each site are also provided, to show the impact on model results.