Introduction
Worldwide, a major proportion of the burden of disease among young people is attributable to the use and misuse of alcohol and other drugs (AOD; Erskine et al. Reference Erskine, Moffitt, Copeland, Costello, Ferrari, Patton and Scott2015; Peacock et al. Reference Peacock, Leung, Larney, Colledge, Hickman, Rehm and Degenhardt2018). Although the minimum legal age to purchase alcohol in Australia is 18 years, representative surveys indicate that one in two students aged between 12 and 17 years will have consumed alcohol in the past year and approximately 44% will have consumed alcohol at risky levels in their lifetime (Guerin & White, Reference Guerin and White2018) Cannabis use among this age group is also concerning, with 15% reporting having used cannabis in the past year (Guerin & White, Reference Guerin and White2018). AOD use has been consistently linked with an array of physical, psychological, social and vocational harms (Hall et al., Reference Hall, Patton, Stockings, Weier, Lynskey, Morley and Degenhardt2016; Peacock et al., Reference Peacock, Leung, Larney, Colledge, Hickman, Rehm and Degenhardt2018). Furthermore, earlier initiation of AOD is associated with an increased likelihood of developing mental health and substance use disorders in adulthood (Behrendt, Wittchen, Hofler, Lieb, & Beesdo, Reference Behrendt, Wittchen, Hofler, Lieb and Beesdo2009; Marel et al., Reference Marel, Sunderland, Mills, Slade, Teesson and Chapman2019). It is therefore critical that effective prevention occurs prior to the onset and acceleration of substance use.
Schools are an ideal location to deliver prevention programs given the relatively low cost and ease at which large universal audiences can be reached (Botvin & Griffin, Reference Botvin and Griffin2016; Cossu et al., Reference Cossu, Cantone, Pintus, Cadoni, Pisano, Otten and Carta2015; Spoth, Randall, Trudeau, Shin, & Redmond, Reference Spoth, Randall, Trudeau, Shin and Redmond2008). While many school-based AOD education prevention programs are available, many fail to demonstrate efficacy in reducing or delaying the uptake of AOD use (Das, Salam, Arshad, Finkelstein, & Bhutta, Reference Das, Salam, Arshad, Finkelstein and Bhutta2016; Foxcroft & Tsertsvadze, Reference Foxcroft and Tsertsvadze2011; Lee, Cameron, Battams, & Roche, Reference Lee, Cameron, Battams and Roche2016). Furthermore, due to poor implementation fidelity, programs that prove efficacious in carefully controlled trials are often ineffective when taken to scale (Bumbarger & Perkins, Reference Bumbarger and Perkins2008). Issues with fidelity include adaptation by teachers, which can result in the intervention losing its effective components (Bumbarger & Perkins, Reference Bumbarger and Perkins2008; Dusenbury, Brannigan, Falco, & Hansen, Reference Dusenbury, Brannigan, Falco and Hansen2003; Giles et al., Reference Giles, Jackson-Newsom, Pankratz, Hansen, Ringwalt and Dusenbury2008) and competing school resources or academic priorities (Parcel & Basen-Engquist, Reference Parcel, Basen-Engquist, MFD, JW, HS, TJ and JN2000).
The Climate Schools drug prevention model was designed to overcome common barriers to implementation and meet set outcomes of the Australian Health and Physical Education curriculum (Newton, Vogl, Teesson, & Andrews, Reference Newton, Vogl, Teesson and Andrews2009, Reference Newton, Vogl, Teesson and Andrews2011). The programs consist of a series of cartoon storylines that impart information via a social influence and harm-minimisation approach to prevention (Botvin & Griffin, Reference Botvin and Griffin2016). The Climate Schools Program was designed to maximise treatment fidelity via a number of mechanisms (Marsch, Bickel, & Badger, Reference Marsch, Bickel and Badger2007). Active intervention content is delivered to students through standardised online modules. These modules are ‘locked’ so that students are required to complete them sequentially and are unable to ‘skip ahead’. This effectively eliminates the possibility of program adaptation and increases intervention adherence. Whilst online content is complemented by classroom activities and lesson summaries, they require minimal teacher training and are designed to complement the online module, but are not required for the program to be successfully delivered (Newton et al., Reference Newton, Vogl, Teesson and Andrews2011).
To date, the effectiveness of the Climate Schools: Alcohol and Cannabis course has been assessed by two cluster randomised controlled trials involving 36 Australian high schools (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015; Newton et al., Reference Newton, Vogl, Teesson and Andrews2009). These trials found the course to be more effective than AOD education as usual in increasing alcohol and cannabis knowledge, reducing the uptake of alcohol use, reducing risky drinking, and reducing the frequency of cannabis use up to 24 months post-baseline (Newton et al., Reference Newton, Teesson, Mather, Champion, Barrett, Stapinski and Slade2018; Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017). While these results are promising, the trials were conducted in one Australian state (New South Wales). Therefore, replication over a greater geographic range is an important next step.
Replication strengthens the empirical foundation of an intervention and is a key criterion for effectiveness in prevention research (Flay et al., Reference Flay, Biglan, Boruch, Castro, Gottfredson, Kellam and Ji2005; Valentine et al., Reference Valentine, Biglan, Boruch, Castro, Collins, Flay and Schinke2011). Recent attention to the value of replication amidst the ‘replication crisis’ (Shrout & Rodgers, Reference Shrout and Rodgers2018) has led to the development of Standards of Evidence, which postulate that prior to broad dissemination, positive findings should be replicated, ideally more than twice (Flay et al., Reference Flay, Biglan, Boruch, Castro, Gottfredson, Kellam and Ji2005; Gottfredson et al., Reference Gottfredson, Cook, Gardner, Gorman-Smith, Howe, Sandler and Zafft2015). Replication in prevention research serves two main purposes: to rule out the possibility that past findings were a result of chance and to determine the generalisability of intervention effects (Flay et al., Reference Flay, Biglan, Boruch, Castro, Gottfredson, Kellam and Ji2005). Moreover, the effectiveness of an intervention can decline over time due to the rapidly evolving social and technological environment (Mohr et al., Reference Mohr, Schueller, Riley, Brown, Cuijpers, Duan and Cheung2015). For example, several countries including Australia have been reporting delays in uptake and/or increases in abstinence among young people (Pape, Rossow, & Brunborg, Reference Pape, Rossow and Brunborg2018). Evaluation of effectiveness and applicability of existing programs amid this changing landscape is essential. Despite the clear need for replication in prevention science, few replications have been conducted, particularly when assessing web-based AOD prevention programs (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015).
The aim of the current study is to conduct a national effectiveness trial to replicate the research of the Climate Schools: Alcohol and Cannabis course among a new and geographically diverse cohort of Australian students. The regions in the trial (Western Australia, New South Wales and Queensland) represent three of the largest and most diverse states in Australia covering populated metropolitan and coastal areas on the east and west coast of the continent (3934 km apart), as well as sparsely populated regional areas. To our knowledge, this study represents the largest replication study to date of a web-based AOD prevention program, including new populations in which the intervention is untested. It is hypothesised that students who receive the Climate Schools: Alcohol and Cannabis course, compared to the active control, will have significantly greater alcohol and cannabis knowledge and significantly less use and misuse of alcohol and cannabis over the 12-month follow-up period.
Method
Participants and design
Data were derived from a larger cluster randomised controlled trial known as the Climate Schools Combined (CSC) Study conducted between 2014 and 2016 (Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017, Reference Teesson, Newton, Slade, Chapman, Birrell, Mewton and Andrews2020). The sample consisted of 6386 students (mean age at baseline = 13.5 years, s.d. = 0.6; 54.8% female) from 71 schools (including both state-funded public and independently-funded private schools) across three Australian states (New South Wales, Western Australia and Queensland). In 2014, Year 8 (Year 9 in Queensland) students in these schools were invited to participate in the study and were included if they consented themselves and received parental consent.
Randomisation and masking
Using block randomisation through the Stata ralloc command, schools were randomly allocated to receive either the Climate Schools: Alcohol and Cannabis course (n = 34) or health education as usual (Control, n = 37) during Year 8, in equal ratios. Students were invited to complete self-report questionnaires in class at baseline, immediately after the intervention (6-months post-baseline) and at 12-months post-baseline. The CONSORT diagram (online Appendix, Fig. A1) summarises participant flow and retention rates for each condition throughout the study.
Trial registration and ethical approvals
This trial was registered with the Australian and New Zealand Clinical Trials Registry (ACTRN 12613000723785). Ethical approval was granted by the University of New South Wales, Curtin University, Queensland University of Technology, the Western Australian and Brisbane Catholic Education Offices, and the New South Wales, Queensland and Western Australian Department of Education and Training.
Interventions
Climate schools
Thirty-four schools delivered the Climate Schools: Alcohol and Cannabis course to their students during Health and Physical Education classes in Year 8. The course consists of 12 × 40-min lessons, comprising a 20-min online cartoon component completed independently by students and 20-min teacher-driven classroom activity. The first six lessons focus on alcohol and were delivered in Term 1, 2014. The remaining six lessons focused on alcohol and cannabis and were delivered approximately 6 months later.
Active control schools
Thirty-seven schools administered health education as usual, including lessons on AOD. In Australia, drug and alcohol education is a mandatory part of the secondary school Health and Physical Education curriculum. Teachers provided a logbook detailing the number and format of these lessons. The number of lessons varied between schools (ranging from five to 20) with the average length of each lesson being 52 min. The majority of teachers (83.5%) reported using a combination of face-to-face and computer-based methods to teach AOD education topics.
Measures
Where appropriate, measures used were consistent with previous Climate Schools trials (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015; Newton, Teesson, Vogl, & Andrews, Reference Newton, Teesson, Vogl and Andrews2010; Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017). Assessments were completed either via paper-and-pencil or online; at the 12-month follow-up, 66% of schools used online surveys, 30% used paper and 4% used both methods. Data were linked over time by a unique identification code to ensure confidentiality.
Demographics
Sociodemographic information was collected to determine the equivalence of the groups. The information included gender, age, country of birth, truancy rates and academic performance.
Alcohol and cannabis knowledge
Alcohol knowledge was assessed by an adapted version of the 16-item ‘Knowledge of Alcohol Index’ (McBride, Farringdon, Muleners, & Midford, Reference McBride, Farringdon, Muleners and Midford2006) and cannabis knowledge was assessed by the 16-item ‘Knowledge about Cannabis Scale’ (Newton et al., Reference Newton, Teesson, Vogl and Andrews2010). These scales asked students to respond either ‘true’ ‘false’ or ‘unsure’ to a series of statements such as ‘A can of regular strength beer contains approximately 1.5 standard drinks’ and ‘Cannabis use improves people's memory and learning.’ The number of correct items for each scale was added to give total scores between 0 and 16 for each participant.
Alcohol use
Alcohol use was measured by a series of questions used in previous Climate Schools trials (Newton et al., Reference Newton, Teesson, Vogl and Andrews2010; Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017). Recent alcohol use was measured by asking participants ‘Have you had a standard alcoholic drink in the past 6 months?’ (responding No/Yes), and heavy episodic drinking measured by asking participants ‘How often did you have 5 or more standard alcoholic drinks on one occasion in the past 6 months?’ (Never/Less than monthly/Once a month/2–3 times a month/ Weekly/Daily or almost daily). Heavy episodic drinking responses were converted to a dichotomous outcome (any heavy episodic drinking in the past 6 months No/Yes).
Alcohol-related harm
The 23-item ‘Rutgers Alcohol Problem Index’ (RAPI) assessed harms related due to alcohol use over the past 6 months. Students were asked to report how often they had experienced a range of harms in the last year while drinking or due to drinking such as ‘neglected my responsibilities’ on a five-point scale ranging from ‘Never’ to ‘More than six times.’ The RAPI has good internal consistency (α = 0.91) and test-retest reliability (r = 0.92) (Miller et al., Reference Miller, Neal, Roberts, Baer, Cressler, Metrik and Marlatt2002; White & Labouvie, Reference White and Labouvie1989). Low prevalence of alcohol use, and therefore alcohol-related harms, in the current sample, lead to a large proportion of zero scores. Therefore, harms were analysed as a binary variable reflecting whether participants reported at least one harm vs. none in the past year. Some participants reported harms despite not reporting any recent drinking, RAPI scores were set to zero for these participants.
Cannabis use
Cannabis use was assessed by two items adapted from the 2010 National Drug Strategy Household Survey (NDSHS; AIHW, 2011). Participants were asked if they had ever tried cannabis (Yes/No) and whether they had tried cannabis in the last 6 months (Yes/No).
Intervention fidelity
All teachers in intervention schools were asked to complete a fidelity logbook detailing the number of lessons and activities they ran with their students. Teachers were also asked to report any adaptations made to classroom activities. To increase adherence and ensure fidelity, students were required to view all online modules in full and were not able to access the next lesson until the previous lesson had been completed.
Sample size calculations
The sample size calculations for the trial aimed to achieve 80% power to detect an effect size [standardised mean difference (SMD)] of 0.15 between groups, based on reviews of existing substance use prevention programs [e.g. see Onrust, Otten, Lammers, and Smit (Reference Onrust, Otten, Lammers and Smit2016)]. Using a sample size method designed for detecting intervention by time interactions in longitudinal cluster-randomised designs (Heo & Leon, Reference Heo and Leon2009), the required sample size to detect between-group differences was seven schools per group, with 100 participants in each school.
Statistical analyses
Analyses were conducted using Stata version 15.1 (StataCorp, 2017). We used multilevel mixed-effects generalised linear models with participant- and school-level random effects to assess Group × Time interactions for each outcome (see online Appendix A2 for details). Substance use knowledge was modelled with a Gaussian distribution and identity link and the binary outcomes of any drinking, heavy episodic drinking, any alcohol-related harms, and cannabis use were modelled with a binomial distribution with a logit link. In line with an Intention To Treat (ITT) framework, we included all available observations for each participant in the groups to which they were originally allocated.
The primary analyses did not adjust for covariates, however, similar to previous analyses (Teesson et al., Reference Teesson, Newton, Slade, Chapman, Birrell, Mewton and Andrews2020), we also conducted sensitivity analyses that adjusted for baseline covariates of sex, school type (public/private), truancy (none/any), drinking (at least one full standard drink), depression (probable diagnosis based on PHQ score), and anxiety (probable diagnosis based on GAD-7 score). These covariates were either considered to be potential confounders or found to be associated with missingness when attrition analyses were conducted. The drinking covariate was not included when modelling alcohol use outcomes. After accounting for these predictors of missingness and the observed outcome values, we assumed that the data were missing at random (MAR). Given that mixed-effects models yield unbiased estimates under the MAR assumption (Hox, Reference Hox2010), comparison of unadjusted and adjusted models allowed us to draw more robust conclusions.
Results
The final sample included 6386 participants from 71 schools across Australia. Online Fig. A1 in the appendix summarises the flow of participants through the trial. Baseline characteristics of the sample are reported in Table 1. A greater proportion of participants in the Climate group were attending public schools (69%) compared to the Control group (36%), and more participants in the Climate group were male (54%) compared to the Control group (36%). A total of 34 teachers from 14 schools (78%) returned their completed fidelity logbooks. Completion rates for each of the 12 lessons ranged from 97% to 98% for module 1 (the first six-lessons) and from 94% to 97% for the module 2 (the remaining six lessons). Examination of teacher logbooks indicated little or minimal adaptation of activities (e.g. discussing cartoons as a class or role-playing the cartoon storylines).
Baseline characteristics of the sample
The majority of the sample (95.2%) completed at least one follow-up survey. Rates of attrition were compared across levels of baseline covariates with χ2 tests. Attrition did not differ significantly by trial group (p = 0.18), school type (p = 0.065), sex (p = 0.23), country of birth (p = 0.12) or possible diagnosis of anxiety (p = 0.055). Attrition was significantly higher among those reporting any truancy (7.9% v. 4.4%, p < 0.001), drinking a full standard drink (9.4% v. 4.4%, p < 0.001) or with a possible diagnosis of depression (6.3% v. 4.2%, p = 0.0055).
Outcomes at each survey occasion are reported in Table 2. Full details of the multilevel mixed models fit for each outcome are found in online Appendix A3. As shown in Figs 1, 2, effects were similar in both the unadjusted models and in the adjusted models that included the baseline covariates of sex, school type, truancy, drinking, depression and anxiety, and the same patterns of significance were seen for the intervention effects.
Estimated change in the Climate group compared to Control by 12 months for knowledge outcomes. Error bars represent 95% confidence intervals. The unadjusted effect was estimated in a model including only Group and Time terms, the adjusted effect was estimated in a model including the baseline covariates of sex, school type (public/private), truancy, drinking, depression, and anxiety.
Estimated change in odds in the Climate group compared to Control by 12 months for substance use outcomes. Error bars represent 95% confidence intervals. The unadjusted effect was estimated in a model including only Group and Time terms, the adjusted effect was estimated in a model including the baseline covariates of sex, school type (public/private), truancy, drinking, depression, and anxiety.
Outcomes at each survey occasion
Values are mean (s.d.) or N (%).
a Cannabis use data was only collected in some schools due to ethical constraints.
Alcohol and cannabis knowledge
Compared to the Control group, the Climate group reported significantly higher alcohol knowledge at baseline: mean difference (MD) 0.52, 95% confidence interval (CI) (0.01–1.02), SMD 0.18, 95% CI (0.00, 0.36), p = 0.044. Adjusting for baseline, the Climate group showed greater increases in knowledge at the 6- [MD 1.82, 95% CI (1.66–1.99), SMD 0.65, 95% CI (0.59–0.71), p < 0.001] and 12-months [MD 1.43, 95% CI (1.25–1.61), SMD 0.51, 95% CI (0.44–0.57), p < 0.001] follow-ups compared to the Control group.
Cannabis knowledge scores were higher in the Climate group compared to the control group at baseline, MD 0.73, 95% CI (0.12–1.34), SMD 0.18, 95% CI (0.03–0.33), p = 0.019. The Climate group showed greater increases in knowledge scores than the Control group at both the 6- [MD 2.61, 95% CI (2.40–2.82), SMD 0.65, 95% CI (0.60–0.71), p < 0.001] and 12-month [MD 1.97, 95% CI (1.73–2.20), SMD 0.49, 95% CI (0.43–0.55), p < 0.001] follow-ups.
Alcohol use
The odds of drinking (full standard drink in the past 6 months) did not differ significantly between Climate and Control at baseline, odds ratio (OR) 1.93, 95% CI (0.90–4.11), p = 0.089. Assessing the change in odds from baseline, the relative increase in odds of drinking was significantly lower in the Climate group compared to the Control group at the 6- [OR 0.79, 95% CI (0.65–0.95), p = 0.014] and 12-month [OR 0.62, 95% CI (0.43–0.91), p = 0.014] follow-ups.
The odds of heavy episodic drinking were significantly higher in the Climate group at baseline, OR 3.23, 95% CI (1.17–8.95), p = 0.024, compared to the Control group. The odds of drinking increased less in the Climate group compared to the Control group at 6- [OR 0.70, 95% CI (0.51–0.95), p = 0.022] and 12-month [OR 0.49, 95% CI (0.26–0.90), p = 0.022] follow-ups.
Alcohol-related harms
Students in the Climate group were significantly more likely to report at least one alcohol-related harm at baseline, OR 2.40, 95% CI (1.10–5.20), p = 0.027 compared to the Control group. There was no strong evidence that the odds of experiencing harms increased at different rates in the Climate group compared to the Control group, by the 6- [OR 0.86, 95% CI (0.69–1.07), p = 0.17] or 12-month [OR 0.73, 95% CI (0.47–1.14), p = 0.17] follow-ups.
Cannabis use
The odds of cannabis use did not differ significantly between Climate and Control groups at baseline, OR 1.94, 95% CI (0.54–6.97), p = 0.31. The change in odds of cannabis use was not significantly different in the Climate group compared to Control by the 6- [OR 0.83, 95% CI (0.32–2.12), p = 0.69] or 12-month [OR 0.57, 95% CI (0.23–1.41), p = 0.22] follow-up.
Discussion
The aim of this study was to replicate research on the Climate Schools: Alcohol and Cannabis course to determine the effectiveness and generalisability of the program among a large and geographically diverse cohort of Australian students. The results provide support for the effectiveness and feasibility of the course with respect to knowledge outcomes and alcohol uptake, but not with respect to cannabis use or alcohol harms.
Consistent with previous trials (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015; Newton, Teesson, Barrett, Slade, & Conrod, Reference Newton, Teesson, Barrett, Slade and Conrod2012; Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017); students who received the Climate Schools intervention demonstrated greater increases in alcohol- and cannabis-related knowledge than those who received education as usual. These increases were evident at both the 6- and 12-month follow-up. While knowledge alone may not reduce AOD uptake, the National Drug Strategy recognises knowledge is key to reducing and minimising the risk of harms related to use (Department of Health, 2017). When considering behaviour change, the program demonstrated effectiveness in reducing the uptake of drinking and heavy episodic drinking at both 6- and 12-month follow-ups. These are consistent with previous trials of the Climate Schools: Alcohol and Cannabis course which found prevention effects on alcohol use and heavy episodic drinking up to 24 months post intervention (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015; Newton et al., Reference Newton, Teesson, Vogl and Andrews2010; Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017). When considered as effect sizes the alcohol use prevention effects at 12 months are 0.26 (uptake of alcohol use) and 0.39 (heavy episodic drinking). Whilst these effect sizes are small to medium they are comparable with and in some cases larger, than other universal AOD prevention program effects (Champion, Newton, Barrett, & Teesson, Reference Champion, Newton, Barrett and Teesson2013; Foxcroft & Tsertsvadze, Reference Foxcroft and Tsertsvadze2012). These are noteworthy results as research suggests that delaying the uptake of alcohol reduces the likelihood of developing mental health and substance use disorders in later life (Behrendt et al., Reference Behrendt, Wittchen, Hofler, Lieb and Beesdo2009; Marel et al., Reference Marel, Sunderland, Mills, Slade, Teesson and Chapman2019).
There was no strong evidence of a change in odds of experiencing alcohol-related harms. As alcohol use becomes more normative and harms increase, these effects may appear, similar to previous studies we have conducted, where differences in alcohol-related harms became apparent at the 36-month follow-up (Slade et al., Reference Slade, Newton, Mather, Barrett, Champion, Stapinski and Teessonunder review). This may be particularly important given evidence in Australia and elsewhere of increasing age of alcohol use initiation and the decreasing prevalence of associated harms among more recent cohorts (Kraus et al., Reference Kraus, Seitz, Piontek, Molinaro, Siciliano, Guttormsson and Hibell2018; Livingston et al., Reference Livingston, Raninen, Slade, Swift, Lloyd and Dietze2016). Nonetheless, refinement of the existing Climate Schools program, or an additional targeted prevention component, may be necessary for students who are at a greater risk of experiencing AOD-related harms in the short term. We have recently evaluated one such approach to this, which combines the universal Climate Schools program with a selective personality-targeted program, Preventure (Teesson et al., Reference Teesson, Newton, Slade, Carragher, Barrett, Champion and Conrod2017). Impacts on alcohol harms were observed among students receiving the targeted program at 24 and 36 months and among students receiving the combined universal and targeted program at 36 months (Slade et al., Reference Slade, Newton, Mather, Barrett, Champion, Stapinski and Teessonunder review).
Similarly, while the odds of cannabis use increased less in the Climate group compared to the controls, this difference was non-significant. This finding is consistent with results from a previous trial assessing the effects of the Climate Schools program on cannabis use (Newton et al., Reference Newton, Teesson, Mather, Champion, Barrett, Stapinski and Slade2018). This may be in part due to insufficient power resulting from a very low rate of cannabis use among this group, which was almost half the prevalence estimate reported elsewhere for this demographic (Guerin & White, Reference Guerin and White2018) as well as ethical issues restricting the collection of cannabis use data in all states. We note, however, that there are several trends with respect to cannabis use and young people that may be reflected in the lack of prevention effects observed in the current trial. Firstly, the average age of initiation of cannabis use has been increasing among recent cohorts and in 2013 when data for this study was being collected it was 18.5 years (AIHW, 2014). It may be that cannabis use prevention may be better delivered later in secondary school, closer to the typical age of initiation or that longer-term follow-up would have been necessary to detect these effects. Secondly, whilst the prevalence of cannabis use has been decreasing among recent cohorts of young people, support for the legalisation of cannabis has been increasing simultaneously, suggesting a dismantling of the expected parallel relationship between cannabis use and legalisation attitudes (Kaur et al., Reference Kaur, Keyes, Hamilton, Chapman, Livingston, Slade and Swiftunder review). Future research and program development could consider the age at which the course is delivered as well as ways in which program content could address attitudes towards legalisation and harms among youth.
Strengths and limitations
A leading strength of this study is the large sample of Government, Independent Private and Catholic secondary schools from three Australian states spanning across the continent. This geographical and economic diversity improves the generalisability of results across Australia (3934 km apart). Furthermore, the fact that replicated effects were achieved in real-world school settings delivered in classrooms as part of the standard school curriculum, without researchers present is notable and is likely due to the program's web-based delivery, which ensures core components of the intervention are retained and the program can be delivered easily and with strong fidelity (Marsch et al., Reference Marsch, Bickel and Badger2007). This was a deliberate strategy of this trial in line with examining effectiveness rather than efficacy. To our knowledge, this study is the largest replication study of a web-based AOD prevention program.
Within this context several limitations need to be considered. Firstly, there were several differences between the Climate and Control groups at baseline. To account for these differences, rigorous statistical methods were employed and when models were adjusted for baseline covariates that differed substantially between groups, the intervention effects for all outcomes were similar to those observed in the primary unadjusted analyses. Furthermore, while research has demonstrated that self-report of AOD use and other transgressive behaviours is a reliable and valid measurement approach (Bandura, Barbaranelli, Caprara, & Pastorelli, Reference Bandura, Barbaranelli, Caprara and Pastorelli1996), there was no corroborating information to support these outcomes. To overcome this limitation, data were collected using only validated measures and participants were asked to refer to an Australian ‘Standard Drinks Chart’ to assist accurate reporting.
Secondly, we acknowledge that multiple methods could be used to operationalise primary outcomes for the current study and there is no agreed-upon method for operationalising outcomes of interest in the field of school-based AOD prevention (Champion et al., Reference Champion, Newton, Barrett and Teesson2013; Champion, Newton, & Teesson, Reference Champion, Newton and Teesson2016). Further, constraints of the data in terms of the observed distribution of scores mean that continuous measures such as frequency and quantity often have to be converted to categorical variables or dichotomised (this was the case for heavy episodic drinking in the current trial) limiting the feasibility of sensitivity analyses. With these limitations in mind, the definitions used in this trial have been used in previous trials of this program (e.g. Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015; Newton et al., Reference Newton, Vogl, Teesson and Andrews2009, Reference Newton, Teesson, Mather, Champion, Barrett, Stapinski and Slade2018) and in other SU prevention trials (Champion et al., Reference Champion, Newton, Barrett and Teesson2013, Reference Champion, Newton and Teesson2016) and are considered to be theoretically important. For example, uptake of alcohol use in adolescence (as measured by consuming at least one full standard drink) has been found to be associated with a number of substance use harms and mental health problems later in life (Behrendt et al., Reference Behrendt, Wittchen, Hofler, Lieb and Beesdo2009; Marel et al., Reference Marel, Sunderland, Mills, Slade, Teesson and Chapman2019).
Thirdly, we note that intervention fidelity and its measurement warrant careful consideration for trial replication. Whilst gold standard measures of treatment fidelity might include methods such as independent observer coding (Schoenwald et al., Reference Schoenwald, Garland, Chapman, Frazier, Sheidow and Southam-Gerow2011), the cost and time requirements of this approach mean it was not considered feasible for a large trial such as the current one. Moreover, the online delivery of the current intervention does not lend itself to observer ratings, which are best suited to an intervention delivered face-to-face. In terms of intervention fidelity in this and previous trials, we were reliant on both the in-built mechanisms of the Climate Schools program designed to maximise intervention fidelity as well as teacher logbooks which indicated program adaptation was rare. Within this context, we do not consider the lack of treatment fidelity to be a threat to the validity of the current trial.
Finally, attrition analyses indicated students lost to the follow-up were at a higher risk of substance use than those retained after baseline. In practice, this meant the data did not meet the ‘missing completely at random’ (MCAR) assumption. Instead, we assumed that the data are missing at random (MAR), conditional on the covariates that predict dropout and the available observations from each participant which were included in sensitivity analyses. Multilevel mixed-effects models yield unbiased estimates under the MAR assumption and we found no differences between unadjusted and adjusted models. Nonetheless, we acknowledge that there is always a possibility that the data are missing not at random (MNAR), and this may introduce bias to the results and limit the generalisability of findings. We also note that the study achieved good retention rates of 83.7% at the 12-month follow-up, and 95.2% of participants completed at least one follow-up survey, due largely to comprehensive follow-up methods including emails, phone calls, and text messages to support retention, further limiting the potential impact of missing data on the findings.
Implications
Amidst a replication crisis, replication trials such as this, are critical to rule out the possibility that past findings were a result of chance and to determine the generalisability of intervention effects (Flay et al., Reference Flay, Biglan, Boruch, Castro, Gottfredson, Kellam and Ji2005). Despite the substantial burden of disease attributable to AOD use, few prevention programs have proven to be effective in reducing or delaying AOD use among young people and even fewer have conducted replication trials (Champion et al., Reference Champion, Newton, Stapinski, Slade, Barrett and Teesson2015). The present study demonstrated support for the Climate Schools: Alcohol and Cannabis course as an effective model for alcohol use prevention. Examination of longer-term follow-ups, beyond 12 months could provide insight into the durability of intervention effects in later years as well as assess whether early prevention can protect against alcohol harms and cannabis use in later years.
Conclusion
Prevention is the key to reducing the considerable burden of disease attributable to AOD use among young people. Schools play a critical role in the delivery of prevention programs and it is vital that teachers are equipped with evidence-based resources that can withstand common barriers to effective implementation. The Climate Schools: Alcohol and Cannabis course is a standardised and scalable program that has been shown to improve knowledge and reduce alcohol uptake among large, diverse samples of adolescents. Future research should consider ways the program could be strengthened to reduce alcohol-related harms, particularly for high-risk individuals and to address some of the changing attitudes and behaviours with respect to cannabis use among young people.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291720001919.
Acknowledgements
The authors would like to acknowledge the schools, students, teachers, health professionals, and research assistants who were involved in this research. The research team also acknowledges the assistance of the New South Wales Department of Education and Communities (SERAP 2013054), the Western Australian Department of Education (D13/0475948), the Western Australian Catholic Education Office, the Brisbane Catholic Education Office, and Queensland Department of Education and Training for access to their schools to conduct this research. This research was funded by the National Health and Medical Research Council (NHMRC) project grant (APP1047291) and a Centre of Research Excellence Grant in Prevention and Early Intervention in Mental Illness and Substance Use (PREMISE, APP1134909). The NHMRC did not influence the design of the study, the collection, analysis, and interpretation of data or the writing of the manuscript.
Author contribution
MT, NN, CC, TS, and GA led the conception and design of the study. MM and TS led the analyses and interpretation of data. LB, LH, NM, SA, MT, NN, CC, LM and TS coordinated the implementation of the trial. NN, AH, LB and CC drafted the article and all authors approved the final version to be published.
Conflict of interest
In addition, MT & NN are two of the developers of the Climate Schools programs and directors of Climate Schools Pty Ltd, a social enterprise established in 2015 to distribute the Climate Schools programs and maximise social well-being.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
