Data sources

We based our meta-regression on a systematic review. A published systematic review of organisational interventions in primary care mental health completed by S.G. was used as the initial source of studies (Reference Gilbody, Whitty and GrimshawGilbody et al, 2003); this review included collaborative care as well as other types of organisational interventions used to improve the management of depression. Searches included Medline, EMBASE, CINAHL, PsycINFO, the Cochrane Library and the Database of Abstracts of Reviews of Effectiveness (DARE), and were run from the inception date of each database to 2003. We updated the comprehensive search strategies from this review (without language restriction) to June 2004 to find recent studies, and then conducted a second update to November 2005 (Fig. 1). Details of the exact search methods and a table of excluded studies are available from the authors upon request. From this comprehensive database, we then identified the subset of collaborative care studies.

Fig. 1

QUOROM (Quality of Reporting Meta-analyses) flow diagram.

Inclusion criteria

The population of interest was patients with depressive symptoms or diagnosed depressive disorders in primary care settings. Primary care was defined as the provision of medical care by professionals who provide first contact and ongoing care to patients, regardless of the patient's age, gender or presenting problem.

Although we have published a broad typology of models of quality improvement which includes collaborative care (Reference Bower and GilbodyBower & Gilbody, 2005), developing precise inclusion criteria for such complex interventions is more problematic, because by definition it is not clear a priori which mechanisms have to be in place in order to define an intervention as ‘collaborative care’. Therefore, any definition is potentially arbitrary, reflected by published reviews of collaborative care that disagree on which studies and interventions are included and excluded (Reference Von Korff and GoldbergVon Korff & Goldberg, 2001; Reference Gilbody, Whitty and GrimshawGilbody et al, 2003; Reference Bijl, van Marwijk and de HaanBijl et al, 2004).

The purpose of the study was to examine the relationship between variation in the interventions within collaborative care studies, and outcomes. Therefore, we used a broad definition, and defined ‘collaborative care’ as a multifaceted organisational intervention, which could include a number of components:

1. (a) the introduction of a new role (case manager) into primary care, to assist in the management of patients with depression through structured and systematic delivery of interventions;

2. (b) the introduction of mechanisms to foster closer liaison between primary care clinicians and mental health specialists (including case managers) around individual patient care;

3. (c) the introduction of mechanisms to collect and share information on the progress of individual patients.

We excluded educational and training interventions and the provision of brief psychological therapy where these were the sole intervention and were not supported by other enhancements of care outlined above. The full list of studies is given in Table 1.

Table 1

Studies included in the review

Study	Reference	Setting	Unit of randomisation	Sample size n	Patient population	Antidepressant use data?	Depressive symptoms data?
Adler 2004	Adler et al (Reference Adler, Bungay and Wilson2004)	USA	Patient	533	Adults with major depression or dysthymia	Yes	Yes
Akerblad 2003	Bungay et al (Reference Bungay, Adler and Rogers2004)	Sweden	General practitioner	1031	Adults with major depression and an indication for antidepressants	Yes	Yes
Araya 2003	Araya et al (Reference Araya, Rojas and Fritsch2003)	Chile	Patient	240	Adult women with major depression	Yes	Yes
Blanchard 1995	Blanchard et al (Reference Blanchard, Waterreus and Mann1995)	UK	Patient	96	Elderly people with depression warranting clinical intervention	Yes	Yes
Brook 2003	Brook et al (Reference Brook, van Hout and Nieuwenhuyse2003a ,Reference Brook, van Hout and Nieuwenhuyse b )	The Netherlands	Patient	147	Adults with depressive complaints, prescribed new antidepressant	No	Yes
Bruce 2004	Coyne et al (Reference Coyne, Brown and Datto2001); Bruce et al (Reference Bruce, Ten Have and Reynolds2004)	USA	Practice	598	Elderly people with major depression, dysthymia and minor depression	Yes	Yes
Callahan 1994	Callahan et al (Reference Callahan, Hendrie and Dittus1994)	USA	Practice	175	Elderly people with newly diagnosed depression	Yes	Yes
Capoccia 2004	Boudreau et al (Reference Boudreau, Capoccia and Sullivan2002); Capoccia et al (Reference Capoccia, Boudreau and Blough2004)	USA	Patient	74	Adults with depression, prescribed a new antidepressant	Yes	Yes
Coleman 1999	Coleman et al (Reference Coleman, Grothaus and Sandhu1999)	USA	Practice	169	Frail elderly people	No	Yes
Datto 2003	Datto et al (Reference Datto, Thompson and Horowitz2003)	USA	Practice	61	Adults with depressive symptoms	No	Yes
Dietrich 2004	Dietrich et al (Reference Dietrich, Oxman and Williams2004a ,Reference Dietrich, Oxman and Williams b )	USA	Practice	405	Adults with major depression and dysthymia, starting/changing treatment	Yes	Yes
Finley 1999	Finley et al (Reference Finley, Rens and Gess1999, Reference Finley, Rens and Pont2003)	USA	Patient	125	Adults with current major depression, prescribed a new antidepressant	Yes	Yes
Hunkeler 2000	Hunkeler et al (Reference Hunkeler, Meresman and Hargreaves2000)	USA	Patient	302	Adults with major depression or dysthymia, prescribed a new antidepressant	Yes	Yes
Katon 1995	Katon et al (Reference Katon, Von Korff and Lin1995); Von Korff et al (Reference Von Korff, Katon and Bush1998)	USA	Patient	217	Adults with depression, prescribed a new antidepressant	Yes	Yes
Katon 1996	Katon et al (Reference Katon, Robinson and Von Korff1996); Von Korff et al (Reference Von Korff, Katon and Bush1998)	USA	Patient	153	Adults with depression, prescribed a new antidepressant	Yes	Yes
Katon 1999	Katon et al (Reference Katon, Von Korff and Lin1999); Simon et al (Reference Simon, Katon and Von Korff2001a )	USA	Patient	228	Adults on antidepressants, at high risk of persistent depression, recurrent depression or dysthymia	Yes	Yes
Katon 2001	Katon et al (Reference Katon, Rutter and Ludman2001a ), Simon et al (Reference Simon, Von Korff and Ludman2002)	USA	Patient	386	Adults, prescribed a new antidepressant, at high risk of relapse	Yes	Yes
Katon 2004	Katon et al (Reference Katon, Von Korff and Lin2003, Reference Katon, Von Korff and Lin2004)	USA	Patient	329	Adults with diabetes with depressive symptoms	No	Yes
Katzelnick 2000	Katzelnick et al (Reference Katzelnick, Simon and Pearson2000); Simon et al (Reference Simon, Manning and Katzelnick2001b )	USA	Practice	407	Adults, high users of services, with depressive symptoms	Yes	Yes
Mann 1998	Mann et al (Reference Mann, Blizard and Murray1998)	UK	Patient	419	Adults with depression	Yes	Yes
Oslin 2003	Oslin et al (Reference Oslin, Sayers and Ross2003)	USA	Physician	97	Adults with depression or dysthymia, at risk drinking	No	Yes
Peveler 1999	Peveler et al (Reference Peveler, George and Kinmonth1999)	UK	Patient	160	Diagnosis of depression, prescribed a new antidepressant	Yes	Yes
Rickles 2003	Rickles (Reference Rickles2003)	USA	Patient	63	Prescribed a new antidepressant	No	Yes
Rost 2001a	Rost et al (Reference Rost, Nutting and Smith2000, Reference Rost, Nutting and Smith2001); Pyne et al (Reference Pyne, Smith and Fortney2003)	USA	Practice	243	Adults with major depression, prescribed a new antidepressant, recently treated	Yes	Yes
Rost 2001b	As above	USA	Practice	189	Adults with major depression, prescribed a new antidepressant, beginning new episode	Yes	Yes
Simon 2000	Simon et al (Reference Simon, Von Korff and Rutter2000)	USA	Patient	392	Adults with depression, prescribed a new antidepressant	Yes	Yes
Simon 2004a	Simon et al (Reference Simon, Ludman and Tutty2004)	USA	Patient	402	Adults with depression, prescribed a new antidepressant	Yes	Yes
Simon 2004b	As above	USA	Patient	393	Adults with depression, prescribed a new antidepressant	Yes	Yes
Swindle 2003	Swindle et al (Reference Swindle, Rao and Helmy2003)	USA	Firm	268	Adults with major depression, dysthymia or partially remitted major depression	Yes	Yes
Unutzer 2002	Unutzer et al (Reference Unutzer, Katon and Williams2001a , Reference Unutzer, Katon and Callahan2003)	USA	Patient	1801	Elderly people with major depression, dysthymia, or both	Yes	Yes
Wells 2000a	Wells et al (Reference Wells, Sherbourne and Schoenbaum2000); Sherbourne et al (Reference Sherbourne, Wells and Duan2001); Schoenbaum et al (Reference Schoenbaum, Unutzer and Sherbourne2001); Unutzer et al (Reference Unutzer, Rubenstein and Katon2001b ); Wells et al (Reference Wells, Sherbourne and Schoenbaum2004)	USA	Practice	867	Adults with major depression or dysthymia	Yes	Yes
Wells 2000b	As above	USA	Practice	932	Adults with major depression or dysthymia	Yes	Yes
Whooley 2000	Whooley et al (Reference Whooley, Stone and Soghikian2000)	USA	Practice	331	Elderly people with depressive symptoms	Yes	Yes
Wilkinson 1993	Wilkinson et al (Reference Wilkinson, Allen and Marshall1993)	UK	Patient	61	Adults with depression, prescribed a new antidepressant	Yes	Yes

Data extraction

Analysis

Analyses were conducted in Stata version 8 for Windows, using the metan and metareg macros. The initial meta-analyses used random effects modelling (Reference Sutton, Abrams and JonesSutton et al, 1998) to provide an overall pooled measure of effect of collaborative care on the two outcomes. However, the main focus of the analysis was on heterogeneity. Heterogeneity was measured using the I ² statistic, which estimates the percentage of total variation across studies that can be attributed to heterogeneity rather than chance. As a guide, I ² values of 25% may be considered low, 50% moderate and 75% high (Reference Higgins, Thompson and DeeksHiggins et al, 2003).

The main analysis used random effects meta-regression, which provided estimates of the relationships between eight intervention content variables and the two outcomes. The permutation test was used to calculate P values (using 1000 Monte Carlo simulations) and to reduce the chance of spurious false-positive findings (Reference Higgins and ThompsonHiggins & Thompson, 2004). The amount of heterogeneity explained by the intervention content variables was examined by reductions in the I ² statistic. Initial univariate analyses (using a criterion of significance of P<0.10) were followed by estimation of a multivariate model. The multivariate model was not based on any automated selection procedure, but involved examination of a number of candidate models involving different combinations of variables. The final model was chosen on the basis of the greatest reduction in heterogeneity. A secondary meta-regression provided an estimate of the relationships between the two outcomes (i.e. whether antidepressant use predicted depressive symptoms).

Meta-analysis

We found a positive effect of collaborative care on antidepressant use (odds ratio 1.92, 95% CI 1.54–2.39; Fig. 2) and depressive outcomes (standardised mean difference 0.24, 95% CI 0.17–0.32; Fig. 3). The I ² estimates of inconsistency were 80% and 54% respectively.

Fig. 2

Meta-analysis of antidepressant use. Note: the Wells (2000) and Simon (2004) studies involved two intervention groups compared against a single control; to avoid double-counting the controls, the sample size and event rate in the control were divided by 2. The Rost 2001 study data are only available analysed in two subgroups, rather than as an overall analysis; in our analysis these subgroups were treated as separate comparisons.

Fig. 3

Meta-analysis of depressive symptoms (see note for Fig. 2).

Meta-regression

Analyses of the effects of intervention content variables are shown in Tables 3 and 4. There was insufficient variability in quality of allocation concealment, as most studies were rated as ‘not clear’, and this variable was not used as a covariate in the final analysis.

Table 3

Univariate analysis of associations between intervention content variables and antidepressant use

Variable	Category 1	Category 2	Log odds ratio regression coefficient (95% CI)	P	I 2 (%)
Study setting	Outside USA	USA	0.076 (-0.558 to 0.710)	0.804	80.2
Patient sample	Patients with depression	Patients with depression willing to take antidepressants	-0.123 (-0.631 to 0.385)	0.647	80.1
Recruitment method	Referral	Systematic identification	0.345 (-0.167 to 0.858)	0.183	78.5
Primary care physician training	No training provided	Training provided	0.328 (-0.163 to 0.818)	0.194	79.5
Case manager background	Non-mental health professional	Mental health professional	0.220 (-0.280 to 0.721)	0.393	78.9
Content of case management	Medication management	Medication management plus psychotherapeutic processes	-0.104 (-0.617 to 0.409)	0.683	80.5
Supervision of case manager	None or variable	Regular and planned	0.039 (-0.549 to 0.627)	0.906	80.4
Case management sessions1			-0.053 (-0.126 to 0.020)	0.151	79.6

Table 4

Univariate analysis of associations between intervention content variables and depressive symptoms

Variable	Category 1	Category 2	Effect size regression coefficient (95% CI)	P	I 2 (%)
Study setting	Outside USA	USA	0.007 (-0.193 to 0.206)	0.930	54.4
Patient sample	Patients with depression	Patients with depression willing to take antidepressants	-0.087 (-0.243 to 0.070)	0.285	52.1
Recruitment method	Referral	Systematic identification	0.146 (-0.014 to 0.306)	0.061	47.8
Primary care physician training	No training provided	Training provided	0.093 (-0.065 to 0.252)	0.237	54.9
Case manager background	Non-mental health professional	Mental health professional	0.187 (0.046 to 0.327)	0.004	42.7
Content of case management	Medication management	Medication management plus psychotherapeutic processes	0.093 (-0.064 to 0.250)	0.206	50.7
Supervision of case manager	None or variable	Regular and planned	0.169 (0.002 to 0.337)	0.033	49.3
Case management sessions1			0.015 (-0.008 to 0.039)	0.174	50.9

None of the intervention content variables was significantly associated with anti-depressant use, and no multivariate model was estimated. Three intervention content variables predicted improvement in depressive symptoms: recruitment by systematic identification (P=0.061), case managers having a specific mental health background (P=0.004) and provision of regular supervision for case managers (P=0.033), which reduced the overall heterogeneity I ² from 54% to 48% and 43 to 49% respectively.

In multivariate analysis, four intervention content variables produced the most robust meta-regression in relation to depressive symptom outcomes. The analysis indicated that non-US studies (P=0.038), recruiting through systematic identification of patients (P=0.081) and using case managers having a specific mental health background (P=0.027) who received regular supervision (P=0.055) were more effective. The combination of these four covariates reduced the overall heterogeneity to 36% (low to moderate between study heterogeneity). The inclusion of ‘setting’ (which was not statistically significant in the univariate analyses) reflects the fact that the multivariate analysis accounts for both the relationships between each intervention content variable and the outcome, and the relationships between intervention content variables (Reference Tabachnick and FidellTabachnick & Fidell, 2001).

The meta-regression of the relationships between antidepressant use and depressive symptoms showed a positive association (β coefficient 0.20, 95% CI 0.02–0.38, P=0.028; Fig. 4).

Fig. 4

Relationship between antidepressant use outcomes and depressive symptoms outcomes.

The results of these analyses were not substantively influenced by the sensitivity analysis using estimates of intracluster correlations of 0.00 and 0.05.

Limitations of the systematic review

As a complex intervention, collaborative care defies simple definition. Our decisions about inclusion and exclusion were informed by our previous conceptual work (Reference Bower and GilbodyBower & Gilbody, 2005), but we took a liberal approach to inclusion precisely because the study focused on the degree to which variability in collaborative care models influenced outcomes. Clearly the inclusion or exclusion of particular studies may have important implications, and thus our findings should be considered exploratory rather than definitive. It should also be noted that most studies were conducted in the USA and the results may not generalise to other contexts. Setting was a significant predictor in the multivariate analysis.

The validity of the coding scheme used to extract data on the interventions has not been confirmed. As noted previously, there were problems of inconsistent reporting and missing data in the published studies. A significant proportion of intervention content variables could not be included as they were not reported consistently, and it is unlikely that it would have been possible to extract data on many additional issues. However, it remains possible that other variables might be more effective predictors than those included in our analyses.

The difficulties encountered in deriving a full description of the interventions echoes traditional problems with poor reporting in randomised trials. There may be a case for adopting a more standardised approach to the reporting of the content of complex interventions (equivalent to CONSORT (Consolidated Standards of Reporting Trials; Reference Moher, Schutz and AltmanMoher et al, 2001) and QUOROM (Quality of Reporting Metaanalyses; Reference Moher, Cook and EastwoodMoher et al, 1999) in order to overcome these problems. The proliferation of web-based journal archives for the presentation of data outside the word limits of the paper-based journals provides an appropriate platform. However, determining the appropriate content and structure of such standardised reports would be challenging, given the potential range of processes involved in complex interventions.

Limitations of the meta-regression technique

The technique of meta-regression has several limitations (Reference Thompson and HigginsThompson & Higgins, 2002). The analysis represents an observational association only, because meta-regression across trials does not have the benefits of randomisation. Equally, statistical power to detect useful associations using meta-regression is limited by (among other things) the number of available studies (Reference Lambert, Sutton and AbramsLambert et al, 2002). Outliers may have a large influence, particularly in the context of a limited sample size. The multivariate model described earlier was found to be sensitive to the particular variables included in the analysis. It should also be noted that the analysis will not be able to detect ‘active ingredients’ that are necessary but do not vary between interventions. Furthermore, it is possible that with certain variables, such as the number of case management sessions, the relationship with average numbers of sessions across trials may not be the same as the relationship within trials. Only individual patient data analysis could overcome this ‘ecological fallacy’ (Reference Thompson and HigginsThompson & Higgins, 2002).

Finally, the analyses were not controlled for quality criteria. The a priori quality criterion (concealment of allocation) showed little variation, as the majority of studies failed to report this adequately. However, it is not clear whether inadequate reporting of concealment always reflects inadequate methods (Reference Soares, Daniels and KumarSoares et al, 2004; Reference Pildal, Chan and HrobjartssonPildal et al, 2005).

Alternatives to meta-regression in the analysis of complex interventions

The controversy over fidelity to assertive community treatment and outcomes (Reference Fiander, Burns and McHugoFiander et al, 2003) indicates that the identification and measurement of ‘active ingredients’ in mental health interventions has important implications for both research and service provision (Reference Marshall and CreedMarshall & Creed, 2000). It is therefore critical to consider the optimal methods of identifying ‘active ingredients’. Our study has shown that the use of meta-regression is feasible but has limitations. The key issue is how well meta-regression compares with the available alternatives, which include clinical expertise, qualitative work, theoretical models and ‘dismantling’ or ‘factorial’ trials.

Clinical expertise is a potentially useful source of hypotheses, and rigorous qualitative work is ideally suited to capture the complexity of care processes, and is especially useful at exploring the perspectives of stakeholders and illuminating context (Reference Weaver, Tyrer and RitchieWeaver et al, 2003; Reference Marshall, Lockwood and LewisMarshall et al, 2004). However, it is unclear whether patients and professionals can reliably identify ‘active ingredients’. Acknowledgement of the limitations of clinical expertise in identifying causal mechanisms is fundamental to evidence-based medicine, and patients will presumably face many of the same challenges as professionals. Insights from theoretical models are another useful source, but few theoretical models within mental health services research are so well validated that they provide a comprehensive description of ‘active ingredients’, and complex mental health issues such as depression will have many competing theories. Although theory is a necessary aspect of the development of a complex intervention, it will rarely be sufficient.

Dismantling and factorial studies test different combinations of ingredients within a randomised comparison. Relevant examples exist in the collaborative care literature. For example, a recent study compared outcomes in patients randomised to a depression care programme (including systematic follow-up) and systematic follow-up alone. There was no difference in outcomes, suggesting that systematic follow-up is critical (Reference Vergouwen, Bakker and BurgerVergouwen et al, 2005). The advantage of such designs is that randomisation is preserved, allowing causal inference. However, the use of such costly designs to identify ‘active ingredients’ may not always be the optimal use of limited research resources.

Clearly comparisons of the different methods are required, and the intervention development currently being conducted by the authors also includes qualitative work which can be compared with the findings of the meta-regression. It is likely that complex interventions will increasingly be required to improve patient care within mental health, and the evaluation of such interventions raises particular challenges. Although there are potential problems with the application of meta-regression, we conclude that the technique has potential in developing useful insights into the active ingredients in complex interventions in mental health, and thus assist in the design and evaluation of future interventions.