Introduction
As one of the most common mental disorders, more than 350 million people suffer from depression globally (World Health Organization, 2012). Aside from being a leading cause of disability worldwide (World Health Organization, 2012), depression is also a significant cause of mortality given its strong links with suicide (Mykletun et al. Reference Mykletun, Bjerkeset, Overland, Prince, Dewey and Stewart2009). Because depression is often chronic, patients depend heavily on both pharmacological and non-pharmacological treatment strategies (Cipriani et al. Reference Cipriani, Barbui, Butler, Hatcher and Geddes2011). There are numerous treatments available for depression and it is important for clinicians to make informed decisions on their use, based on the best available clinical evidence. As a major approach for synthesising results from clinical trials, systematic reviews (SRs) use explicit and reproducible methods to identify, critically appraise and summarise all clinical trial results that meet pre-defined eligibility criteria (Higgins & Green, Reference Higgins and Green2011). It is a well-established method in evidence-based healthcare, distinguishing itself from narrative reviews which are prone to bias due to the lack of transparency (Higgins & Green, Reference Higgins and Green2011). Within a SR, meta-analysis is a common statistical method that allows quantitative synthesis of trial results, allowing estimation of the treatment effect with higher precision and permitting exploration of heterogeneity in patient outcomes (Higgins & Green, Reference Higgins and Green2011). A methodologically rigourous and up-to-date SR can provide the best clinical evidence for supporting treatment decision-making (Mulrow et al. Reference Mulrow, Cook and Davidoff1997).
Despite high publication volume and wide acceptance among healthcare professionals (Moher et al. Reference Moher, Tetzlaff, Tricco, Sampson and Altman2007), SRs should not be regarded as the ‘best’ sources of evidence without critical appraisal of included studies. A SR with methodological deficiencies might overestimate or underestimate treatment effect, draw biased conclusions and subsequently mislead decision-making (Mulrow et al. Reference Mulrow, Cook and Davidoff1997). Evidence-users, including clinicians, policy-makers, managers and patients are advised not to accept the results and conclusions from SRs without critical appraisal of included evidence (Fleming et al. Reference Fleming, Koletsi, Seehra and Pandis2014). At present the quality of summary evidence to guide decision-makers in this area is unknown as no previous work has comprehensively examined the methodological quality of SRs focused on treatment for depression. This work will inform the state-of-science in the field and provide insight on how methodological quality may be improved in the future.
In this cross-sectional study, we aim to: (1) describe the bibliographical characteristics of a representative sample of SRs on depression treatments; (2) appraise the methodological quality of a representative sample of SRs on depression treatments using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool (Shea et al. Reference Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher and Bouter2007); and (3) examine the relationship between methodological rigour and bibliographical features.
Materials and methods
Eligibility criteria
All SRs that performed at least one meta-analysis of any depression treatment were eligible, without restrictions on publication year. We only included SRs with at least one meta-analysis in order to ensure that all 11 AMSTAR items were applicable during the critical appraisal process (Burda et al. Reference Burda, Holmer and Norris2016). SRs must have pooled primary studies recruiting patients with depression who were diagnosed using an established criteria or questionnaire [e.g., the Diagnostic and Statistical Manual of Mental Disorders, Third edition (American Psychiatric Association, 1980) or the Hamilton Rating Scale for Depression (Hamilton, Reference Hamilton1960)]. Narrative reviews, protocols, network meta-analyses, SRs without meta-analysis or SRs focused on the diagnosis of depression were excluded. The most up-to-date version was selected when a SR had been updated.
Literature search
We applied the keyword ‘depress*’ in the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE), of which the former contains all Cochrane reviews and the later indexes non-Cochrane SRs (Cochrane Community (beta) 2015). Databases were searched from inception until February 2017, with no language restriction.
Literature screening, selection and data extraction
The title and abstract of retrieved citations was screened independently by two authors (Feng and Ho). Full texts of potentially eligible citations were then retrieved, and eligibility was further assessed by these two authors, independently. Data extraction was again performed independently by two reviewers (Feng and Ho), with discrepancy resolved by referring to the original publications. Persistent disagreements were resolved by consensus adjudication led by a third author (Chung).
Methodological quality assessment
The methodological quality of included SRs was assessed with the AMSTAR (Shea et al. Reference Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher and Bouter2007). The 11 item AMSTAR is an established instrument for appraising the methodological quality of SRs (Shea et al. Reference Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher and Bouter2007). AMSTAR is a well-evaluated assessment tool with good agreement, construct validity and feasibility, as demonstrated in validation studies (Shea et al. Reference Shea, Hamel, Wells, Bouter, Kristjansson, Grimshaw, Henry and Boers2009). It has been applied to evaluate the methodological quality of SRs in many clinical areas, including diabetes mellitus (Wu et al. Reference Wu, Lam, Yu, Ho, Feng, Wong, Yip, Tsoi, Wong and Chung2016b), hypertension (Wu et al. Reference Wu, Du, Ho, Lee, Yip, Wong, Wong and Chung2016a), Chinese herbal medicine (Chung et al. Reference Chung, Ho, Wu, Fung, Lai, Wu and Wong2015), chronic obstructive pulmonary disease (Ho et al. Reference Ho, Wu, Yuan, Liu, Lai, Wong and Chung2015), influenza vaccination (Remschmidt et al. Reference Remschmidt, Wichmann and Harder2014), orthodontics (Papageorgiou et al. Reference Papageorgiou, Papadopoulos and Athanasiou2011) and oral healthcare interventions (Sequeira-Byron et al. Reference Sequeira-Byron, Fedorowicz, Jagannath and Sharif2011). Performance of the included SRs on each of the 11 items is reported separately, with no calculation of the overall score, as there is no empirical evidence on how to assign an appropriate weighting for each item (Shea et al. Reference Shea, Hamel, Wells, Bouter, Kristjansson, Grimshaw, Henry and Boers2009). The detailed assessment form for AMSTAR is presented in online Appendix A. Judgements were assigned for each SR according to standard guidelines and each item was classed as ‘yes’, ‘no’ or ‘cannot answer/not reported’, as appropriate. The methodological quality was assessed independently by two assessors (Feng and Ho). In case of discrepancy, a third researcher (Chung) was consulted.
Data analysis
Bibliographical characteristics and the AMSTAR assessment results were analysed with descriptive statistics. Data were summarised as frequencies, percentages, medians and ranges. The associations between bibliographical characteristics and scoring on AMSTAR items were analysed using logistic regression analysis. Adjusted odds ratio (AOR) was calculated to measure the association between bibliographical characteristics and scoring on AMSTAR items. Seven potential predictors were examined for SR quality: Cochrane review v. non-Cochrane review, update review v. original review; year of publication (continuous variable); impact factor of the journal at the year before the SR published (continuous variable); number of authors (continuous variable); location of corresponding author Europe v. non-Europe and pharmacological v. non-pharmacological treatment. We used the ‘Enter’ method to include all seven predictors in the regression model. Predictors that showed statistical significance were reported in Table 3, results on remaining predictors will be available on request. A p-value <0.05 was considered as reaching statistical significance. The Hosmer–Lemshow (HL) test was applied to evaluate the degree of model-fitting for logistic regression, with a p-value >0.1 indicating good model-fitting. All the statistical analyses were conducted with the Statistical Package for Social Sciences (SPSS) 21.0.
Results
Bibliographical characteristics of SRs
We identified 1653 records through literature searches, of which 1196 were excluded as ineligible. The remaining 358 were included in this study and the literature selection flow can be found in Fig. 1. The 358 SRs included a total of 8034 primary studies and 877 010 participants, and a full list of included SRs can be found in online Appendix B. The included SRs were published between 1995 and 2016, with a median publication year of 2011. Bibliographical characteristics of the included SRs are presented in Table 1. More than half (54.5%) of the SRs focused on non-pharmacological treatments, the remaining 163 SRs (45.5%) reporting on pharmacological treatments for depression. Of the pharmacological and non-pharmacological treatment SRs, 34.4 and 71.3% did not report treatment harms, respectively.
Sampling of meta-analyses on depression treatments: flow chart.
Bibliographical characteristics of 358 included systematic reviews on depression treatments
* Values are n (%), median (range) or n (mean ± s.d.).
RCT, randomised controlled trial; SR, systematic review; PRISMA, The Preferred Reporting Items for Systematic Reviews and Meta-analyses.
Sixty-two SRs (17.3%) were Cochrane reviews and 56 (15.6%) SRs were an update of a previous SR. Most SRs (72.6%) were published in journals with impact factors lower than 5 and only 12 (3.4%) were accepted by journals with impact factors of higher than 10. The median review team size was 4 (range 1–13). In 46.9% of the SRs, the corresponding authors were based in Europe, and around one quarter (22.3%) were from America. Funding source locations had a similar distribution to that of the corresponding authors, but such information was not reported in more than half of the SRs.
The majority of SRs (96.6%) searched English databases (that also index non-English articles) and 53.1% of SRs report including articles of any language. Only 58 (16.2%) SRs conducted searches in non-English databases. English databases were defined as those operating in English but which may index articles with full-text articles not in English (such as DARE). Non-English databases include any that are not operated in English. Furthermore, 74 SRs (20.7%) reviewed English publications exclusively. Most SRs (91.3%) included randomised controlled trials (RCTs) alone, 18 (5.0%) included RCTs and observational studies and only six (1.7%) SRs included solely observational studies (Table 1).
Methodological quality of SRs
Results of the methodological quality assessment are reported according to each AMSTAR item in Table 2. The majority of SRs (95.8%) described characteristics of included studies and conducted comprehensive literature searches (88.5%). Although 75.7% of SRs assessed and documented the risk of bias in the primary studies, only 31.3% took the risk of primary study bias into account when drawing conclusions. Around one quarter (25.7%) of SRs used inappropriate methods to combine findings during meta-analyses. Furthermore, most SRs had poor performance in the following three methodological aspects: (i) declaring conflict of interests for all included primary studies and the SR itself (31.6%); (ii) providing an ‘a priori’ published protocol (26.0%); and (iii) providing lists for both included and excluded studies (29.1%). Common shortcomings on reporting of SR methodology were: one third (33.2%) of the SRs did not report sufficient information to determine whether duplicate study selection and data extraction were performed, and 20.4% did not clearly state whether they included grey literature or not.
Methodological quality of 358 included systematic reviews on depression treatments
AMSTAR, Assessing the Methodological Quality of Systematic Reviews; NA, not applicable.
Predictors of methodological quality
In logistic regression analyses, significant associations between bibliographical characteristics and methodological quality were observed in five AMSTAR items (Table 3), however the majority of characteristics were not associated with any other AMSTAR items. SRs published more recently were more likely to provide an ‘a priori’ design (AOR for each additional year: 1.31, 95% confidence interval (CI) 1.09–1.57), to describe characteristics of the included studies (AOR 1.16, 95% CI 1.06–1.28) and to assess publication bias (AOR 1.13, 95% CI 1.06–1.19). However, they were less likely to list both included and excluded studies (AOR 0.86, 95% CI 0.81–0.92). Three predictors were identified for better performance in publication bias assessment: higher impact factor (AOR per 1 impact factor unit increase: 1.14, 95% CI 1.04–1.25), SRs completed by more review authors (AOR for each additional author: 1.12, 95% CI 1.01–1.24) and SRs on non-pharmacological treatments (AOR 1.62, 95% CI 1.01–2.59). SRs completed by teams with a European corresponding author (AOR 3.90, 95% CI 1.70–8.97) were more likely to provide an ‘a priori’ design, and to combine the results using appropriate statistical methods (AOR 1.76, 95% CI 1.05–2.93). Finally, SRs on non-pharmacological treatments were less likely to provide list of included and excluded studies (AOR 0.42, 95% CI 0.20–0.86). The p-values of all HL tests were >0.1, indicating good model fit for all logistic regression analyses.
Association between characteristics of depression treatments systematic reviews and methodological quality: Logistic regression analysis
AMSTAR, Assessing the Methodological Quality of Systematic Reviews; CI, confidence interval; OR, odds ratio.
* The p values of all HL tests were >0.1, indicating good model fit for all logistic regression analyses.
† As an editorial policy of the Cochrane Library, all Cochrane reviews are required to provide protocol (item 1) and lists of included and excluded studies, and therefore all of them will fulfil requirements of AMSTAR Item 1 and 5. It appears to be methodologically unsound to included Cochrane reviews in regression analyses on these two items (we have attempted to do some and results are shown in online Appendix C). Hence, in this Table, regression analyses for Item 1 and 5 were only performed in non-Cochrane reviews.
‡ Corresponding author from areas other than Europe were used as reference.
§ Pharmacological treatment was used as the reference.
Discussion
We have included a representative sample of SRs (n = 358) published between 1995 and 2016, which had synthesised evidence on the effectiveness of depression treatments. This reflects an acceptance of the evidence-based approach in making decisions on depression treatments. More than half of these SRs were conducted by teams led by European researchers. However, the methodological quality of these SRs is low, especially in the aspects of: (i) ensuring comprehensive literature search and being up to date; (ii) considering risk of bias when drawing conclusions and use of appropriate statistical approaches; (iii) declaring conflict of interests transparently; and (iv) the reporting of methodological details and treatment harms.
Ensuring comprehensive literature search
With regards to literature search, SR limitations may not be apparent at the first instance as 88.5% had conducted a ‘comprehensive’ search according to the AMSTAR standard. However, our data showed that only 16.2% performed searches in non-English databases. Although most of the included SRs searched two or more databases indexing non-English articles, we cannot exclude the possibility of language bias in study selection, as 20.7% included only English publications, and 26.3% did not describe eligibility criteria with regards to language. Empirical evidence has suggested that inclusion of only English publications in SRs can increase the risk of overestimation (Egger et al. Reference Egger, Zellweger-Zähner, Schneider, Junker, Lengeler and Antes1997) or underestimation (Jüni et al. Reference Jüni, Holenstein, Sterne, Bartlett and Egger2002) of treatment effects. Furthermore, the addition of non-English studies will improve generalisability and precision of pooled estimates (Morrison et al. Reference Morrison, Polisena, Husereau, Moulton, Clark, Fiander, Mierzwinski-Urban, Clifford, Hutton and Rabb2012), as searches in non-English databases may allow identification of a large number of additional primary studies conducted in non-English-speaking populations (Wu et al. Reference Wu, Tang, Mao, Yuan, Qin and Chung2013; Cohen et al. Reference Cohen, Korevaar, Wang, Spijker and Bossuyt2015). Future SRs should set no eligibility criteria based on language, and English-speaking reviewers may reduce their burden by using recommended screening strategies (Busse et al. Reference Busse, Bruno, Malik, Connell, Torrance, Ngo, Kirmayr, Avrahami, Riva, Ebrahim, Struijs, Brunarski, Burnie, LeBlanc, Coomes, Steenstra, Slack, Rodine, Jim, Montori and Guyatt2014).
Another methodological limitation is the possible exclusion of grey literature (Shea et al. Reference Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher and Bouter2007). Among included SRs, 26.3% did not include grey literature and 20.4% did not provide information on this issue. Current evidence suggested that exclusion of grey literature causes underrepresentation of negative results, and subsequently leads to overestimation of the treatment effects (Dwan et al. Reference Dwan, Altman, Arnaiz, Bloom, Chan, Cronin, Decullier, Easterbrook, Von Elm, Gamble, Ghersi, Ioannidis, Simes and Williamson2008). While SRs conducted by more authors, published more recently, published in higher impact factor journals or focused on non-pharmacological treatments are more likely to detect publication bias stemming from this problem, 45.3% of the SRs did not assess publication bias and thus their reported effect size may be questionable. Aside from avoiding language and publication bias, trustworthiness of SR results also depends on whether the search is up-to-date (Moher & Tsertsvadze, Reference Moher and Tsertsvadze2006).
Considering risk of bias and use of appropriate statistics
In SRs, assessing risk of bias of primary studies is of crucial importance as this will gauge the validity of meta-analytic results. Unfortunately, 24.3% of SRs did not appraise their included primary studies. Furthermore, only 31.3% explicitly consider risk of bias of primary studies when drawing conclusions. This implies that the ‘garbage in, garbage out’ (Egger et al. Reference Egger, Smith and Sterne2001) problem could be prevalent among SRs on depression treatments, echoing studies in other fields (Moja et al. Reference Moja, Telaro, D'Amico, Moschetti, Coe and Liberati2005; Ho et al. Reference Ho, Wu, Yuan, Liu, Lai, Wong and Chung2015). Use of an inappropriate statistical method for pooling findings in 25.7% of the SRs represents another threat to validity. It is recommended that a multidisciplinary team should be built to ensure availability of statistical expertise.
Declaring conflict of interests explicitly
It is well acknowledged that SRs funded commercially are less transparent, and are more likely to draw favourable conclusions to support the interventions (Lexchin et al. Reference Lexchin, Bero, Djulbegovic and Clark2003; Jørgensen et al. Reference Jørgensen, Hilden and Gøtzsche2006). Among included SRs, only 31.6% declared conflict of interests for both the included primary studies and the SR itself. While many journals now request that authors state conflict of interests for the SR, similar requirements do not always apply for the primary studies included in SRs. Lack of such information hinders assessment on possible conflict of interests stemming from primary studies. Adding such a requirement in the next version of PRISMA reporting guideline could further prevent this problem (Moher et al. Reference Moher, Liberati, Tetzlaff and Altman2009).
Comparisons with similar studies
Overall, quality of SRs on depression treatments is variable when compared with other clinical areas. A lower proportion of our sample reported PRISMA-like flow diagrams and treatment harms when compared to a sample of SRs from MEDLINE indexed in February 2014 (Page et al. Reference Page, Shamseer, Altman, Tetzlaff, Sampson, Tricco, Catalá-López, Li, Reid, Sarkis-Onofre and Moher2016). However, when compared to SRs on hypertension treatments (Wu et al. Reference Wu, Du, Ho, Lee, Yip, Wong, Wong and Chung2016a), our sample performed better in the majority of AMSTAR items except for providing a protocol, conducting duplicate study selection and data extraction, and in considering risk of bias when drawing conclusions. Our sample also did better in stating sources of support than SRs on hypertension treatments (Wu et al. Reference Wu, Du, Ho, Lee, Yip, Wong, Wong and Chung2016a), diabetes treatments (Wu et al. Reference Wu, Lam, Yu, Ho, Feng, Wong, Yip, Tsoi, Wong and Chung2016b) and chronic obstructive pulmonary disease (Ho et al. Reference Ho, Wu, Yuan, Liu, Lai, Wong and Chung2015), while appeared inferior to SRs on Chinese herbal medicine (Chung et al. Reference Chung, Ho, Wu, Fung, Lai, Wu and Wong2015).
Strengths and limitations of this study
This study has comprehensively assessed the methodological quality of SRs on depression treatments using the validated AMSTAR instrument (Shea et al. Reference Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher and Bouter2007, Reference Shea, Hamel, Wells, Bouter, Kristjansson, Grimshaw, Henry and Boers2009). The results on each AMSTAR item were separated so as to inform future areas for improvement, as well as to alert evidence-users on potential bias associated with existing SRs. Meanwhile, our assessment result may be limited by varied reporting quality of the included SRs. For instance, 33.2% of the included SRs did not describe whether duplicate study selection and data extraction were performed, and from the publication alone it is hard to ascertain whether such a procedure has been implemented (Wegewitz et al. Reference Wegewitz, Weikert, Fishta, Jacobs and Pieper2016). Secondly, subgroup analysis and sensitivity analysis, which are important methodological aspects of SRs (Fleming et al. Reference Fleming, Koletsi, Seehra and Pandis2014), are not covered by the AMSTAR, thus, we did not assess those two domains in this study. The addition of these areas to AMSTAR has been proposed (Burda et al. Reference Burda, Holmer and Norris2016).
Implications
Despite a vast and growing body of summary literature relating to treatments for depression and despite a trend for improved reporting quality among SRs, many recently published SRs still do not meet minimum reporting criteria. Clinical decisions may not be made using the highest quality of evidence and subsequently, treatment decisions may be suboptimal for patients. Review authors should adhere to reporting standards for their work and use of these should be enforced by journal editors.
Conclusion
In conclusion, this study demonstrated that the methodological quality of SRs on depression treatments is low and variable. Areas that require urgent improvements include consideration of risk of bias among included studies when formulating conclusions, explicit declaration of conflict of interests, prevention of language and publication biases, using appropriate statistical techniques and ensuring timely updates of existing SRs. It is noteworthy that treatment harms are not reported in one third of the SRs on pharmacological treatments. Readers are encouraged to critically appraise quality of SRs before applying the results in decision-making. Peer reviewers and editors have a crucial role in promoting progress of SR quality by adhering to existing methodological and reporting standards promulgated by AMSTAR and PRISMA.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S2045796017000208.
Acknowledgements
The authors would like to acknowledge the significant contribution of other members of the research team who worked hard to identify, screen and select studies for inclusion in this work and who were also responsible for extracting data from the original manuscripts: CHL Wong, JTM Chow, WKW Cheung, YF Yu, BHK Yip and RWS Sit.
Financial Support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Conflicts of Interest
None.
Authorship statement
All authors listed meet the authorship criteria according to the latest guidelines of the International Committee of Medical Journal Editors. VC and RH involved in study concept and design; YF in literature search. YF and RH helped in literature selection, data extraction and methodological quality assessments. XYW, SW and DT contributed in analysis and interpretation of data. VC and XYW drafted the manuscript. VC, DT, SW and XYW contributed to critical revision of the manuscript for important intellectual content. All authors are in agreement with the manuscript.
Availability of Data and Materials
Data set used for data analysis in this study is available from the corresponding author in request.
