The principle of having the psychiatric nomenclature based on research evidence goes back more than 40 years. In the early 1970s, researchers at Washington University in St Louis developed a set of research diagnostic criteria (called the ‘Feighner criteria’) for 15 diagnostic conditions which they considered had the most evidence of validity based on an empirical review of over 1000 articles. The paper concluded with the statement that ‘what we now present is our synthesis of existing information, a synthesis based on data rather than on opinion and tradition’ (Feighner et al. Reference Feighner, Robins, Guze, Woodruff, Winokur and Munoz1972, p. 62). The Feighner criteria formed the basis for a larger set of research criteria [known as the Research Diagnostic Criteria (RDC); Spitzer et al. Reference Spitzer, Endicott and Robins1978] which were developed as part of a collaborative project on the psychobiology of depression sponsored by the Clinical Research Branch of the NIMH. Diagnostic criteria for 10 additional disorders were added but the limited available evidence necessitated that their development be guided by clinical wisdom rather than empirical data (Spitzer et al. Reference Spitzer, Endicott and Robins1975), setting the precedent for the adoption of an expert clinical consensus model in the development of psychiatric diagnoses, particularly for diagnostic categories for which little to no informative empirical data was available.

From DSM-III onward, successive editions of the DSM have strived to ground the diagnostic definitions in empirical evidence. The Introduction to DSM-III states, for example, that ‘in attempting to resolve diagnostic issues, the [DSM-III] Task Force relied as much as possible on research evidence relevant to various kinds of diagnostic validity’ (APA, 1980, p. 3). The introduction to DSM-III-R goes into even further detail about the role of empirical evidence in the revision process. For example, in describing the criteria for evaluating proposals for revisions, ‘the greatest weight was given to the presence of empirical support from well-conducted research studies’ (APA, 1987, p. xxi). Unfortunately, however, the relatively limited evidence base available to those developing DSM-III and DSM-III-R necessitated relying on a consensus of the experts working on the diagnostic revision. As noted in DSM-III-R, given the lack of empirical data for most proposals, the DSM-III-R advisory committee members gave ‘primary importance … to other considerations such as clinical experience [and] a judgment as to whether the proposal was likely to increase the reliability and validity of the diagnosis under consideration’ (APA, 1987, p. xxi).

DSM-III and DSM-III-R stimulated a substantial amount of research into mental disorder diagnoses. As a result, it was decided that the development of DSM-IV ‘need not rely so exclusively on the consensus of a committee of experts and can instead be more comprehensively informed by, and explicitly based on, empirical data’ (Widiger et al. Reference Widiger, Frances, Pincus, Davis and First1991, p. 282). Consequently, ‘the major emphasis in the development of DSM-IV [was] to maximize the impact of research on the deliberations of the work groups and to document the empirical support for any revisions that [were] implemented’ (Widiger et al. Reference Widiger, Frances, Pincus, Davis and First1991, p. 282). This was accomplished by establishing a three-stage process of empirical review, consisting of comprehensive and systematic literature reviews, secondary analyses of existing data sets, and field trials to provide reliability and validity data for those proposals that were the most substantial or controversial (Frances et al. Reference Frances, Pincus, Widiger, Davis and First1990; Widiger et al. Reference Widiger, Frances, Pincus, Davis and First1991). An explicit requirement of the DSM-IV revision process was that the work group members review the empirical literature ‘to document explicitly the evidence supporting the text and criteria published in DSM-IV and to ensure that the decisions of the Work Groups and Task Force were fully informed by the relevant research’ (Frances et al. Reference Frances, Widiger and Pincus1989, p. 374). The work group members were explicitly instructed that the goal of the literature reviews was not to generate the data to argue for a certain position but rather to provide a fair, balanced, and descriptive summary of the literature, with the ideal review ‘reaching the conclusions that a person with no fixed preconceptions (a consensus scholar) would discover from a comprehensive overview of the entire research literature’ (p. 374). Moreover, to insure that the literature reviews were in fact unbiased and comprehensive, draft versions of the literature reviews were themselves reviewed ‘by all of the Work Group members and a number of advisors to check for gaps in coverage and inaccuracies in the reporting of studies’, with input being solicited in particular ‘from persons who would likely be critical of the conclusions of the review’ (Widiger et al. Reference Widiger, Frances, Pincus and Davis1990, p. 199).

In their description of the DSM-5 empirical review process, Kendler & Solomon (Reference Kendler and Solomon2016), in their article, note that ‘there was no requirement that DSM Work Groups do (or commission) a systematic evidence review. For DSM-5, some Work Groups produced systematic evidence reviews, and some did not.’ This statement echoes Kendler's depiction of the expert consensus process during the DSM-5 revision, in which ‘discussions typically focused on the opinions of individual members about what diagnostic features should be changed. These opinions were based on a wide range of factors including extensive clinical experience of the members, their conceptual differences with the current criteria and relevant research findings. While detailed literature reviews were sometimes commissioned and utilized in deliberations, research results were typically used less systematically, usually to support particular positions advocated by Work Group members’ (Kendler, Reference Kendler2013, pp. 1794–1795). Kendler & Solomon recommend that, moving forward, ‘thorough and objective evidence-based reviews play a central part in the [DSM revision] process’.

However, as is evident in the above description of the DSM-IV empirical review process, the DSM-5 revision represents a significant regression in terms of the centrality of empirical review in the DSM revision process, something which is at odds with the stated goals of DSM-5. For example, in their paper discussing the development of DSM-5, Kupfer et al. (Reference Kupfer, Regier and Kuhl2008) noted that one of the four principles guiding the DSM-5 revision process is that ‘recommendations will be grounded in empirical evidence’ (p. 5). However, as noted by Kendler & Solomon, in contrast to the emphasis on systematic reviews in medicine which is a manifestation of the evidence-based medicine movement, such systematic evidence-based reviews have not been consistently integrated into the development of DSM-5, raising questions about the empirical rigor underlying the DSM-5 revision. Indeed, in their paper reviewing the DSM-5 process, Widiger & Crego (Reference Widiger and Crego2015) point out the inadequately documented empirical basis for a number of DSM-5 changes, including the placement of behavioral addiction disorders in the substance section, the proposed hybrid dimensional-categorical classification of personality disorders, the inclusion of disruptive mood dysregulation disorder, the collapsing of hypochondriasis, pain disorder, and somatization disorder into a single diagnostic category, and changes in the diagnostic criteria for cyclothymic disorder.

Moving forward, what can be done to insure that future DSM revisions ground proposals for change more firmly on empirical evidence? It should be noted that the planned adoption of a continuous improvement model during the (likely lengthy) interval leading up to the next major DSM revision (i.e. DSM-6) should solve this problem for the time being given its requirement that the web-based proposals for changes be accompanied by extensive empirical justifications (First, Reference First2016). In contrast to prior DSM revision efforts, proposals for change will primarily originate from interested persons or organizations that are external to the American Psychiatric Association committee overseeing the continuous revision process. Those proposing significant changes to diagnostic criteria or proposals for new disorders will be required to provide supportive information in a structured format which must include data documenting improvements in validity across a range of validators. Moreover, the DSM Steering Committee has established fairly stringent criteria for approval of proposals in terms of the requirements for supportive empirical data. Thus, submissions for continuous improvements to DSM-5 will, by design, be strongly rooted in solid empirical evidence.

Although planning for the next wholesale DSM revision is likely many years in the future, it is instructive to examine factors that might have been responsible for the reduced focus on empirical data in the DSM-5 revision process. Given that DSM-IV and DSM-5 explicitly shared the goal of having changes grounded in empirical evidence, it appears that the biggest difference concerns the emphasis (or lack thereof) on the importance of conducting a robust empirical review during the revision process. In the many papers describing the DSM-IV revision, the developers of DSM-IV repeatedly stated that the implementation of a comprehensive empirical review process was the main focus of the DSM-IV revision (Frances et al. Reference Frances, Widiger and Pincus1989, Reference Frances, Pincus, Widiger, Davis and First1990; Widiger et al. Reference Widiger, Frances, Pincus and Davis1990, Reference Widiger, Frances, Pincus, Davis and First1991; Widiger & Trull, Reference Widiger, Trull, Costa De Silva and Nadelson1993), culminating in the publication of the four-volume DSM-IV Sourcebook which provided extensive documentation of the empirical review (Widiger et al. Reference Widiger, Frances, Pincus, First, Ross and Davis1994, Reference Widiger, Frances, Pincus, Ross, First and Davis1996, Reference Widiger, Frances, Pincus, Ross, First and Davis1997, Reference Widiger, Frances, Pincus, Ross, First, Davis and Kline1998). Moreover, there were two methods conferences that kicked off the DSM-IV revision which focused on educating members of the Task Force on various aspects of the empirical review process, including procedures for aggregating and analyzing data from different studies to maximize a comprehensive and objective consideration of the empirical evidence as well as procedures to be used in reviewing research, selecting the validators for items and criteria sets, revising criteria sets, and conducting field trials (Widiger et al. Reference Widiger, Frances, Pincus, Davis and First1991).

In contrast, the major focus of the DSM-5 revision effort was on trying to affect a paradigm shift (Kupfer et al. Reference Kupfer, First, Regier, Kupfer, First and Regier2002; Regier et al. Reference Regier, Narrow, Kuhl and Kupfer2009; First, Reference First2010; Kendler & First, Reference Kendler and First2010) in terms of moving the DSM away from its categorical descriptive approach towards a more etiological dimensional approach. Although such a shift ultimately did not occur, it is likely that the effort spent on trying to achieve a paradigm shift came at the expense of focusing on the hard work of conducting systematic reviews of the empirical literature. Besides stating that recommendations for revisions should be guided by research evidence, there is no mention of any details of the empirical review process either in the introduction to DSM-5 or in the few papers about the revision process that were written during the DSM-5 process itself (Kupfer et al. Reference Kupfer, Kuhn and Regier2009; Regier et al. Reference Regier, Narrow, Kuhl and Kupfer2009, Reference Regier, Narrow, Kuhl and Kupfer2011, Reference Regier, Kuhl, Narrow and Kupfer2012). Moreover, the only methods conference conducted in association with DSM-5 development (Kraemer et al. Reference Kraemer, Shrout and Rubio-Stipec2007) was directed towards the participants in the 11 DSM-5 pre-planning conferences (who for the most part were not involved in the development of the DSM-5 itself) and focused on providing ‘attendees with an overview of cutting edge statistical approaches to apply to analyses of existing and emerging data from diverse fields’ (APA, 2004). Finally, a document entitled ‘Guidelines for making changes to DSM-V’ (Kendler et al. Reference Kendler, Kupfer, Narrow, Phillips and Fawcett2009) which provided guidelines for DSM Work Group members regarding the kinds of supportive information required (e.g. the how to use validators to compare the validity of criteria sets) was only made available to DSM-5 Work Group members in 2009, some 2 years into the revision process formally began in 2007.

In the current era of evidence-based medicine, the centerpiece of the DSM must be its reliance on solid empirical data. In order for the DSM to continue to remain credible, it is essential that the developers of future editions of the DSM avoid taking their eyes off the empirical ball and insure that the manual remains grounded in solid empirical evidence.