Parsons, Kruijt, and Fox (Reference Parsons, Kruijt and Fox2019) recently asserted that ‘Cumulative science rests on the foundations of measurements’. Surely any psychological scientist would agree with this axiomatic statement. Indeed, the notion that measurement is foundational would seem especially crucial in psychology, which generally concerns itself with the measurement of constructs, complex concepts that synthesize simpler elements into a cohesive unit to capture processes that are not directly observable but are instead inferred from data. Given the fundamental challenges associated with measuring psychological constructs, critical evaluation of the construct validity of our measures is essential toward avoiding erroneous inferences about human behavior (Flake & Fried, Reference Flake and Fried2020). Crucially, the property of validity cannot be established definitively as present or absent; rather, it is an ongoing question of degree that must be continually revisited based on advances in measurement research and theory (Messick, Reference Messick1987), a daunting complication discussed later.
Construct validity theory has generated an incredibly rich and detailed literature ranging from seminal work establishing the concept (Cronbach & Meehl, Reference Cronbach and Meehl1955) to subsequent work (e.g. Borsboom, Mellenbergh, & van Heerden, Reference Borsboom, Mellenbergh and van Heerden2004; Messick, Reference Messick1987; Sechrest, Reference Sechrest2005) that departs from classical approaches to construct validation. The relevant literature is complex and continues to develop to this day (Flake et al., Reference Flake, Davidson, Wong and Pek2022). Summarizing this work is beyond the scope of the current paper, in which I aim to bridge the divide between measurement theory and research on the etiology of psychopathology by focusing on pragmatic concerns surrounding the validity of putative vulnerabilities to psychopathology and their measurement. Accordingly, I use the pragmatic two-part conceptualization of validity proposed by Borsboom et al. (Reference Borsboom, Mellenbergh and van Heerden2004), who state that (1) a construct must in fact exist as an attribute if it is to be measured and that (2) measurement tools must reflect systematic variation in the attribute to be considered valid. I adopt this account for several reasons: first, I agree with Borsboom et al.'s (Reference Borsboom, Mellenbergh and van Heerden2004) point that it seems likely that most researchers construe construct validity in these more casual terms, rather than thinking about it in the context of formalized theory. Second, this framework highlights the salient issues I hope to address in this manuscript, namely, the need for theory-driven, consensus definitions of vulnerability constructs (i.e. the attributes) and parallel measures that are validated as thoroughly as possible prior to their implementation in etiological model testing. Finally, while this definition distinguishes between the validity of the constructs themselves and the validity of their measures, it highlights the interconnected nature of construct validity and measurement in psychological science.
In this paper, I argue that, for the field to make greater progress in understanding the causes of psychopathology, we must be more systematic and unified in how we conceptualize and measure putative etiological influences, prior to testing these influences in causal models. To make this argument, I briefly discuss the construct validity of our indices of psychopathology, the constructs we aim to predict, as a point of contrast to how we currently approach the measurement of etiological influences. This latter topic is then discussed at length, along with illustrative examples that suggest that the piecemeal approach to assessing vulnerability that currently characterizes the field is a key contributor to poor replicability that cannot be remediated by open science practices. I conclude with several recommendations aimed at advancing progress in our understanding of the development of psychopathology.
Construct validity and measurement in contemporary psychopathology research
Research on the etiology of psychopathology is often concerned with associations between vulnerability constructs and clinical outcomes represented by disorder constructs, which may be diagnoses or symptom clusters. In terms of psychopathological outcomes, disorder constructs, such as major depression, anxiety disorders, and others described in relatively recent editions of the Diagnostic and Statistical Manual of Mental Disorders (DSM) have driven decades of etiological research. The development of the DSM has been described thoroughly elsewhere (Lilienfeld, Smith, & Watts, Reference Lilienfeld, Smith, Watts, Craighead, Miklowitz and Craighead2014); for the current discussion, it is sufficient to know that, beginning with the release of DSM-III (APA, 1980), mental disorders have been formally instantiated by diagnostic algorithms that draw upon standardized diagnostic criteria reflecting the signs and symptoms thought pertinent to each disorder. The codification of diagnostic criteria in the DSM-III, alongside the development of structured interview-based measures designed to minimize information variance between clinicians, was done to provide a much-needed common language for research on the etiology and treatment of mental disorders as well as their assessment (Lilienfeld & Treadway, Reference Lilienfeld and Treadway2016).
A large body of research indicates the fallibility of DSM constructs in terms of their construct validity. Some key problems with diagnostic constructs include their failure to map clearly onto etiological factors as well as their misrepresentation of what are primarily dimensional symptom phenomena as discrete conditions reflecting states of health v. illness (Hyman, Reference Hyman2021). Even widely studied disorder constructs such as depression are subject to ongoing debate regarding their validity (Fried, Flake, & Robinaugh, Reference Fried, Flake and Robinaugh2022; Kendler, Reference Kendler2016, Reference Kendler2017). Nevertheless, an unsystematic review of relevant literature suggests that the field continues to rely heavily on DSM constructs and symptom content, despite their flaws. Perhaps ironically, many studies documenting the flaws of the DSM approach cite the disease burden associated with these disorders as defined by the DSM to motivate improvements in their construct validity (e.g. Fried et al., Reference Fried, Flake and Robinaugh2022). Models developed as alternatives to the DSM approach, such as the hierarchical taxonomy of psychopathology (HiTOP, Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017) and network-based models (e.g. Roefs et al., Reference Roefs, Fried, Kindt, Martijn, Elzinga, Evers and Jansen2022), nevertheless use DSM symptoms as their foundation.
Clearly, despite significant efforts to move away from DSM disorder constructs, we continue to rely heavily on them. Why might this be? Significant weaknesses aside, contemporary versions of the DSM and the measurement initiatives that accompanied them (Spitzer, Reference Spitzer1983) provided a foundation for the study of psychopathology etiology by operationalizing the outcomes of interest as disorder constructs and by providing parallel instrumentation designed to assess these constructs. The result is that, despite pervasive skepticism regarding the DSM, we know more about how putative vulnerability constructs relate to diagnoses and diagnostic symptoms than other, potentially more valid disorder constructs.
Variation in definitions and measures of vulnerability constructs leads to mixed findings: some examples
Unfortunately, efforts made to unify the definition and measurement of disorder constructs stand in sharp contrast to how our field has approached vulnerability constructs, which have not generally been subject to a fulsome discussion of their defining features or their measurement prior to their inclusion in etiological models. Examples include constructs such as resilience, stress sensitivity, self-regulation, and emotion regulation, highly complex constructs that generally lack a working consensus definition. For example, in developmental psychopathology, we lack a strong consensus regarding how to best conceptualize resilience, which has been treated as a personal trait, a process, and an outcome, depending on the study in question (Hayden & Durbin, Reference Hayden, Durbin, Ollendick, White and White2019; Hayden & Harkness, Reference Hayden, Harkness, Harkness and Hayden2020). In the case of emotion regulation, embedded within this construct is the notion of an emotional response which is then consciously or unconsciously subjected to other processes in service of either mitigating or intensifying the initial emotional reaction (Campos, Frankel, & Camras, Reference Campos, Frankel and Camras2004). Thus, according to some authors (Cole, Martin, & Dennis, Reference Cole, Martin and Dennis2004; see also Campos et al., Reference Campos, Frankel and Camras2004) the construct itself is comprised of multiple, presumably somewhat orthogonal facets. There may be widespread, expert consensus regarding the existence of these constructs [attributes, in Borsboom et al.'s (Reference Borsboom, Mellenbergh and van Heerden2004) terms] and that they are potentially important determinants of human behavior and adaptation; in fact, a large research literature may have emerged concerning the causal roles these constructs play in typical and atypical development, despite the lack of consensus on their definition and the absence of well-validated measures.
The large literature on the role of stress and depression provides another example of how measurement challenges complicate our ability to draw causal conclusions. The term stress has been used to refer both to the environmental stressor itself and individual differences in responses to it (Harkness & Monroe, Reference Harkness and Monroe2016). This is problematic in and of itself; however, even when researchers clearly specify that their usage of the term refers specifically to stress exposures, how these are measured poses significant challenges to understanding the causal role of stress in depression. Specifically, methods that rely solely on self-reported life event checklists conflate exposure to stress with individual difference factors, such as trait neuroticism or depression itself, that are related to stress perception and reactivity. The implications of this misspecification for both theory and intervention are striking, as this lack of measurement precision means that we are unable to parse environmental from within-person influences on depression. While individual studies can address this issue through measurement strategies equipped to disentangle life events from individuals' responses to them, this is insufficient given that we draw conclusions regarding the nature of psychopathology etiology based on a cumulative literature, a literature currently rife with wide variability in how rigorously stress is defined and assessed.
Attempts to explain disparate findings in studies of vulnerability to psychopathology have often focused on developmental and environmental moderators of associations between vulnerabilities and outcomes. The development of psychopathology is inherently complex, so associations between vulnerabilities and psychopathology are almost certainly moderated by other factors, such as other within-person characteristics, environmental influences, and so on. However, it is impossible to disentangle the developmental, environmental, and other substantive influences that contribute to multifinality from the influence of variability in how vulnerabilities are conceptualized and measured in different studies. Even though psychopathology researchers may not realize it, we have apparently glossed over the substantive and structural phases of construct validation in our rush to the external stage (Flake, Pek, & Hehman, Reference Flake, Pek and Hehman2017; Loevinger, Reference Loevinger1957), in which predictive validity for criterion variables, clinical outcomes in the current context, is emphasized. As noted elsewhere (Flake et al., Reference Flake, Pek and Hehman2017), conclusions made in the external stage may not be valid if the substantive and structural phases have been neglected.
Significantly, these initial stages of validity evidence are accomplished through careful consideration of definitional issues (i.e. the substantive phase) and through thorough psychometric scrutiny of the properties of relevant measures of the construct (i.e. the structural phase). Limited measurement validity constrains even the most sophisticated studies of etiology as it cannot be addressed through statistical methods applied after the fact (Flake et al., 2022; Westfall & Yarkoni, Reference Westfall and Yarkoni2016). For example, as discussed by Westfall and Yarkoni (Reference Westfall and Yarkoni2016), measurement (un)reliability poses serious constraints on the ability to address potential confounds through covariates. Similar measurement-related complexities exist in the context of tests of mediation, which are frequently used to test causal models of psychopathology (Bullock, Green, & Ha, Reference Bullock, Green and Ha2010). The conclusion is that enthusiasm for integrating vulnerability constructs in tests of causal models of psychopathology has outpaced efforts to define and measure them well.
Current efforts to strengthen psychological science are incomplete solutions
The poor progress made with respect to discovering causes of psychopathology is likely partially related to undesirable research practices outside the realm of measurement concerns. Much has been written in the last decade or so, in the context of the open science movement, about the failure of psychology to produce reliable findings (e.g. Simmons, Nelson, & Simonsohn, Reference Simmons, Nelson and Simonsohn2011) in domains such as social priming, ego depletion, and facial feedback (see Świątkowski & Dompnier, Reference Świątkowski and Dompnier2017, for a review). In response, psychological scientists have advocated for increased transparency in research practices, including preregistration, in which research questions, study design, and data analytic plans are made publicly available prior to conducting the study in question. The goal is to reduce ‘researcher degrees of freedom’, or to constrain the extent to which data analytic and other decisions are made based not on scientific principles but in service of producing statistically significant findings.
Open science initiatives are undoubtedly useful tools toward building a more robust understanding of the causes of psychopathology. However, these are partial solutions. Crucially, open science practices as they are currently implemented are most compatible with hypothesis-driven research (Thompson, Wright, & Bissett, Reference Thompson, Wright and Bissett2020). However, well-specified hypotheses are grounded in a cohesive, robust research literature, which has very little chance of developing unless the constructs under investigation have been measured reliably and validly. Researcher ‘degrees of freedom’ and other potentially problematic practices unfold after measures have been chosen and data have been collected. However, if the foundation of our science truly rests on its measures, attending to the reliability and validity of our measures prior to data collection is crucial (Flake et al., Reference Flake, Pek and Hehman2017).
Making progress toward understanding vulnerability to psychopathology: recommendations
In the sections that follow, I provide recommendations aimed at addressing the problems of the lack of consensus regarding (1) the nature of vulnerability constructs and (2) how to best measure them. I acknowledge that I have not described these solutions in sufficient depth for them to be implemented, due in part to limitations of space and of my own expertise. More importantly, however, is the fact that useful solutions will require the efforts of multidisciplinary teams supported by leading scientific organizations in the field, as well as funding agencies that support research on psychopathology etiologies (Table 1).
Table 1.Making progress toward understanding vulnerability to psychopathology: obstacles and potential solutions
Develop theory-driven, working consensus definitions of vulnerability constructs
The field would benefit greatly from consensus statements regarding current knowledge of specific vulnerability constructs (Sechrest, Reference Sechrest2005). This would seem possible, given that the field has a history of developing consensus approaches to constructs, including the DSM-III and the Research Domain Criteria (RDoC; Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010) initiative. Such consensus statements could emerge through ‘white papers’, consortium research groups (e.g. HiTOP; Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017), through organized efforts led by leading scientific organizations in the field, as was done in the case of the DSM-III (Lilienfeld & Treadway, Reference Lilienfeld and Treadway2016), or through government-funded initiatives like those that supported the development of RDoC (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010). These efforts would need to not only fully describe the constructs of interest but to also position these constructs in terms of their interrelations with other variables (e.g. Flake et al., Reference Flake, Pek and Hehman2017), thereby laying the groundwork for validation research involving data collection. These definitions should be regarded as fallible ‘works in progress’ to be revisited periodically based on the accrual of relevant data. As noted by Flake and Fried (Reference Flake and Fried2020), we need transparency in our construct definitions and operationalizations for many reasons, including the need to reduce the problem of jingle-jangle fallacies,Footnote †Footnote 1 phenomena that can greatly reduce clarity regarding interrelationships between constructs and lead to misspecification of causal models.
We have an abundance of vulnerability constructs that would clearly benefit from intensive conceptual discussion. Thus, setting priorities may be useful. We might first focus on improving our measures of constructs that seem to have large or at least consistent effect sizes for multiple psychopathological outcomes of interest, such as the early environment, given that relevant measures are presumably more tolerant of measurement error given a strong effect size. For example, the development and optimization of multimethod measures of early environmental factors, such as caregiving and other early environmental variables (e.g. the home environment itself), would be of great benefit to the field. Perhaps surprisingly, although there may be good individual measures of early care during a specific developmental period, we lack well-vetted, multimethod measures of the environment that cover the broader developmental period of childhood and early adolescence,Footnote 2 important periods for the development of psychopathology. Widely used, older measures (e.g. Robinson, Mandleco, Olsen, & Hart, Reference Robinson, Mandleco, Olsen and Hart1995) need updating and reexamination of their psychometric properties, especially given societal shifts in familial roles, norms, and home environments.
If measuring the environment should be a priority, what might we wish to de-emphasize, at least for the time being? This is controversial, but we might reconsider our embrace of at least some biological indices of risk, given their unknown or questionable psychometric properties, cost, complexity, and the potentially small effect size of these constructs for predicting human behavior. In the rush to embrace novel technologies as tools for etiological discovery, and in our tendency to reify biological levels of analysis (Berenbaum, Reference Berenbaum2013), we may be neglecting the study of constructs that provide the most traction for preventative and early intervention efforts.
Develop and validate a multimethod toolbox for assessing vulnerability constructs
The development of working consensus definitions for putatively etiologically relevant constructs should be done alongside the selection and development of parallel, multimethod measures of constructs that can be widely used across research groups.Footnote 3 The determination of criteria for measure inclusion can draw upon existing frameworks for validity [e.g. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, & NCME), 2014; Campbell & Fiske, Reference Campbell and Fiske1959; Feighner et al., Reference Feighner, Robins, Guze, Woodruff, Winokur and Munoz1972]. Rather than developing a static list of measures that are used without further thought or scrutiny, the primary goal of this initiative would be to identify and facilitate the use of promising measurement tools to lay the groundwork for additional measurement improvements based on emerging data. Instead of ‘starting from scratch’, we may wish to reexamine the psychometric properties of older measurement tools that appear valid based on previous work, using modern standards for enhancing their reliability and validity (e.g. as has been done with the dot-probe paradigm; Evans & Britton, Reference Evans and Britton2018). Although we should avoid conflating measures with the constructs themselves, the process of construct validation entails both theory validation and measure validation (Strauss & Smith, Reference Strauss and Smith2009), such that one cannot be validated without the other. Thus, this measurement work should occur alongside the conceptual definitional work proposed in my first recommendation.
While this approach could be seen as having the potential to stifle measurement innovation, there should be provisions made for investigator-driven innovations in developing new measures that can be added to the set of recommended measures based on evidence of their validity. The development of a set of recommended measures would also free up investigators who study etiological influences (who may not be experts in quantitative methods or measurement development) to focus on innovations in other aspects of study design. Investigators would be free to use measures with limited validity evidence, but the existence of a list of recommended measures would ideally motivate serious consideration and justification of such decisions.
Develop open-access databases that serve as repositories for granular measurement data that can be used to evaluate and improve extant measures
As noted previously, validity is a conditional phenomenon, not a static property of a measure. If taken seriously, this is daunting to investigators who are not theorists or experts in quantitative methods; for example, the measurement invariance of relevant measures in a given population (e.g. based on gender; Klein Haneveld, Molenaar, de Vogel, Smid, & Kamphuis, Reference Klein Haneveld, Molenaar, de Vogel, Smid and Kamphuis2022) would need to be established prior to their implementation in a study of psychopathology. We should maintain standards in our expectations that investigators report at least minimum validity evidence for their measures (Flake et al., 2022); however, it is probably unrealistic to expect that psychopathologists interested in implementing an index of a vulnerability process in a study of etiology will halt their research programs to do a thoroughgoing scrutiny of their measures, especially given that they may lack the relevant expertise and the options for moving ahead with their research are unclear, should a measure not hold up under close scrutiny. However, given current appreciation for the benefits of data-sharing initiatives, an open-access database that provided granular data, ideally from studies that use the consensus-based, multimethod measures of vulnerabilities as described in the previous recommendation, could be an extremely valuable resource for trainees interested in measurement and quantitative researchers.
Incentivize ‘pre-clinical’ research on measurement development
In psychopathology research, longitudinal methods that study individuals with vulnerabilities prior to the onset of disorder are often viewed as superior to cross-sectional work, which cannot speak clearly to causal processes. However, as noted earlier, longitudinal work often skips over the admittedly less glamorous phases of substantive and structural validity, phases which are nevertheless essential toward generating robust findings. Thus, from a measurement development standpoint, there is much to be gained from descriptive, cross-sectional studies that include clinically ill individuals (i.e. those with diagnoses) as a means of delving deeply into the structural phase of construct validation (Loevinger, Reference Loevinger1957). This approach facilitates the use of psychopathology symptoms as a scaffold for the iterative development of better measures of relevant vulnerability processes (i.e. those with probable etiological significance) using modern standards for psychometric investigations. Individuals with clinical disorder clearly have ‘risk’ for that disorder and therefore have relevant vulnerabilities that can be characterized via intensive measurement development. This is not to disregard the very real limitations of cross-sectional data to identify etiological processes that unfold over time (Markus & Borsboom, Reference Markus and Borsboom2013), but to identify a reasonably parsimonious place from which to begin this important research. If we develop and refine multimethod assessment tools that validly characterize vulnerabilities with strong predictive power for symptoms and clinically significant impairment, we will have a much stronger foundation for the study of etiological processes that rely on longitudinal methods and experimental manipulations to better establish causality.
As an initial step in this process, we might focus on the characterization of individuals with prototypical manifestations of disorder constructs; while doing so admittedly reduces generalizability, measurement development will benefit from greater homogeneity of samples, permitting stronger claims regarding the nature of putative vulnerabilities relevant to specific disorders. Diagnostic phenotypes, while admittedly imperfect, are still the best understood phenotypes that we have in psychopathology; thus, the relevant literature, mixed though it may be, is better positioned to provide initial clues regarding which vulnerabilities should be looked at and how they might be optimally measured. Furthermore, if part of the problem with DSM diagnoses is that they do not map clearly onto etiology, it could be that improving our measures of putative etiological influences will ultimately help improve diagnoses (Cronbach & Meehl, Reference Cronbach and Meehl1955).
Some might argue that the RDoC (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010) provide a better vantage point than DSM symptoms and diagnoses for the study of etiological mechanisms, given that they were designed to map more closely onto etiological processes than diagnosis does. Unfortunately, there is evidence that the RDoC initiative has not been as useful of a framework for the study of etiology as we might have hoped. For example, the theoretical domains articulated within RDoC do not cohere following the hypothesized latent structure (e.g. positive and negative valence systems; Olino, McMakin, & Forbes, Reference Olino, McMakin and Forbes2018; Peng et al., Reference Peng, Knotts, Taylor, Craske, Stein, Bookheimer and Paulus2021). While RDoC domains and indicators were subject to significant discussion regarding their definition, these decisions were not embedded within psychometric consideration of measures of RDoC content. Ironically, we might ultimately gain better traction in attaining RDoC's goal of delineating etiological processes if we used clinical symptoms from the DSM as the scaffold for the development of good measures of RDoC constructs.
On a smaller scale, as reviewers of grants and manuscripts, we should be cognizant of the limitations of research on psychopathology etiology that lacks a firm measurement foundation, as well as the merits of research focused on measurement development, and evaluate research accordingly. Sechrest (Reference Sechrest2005) noted the importance of training in measurement in both graduate and continuing education domains; this point is still relevant, and the current state of the field may reflect limitations in awareness of the centrality of validity and how to evaluate it when reviewing grants and manuscripts. In evaluating research on psychopathology, a premium should be placed on the methodological strengths of research and on the provision of validity evidence for measures of constructs of interest (Flake et al., Reference Flake, Pek and Hehman2017). Rewarding measurement-intensive research may help curtail the practice of developing ‘ad-hoc’ measures to use in specific studies (see Flake & Fried, Reference Flake and Fried2020, for a discussion of ‘on the fly’ measurement), which causes problems by leading to the proliferation of new measures that are used by only a handful of research groups, hampering our ability to synthesize research literatures.
Explore the use of technological innovations that facilitate the collection of quality data
Earlier, I noted the need for multimethod approaches to measuring important constructs. While the literature on child development has a strong tradition of including observation approaches, many studies of both youth and adults rely heavily on questionnaire measures rather than observational data, despite their limitations (Durbin & Wilson, Reference Durbin and Wilson2012). This is likely due to the very real expense of collecting and coding observational data, which may require aggregation over many samples of behavior to yield reliable estimates (Epstein, Reference Epstein1983). Technological advances are increasingly capable of reducing the expense of observational data; for example, automated facial affect and linguistic coding are increasingly feasible, albeit less well developed for use with children. Such methods may have the potential to reduce the significant burden associated with observational data (Iddiols, Daoust, Mullen, & Hayden, Reference Iddiols, Daoust, Mullen and Hayden2022). While there is much work to be done toward validating these tools, to the extent research groups can collect high-quality observational data, it will be increasingly possible to offset the expense associated with these data by using automated methods of coding observational data. Conversely, automated approaches cannot salvage studies that use measures without construct validity; as noted recently by Flake et al. (2022), measurement decisions are foundational to data quality and the construct validity of poor measures cannot be salvaged after the fact.
Conclusion
Progress in understanding vulnerability to psychopathology has been stymied by an unsystematic approach to defining, measuring, and validating putative vulnerability factors. The complexity inherent to the development of psychopathology indicates that systematic development of consensus measures of vulnerabilities should occur prior to their implementation in etiological models. Although this approach will require significant collaboration and investment of time and resources, it should ultimately yield greater progress than the current approach, in which oftentimes underdeveloped constructs and measures are embedded within complex causal models. Returning our attention to issues of construct validity and measurement development in service of refining our indices of vulnerability is crucial toward building a robust understanding of the causes of psychopathology.
You have
Access
Parsons, Kruijt, and Fox (Reference Parsons, Kruijt and Fox2019) recently asserted that ‘Cumulative science rests on the foundations of measurements’. Surely any psychological scientist would agree with this axiomatic statement. Indeed, the notion that measurement is foundational would seem especially crucial in psychology, which generally concerns itself with the measurement of constructs, complex concepts that synthesize simpler elements into a cohesive unit to capture processes that are not directly observable but are instead inferred from data. Given the fundamental challenges associated with measuring psychological constructs, critical evaluation of the construct validity of our measures is essential toward avoiding erroneous inferences about human behavior (Flake & Fried, Reference Flake and Fried2020). Crucially, the property of validity cannot be established definitively as present or absent; rather, it is an ongoing question of degree that must be continually revisited based on advances in measurement research and theory (Messick, Reference Messick1987), a daunting complication discussed later.
Construct validity theory has generated an incredibly rich and detailed literature ranging from seminal work establishing the concept (Cronbach & Meehl, Reference Cronbach and Meehl1955) to subsequent work (e.g. Borsboom, Mellenbergh, & van Heerden, Reference Borsboom, Mellenbergh and van Heerden2004; Messick, Reference Messick1987; Sechrest, Reference Sechrest2005) that departs from classical approaches to construct validation. The relevant literature is complex and continues to develop to this day (Flake et al., Reference Flake, Davidson, Wong and Pek2022). Summarizing this work is beyond the scope of the current paper, in which I aim to bridge the divide between measurement theory and research on the etiology of psychopathology by focusing on pragmatic concerns surrounding the validity of putative vulnerabilities to psychopathology and their measurement. Accordingly, I use the pragmatic two-part conceptualization of validity proposed by Borsboom et al. (Reference Borsboom, Mellenbergh and van Heerden2004), who state that (1) a construct must in fact exist as an attribute if it is to be measured and that (2) measurement tools must reflect systematic variation in the attribute to be considered valid. I adopt this account for several reasons: first, I agree with Borsboom et al.'s (Reference Borsboom, Mellenbergh and van Heerden2004) point that it seems likely that most researchers construe construct validity in these more casual terms, rather than thinking about it in the context of formalized theory. Second, this framework highlights the salient issues I hope to address in this manuscript, namely, the need for theory-driven, consensus definitions of vulnerability constructs (i.e. the attributes) and parallel measures that are validated as thoroughly as possible prior to their implementation in etiological model testing. Finally, while this definition distinguishes between the validity of the constructs themselves and the validity of their measures, it highlights the interconnected nature of construct validity and measurement in psychological science.
In this paper, I argue that, for the field to make greater progress in understanding the causes of psychopathology, we must be more systematic and unified in how we conceptualize and measure putative etiological influences, prior to testing these influences in causal models. To make this argument, I briefly discuss the construct validity of our indices of psychopathology, the constructs we aim to predict, as a point of contrast to how we currently approach the measurement of etiological influences. This latter topic is then discussed at length, along with illustrative examples that suggest that the piecemeal approach to assessing vulnerability that currently characterizes the field is a key contributor to poor replicability that cannot be remediated by open science practices. I conclude with several recommendations aimed at advancing progress in our understanding of the development of psychopathology.
Construct validity and measurement in contemporary psychopathology research
Research on the etiology of psychopathology is often concerned with associations between vulnerability constructs and clinical outcomes represented by disorder constructs, which may be diagnoses or symptom clusters. In terms of psychopathological outcomes, disorder constructs, such as major depression, anxiety disorders, and others described in relatively recent editions of the Diagnostic and Statistical Manual of Mental Disorders (DSM) have driven decades of etiological research. The development of the DSM has been described thoroughly elsewhere (Lilienfeld, Smith, & Watts, Reference Lilienfeld, Smith, Watts, Craighead, Miklowitz and Craighead2014); for the current discussion, it is sufficient to know that, beginning with the release of DSM-III (APA, 1980), mental disorders have been formally instantiated by diagnostic algorithms that draw upon standardized diagnostic criteria reflecting the signs and symptoms thought pertinent to each disorder. The codification of diagnostic criteria in the DSM-III, alongside the development of structured interview-based measures designed to minimize information variance between clinicians, was done to provide a much-needed common language for research on the etiology and treatment of mental disorders as well as their assessment (Lilienfeld & Treadway, Reference Lilienfeld and Treadway2016).
A large body of research indicates the fallibility of DSM constructs in terms of their construct validity. Some key problems with diagnostic constructs include their failure to map clearly onto etiological factors as well as their misrepresentation of what are primarily dimensional symptom phenomena as discrete conditions reflecting states of health v. illness (Hyman, Reference Hyman2021). Even widely studied disorder constructs such as depression are subject to ongoing debate regarding their validity (Fried, Flake, & Robinaugh, Reference Fried, Flake and Robinaugh2022; Kendler, Reference Kendler2016, Reference Kendler2017). Nevertheless, an unsystematic review of relevant literature suggests that the field continues to rely heavily on DSM constructs and symptom content, despite their flaws. Perhaps ironically, many studies documenting the flaws of the DSM approach cite the disease burden associated with these disorders as defined by the DSM to motivate improvements in their construct validity (e.g. Fried et al., Reference Fried, Flake and Robinaugh2022). Models developed as alternatives to the DSM approach, such as the hierarchical taxonomy of psychopathology (HiTOP, Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017) and network-based models (e.g. Roefs et al., Reference Roefs, Fried, Kindt, Martijn, Elzinga, Evers and Jansen2022), nevertheless use DSM symptoms as their foundation.
Clearly, despite significant efforts to move away from DSM disorder constructs, we continue to rely heavily on them. Why might this be? Significant weaknesses aside, contemporary versions of the DSM and the measurement initiatives that accompanied them (Spitzer, Reference Spitzer1983) provided a foundation for the study of psychopathology etiology by operationalizing the outcomes of interest as disorder constructs and by providing parallel instrumentation designed to assess these constructs. The result is that, despite pervasive skepticism regarding the DSM, we know more about how putative vulnerability constructs relate to diagnoses and diagnostic symptoms than other, potentially more valid disorder constructs.
Variation in definitions and measures of vulnerability constructs leads to mixed findings: some examples
Unfortunately, efforts made to unify the definition and measurement of disorder constructs stand in sharp contrast to how our field has approached vulnerability constructs, which have not generally been subject to a fulsome discussion of their defining features or their measurement prior to their inclusion in etiological models. Examples include constructs such as resilience, stress sensitivity, self-regulation, and emotion regulation, highly complex constructs that generally lack a working consensus definition. For example, in developmental psychopathology, we lack a strong consensus regarding how to best conceptualize resilience, which has been treated as a personal trait, a process, and an outcome, depending on the study in question (Hayden & Durbin, Reference Hayden, Durbin, Ollendick, White and White2019; Hayden & Harkness, Reference Hayden, Harkness, Harkness and Hayden2020). In the case of emotion regulation, embedded within this construct is the notion of an emotional response which is then consciously or unconsciously subjected to other processes in service of either mitigating or intensifying the initial emotional reaction (Campos, Frankel, & Camras, Reference Campos, Frankel and Camras2004). Thus, according to some authors (Cole, Martin, & Dennis, Reference Cole, Martin and Dennis2004; see also Campos et al., Reference Campos, Frankel and Camras2004) the construct itself is comprised of multiple, presumably somewhat orthogonal facets. There may be widespread, expert consensus regarding the existence of these constructs [attributes, in Borsboom et al.'s (Reference Borsboom, Mellenbergh and van Heerden2004) terms] and that they are potentially important determinants of human behavior and adaptation; in fact, a large research literature may have emerged concerning the causal roles these constructs play in typical and atypical development, despite the lack of consensus on their definition and the absence of well-validated measures.
The large literature on the role of stress and depression provides another example of how measurement challenges complicate our ability to draw causal conclusions. The term stress has been used to refer both to the environmental stressor itself and individual differences in responses to it (Harkness & Monroe, Reference Harkness and Monroe2016). This is problematic in and of itself; however, even when researchers clearly specify that their usage of the term refers specifically to stress exposures, how these are measured poses significant challenges to understanding the causal role of stress in depression. Specifically, methods that rely solely on self-reported life event checklists conflate exposure to stress with individual difference factors, such as trait neuroticism or depression itself, that are related to stress perception and reactivity. The implications of this misspecification for both theory and intervention are striking, as this lack of measurement precision means that we are unable to parse environmental from within-person influences on depression. While individual studies can address this issue through measurement strategies equipped to disentangle life events from individuals' responses to them, this is insufficient given that we draw conclusions regarding the nature of psychopathology etiology based on a cumulative literature, a literature currently rife with wide variability in how rigorously stress is defined and assessed.
Attempts to explain disparate findings in studies of vulnerability to psychopathology have often focused on developmental and environmental moderators of associations between vulnerabilities and outcomes. The development of psychopathology is inherently complex, so associations between vulnerabilities and psychopathology are almost certainly moderated by other factors, such as other within-person characteristics, environmental influences, and so on. However, it is impossible to disentangle the developmental, environmental, and other substantive influences that contribute to multifinality from the influence of variability in how vulnerabilities are conceptualized and measured in different studies. Even though psychopathology researchers may not realize it, we have apparently glossed over the substantive and structural phases of construct validation in our rush to the external stage (Flake, Pek, & Hehman, Reference Flake, Pek and Hehman2017; Loevinger, Reference Loevinger1957), in which predictive validity for criterion variables, clinical outcomes in the current context, is emphasized. As noted elsewhere (Flake et al., Reference Flake, Pek and Hehman2017), conclusions made in the external stage may not be valid if the substantive and structural phases have been neglected.
Significantly, these initial stages of validity evidence are accomplished through careful consideration of definitional issues (i.e. the substantive phase) and through thorough psychometric scrutiny of the properties of relevant measures of the construct (i.e. the structural phase). Limited measurement validity constrains even the most sophisticated studies of etiology as it cannot be addressed through statistical methods applied after the fact (Flake et al., 2022; Westfall & Yarkoni, Reference Westfall and Yarkoni2016). For example, as discussed by Westfall and Yarkoni (Reference Westfall and Yarkoni2016), measurement (un)reliability poses serious constraints on the ability to address potential confounds through covariates. Similar measurement-related complexities exist in the context of tests of mediation, which are frequently used to test causal models of psychopathology (Bullock, Green, & Ha, Reference Bullock, Green and Ha2010). The conclusion is that enthusiasm for integrating vulnerability constructs in tests of causal models of psychopathology has outpaced efforts to define and measure them well.
Current efforts to strengthen psychological science are incomplete solutions
The poor progress made with respect to discovering causes of psychopathology is likely partially related to undesirable research practices outside the realm of measurement concerns. Much has been written in the last decade or so, in the context of the open science movement, about the failure of psychology to produce reliable findings (e.g. Simmons, Nelson, & Simonsohn, Reference Simmons, Nelson and Simonsohn2011) in domains such as social priming, ego depletion, and facial feedback (see Świątkowski & Dompnier, Reference Świątkowski and Dompnier2017, for a review). In response, psychological scientists have advocated for increased transparency in research practices, including preregistration, in which research questions, study design, and data analytic plans are made publicly available prior to conducting the study in question. The goal is to reduce ‘researcher degrees of freedom’, or to constrain the extent to which data analytic and other decisions are made based not on scientific principles but in service of producing statistically significant findings.
Open science initiatives are undoubtedly useful tools toward building a more robust understanding of the causes of psychopathology. However, these are partial solutions. Crucially, open science practices as they are currently implemented are most compatible with hypothesis-driven research (Thompson, Wright, & Bissett, Reference Thompson, Wright and Bissett2020). However, well-specified hypotheses are grounded in a cohesive, robust research literature, which has very little chance of developing unless the constructs under investigation have been measured reliably and validly. Researcher ‘degrees of freedom’ and other potentially problematic practices unfold after measures have been chosen and data have been collected. However, if the foundation of our science truly rests on its measures, attending to the reliability and validity of our measures prior to data collection is crucial (Flake et al., Reference Flake, Pek and Hehman2017).
Making progress toward understanding vulnerability to psychopathology: recommendations
In the sections that follow, I provide recommendations aimed at addressing the problems of the lack of consensus regarding (1) the nature of vulnerability constructs and (2) how to best measure them. I acknowledge that I have not described these solutions in sufficient depth for them to be implemented, due in part to limitations of space and of my own expertise. More importantly, however, is the fact that useful solutions will require the efforts of multidisciplinary teams supported by leading scientific organizations in the field, as well as funding agencies that support research on psychopathology etiologies (Table 1).
Making progress toward understanding vulnerability to psychopathology: obstacles and potential solutions
Develop theory-driven, working consensus definitions of vulnerability constructs
The field would benefit greatly from consensus statements regarding current knowledge of specific vulnerability constructs (Sechrest, Reference Sechrest2005). This would seem possible, given that the field has a history of developing consensus approaches to constructs, including the DSM-III and the Research Domain Criteria (RDoC; Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010) initiative. Such consensus statements could emerge through ‘white papers’, consortium research groups (e.g. HiTOP; Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017), through organized efforts led by leading scientific organizations in the field, as was done in the case of the DSM-III (Lilienfeld & Treadway, Reference Lilienfeld and Treadway2016), or through government-funded initiatives like those that supported the development of RDoC (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010). These efforts would need to not only fully describe the constructs of interest but to also position these constructs in terms of their interrelations with other variables (e.g. Flake et al., Reference Flake, Pek and Hehman2017), thereby laying the groundwork for validation research involving data collection. These definitions should be regarded as fallible ‘works in progress’ to be revisited periodically based on the accrual of relevant data. As noted by Flake and Fried (Reference Flake and Fried2020), we need transparency in our construct definitions and operationalizations for many reasons, including the need to reduce the problem of jingle-jangle fallacies,Footnote †Footnote 1 phenomena that can greatly reduce clarity regarding interrelationships between constructs and lead to misspecification of causal models.
We have an abundance of vulnerability constructs that would clearly benefit from intensive conceptual discussion. Thus, setting priorities may be useful. We might first focus on improving our measures of constructs that seem to have large or at least consistent effect sizes for multiple psychopathological outcomes of interest, such as the early environment, given that relevant measures are presumably more tolerant of measurement error given a strong effect size. For example, the development and optimization of multimethod measures of early environmental factors, such as caregiving and other early environmental variables (e.g. the home environment itself), would be of great benefit to the field. Perhaps surprisingly, although there may be good individual measures of early care during a specific developmental period, we lack well-vetted, multimethod measures of the environment that cover the broader developmental period of childhood and early adolescence,Footnote 2 important periods for the development of psychopathology. Widely used, older measures (e.g. Robinson, Mandleco, Olsen, & Hart, Reference Robinson, Mandleco, Olsen and Hart1995) need updating and reexamination of their psychometric properties, especially given societal shifts in familial roles, norms, and home environments.
If measuring the environment should be a priority, what might we wish to de-emphasize, at least for the time being? This is controversial, but we might reconsider our embrace of at least some biological indices of risk, given their unknown or questionable psychometric properties, cost, complexity, and the potentially small effect size of these constructs for predicting human behavior. In the rush to embrace novel technologies as tools for etiological discovery, and in our tendency to reify biological levels of analysis (Berenbaum, Reference Berenbaum2013), we may be neglecting the study of constructs that provide the most traction for preventative and early intervention efforts.
Develop and validate a multimethod toolbox for assessing vulnerability constructs
The development of working consensus definitions for putatively etiologically relevant constructs should be done alongside the selection and development of parallel, multimethod measures of constructs that can be widely used across research groups.Footnote 3 The determination of criteria for measure inclusion can draw upon existing frameworks for validity [e.g. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA, APA, & NCME), 2014; Campbell & Fiske, Reference Campbell and Fiske1959; Feighner et al., Reference Feighner, Robins, Guze, Woodruff, Winokur and Munoz1972]. Rather than developing a static list of measures that are used without further thought or scrutiny, the primary goal of this initiative would be to identify and facilitate the use of promising measurement tools to lay the groundwork for additional measurement improvements based on emerging data. Instead of ‘starting from scratch’, we may wish to reexamine the psychometric properties of older measurement tools that appear valid based on previous work, using modern standards for enhancing their reliability and validity (e.g. as has been done with the dot-probe paradigm; Evans & Britton, Reference Evans and Britton2018). Although we should avoid conflating measures with the constructs themselves, the process of construct validation entails both theory validation and measure validation (Strauss & Smith, Reference Strauss and Smith2009), such that one cannot be validated without the other. Thus, this measurement work should occur alongside the conceptual definitional work proposed in my first recommendation.
While this approach could be seen as having the potential to stifle measurement innovation, there should be provisions made for investigator-driven innovations in developing new measures that can be added to the set of recommended measures based on evidence of their validity. The development of a set of recommended measures would also free up investigators who study etiological influences (who may not be experts in quantitative methods or measurement development) to focus on innovations in other aspects of study design. Investigators would be free to use measures with limited validity evidence, but the existence of a list of recommended measures would ideally motivate serious consideration and justification of such decisions.
Develop open-access databases that serve as repositories for granular measurement data that can be used to evaluate and improve extant measures
As noted previously, validity is a conditional phenomenon, not a static property of a measure. If taken seriously, this is daunting to investigators who are not theorists or experts in quantitative methods; for example, the measurement invariance of relevant measures in a given population (e.g. based on gender; Klein Haneveld, Molenaar, de Vogel, Smid, & Kamphuis, Reference Klein Haneveld, Molenaar, de Vogel, Smid and Kamphuis2022) would need to be established prior to their implementation in a study of psychopathology. We should maintain standards in our expectations that investigators report at least minimum validity evidence for their measures (Flake et al., 2022); however, it is probably unrealistic to expect that psychopathologists interested in implementing an index of a vulnerability process in a study of etiology will halt their research programs to do a thoroughgoing scrutiny of their measures, especially given that they may lack the relevant expertise and the options for moving ahead with their research are unclear, should a measure not hold up under close scrutiny. However, given current appreciation for the benefits of data-sharing initiatives, an open-access database that provided granular data, ideally from studies that use the consensus-based, multimethod measures of vulnerabilities as described in the previous recommendation, could be an extremely valuable resource for trainees interested in measurement and quantitative researchers.
Incentivize ‘pre-clinical’ research on measurement development
In psychopathology research, longitudinal methods that study individuals with vulnerabilities prior to the onset of disorder are often viewed as superior to cross-sectional work, which cannot speak clearly to causal processes. However, as noted earlier, longitudinal work often skips over the admittedly less glamorous phases of substantive and structural validity, phases which are nevertheless essential toward generating robust findings. Thus, from a measurement development standpoint, there is much to be gained from descriptive, cross-sectional studies that include clinically ill individuals (i.e. those with diagnoses) as a means of delving deeply into the structural phase of construct validation (Loevinger, Reference Loevinger1957). This approach facilitates the use of psychopathology symptoms as a scaffold for the iterative development of better measures of relevant vulnerability processes (i.e. those with probable etiological significance) using modern standards for psychometric investigations. Individuals with clinical disorder clearly have ‘risk’ for that disorder and therefore have relevant vulnerabilities that can be characterized via intensive measurement development. This is not to disregard the very real limitations of cross-sectional data to identify etiological processes that unfold over time (Markus & Borsboom, Reference Markus and Borsboom2013), but to identify a reasonably parsimonious place from which to begin this important research. If we develop and refine multimethod assessment tools that validly characterize vulnerabilities with strong predictive power for symptoms and clinically significant impairment, we will have a much stronger foundation for the study of etiological processes that rely on longitudinal methods and experimental manipulations to better establish causality.
As an initial step in this process, we might focus on the characterization of individuals with prototypical manifestations of disorder constructs; while doing so admittedly reduces generalizability, measurement development will benefit from greater homogeneity of samples, permitting stronger claims regarding the nature of putative vulnerabilities relevant to specific disorders. Diagnostic phenotypes, while admittedly imperfect, are still the best understood phenotypes that we have in psychopathology; thus, the relevant literature, mixed though it may be, is better positioned to provide initial clues regarding which vulnerabilities should be looked at and how they might be optimally measured. Furthermore, if part of the problem with DSM diagnoses is that they do not map clearly onto etiology, it could be that improving our measures of putative etiological influences will ultimately help improve diagnoses (Cronbach & Meehl, Reference Cronbach and Meehl1955).
Some might argue that the RDoC (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn and Wang2010) provide a better vantage point than DSM symptoms and diagnoses for the study of etiological mechanisms, given that they were designed to map more closely onto etiological processes than diagnosis does. Unfortunately, there is evidence that the RDoC initiative has not been as useful of a framework for the study of etiology as we might have hoped. For example, the theoretical domains articulated within RDoC do not cohere following the hypothesized latent structure (e.g. positive and negative valence systems; Olino, McMakin, & Forbes, Reference Olino, McMakin and Forbes2018; Peng et al., Reference Peng, Knotts, Taylor, Craske, Stein, Bookheimer and Paulus2021). While RDoC domains and indicators were subject to significant discussion regarding their definition, these decisions were not embedded within psychometric consideration of measures of RDoC content. Ironically, we might ultimately gain better traction in attaining RDoC's goal of delineating etiological processes if we used clinical symptoms from the DSM as the scaffold for the development of good measures of RDoC constructs.
On a smaller scale, as reviewers of grants and manuscripts, we should be cognizant of the limitations of research on psychopathology etiology that lacks a firm measurement foundation, as well as the merits of research focused on measurement development, and evaluate research accordingly. Sechrest (Reference Sechrest2005) noted the importance of training in measurement in both graduate and continuing education domains; this point is still relevant, and the current state of the field may reflect limitations in awareness of the centrality of validity and how to evaluate it when reviewing grants and manuscripts. In evaluating research on psychopathology, a premium should be placed on the methodological strengths of research and on the provision of validity evidence for measures of constructs of interest (Flake et al., Reference Flake, Pek and Hehman2017). Rewarding measurement-intensive research may help curtail the practice of developing ‘ad-hoc’ measures to use in specific studies (see Flake & Fried, Reference Flake and Fried2020, for a discussion of ‘on the fly’ measurement), which causes problems by leading to the proliferation of new measures that are used by only a handful of research groups, hampering our ability to synthesize research literatures.
Explore the use of technological innovations that facilitate the collection of quality data
Earlier, I noted the need for multimethod approaches to measuring important constructs. While the literature on child development has a strong tradition of including observation approaches, many studies of both youth and adults rely heavily on questionnaire measures rather than observational data, despite their limitations (Durbin & Wilson, Reference Durbin and Wilson2012). This is likely due to the very real expense of collecting and coding observational data, which may require aggregation over many samples of behavior to yield reliable estimates (Epstein, Reference Epstein1983). Technological advances are increasingly capable of reducing the expense of observational data; for example, automated facial affect and linguistic coding are increasingly feasible, albeit less well developed for use with children. Such methods may have the potential to reduce the significant burden associated with observational data (Iddiols, Daoust, Mullen, & Hayden, Reference Iddiols, Daoust, Mullen and Hayden2022). While there is much work to be done toward validating these tools, to the extent research groups can collect high-quality observational data, it will be increasingly possible to offset the expense associated with these data by using automated methods of coding observational data. Conversely, automated approaches cannot salvage studies that use measures without construct validity; as noted recently by Flake et al. (2022), measurement decisions are foundational to data quality and the construct validity of poor measures cannot be salvaged after the fact.
Conclusion
Progress in understanding vulnerability to psychopathology has been stymied by an unsystematic approach to defining, measuring, and validating putative vulnerability factors. The complexity inherent to the development of psychopathology indicates that systematic development of consensus measures of vulnerabilities should occur prior to their implementation in etiological models. Although this approach will require significant collaboration and investment of time and resources, it should ultimately yield greater progress than the current approach, in which oftentimes underdeveloped constructs and measures are embedded within complex causal models. Returning our attention to issues of construct validity and measurement development in service of refining our indices of vulnerability is crucial toward building a robust understanding of the causes of psychopathology.
Acknowledgements
The author is grateful for the thoughtful feedback received from Daniel N. Klein, Thomas M. Olino, Anna Weinberg, Kasey Stanton, the LEAP Lab, an anonymous reviewer, and Eiko Fried (self-identified reviewer) on this manuscript. Support for the author's research program and the development of this paper comes from the Social Sciences and Humanities Research Council, the Natural Sciences and Engineering Research Council, and BrainsCAN.
Conflict of interest
None.