General measurement perspective

Creating a reliable and valid measure of any psychological construct requires a set of relatively well-known methodological procedures (e.g., Borsboom, Mellenbergh, & van Heerden, Reference Borsboom, Mellenbergh and van Heerden2004; Campbell & Fiske, Reference Campbell and Fiske1959; Cronbach, Reference Cronbach1970; Cronbach & Meehl, Reference Cronbach and Meehl1955; McDonald, Reference McDonald1999; Nunnally, Reference Nunnally1967). This process starts with a theory that specifies critical features of the construct. This theory defines the nature of, or the elements of, the psychological construct in the population of individuals in which the measure will be administered. This feature of the theory about the psychological construct provides the basis for the generation of the items or observations to be included in the operational definition (i.e., the measure) of that psychological construct.

Furthermore, it is essential that the theory describing the nature of the construct is culturally informed or specific to the populations being sampled (e.g., Knight, Roosa, & Umaña-Taylor, Reference Knight, Roosa and Umaña-Taylor2009; Poortinga, Reference Poortinga2016; White, Knight, & Roosa, Reference White, Knight, Roosa, Caldera and Lindsey2015). For example, in developing a measure of depression for use in a White middle-class sample in the United States, it would generally be reasonable to include items or observations reflecting, among other things, suicidal ideation because this is a known indicator of depression in this population. To create an unbiased assessment of depression for use in a White middle-class population in the United States, it is desirable to select a sample of items/observations that is representative of the population of items/observations defining the construct in this specific population of persons. That is, if depression is reflected by suicidal ideation, crying, and sadness in this population of persons, then the measure of depression should have items or observations of these thoughts and behaviors. As the set of items becomes less representative (i.e., becomes a select subset) of the population of depression items/observations in the group of interest, it becomes less likely that the set of items or observations accurately assesses depression in this population. Hence, the accuracy of scientific inferences from research is enhanced by the use of measures that have a representative sampling of the items or observations that are indicators of the psychological constructs of interest (whether a cultural, developmental, or a psychopathological construct) administered to a representative sample of the population of interest.

In addition, theory also defines how the psychological construct of interest relates to other constructs, or fits within the nomological net of construct validity relations. The presumption here is that researchers primarily know what any particular measuring system is truly assessing by examining the degree to which the numerical scores produced by that system covary with scores produced by measures of theoretically related and theoretically unrelated psychological constructs. For example, it is generally reasonable to expect Mexican American adolescents’ cultural orientation to be associated with their parents’ degree of ethnic and mainstream culture socialization and the cultural nature of the context in which they live. Such a finding represents evidence of convergent construct validity. In contrast, it would not be reasonable to expect Mexican American adolescents’ cultural orientation to be associated with an unbiased measure of general intelligence. Finding a nonsignificant association between the scores from measures of these two constructs represents evidence of discriminant construct validity. To the degree that these kinds of observed associations between the scores on measures of psychological constructs conform to expectations, based upon the understanding of the theoretically driven nomological net of expected associations with other psychological constructs, researchers begin to believe that they are measuring the construct they intended to measure.

Again, it is essential that the observed construct validity associations conform to the construct validity relation expectations for the psychological construct in the population of interest (e.g., Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009; White et al., Reference White, Knight, Roosa, Caldera and Lindsey2015). As the observed construct validity relations deviate from the culturally informed expectation (i.e., based on knowledge and understanding of specific concepts that relate to group membership), it becomes less likely that the measure is assessing the intended psychological construct in the population of interest, and the scientific inferences made from the research using such a measure become less credible. Hence, the accuracy of scientific inferences from research is enhanced by the use of measures that produce scores that conform to the nomological net of expected construct validity relations (e.g., Cronbach & Meehl, Reference Cronbach and Meehl1955) for the psychological constructs of interest (whether a cultural, developmental, or a psychopathological construct) when administered to a representative sample of the population of interest.

An example of a valid and reliable measure of one dimension of ethnic and mainstream social identities for Mexican Americans is the Mexican American Cultural Values Scale (MACVS; Knight et al., Reference Knight, Gonzales, Saenz, Bonds, Germán, Deardorff and Updegraff2010). Previous qualitative and quantitative research has supported the construct validity and reliability of the measure for use with Mexican American participants (Cruz, King, Cauce, Conger, & Robins, Reference Cruz, King, Cauce, Conger and Robins2017; Knight et al., Reference Knight, Gonzales, Saenz, Bonds, Germán, Deardorff and Updegraff2010). This scale assesses Mexican American individuals’ degree of endorsement with values relatively more associated with the Mexican American culture and values relatively more associated with the mainstream culture. The Mexican American values scale consists of five correlated subscales from MACVS: familism-support (e.g., “Parents should teach their children that the family always comes first”); familism-obligation (e.g., “If a relative is having a hard time financially, one should help them out if possible”); familism-referent (e.g., “A person should always think about their family when making important decisions”); respect (e.g., “Children should always honor their parents and never say bad things about them”); and religiosity (e.g., “One's belief in God gives inner strength and meaning to life”). The mainstream values scale consists of three substantially correlated subscales from the MACVS: material success (e.g., “The best way for a person to feel good about himself/herself is to have a lot of money”); independence and self-reliance (e.g., “As children get older their parents should allow them to make their own decisions”); and competition and personal achievement (e.g., “One must be ready to compete with others to get ahead”).

The influence of measurement reliability and validity on observed associations

Unfortunately, limitations in reliability and construct validity can impact observed associations between cultural orientation, development, and psychopathology in at least two ways, and in turn, cloud accurate understanding of the role of cultural orientation in development and psychopathology. The first concern is measurement error. Although the measures of constructs are designed to produce numerical values that reflect the degree, intensity, or magnitude of the construct in the individual, an important premise of the psychometric conceptualization of measurement is that any measurement system includes inaccuracies. When these inaccuracies produce random variance in the scores generated by the measure, they reflect measurement error. In the ideal case, these inaccuracies in the measurement of a construct are not only random but also relatively small. Because these errors are random, the variance in a measure of a construct attributable to measurement error cannot be related to any measure of another construct. Therefore, these random errors limit the degree to which the scores on a target measure can be associated (e.g., correlated) with scores on other measures, including the same measure administered at another point in time. Thus, these types of errors influence the degree of unreliability associated with a target measure as well as the validity coefficients associated with a measure (i.e., the associations with scores on measures of other theoretically related constructs). Because this variance is random, these measurement errors can only result in the validity coefficients being attenuated (i.e., biased downward) because this variance reduces the proportion of systematic variance in the score. Therefore, if a measure includes substantial random measurement error (i.e., the scores are not highly reliable), the validity coefficients will be a substantial underestimate of the degree of covariation between the two constructs in the population of interest.

Although it is theoretically possible for the scores from the measurement of two constructs to have a correlation of 1.00, measurement error limits the observed association to the product of the square roots of reliability coefficients of the two measures (Knight, Reference Knight, Kieras and Just1984). Assuming that the same cultural orientation indicator is assessed twice (either at two different [developmentally relevant] points in time, or using two different measurement systems) and that the reliability coefficients for each assessment are .70, then the best possible observed correlation between these two sets of scores is .70 (i.e., the square root of .70 times the square root of .70), rather than the expected correlation of 1.00. This is because the reliability coefficient of .70 indicates that only 49% of the variance in the observed scores is systematic variance due to cultural orientation. Further, barring any other measurement issues, the reliability coefficient of .70 indicates that the remaining 51% of the variances in the scores is random error variance that cannot be associated with any other scores.

Figure 2 provides an example of this simple case in which the scores produced by both a measure of cultural orientation and a measure of psychopathology include systematic construct variance and random error variance. This figure assumes that the expected true association between cultural orientation and psychopathology because adaptive and maladaptive responses are multiply determined by numerous causal constructs. This .30 correlation indicates that these two constructs share 9% of their systematic variance, which is represented by the area of overlap of the two Venn circles relative to the area represented in the systematic variance of the psychopathology construct. Unfortunately, the best possible observed correlation is .21 if the reliability coefficients for each of the two sets of scores is only .70 (the true correlation times the product of the square roots of the two reliability coefficients). Hence, as the expected true correlation is smaller in magnitude, the observed correlation gets smaller in magnitude too. However, most important, as the measures get less reliable (i.e., approach .60 and indicate that only 36% of the observed variance is systematic construct variance), the observed correlation becomes an even greater underestimate of the true correlation. Because the effects of measurement errors on observed associations is relatively straightforward, there are analytical procedures (e.g., mathematical correction for attenuation, structural equation modeling, etc.) that may allow researchers to determine, or adjust for, the degree to which covariation among observed scores underestimates the true covariation among the underlying constructs. Nevertheless, measurement error can create substantial difficulties in making accurate scientific inferences. In extreme cases, the measures of cultural orientation or psychopathology may be so unreliable that is very unlikely for a specific study to detect an association between these constructs if one truly exists. Hence, the attenuation of observed associations may lead to considerable inconsistencies in the observed findings reported in the scientific literature.

Figure 2.

The influence of random measurement error on the observed association of cultural orientation with psychopathology.

The second concern is measurement bias. Unfortunately, often the variance in a set of scores produced by a measure is influenced by more than one construct as well as random measurement errors. That is, there is often systematic variance that results in the numerical representation inaccurately estimating the desired psychological construct because the responses on that measure are influenced by secondary constructs other than the target construct of interest. These secondary constructs add systematic variance, or measurement bias, in the scores (in contrast to the random variance contributed by measurement error) that are attributable to secondary constructs adding to the overall variance in the scores and representing a validity problem.

Often these secondary constructs are response biases, or other methodological artifacts, that are undesirable and sometimes unavoidable. For example, there is evidence that Latinos are more likely to use the extreme response alternatives on Likert-type response scales (i.e., “strongly agree/disagree” vs. “agree/disagree somewhat”) compared to non-Latino Whites (Hui & Triandis, Reference Hui and Triandis1989; Marín, Gamba, & Marín, Reference Marín, Gamba and Marín1992), and that more enculturated and less acculturated (i.e., stronger ethnic culture social identity and weaker mainstream culture social identity) Latinos more often have an extreme response alternative bias than more acculturated (i.e., stronger mainstream culture social identity) Latinos (Marín et al., Reference Marín, Gamba and Marín1992). Similarly, there is evidence that negatively worded items are relatively difficult for children to understand, particularly when translated into some languages (e.g., Nair, White, Roosa, & Knight, Reference Nair, White, Roosa and Knight2009; White, Umaña-Taylor, Knight, & Zeiders, Reference White, Umaña-Taylor, Knight and Zeiders2011). The impact of secondary constructs, such as these examples, may be quite problematic because the variance they add to the scores is not random and not the result of the target psychological construct one is trying to assess.

The systematic variance contributed by a secondary construct increases the variance in the scores derived from a measure and can lead the validity coefficients (i.e., associations between theoretically linked psychological constructs) to be either upwardly or downwardly biased. If the measures of the two theoretically related psychological constructs both introduce variance from the same (or very similar) secondary constructs (i.e., both measures have the same or similar measurement bias), then the association between the two sets of scores may overestimate the degree of covariation between the target construct and the validity construct in the population. That is, as the squared validity coefficient (e.g., squared correlation) is an estimate of the proportion of shared variance between the two sets of scores, increasing the variance of each set of scores with similar (i.e., highly correlated) systematic secondary influences increases the correlation between the two sets of scores by increasing the proportion of shared variance between the scores.

Figure 3 provides an example of this more complex case, in which the scores produced by both a measure of cultural orientation and a measure of psychopathology include systematic construct of interest variance as well as systematic shared method variance and random error variance. For example, if the measure of social identities and adaptive or maladaptive outcomes are both assessed with self-report measures, the scores on these assessments may include shared method variance. In this case, the response bias adds systematic variance in each set of scores that represents additional shared method variance beyond the shared variance between the cultural orientation and psychopathology constructs. That is, the correlation between cultural orientation and psychopathology absent the shared method variance (represented by section a in Figure 3) becomes inflated as the shared method variance increases the total shared variance between the cultural orientation and psychopathology scores (represented by sections a + b in Figure 3). However, this upward bias to the observed association between the measure of social identities and psychopathology is mitigated to some degree by the underestimation created by the random measurement errors in each set of scores. Ultimately, as the two sets of scores become highly reliable (i.e., have a reliability coefficient of .90 or greater with relatively little random measurement error, as in Figure 3), or as the proportion of shared method variance gets greater (i.e., section b in Figure 3), the upward bias of the observed correlation will be greater than the underestimation produced by the random measurement error, and the observed correlation will overestimate the true correlation between cultural orientation and psychopathology. Unfortunately, the measurement of social identities has relied heavily on self-report measures and there has been little effort to examine a multitrait-multimethod (Campbell & Fiske, Reference Campbell and Fiske1959) matrix of the influence of these social identities on psychopathology. Hopefully, future research will lead to the development of more observational or other-report measures of these social identities. Until such measures are developed researchers should, at least, strive to use measures of adaptive and maladaptive outcomes with non-self-report measures to address the issue of common reporter bias to some extent.

Figure 3.

The influence of random measurement error and measurement bias on the observed association of cultural orientation with psychopathology.

It is also possible, however, that the measure of social identities and the psychopathology measure introduce very different systematic secondary constructs (i.e., the two measures may have very different and unrelated measurement biases). In this case, the observed correlation between the two sets of scores will lead to underestimate the degree of covariation between cultural orientation and psychopathology because the unrelated systematic variance added by the response biases operate much like the random variance introduced by measurement error. For example, the correlation between cultural orientation scores in a Latino sample using a 5-point Likert-type self-report response scale and psychopathology scores based upon observations by trained non-Latino observers will likely underestimate the true correlation because the extreme response alternative bias of less acculturated Latinos (Marín et al., Reference Marín, Gamba and Marín1992) adds systematic variance to the cultural orientation scores that is not shared with the observed psychopathology scores. In essence, the proportion of shared variance between cultural orientation and psychopathology is decreased by the addition of unrelated method variance in much the same way that random measurement error adds unrelated variance. Hence, the underestimation of the true correlation created by the inclusion of unrelated method variance in the cultural orientation scores and the psychopathology scores is further attenuated by the random measurement error variance created by the measures. Furthermore, the situation in this example is further complicated by the likelihood that the measure of psychopathology also includes some unrelated systematic, but unrelated, measurement variance.

Unfortunately, researchers often have little knowledge of the specific secondary constructs being assessed by their measures. This makes it very difficult to (a) determine if the observed correlations are overestimates or underestimates of the associations among the underlying constructs in the population; and (b) adjust the magnitude of the observed correlations by controlling for these systematic sources of variability. Hence, the limited information usually available regarding the existence and nature of measurement biases has the potential to have important consequences regarding the accuracy of the scientific inferences made with measurement systems that do not exhibit strong psychometric properties (i.e., strong reliability and validity).

Best practice #1: Reliance on a multi-identity cultural orientation approach

We recommend researchers develop an explicit awareness that the indicators that represent cultural orientation encompass a wide range of psychological dimensions (i.e., knowledge, behaviors and expectations, attitudes and beliefs, and values; Gonzales et al., Reference Gonzales, Knight, Morgan-Lopez, Saenz, Sirolli, Contreras, Kerns and Neal-Barnett2002; Knight et al., Reference Knight, Gonzales, Saenz, Bonds, Germán, Deardorff and Updegraff2010; Schwartz et al., Reference Schwartz, Unger, Zamboanga and Szapocznik2010) across multiple social identities. This awareness and the associated research design implications are essential for advancing scientific understanding of the role of the development of the cultural self (Causadias, Reference Causadias2013) in individuals’ adaptation and maladaptation. Given that the multiple psychological dimensions of one's social identities are not likely to play an equal, or essentially similar, role in all aspects of one's development, researchers should consider a more nuanced vision of the existing literature. This is further complicated by the embedded multi-identity nature of cultural orientation (i.e., converging scripts associated with, at least, the ethnic and mainstream social identities; e.g., Berry, Reference Berry, Chun, Organista and Marin2003; Gonzales et al., Reference Gonzales, Knight, Morgan-Lopez, Saenz, Sirolli, Contreras, Kerns and Neal-Barnett2002; Tsai et al., Reference Tsai, Chentsova-Dutton, Wong, Hall and Okazaki2002). This framework of multiple psychological dimensions embedded within multiple social identities creates the need for a relatively broad literature that reports findings from a wide range of measures of specific dimensions of at least the social identities associated with dual cultural adaptation (i.e., acculturation and enculturation). There are at least two limitations associated with the ways in which some researchers have interpreted the current literature (overgeneralization and compartmentalization).

To date, some studies of the role of these social identities on shaping individuals’ cultural orientations have often included only a very limited range of assessments of the social identities and embedded psychological dimensions. For example, some measures labeled as “acculturation measures” (e.g., Cuellar, Arnold, & Maldonado, Reference Cuellar, Arnold and Maldonado1995; Szapocznik, Kurtines, & Fernandez, Reference Szapocznik, Kurtines and Fernandez1980) largely overrepresent assessments of the behavioral and attitudinal dimensions of ethnic and mainstream social identities, particularly language use and affiliation patterns (Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). Unfortunately, the scientific inferences regarding the roles of these social identities and derived cultural orientations in development and psychopathology have often not been adequately constrained by reference to the nature of the specific assessments. That is, although a study may include only assessments of a limited subset of the indicators of a specific dimension of ethnic and/or mainstream social identities (i.e., language use or preference, and/or affiliation patterns), researchers often overgeneralize by making scientific inferences about cultural orientation more broadly.

At the same time, researchers often study different, but much related, indicators and dimensions of ethnic and/or mainstream social identities without consistently and effectively integrating the findings from these literatures (Schwartz et al., Reference Schwartz, Unger, Zamboanga and Szapocznik2010). For example, there are at least two research literatures that focus on ethnic and mainstream social identities that are referred to by quite different labels. One is the literature on ethnic–racial or mainstream identity (e.g., affirmation, exploration, and resolution; Phinney Reference Phinney1992; Roberts el al., Reference Roberts, Phinney, Masse, Chen, Roberts and Romero1999; Umaña-Taylor, Yazedjian, & Bámaca-Gómez, Reference Umaña-Taylor, Yazedjian and Bámaca-Gómez2004), and the other is the literature on acculturation and enculturation (e.g., individuals’ preference and engagement in culture-related behaviors and with members from the mainstream and the ethnic culture; Cuellar et al., Reference Cuellar, Arnold and Maldonado1995; Szapocznik et al., Reference Szapocznik, Kurtines and Fernandez1980; Unger et al., Reference Unger, Gallaher, Shakib, Ritt-Olson, Palmer and Johnson2002). A better understanding of cultural orientation is likely to result from attending to both of these, and other related sets of findings. Unfortunately, compartmentalization sometimes leads to inconsistent efforts to link these related lines of research in the broader literature, and only some researchers attend to these related sets of findings.

Hence, a recommended practice for researchers is to focus on the role of the social identities influencing individual's cultural orientation in development and psychopathology by including assessments of relevant indicators of the different social identity dimensions, as long as those indicators and dimensions are appropriate for the developmental status of the participants in the study sample and relevant to the research questions of interest. In addition, we recommend researchers develop a more nuanced understanding of the literature (e.g., Bornstein, Reference Bornstein2017) by organizing findings by the social identity dimensions used as indicators of cultural orientation in each individual study. For example, it may be important to intentionally and occasionally link up, theoretically, to the broader multi-identity construct of cultural orientation that we are advancing herein to further advance scientific understanding. It is, however, simultaneously critical that researchers acknowledge, with a high degree of specificity, what aspects of cultural orientation (e.g., which indicators and dimensions of which social identities) have and have not been assessed.

It is likely the case that a more nuanced organization of the literature will be essential to our understanding of the role of cultural orientation in development and psychopathology. For example, the evidence that some ethnic culture-related behaviors (e.g., Spanish language use) decline substantially across generation of immigration among Latinos while important ethnic culture-related attitudes, such as ethnic identity affirmation and ethnic culture pride, do not (Keefe & Padilla, Reference Keefe and Padilla1987), suggests that different social identity dimension indicators of cultural orientation may have different developmental courses and associations with adaptation and maladaptation. For example, when summarizing the relevant research findings, the tendency to equate two measures that produce different types of “acculturation scores” (e.g., behavioral acculturation vs. attitudinal acculturation) because they assess somewhat different indicators of different social identity dimensions of cultural orientation may limit scientific understanding of the role of cultural orientation in development and psychopathology. In all likelihood, different social identity dimensions of cultural orientation may be differentially associated with adaptation and maladaptation.

Best practice #2: Reliance on a contextual cultural orientation approach

We recommend researchers be very explicit about the settings in which social identities are being assessed in the research process because the process of contextual interpretation may influence the degree of activation and impact of the individuals’ different social identities on their cultural orientation. Specifically, because an individual's cultural orientation is a function of differentially activated social identities based on the process of contextual interpretation, the settings in which researchers assess participants’ social identities become central aspects of the research design. As noted earlier, features of the immediate assessment context may foster the activation of specific social identities. Evidence of this potential comes directly from the research literature focusing upon cultural priming. That is, the research on frame switching has demonstrated that very simple features of the experimental manipulations to which individuals are exposed impact their attributions about behaviors in those assessment contexts (Benet-Martinez, Leu, Lee, & Morris, Reference Benet-Martinez, Leu, Lee and Morris2002; Hong, Morris, Chiu, & Benet-Martinez, Reference Hong, Morris, Chiu and Benet-Martinez2000). Because the process of contextual interpretation influences the association between social identities and cultural orientation (Figure 1), researchers focusing on the role of cultural orientation in individuals’ adaptation and maladaptation need to carefully consider and describe the settings or contexts in which they are assessing participants’ social identities, as well as the instrumentation. In addition, researchers summarizing the empirical literature on the role of cultural orientation in adaptation and maladaptation need to be sensitive to the potential impact of the assessment context (including both setting and instrumentation) on findings. In both examples, we are borrowing from research on investigator positionality (Bourke, Reference Bourke2014) to suggest that setting positionality (the characteristics of the context in which assessment occurs and the instrumentation used) becomes an important aspect of the research process to report when studying cultural orientations.

Best practice #3: Reliance on developmentally appropriate indicators of cultural orientation

We recommend researchers carefully select measure(s) of the particular indicators of the social identity dimensions being assessed that are appropriate for their specific research question(s) and the developmental status of their participants. At any given developmental status, there are multiple indicators of the different social identity dimensions of cultural orientation that are developmentally relevant. For example, if the specific research question is focused on the association of the pace of the development of cultural orientation (e.g., cultural knowledge) to outcomes during early childhood, a measure of ethnic culture constancy may be quite appropriate because the changes in cognitive capabilities during this age range result in individual differences in the specific timing of the development of ethnic culture constancy. Note, however, ethnic culture constancy is only one of many indicators that are developmentally appropriate indicators of cultural orientation during early childhood.

In contrast, most researchers would be aware that administering a measure of ethnic culture constancy in a sample of adolescents and young adults to address almost any research question would not be appropriate because there would be little meaningful variance in the scores produced by these measures at these ages. That is, by adolescence nearly all youth should have developed a reasonably thorough awareness of the permanency of their ethnic culture group membership (Aboud, Reference Aboud1984), unless they are experiencing either a cognitive delay or an environmental context in which ethnicity and race are not salient (which is rare in the United States and increasingly rare in the context of globalization). There is an overreliance in the literature on measures of cultural orientation that emphasize language use (e.g., Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009) and affiliation patterns to the near exclusion of other important indicators (e.g., cultural values, etc.), even though patterns of language use may be pretty well established at a relatively young age. This combined with the evidence (Keefe & Padilla, Reference Keefe and Padilla1987) suggesting that some behaviors, such as language use, may not be the best indicator of cultural orientation among later generations, at least in Mexican American youth, may limit our understanding of the role of cultural orientation in development and psychopathology. Understanding the role of cultural orientation in development and psychopathology would strongly benefit from studies that assess the full range of dimensions that reflect cultural orientation that are appropriate given the research question(s) and the developmental status of the participants.

Best practice #4: Reliance on a developmental cultural orientation approach

We recommend researchers implement sophisticated longitudinal research designs to enhance our understanding of the role of cultural orientation in development and psychopathology because of the age-related changes in the social identities that occur through normative developmental processes and developmentally tailored socialization experiences. In addition to providing evidence of a static or cross-sectional association between cultural orientation and psychopathology, longitudinal findings may provide stronger evidence regarding how developmental changes in the social identities that shape cultural orientations relate to adaptation or maladaptation. Most importantly, longitudinal findings can provide evidence supporting the causal linkage between the development of cultural orientation and adaptive and maladaptive responses.

For example, cross-lagged panel models (e.g., Gonzales et al., Reference Gonzales, Germán, Kim, George, Fabrett, Millsap and Dumka2008; Tropp, Hawi, Van Laar, & Levin, Reference Tropp, Hawi, Van Laar and Levin2012) may help determine if the development of specific forms or patterns of social identities (e.g., a cultural orientation comprised by strong ethnic and mainstream social identities, and strong gender and professional identities, and their associated scripts) leads to adaptation or maladaptation, if adaptive or maladaptive responses lead to the development of specific forms or patterns of social identities, or if the associations between social identities and adaptation and maladaptation are bidirectional. These cross-lagged panel model analyses require longitudinal assessment of both the presumed independent variable and dependent variables over three, and preferably more, time points. Similarly, prospective longitudinal models (e.g., Knight, Mazza, & Carlo, Reference Knight, Mazza and Carlo2017; Schwartz et al., Reference Schwartz, Unger, Baezconde-Garbanati, Benet-Martínez, Meca, Zamboanga and Soto2015) in which the developmental trajectories of mainstream and ethnic identities (i.e., acculturation and enculturation processes), shaping cultural orientations, across a specific age span are associated with adaptation or maladaptation at a later age, after controlling for earlier levels of the participants’ adaptive or maladaptive outcomes (perhaps preferably at the end of the age span during which the trajectories were identified), provide evidence regarding whether the development of specific social identities likely is causally associated with adaptation or maladaptation. These prospective model analyses require longitudinal assessment over at least three time points and with assessments of presumed mediator variables and independent variables over multiple and selective time points.

In addition, the reliance on longitudinal data can also enhance our understanding of the likely causal linkages in studies evaluating a mediational model (e.g., Oppedal, Røysamb, & Sam, Reference Oppedal, Røysamb and Sam2004; Umaña-Taylor & Updegraff, Reference Umaña-Taylor and Updegraff2007) of the association between social identities and adaptation or maladaptation. A single-point-in-time assessment could be used to engage in a structural equation model test of the hypothesis that a bicultural orientation (i.e., converging strong ethnic culture social identity and strong mainstream culture social identity in a manner that allows the individual to function appropriately in a wide variety of cultural contexts) is associated with general success in ethnic and mainstream contexts, and in turn, associated with adaptive outcomes.

However, longitudinal data would allow for a strong test of this mediational model (Roth & MacKinnon, Reference Roth, MacKinnon, Newsom, Hofer and Nelson-Jones2012) by examining the association of bicultural orientation at Time 1 with general success in the ethnic and mainstream contexts at Time 2 (controlling for this general success construct at Time 1), and the association of general success at Time 2 with adaptive outcomes at Time 3 (controlling for these adaptive outcomes at Time 2). Each of these three more sophisticated longitudinal analytical procedures requires an independent or predictor variable (i.e., presumed causal variable) measured at an earlier time point to be associated with a presumed dependent or mediator variable at a later time point and after the variance in the scores on that dependent/mediator variable have been residualized by the earlier time point assessment of the dependent/mediator variable (i.e., after the shared variance between the Time 1 and Time 2 dependent/mediator variable scores have been removed from the Time 2 dependent/mediator variable scores). In these analyses it is difficult or impossible to envision a mechanism through which the individuals’ Time 2 residualized dependent/mediator scores could cause the individuals’ Time 1 independent variable scores, thereby increasing one's confidence in the inference that the causal associations flow from the presumed independent variable to the presumed dependent/mediator variables (as well as between the presumed mediator to the presumed dependent variable; Roth & MacKinnon, Reference Roth, MacKinnon, Newsom, Hofer and Nelson-Jones2012).

Such longitudinal analyses also aid in the consideration of the role of common method variance in the association between social identity scores and psychopathology scores. That is, by residualizing the psychopathology scores measured subsequent to the assessment of social identity by the psychopathology scores measured at the same time as social identity, the shared method variance between the psychopathology scores at the two time points is controlled. Even if this shared method variance also exists in the social identity scores, this method variance does not inflate the observed association between the social identity scores and the residualized psychopathology scores because the common method variance has already been statistically removed from the psychopathology scores.

Measurement implications of longitudinal analyses

Conducting these types of sophisticated longitudinal analyses has significant measurement implications. Specifically, it is imperative that researchers have high-quality measures (i.e., high validity and reliability) of the psychological constructs in these analyses and that the scores on these measures exhibit relatively strong measurement equivalence across ages. This is essential because any observed age differences in cultural orientation or development and psychopathology could be a function of measurement or third variable effects across time, rather than developmental changes experienced by the individual. Hence, researchers interested in studying cultural orientation (as indexed by social identities) and development and psychopathology need to be aware of how to evaluate the longitudinal measurement equivalences of their assessments. A number of different types of measurement equivalence, and the procedures for assessing these, have been considered in the literature (e.g., Hughes, Seidman, & Williams, Reference Hughes, Seidman and Williams1993; Hui & Triandis, Reference Hui and Triandis1985; Knight & Hill, Reference Knight, Hill, McLoyd and Steinberg1998; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009; Little, Preacher, Selig, & Card, Reference Little, Preacher, Selig and Card2007; Malpass & Poortinga, Reference Malpass, Poortinga, Lonner and Berry1986; Millsap & Meredith, Reference Millsap, Meredith, MacCallum and Cudeck2007; Reise, Widaman, & Pugh, Reference Reise, Widaman and Pugh1993; Vandenberg & Lance, Reference Vandenberg and Lance2000; Widaman & Reise, Reference Widaman, Reise, Bryant, Windle and West1997). Hui and Triandis (Reference Hui and Triandis1985) have organized these notions of equivalence into several categories including item equivalence, functional equivalence, and scalar equivalence.

Item equivalence exists when the items on a measure have the same meaning across ages (Hui & Triandis, Reference Hui and Triandis1985; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). The evaluation of item equivalence is largely addressed during the examination of factorial invariance or the similarity of item functioning. For example, the assessment of the factorial invariance of a familism measure is generally based upon either the correlations among the individual items or observations in the familism measure, or upon the relation of each individual item or observation to the total score or scale score generated from the total set of items. If the responses to the familism items or observations are intercorrelated in a manner consistent with the culturally informed theory, and similarly across ages when this theory proscribes similarity, then item equivalence is likely.

The evaluation of the functional equivalence and scalar equivalence of measures across ages are more focused upon analyses examining the functioning of the total score or scale score derived from the combination of multiple items or observations in a measurement system (Hui & Triandis, Reference Hui and Triandis1985; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). For example, the assessment of the functional and scalar invariance of a familism measure is generally based upon the relations of the total score or scale score generated from the total set of familism items/observations to the total score or scale score for measures of other constructs (e.g., ethnic culture socialization). If the familism total/scale scores are related to ethnic culture socialization scores (as well as many other construct validity variables) in a manner consistent with the culturally informed theory, and similarly across ages when this theory proscribes similarity, then functional and scalar equivalence likely exists. Hence, assessments of functional and scalar equivalence are used to evaluate the construct validity equivalence of measures.

Functional equivalence exists when the total scores or scale scores generated by a measure have similar precursors, consequents, and correlates across ages (Hui & Triandis, Reference Hui and Triandis1985; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). That is, if the scores on a measure relate to scores on other measures in a manner consistent with culturally informed theory, and if these relations are similar across ages when this theory proscribes similarity, then there is some evidence of functional equivalence. Scalar equivalence exists when a given score on a measure refers to the same degree, intensity, or magnitude of the construct across ages (Hui & Triandis, Reference Hui and Triandis1985; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). It is important to note that a measure could have similar precursors, consequents, and correlates (i.e., display functional equivalence), but a given scale score can have quite different meaning at different ages. For example, the appropriate diagnostic cutoff for inferring clinical depression from a standard survey measure of depression may be different depending on the age and developmental status of the participant. In comparison to the other forms of equivalence, scalar equivalence is the most important to achieve because it is necessary to ensure the scientific credibility of the inferences researchers make from the types of data analyses they most often conduct. Scalar equivalence is, in many ways, also the most evidentiary demanding form of equivalence to demonstrate. Further, the assessment of the functional and scalar equivalence of a measure requires scale-level analyses in addition to the item-level analyses necessary for examining item equivalence.

Unfortunately, the representative indicators of a psychological construct may vary slightly, or greatly, at different levels of developmental status. For example, there has been some evidence that the indicators of depression vary by developmental status (e.g., Fried, Reference Fried2017; Fried & Nesse, Reference Fried and Nesse2015; Fried et al., Reference Fried, van Borkulo, Epskamp, Schoevers, Tuerlinckx and Borsboom2016; Santor, Gregus, & Welch, Reference Santor, Gregus and Welch2006). Further, changes in the indicators of developmental psychological constructs, including social identity constructs, with development are not uncommon. If the researcher's longitudinal assessments cross developmental boundaries, and there is variability in the representativeness of the indicators of a psychological construct, then relying on factorial invariance analyses will not be quite appropriate. If there are few indicator changes, or if these changes are very subtle, the researcher may need to rely on measures that include some modest variability in the indicators across the longitudinal assessments. Not only does the heterotypic development of social identities create difficulties in allowing measurements that are equivalent across developmental levels, but these issues also arise given the heterotypic nature of the development of psychopathology (see Wichstrom et al., Reference Wichstrom, Belsky and Steinsbekk2016, for an example). If the indicators are mostly the same across assessments, and there are a few different indicators at different developmental levels, perhaps the best one can do to evaluate longitudinal measurement equivalence is to conduct tests of partial factorial invariance (for a more detailed discussion see Knight & Hill, Reference Knight, Hill, McLoyd and Steinberg1998; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). Finding the appropriate partial factorial invariance along with strong evidence of construct validity equivalence (i.e., functional and scalar equivalence) may be the best one can do to support the longitudinal equivalence of a measure. If the indicators vary greatly across the participants’ developmental status at the difference longitudinal assessment points, item factorial invariance analyses may be completely inappropriate and examination of construct validity equivalence may be all that is possible.

Assessing cultural orientation with “proxy variables”

We recommend researchers consider the limitations associated with the use of “proxy variables” as indexes of cultural orientation (Causadias, Reference Causadias2013). The early research on the role of cultural orientation in development and psychopathology sometimes relied on the measurement of demographic types of variables as indices of participants’ cultural orientation. For example, Knight and Kagan (Reference Knight and Kagan1977) examined the prosocial and competitive choices of Mexican American children and used generation of family immigration as an index of children's cultural orientation. Among the types of proxy variables sometimes assessed to index cultural orientation are measures of generation of immigration, nativity (i.e., place of birth), and years living in country of immigration. Please note that these types of demographic variables are legitimately used for many research purposes. It is when the measurement of these types of variables is used as an index of the cultural orientation of individuals that researchers should be aware of the limitations.

Although these proxy variables are relatively easy to measure with a high degree of reliability, they create indices of cultural orientation that produce scores with a great restriction of range. For example, for most families for which generation of migration might be assessed, the possible scores typically have a range of four levels at best (i.e., first, second, third, and fourth generation). Further, this restricted range does not allow or assess the variability in cultural orientation that may occur within participants within the same generational status. That is, all participants with the same generational status are assumed to have the same cultural orientation even though there may be great differences in the ethnic and mainstream contexts in which these participants live and the experiences they might have with the mainstream and the ethnic culture groups. The restricted range of variances, and the inability to make fine discriminations between the cultural orientations of participants with a similar status on one of these proxy variable measures, likely attenuates the observed associations between the scores on these types of indices of cultural orientation and psychopathology. These attenuations may limit the accuracy of the scientific inferences researchers make from the research findings using these types of proxy measures.

More recently, and reflecting prevailing guidelines for integrating culture into development and psychopathology (Causadias, Reference Causadias2013), cultural researchers have moved to assessing cultural orientation as a psychological construct by developing and administering more direct measures of the different psychological dimensions of social identities. These more direct measures of social identities (e.g., behavioral indicators; Telzer, Masten, Berkman, Lieberman, & Fuligni, Reference Telzer, Masten, Berkman, Lieberman and Fuligni2011; and daily diaries; Yip & Fuligni, Reference Yip and Fuligni2002) are not as likely to produce scores that have a restriction of variance, and these measures capture individual differences in participants’ social identities. Unfortunately, a small segment of the research community continues to rely on measures of proxy variables to index participants’ cultural orientations. In addition, some expert reviewers still encourage authors to include measures of proxy variables as key indices of participants’ cultural orientation, even when researchers measure cultural orientation with one or more of the psychological construct measures. This is not to suggest that we should not include measures of proxy variables in our research. The scores produced by these measures should certainly be included for sample description purposes and for examining the linkage between these and measures of cultural orientation as a social identity psychological construct for validity purposes. However, these proxy measures should be subservient to the use of more direct, reliable, and valid psychological measures when examining the association of social identities and derived cultural orientations with adaptive and maladaptive development.

Assessing cultural orientation with measures in multiple languages

We recommend researchers consider the use of measures that have instructions, directions, and items or observations in multiple languages, particularly when research on the role of cultural orientation in development and psychopathology includes studies of ethnic groups for which there are recent immigrant families. For example, if one is studying how some indicators or dimensions of ethnic and mainstream social identities are associated with development and psychopathology in a representative sample of Mexican Americans, it is likely to be necessary to have measures that are administered in Spanish as well as in English. Furthermore, if this research is longitudinal, individual participants’ may prefer to complete measures in a different language at different time point assessments, depending on their capabilities and preferences at the different time points. The reliance on communications with participants in different languages can easily be a source of increased measurement error and/or measurement bias in the assessments of psychological constructs.

Fortunately, there have been considerable advances in the procedures for translating measures (e.g., Erkut, Reference Erkut2010; Erkut, Alarcón, Garcia-Coll, Tropp, & García, Reference Erkut, Alarcón, Garcia-Coll, Tropp and García1999; Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009; Prieto, Reference Prieto1992; Wang, Lee, & Fetzer, Reference Wang, Lee and Fetzer2006). In part, these advancements stem from the concerns that the simple translation and back translation procedures historically used in cultural research (i.e., Brislin, Reference Brislin1970) are limited because the bilingual translators are not comparable to monolingual participants in research samples. That is, bilingual translators used in the translation/back translation procedures are very often more highly educated, from a higher socioeconomic class, and have additional cognitive resources (i.e., greater frame-switching capabilities) compared to monolingual research participants (Knight, Roosa, et al., Reference Knight, Roosa and Umaña-Taylor2009). In addition, Knight et al. have suggested that the earlier mentioned analytical procedures designed to evaluate measurement equivalence provide useful tools for empirically evaluating the quality of translated measures (e.g., White et al., Reference White, Umaña-Taylor, Knight and Zeiders2011). Hence, to minimize the impact of any added measurement error or measurement bias on their research findings, researchers interested in examining the role of cultural orientation in development and psychopathology should consider the literature on the procedures for producing high-quality equivalent translations (i.e., reliable and valid). Further, in addition to considerations of the longitudinal measurement equivalence of assessments, cross-language measurement equivalence becomes an important empirical issue to address.

Assessing cultural orientation among “panethnic” (“panrace”) samples

We recommend researchers consider the limitations associated with the use of “panethnic” (“panrace”) samples and assessments. The currently available research examining the role of social identity indices and derived cultural orientations on development and psychopathology includes studies that sample one or more specific ethnic groups (e.g., Mexican Americans, Cuban Americans, Japanese Americans, Korean Americans, etc.) and that examine this association within each ethnic sample. There are also numerous studies that examine this association in panethnic samples (e.g., Latino/a Americans, Asian Americans, etc.). Combining participants with different cultural social identities into one group has important measurement implications. For example, if the focus is on the association of ethnic culture-related values (as an indicator of the influence of one of the dimensions of ethnic social identity on individuals’ cultural orientation) to development and psychopathology, the presumption of studies examining panethnic samples is that the cultural values associated with the different countries and cultural backgrounds that are combined into a panethnic sample are basically the same. Any differences in the cultural values associated with the social-level cultural scripts of the specific groups and the cultural social identities developed by participants from these groups are unmeasured, and likely increase the error variance in the observed scores. This, in turn, is likely to limit the reliability of the measure of cultural values administered to the participants in the panethnic sample. In addition, these unmeasured cultural values that may be appropriate for only some members of a panethnic sample result in the set of items or observations being differentially representative of the population of cultural values for the different specific group members in the panethnic sample. This, in turn, is likely to limit the validity of the measure of cultural values administered to the participants in the panethnic sample. For a similar discussion about limitations associated with measuring racial discrimination across diverse groups and the need to capture experiences unique to specific ethnic and racial groups, see Seaton, Gee, Neblett, and Spanierman (Reference Seaton, Gee, Neblett and Spanierman2018).

Sometimes the reliance on panethnic samples also leads to the modification of the items used for assessing cultural orientation. For example, if the focus of the research is on the association of ethnic culture pride (as an indicator of the influence of one of the dimensions of ethnic social identity on individuals’ cultural orientation) to development and psychopathology, researchers may elect to use panethnic labels in their items, even if the measure was developed with the use of specific ethnic group labels. In this case, an item for which participants are asked about their degree of agreement, such as “I am proud to be a Mexican American,” might be modified to “I am proud to be a Latino/a.” Although this seems like a relatively simple straightforward adjustment, participants might attach different meanings to each of these labels and might prefer to use one over the other depending on age, immigrant generation, nativity, life circumstances and experiences, and the contexts in which they are embedded (Pasco & White, Reference Pasco and White2018). Furthermore, some participants might only identify with one of these labels; therefore, asking participants who might identify as Mexican Americans but not as Latinos/as if they are proud to be Latino will not provide any information of those participants’ cultural orientation but will introduce error variance. Similarly, the same is true for participants of Mexican descent who might identify as Latinos/as but not as Mexican Americans when asked if they are proud to be Mexican Americans.

A more serious concern occurs when the modification of this item is designed to make the item fit multiple specific ethnic groups in multiple panethnic samples. For example, this item may be modified to a nonspecific ethnic label such as “I am proud to be a member of my ethnic group.” In this case, more participants may not understand the meaning of “my ethnic group” or may not refer to this label in the same way that the researcher assumes they will, particularly if they are from a multiethnic family. This is not to suggest that all participants will understand the specific ethnic group label (i.e., Mexican American) or see themselves as members of that specific ethnic group, or that these participants’ responses will not add error variance to the scores from these measures. This will most likely happen. However, it is to suggest that the modification to, or the use of, a more generic label, and particularly a nonspecific ethnic label, is likely to lead to greater error variance in the scores produced by such measures. This, in turn, is likely to limit the reliability of the measure of ethnic culture pride administered to the participants in the panethnic sample. In addition, if the difficulties in understanding the term “my ethnic group” are in any way associated with the individuals’ level of cultural orientation, this association is also likely to limit the validity of the measure of cultural values administered to the participants in the panethnic sample.

Although many studies of specific ethnic groups may require the use of measures in the English language and at least one other language, the reliance on panethnic samples can exacerbate the measurement difficulties associated with administering measures in multiple languages (as noted earlier). For example, although the multiple ethnic groups included in the panethnic Latino/a population generally have a family history of the use of Spanish, there are cultural and geographic variations in Spanish, in the manner it is spoken and the meaning associated with specific words. Furthermore, there are many indigenous languages spoken as a primary language among members of the Latino/a population. Similarly, the multiple ethnic groups included in the panethnic Asian American population not only have a different family history of the specific dialects and word meaning when they use the same non-English language but also may use a completely different non-English language. Hence, unless such studies restrict the representativeness of their ethnic populations participants who use a single language (usually the English language), the measurement demands (as described earlier) associated with the use of panethnic samples are likely to lead to lowered reliability and validity of measures administered even more than studies that sample one or more specific ethnic populations.

The lower reliability and validity (e.g., resulting from the addition of measurement error and measurement bias of measures), likely to be produced by the sometimes subtle, sometimes substantial, ethnic group differences in the indicator of cultural orientation, label modifications to measures, and multiple language use in studies of panethnic samples, may limit the accuracy of the scientific inferences researchers make from these panethnic studies.

Assessing cultural orientation using a broadly representative sample of the target ethnic population

We recommend researchers consider the implications of their sampling of participants in studying the role of cultural orientation in psychopathology. Recently, there are increasing studies examining the role of cultural orientation and its association with maladaptation and adaptation engaging representative samples of the respective ethnic population being studied and using valid and reliable assessment of cultural orientation (e.g., Gonzales et al., Reference Gonzales, Germán, Kim, George, Fabrett, Millsap and Dumka2008; Knight et al., Reference Knight, Mazza and Carlo2017; Umaña-Taylor & Updegraff, Reference Umaña-Taylor and Updegraff2007). Such research designs greatly align with the theoretical framework advanced and many of the aforementioned recommendations.

There are, however, other types of research designs ranging from studies of small convenience samples to nationally representative samples that have been used to examine the role of cultural orientation in psychopathology and development. For example, some studies examine the association between social identity dimensions and adaptation or maladaptation in samples that have been identified as at high risk for mental health problems (e.g., Knight, Vargas-Chanes, et al., Reference Knight, Vargas-Chanes, Losoya, Cota-Robles, Chassin and Lee2009; Lau et al., Reference Lau, McCabe, Yeh, Garland, Wood and Hough2005) or in subsamples of national data sets (e.g., Abraido-Lanza, Chao, & Gates, Reference Abraído-Lanza, Chao and Gates2005; Gordon-Larsen, Harris, Ward, & Popkin, Reference Gordon-Larsen, Harris, Ward and Popkin2003). Both of these types of research designs have important uses. A multigroup convenience sample selected from participants in a clinical program can be quite useful in estimating the degree of underutilization of clinical resources in some ethnic or racial groups. Similarly, a national sample may be quite useful for studying some type of phenomena (e.g., genetic effects) that require very large samples.

However, there are at least a couple of issues researchers should consider before relying on these samples for studying the role of cultural orientation, represented by social identities, in psychopathology in a target ethnic community. For example, some ethnic populations include a substantial proportion of recent immigrants who may not be very fluent in the English language or who may be uncomfortable using the English language. Studies that require the use of English may not include select subsets of the ethnic population and limited English-capable ethnic population members may not be willing to participate in these studies. Even if the sampling strategy is school based and includes older children and adolescents who are very often relatively proficient in the English language, non- or limited-English speaking parents may be less willing to provide their consent for their youths’ participation. In addition, there may be disproportionate schooling in nonpublic schools (e.g., religiously affiliated private schools or culturally focused charter schools) in some communities by some ethnic group families.

Further, if participants with limited English proficiency are included in these samples and required to complete the measures in English, the assessments of this subset of the ethnic subsample are not likely to be as reliable, valid, or equivalent across ethnic subgroups (e.g., recent immigrants vs. US-born participants). The scores generated from their responses may include measurement bias (in addition to increased random measurement error) if the participants’ limited English capabilities are associated with important indicators of cultural processes (e.g., differential use of response scales; Hui & Triandis, Reference Hui and Triandis1989; Marín et al., Reference Marín, Gamba and Marín1992), which may influence the scientific inferences. This latter consideration may also apply to studies that allow trained interviewers to use informal translation procedures when limited English speaking participants have difficulty with the assessments. These possibilities may limit the scientific inferences about the role of cultural orientation in the development and psychopathology based on the data available in these studies.

Selectively incorporating control variables in studies of cultural orientation

We recommend researchers carefully consider the ways in which variables can be incorporated as statistical controls into analyses examining the role of cultural orientation in development and psychopathology. Statistical control variables are often labeled control variables, confounds, and/or covariates without a consensus on the use of these terms (Christenfeld, Sloan, Carroll, & Greenland, Reference Christenfeld, Sloan, Carroll and Greenland2004; Greenland, Reference Greenland1980). Regardless of the term used, the use of statistical controls can be problematic. From a pragmatic perspective, the best control variables are those that are associated with the presumed dependent variable but not associated with the presumed independent variable. That is, in the study of the role of cultural orientation (e.g., focused on ethnic and mainstream culture pride) on development and psychopathology (e.g., drug abuse and addiction), an ideal variable for use as a statistical control is one that is associated with the psychopathology scores but not associated with the cultural orientation scores (e.g., family history of drug abuse and addiction). In this case, variance in the psychopathology scores that is associated with the control variable scores is treated as systematic explained variance rather than as error variance, thereby allowing a more powerful test of the association between the cultural orientation scores and the psychopathology scores. That is, the variance in the psychopathology scores that is associated with the control variable scores would have been treated as error variance without the inclusion of the control variable in the analyses, making larger the error term in the significance test of the cultural orientation association with psychopathology (given adjustments for degrees of freedom) and less likely to detect an effect (i.e., reject the null hypothesis) if one exists. Hence, another use of an appropriate control variable is to allow a more powerful significance test of the association between the presumed independent and dependent variables.

If, however, scores on a control variable are associated with the cultural orientation scores (i.e., the presumed independent variable) as well as the psychopathology scores (i.e., the presumed dependent variable), the use of this control variable as a statistical control may cloud the true association between the cultural orientation and psychopathology scores. For example, occasionally researchers have utilized a proxy variable (i.e., generation of migration) or a variable that may be another indicator of cultural orientation (i.e., language preference) as a statistical control variable. In this case, the observed association between the social identity scores and the psychopathology scores may be significantly biased because of residualization by the control variable. Lost in this case is the possibility that there exists a mediational relationship between the three sets of scores that may indicate a more substantial, and more complex, relation between the social identity and psychopathology construct. Further, when the control variable is a proxy variable or another indicator of cultural orientation for which the scores are associated with the presumed independent variable cultural orientation scores but not the psychopathology (i.e., the presumed dependent variable) scores, residualizing the independent variable cultural orientation scores leads to a lowered estimate of the association between cultural orientation and psychopathology. In either of these latter cases (i.e., the control variable associated with both the presumed independent and dependent variables, or the control variable associated with the presumed independent variable only), the observed associations may be biased and lead to scientific inferences that are not entirely accurate.

In addition, researchers may misinterpret the impact of using a statistical control variable in data analysis. For example, occasionally a researcher will examine the association between the scores on a measure of cultural orientation and scores on a measure of psychopathology while statistically controlling for a proxy variable indicator (e.g., generational status), and conclude that this control allows for the inferences that the observed association between the cultural orientation and psychopathology scores is the same regardless of individuals’ generations. This is inaccurate. One often unconsidered requirement of the use of a variable as a statistical control is that the scores from the control variable measure must not interact with (i.e., must not moderate) the association between the presumed independent and dependent variables. Hence, the use of generational status as a control variable requires an empirical test demonstrating that the generational status scores do not moderate the association between the cultural orientation and psychopathology scores (see White, Knight, et al., Reference White, Knight, Jensen and Gonzales2018, for an example). As the empirical evidence of nonsignificant moderation by generational status allows the inferences that the observed association between the cultural orientation and psychopathology scores is the same regardless of the individuals’ generational status, it is the empirical test of moderation that is often more appropriate given the researcher's interests.