The research of Nicki R. Crick has been instrumental in advancing our understanding of aggressive behavior. In particular, Crick's work has been critical for defining the various ways that aggression can be expressed in children and adolescents, especially in terms of its forms and functions. While physical aggression has long been a construct of interest in the social sciences because, by definition, it leads to physical harm to its victims (Berkowitz, Reference Berkowitz1993), Crick's work was influential in drawing attention to another form of aggression in which the victim's relationships are harmed (Crick, Ostrov, & Kawabata, Reference Crick, Ostrov, Kawabata, Flannery and Vazsonyi2007). Relational aggression consists of behaviors such as gossiping about others, excluding children from a peer group, spreading rumors, or telling others not to be friends with a child (Björkqvist, Lagerspetz, & Kaukiainen, Reference Björkqvist, Lagerspetz and Kaukiainen1992; Crick & Grotpeter, Reference Crick and Grotpeter1995; Lagerspetz, Björkqvist, & Peltonen, Reference Lagerspetz, Björkqvist and Peltonen1988). This form of aggression is associated with a host of social and psychological problems in both the victims and the perpetrators of the aggressive behavior (for a review, see Marsee & Frick, Reference Marsee, Frick, Arsenio and Lemerise2010). Crick's work has also been instrumental in elucidating the different functions served by aggressive behavior, whether physical or relational (Crick & Dodge, Reference Crick and Dodge1996; Mathieson & Crick, Reference Mathieson and Crick2010; Ostrov & Crick, Reference Ostrov and Crick2007). Specifically, reactive aggression occurs as an angry response to real or perceived provocation or threat, whereas proactive aggression is typically unprovoked and is often used for instrumental gain or dominance over others (Dodge, Reference Dodge, Garber and Dodge1991; Dodge & Coie, Reference Dodge and Coie1987).
Broadening and refining definitions of aggressive behavior to consider these different forms and functions has had important implications for understanding gender differences in the way aggression may be expressed (Cullerton-Sen et al., Reference Cullerton-Sen, Cassidy, Murray-Close, Cicchetti, Crick and Rogosch2008) and in understanding the different causal processes that underlie aggression (Dodge & Pettit, Reference Dodge and Pettit2003; Marsee & Frick, Reference Marsee, Frick, Arsenio and Lemerise2010). However, there are a number of issues arising from this research that could use clarification. These issues are relevant for advancing Crick's seminal work on the causes of aggression and for the development of effective treatments to reduce aggressive behavior in children and adolescents (Leff & Crick, Reference Leff and Crick2010). In this paper, we attempt to address several of these important outstanding issues.
The first issue is clarifying the association between gender and the different forms of aggression. The construct of relational aggression was developed to tap methods of harming others that may be preferred by girls. This preference may be due to cultural prohibitions over the expression of physical aggression in girls as well as the greater importance of relationships to girls, which may result in attempts to harm relationships being more hurtful to them (Crick, Reference Crick1996; Crick & Grotpeter, Reference Crick and Grotpeter1995; Galen & Underwood, Reference Galen and Underwood1997). In a comprehensive meta-analytic review of 148 studies, Card, Stucky, Sawalani, and Little (Reference Card, Stucky, Sawalani and Little2008) reported that the association between gender and physical aggression (r = .29) was positive and significant (with boys showing more physical aggression), whereas the association between gender and relational aggression was significantly different from zero, but too small to be considered meaningful (r = –.03). These findings suggest that the male predominance in aggression is largely confined to physical aggression. However, these findings do not support the contention that girls show more relational aggression than do boys but instead suggest that boys and girls show equivalent levels of this form of aggression.
Card et al. (Reference Card, Stucky, Sawalani and Little2008) considered whether the method of assessing physical aggression influenced associations with gender, with parent reports and self-reports yielding the smallest associations (r = .15 and .21, respectively) and peer reports yielding the largest (r = .37). For relational aggression, parent and teacher reports resulted in effects of girls showing more relational aggression than boys did (r = –.08 and –.07 for parent and teacher report, respectively), whereas boys self-reported slightly more relational aggression (r = .03); however, all of these effects were small in magnitude. There are two possible influences on the relationship between gender and the forms aggression that were not considered in the meta-analysis. One is whether the measures of physical and relational aggression were equivalent in their coverage of relevant aggressive behaviors. For example, items related to relational aggression may be worded to detect less severe and more normative types of harmful behavior than are items assessing physical aggression, and such differences in severity could influence the associations with gender. A second possible influence is the type of sample studied, such that it is not clear whether gender differences in type of aggression are similar across samples that may differ on their base rate of aggression. Thus, in the current study, we examined gender differences in the forms of aggression using a self-report measure designed to have similar items (similar in number, rating format, and severity) assessing physical aggression and relational aggression. Further, we examined possible gender differences in three samples of adolescents in various settings (i.e., community, residential treatment, and detained) to examine the robustness of associations with gender across types of samples using the same measure.
Another consideration in exploring gender differences in the forms of aggression is whether the forms differ in their incremental utility in predicting problematic outcomes in boys and girls. Specifically, boys and girls may not differ in their level of relational aggression, but this type of aggression may predict problems in adjustment for girls more than for boys, even when controlling for level of physical aggression. In support of this possibility, several studies have found that relational aggression predicts social–psychological maladjustment above and beyond overt aggression more consistently for girls than for boys (Crick, Reference Crick1996; Crick & Grotpeter, Reference Crick and Grotpeter1995; Cullerton-Sen et al., Reference Cullerton-Sen, Cassidy, Murray-Close, Cicchetti, Crick and Rogosch2008; Marsee & Frick, Reference Marsee and Frick2007; Prinstein, Boergers, & Vernberg, Reference Prinstein, Boergers and Vernberg2001).
A related question is whether there are children who show high rates of relational but not physical aggression who also show problems in adjustment that may warrant intervention. This question is critical for determining whether relational aggression should be considered in criteria for mental health conditions because it would indicate that criteria focused only on physical aggression may miss impaired children in need of treatment (Frick & Nigg, Reference Frick and Nigg2012). Further, it is essential to investigate whether these profiles of aggression differ by gender, such that boys may exhibit both physical and relational aggression when aggressive, whereas girls may be more likely to show only relational aggression. Addressing this question requires a person-centered approach to data analyses that has not been commonly used in past work. In one notable exception, Crapanzano, Frick, and Terranova (Reference Crapanzano, Frick and Terranova2010) used cluster analyses to study the patterns of aggressive behavior displayed in a sample of middle school students and found a cluster of girls who were high on relational aggression but who showed normative levels of physical aggression. These relationally aggressive girls also showed problems in adjustment, such as higher rates of anger, impulsivity, and bullying compared to girls with normative levels of aggression. Unfortunately, this study did not report whether there was a group of purely relationally aggressive boys who also showed problems in adjustment. Thus, in the current study we examined profiles of physical and relational aggression (i.e., forms of aggression) across three adolescent samples to determine whether purely relationally aggressive groups of both boys and girls emerged and, if so, whether they showed problems in adjustment.
The final issue investigated in the present study was whether profiles of reactive and proactive aggression (i.e., the functions of aggression) varied across the different study samples. Reactive and proactive aggression have emerged as separate dimensions in factor analyses (Little, Jones, Henrich, & Hawley, Reference Little, Jones, Henrich and Hawley2003; Poulin & Boivin, Reference Poulin and Boivin2000), and research has documented differences in their emotional and cognitive correlates. Specifically, reactive aggression has been linked to low frustration tolerance, poorly regulated emotional responses to provocation, impulsivity, and a tendency to misinterpret ambiguous behaviors as hostile provocation (Atkins, Osborne, Bennett, Hess, & Halperin, Reference Atkins, Osborne, Bennett, Hess and Halperin2001; Muñoz, Frick, Kimonis, & Aucoin, Reference Muñoz, Frick, Kimonis and Aucoin2008; Phillips & Lochman, Reference Phillips and Lochman2003). In contrast, proactive aggression has been associated with the tendency to view aggression as an effective means to reach goals (i.e., positive outcome expectancies), reduced emotional responsiveness to negative emotional stimuli, and a callous–unemotional (CU) interpersonal style (i.e., lacking guilt and empathy or a callous manipulation of others; Crick & Dodge, Reference Crick and Dodge1996; Frick, Cornell, Barry, Bodin, & Dane, Reference Frick, Cornell, Barry, Bodin and Dane2003; Hubbard et al., Reference Hubbard, Smithmyer, Ramsden, Parker, Flanagan and Dearing2002). Although the vast majority of these studies of the different functions of aggression have focused on physical aggression, there is evidence that relational aggression can also be divided into both reactive and proactive types (Little et al., Reference Little, Jones, Henrich and Hawley2003; Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011) and that these two types of relational aggression show different associations with several theoretically important variables (Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011; Marsee & Frick, Reference Marsee and Frick2007; Mathieson & Crick, Reference Mathieson and Crick2010). For example, Marsee and Frick (Reference Marsee and Frick2007) reported that reactive relational aggression was uniquely associated with poorly regulated emotion and anger to perceived provocation, whereas proactive relational aggression was uniquely associated with CU traits and positive outcome expectations for aggression in a detained sample of girls. Thus, causal theories of aggression must consider these different functions of aggression in order to fully explain the construct.
Causal theories must also account for the high correlation between the two types of aggression, which ranges from .40 to .90 across samples of youth with the typical estimate being about .70 (Little et al., Reference Little, Jones, Henrich and Hawley2003; Poulin & Boivin, Reference Poulin and Boivin2000). Further, research has consistently shown an asymmetry in the overlap between the two types of aggression. Specifically, there appears to be a significant number of children who only show reactive aggression, whereas most children who show high levels of proactive aggression also show high rates of reactive aggression (Brown, Atkins, Osborne, & Milnamow, Reference Brown, Atkins, Osborne and Milnamow1996; Dodge & Coie, Reference Dodge and Coie1987; Frick et al., Reference Frick, Cornell, Barry, Bodin and Dane2003; Muñoz et al., Reference Muñoz, Frick, Kimonis and Aucoin2008; Pitts, Reference Pitts, Raine, Brennan, Farrington and Mednick1997). The high correlation between the two types of aggression and the fact that the combined aggressive group is typically more aggressive overall has led some researchers to question whether the two functions of aggression reflect different patterns of behavior with unique causal factors (Bushman & Anderson, Reference Bushman and Anderson2001; Walters, Reference Walters and Morgan2005). That is, an alternative way of interpreting these findings is that proactive aggression is simply a marker of a more severe pattern of aggression, and not a different type of aggression. Thus, in the current study, we test whether distinct profiles of reactive and proactive behavior emerge across samples, across gender, and across the different forms of aggression when using the same measure. Further, we test whether the profiles differ on severity (e.g., the combined proactive and reactive groups showing more problems in adjustment) and/or on types of risk factors (e.g., the pure reactive group showing more problems in emotional regulation and the combined group showing more CU traits).
To summarize, in the current study we address three important issues for understanding the forms and functions of aggression across three different samples that likely vary in their base rate of aggression. First, we address the question of whether gender differences in physical and relational aggression are consistent across these different types of samples when using the same measure of aggression designed to have similar items assessing the different forms of aggression. Second, we address the question of whether a purely relationally aggressive group (i.e., low on physical aggression) of both boys and girls emerges across samples and whether this group shows signs of psychosocial impairment. Third, we address the question of whether profiles of aggressive behaviors differing in their function that have been found in past studies (e.g., a group moderately high on reactive aggression only and a group high on both reactive and proactive aggression) can be consistently replicated across the different samples using the same measure of aggression. We test differences across groups on important variables to determine whether adolescents with distinct profiles of aggressive behavior differ on severity of impairment and/or type of risk factors relevant to causal theory.
Method for Sample 1: Community
Participants
There were 307 adolescent participants (132 boys, 171 girls) between the ages of 11 and 18 years (M = 14.29, SD = 1.84). Four participants (1.3%) were missing gender information, and 12 (3.9%) were missing age information. Two samples of youth were included in this study: students recruited from high schools (n = 166) and volunteers from the community recruited as part of a larger study on parenting and adolescent behavior (n = 141). The sample was primarily Caucasian (57%) and African American (27.4%), with a small percentage of Hispanic (3.9%), Asian (2.0%), Native American (2.3%), and “other” ethnicities (5.5%). Approximately 2% of the sample did not report ethnicity.
Participants in the high school sample were recruited from two suburban public schools in the southeastern United States. Students were in the 9th–12th grades (n = 166), with a mean age of 14.97 (SD = 1.10) years (for a more detailed description of this sample, see Marsee, Reference Marsee2008). Participants in the general community sample were recruited as part of a larger study of parents and adolescents in the southeastern United States using several strategies, including announcements and flyers posted around a university campus and the general community and classified ads placed on the Internet. Volunteer adolescents (n = 141) had a mean age of 13.55 (SD = 2.18) years (for a more detailed description of this sample, see Marsee, Lau, & Lapré, Reference Marsee, Lau and Lapré2013).
Measures
Demographic information
Basic demographic information was collected through self-report and included arrest history (dichotomized as 0 = never arrested, 1 = arrested at least once), age, gender, and ethnicity.
Peer Conflict Scale (PCS)
The PCS (Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011) is a 40-item self-report measure developed to assess both the forms and functions of aggression, using the same number of items, similar rating formats, and the same level of severity across the different types of aggression. In an attempt to have similar levels of severity, each relational aggression item (e.g., “I say mean things about others, even if they have not done anything to me”) was carefully worded to match the wording of a physical aggression item (e.g., “I am deliberately cruel to others, even if they haven't done anything to me”). This was done in an effort to equate the perception of harmfulness across the items. In addition, the PCS was developed to have adequate coverage of each form and function of aggression. Specifically, the PCS includes 20 items assessing reactive aggression (10 reactive physical items: “When someone hurts me, I end up getting into a fight” and 10 reactive relational items: “If others make me mad, I tell their secrets”) and 20 items assessing proactive aggression (10 proactive physical items: “I start fights to get what I want” and 10 proactive relational items: “I gossip about others to become popular”). Items are rated on a 4-point scale (0 = not at all true, 1 = somewhat true, 2 = very true, and 3 = definitely true), and scores are calculated by summing the items on all four subscales.
The factor structure of the PCS has been supported in a large sample of older children and adolescents (N = 855; age range = 12–18; Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011). Specifically, confirmatory factor analysis showed that a hierarchical four-factor model best fit the data. Supporting the validity of PCS scores, subscales were significantly correlated with a laboratory measure of aggressive behavior, and reactive and proactive subtypes showed different responses to provocation (e.g., reactive aggression was associated with aggressive responses to low provocation) in a detained sample of boys (Muñoz et al., Reference Muñoz, Frick, Kimonis and Aucoin2008). In a detained sample of girls, the reactive and proactive subscales for both relational and physical aggression showed differential correlations with important external criteria (i.e., reactive being correlated with measures of emotional dysregulation and proactive being correlated with measures of CU traits and positive outcome expectancies for aggression; Marsee & Frick, Reference Marsee and Frick2007). In this community sample, the internal consistency was satisfactory for each scale (total physical α = 0.89; total relational α = 0.87; reactive physical α = 0.87; reactive relational α = 0.77; proactive physical α = 0.79; proactive relational α = 0.76). The PCS subscales were significantly intercorrelated ranging from r = .48 (reactive physical and reactive relational) to r = .76 (proactive relational and reactive relational), all p < .001.
Inventory of Callous–Unemotional Traits (ICU)
The ICU (Kimonis et al., Reference Kimonis, Frick, Skeem, Marsee, Cruise and Muñoz2008) is a 24-item self-report scale designed to assess CU traits in youth. Each item (e.g., “I feel bad or guilty when I do something wrong” or “I do not show my emotions to others”) is rated on a 4-point scale (0 = not at all true, 1 = somewhat true, 2 = very true, and 3 = definitely true). Scores are calculated by reverse scoring the positively worded items (12 of 24 items; 50%) and then summing all items to obtain a total score. The ICU total score is associated with aggression, delinquency, and both psychophysiological and self-report indices of emotional reactivity in detained and incarcerated samples of youth (Kimonis et al., Reference Kimonis, Frick, Skeem, Marsee, Cruise and Muñoz2008), as well antisocial behavior, impairment, and sensation seeking in a large community sample of adolescents (Essau, Sasagawa, & Frick, Reference Essau, Sasagawa and Frick2006). Internal consistency of ICU scores in the current sample was good (α = 0.80).
Self-Report of Delinquency (SRD)
The SRD (Elliott, Huizinga, & Ageton, Reference Elliott, Huizinga and Ageton1985) is a structured interview that measures delinquent behavior in youth by assessing whether the youth has engaged in 36 delinquent acts (e.g., destroying property, stealing, carrying weapons, selling drugs, hitchhiking, physical fighting, rape, or alcohol and drug use). Krueger et al. (Reference Krueger, Schmutte, Capsi, Moffitt, Campbell and Silva1994) reported significant correlations between the SRD and informant report of delinquency (i.e., friends or family who reported on youth's antisocial behavior during the past 12 months; r = .48, p < .01), police contacts (r = .42, p < .01), and court convictions (r = .36, p < .01). In the community sample, an 18-item brief version of the SRD was given, omitting questions relating to sexual behavior, nonviolent delinquency, and drug use. A total delinquency score was calculated with a possible range of 0–18 that had good internal consistency (α = 0.82).
Abbreviated Dysregulation Inventory (ADI)
The ADI (Mezzich, Tarter, Giancola, & Kirisci, Reference Mezzich, Tarter, Giancola and Kirisci2001) is a 30-item self-report questionnaire used to measure three aspects of dysregulation (emotional/affective, behavioral, and cognitive) in youth. Each item on the ADI is rated on a 4-point scale (0 = never true, 3 = always true). The emotional dysregulation subscale of the ADI has been shown to be uniquely associated with reactive aggression in detained adolescent girls (Marsee & Frick, Reference Marsee and Frick2007) and high school students (Marsee, Reference Marsee2008), while controlling for levels of proactive aggression. The emotional dysregulation and behavioral dysregulation subscales of the ADI were used in analyses, and both exhibited good internal consistency (behavioral dysregulation α = 0.87, emotional dysregulation α = 0.87).
Procedures
Institutional review board approval was obtained for the study prior to data collection. For the high school data collection, parental consent forms and invitations to participate in the study were distributed to first-period teachers for all students in Grades 9 through 12 at the target schools. Only students who received permission from their parents and who provided assent were allowed to participate. After parental permission was obtained, students were assessed in groups during their free period at school. Instructions for completing study measures were read aloud. Each student received a coupon redeemable at a fast-food restaurant for a free snack for their participation.
For the general community sample, participants were scheduled to complete a battery of questionnaires and computer tasks in a university campus laboratory. When participants arrived for their scheduled assessment, a research assistant reviewed the consent/assent forms with the parents and youth. The forms were administered in individual sessions and were read aloud to each participant. Each participant received $25 in compensation for completing the study.
Results for Sample 1: Community
Gender differences
Bivariate correlations between gender (coded as 0 = boys, 1 = girls) and each of the PCS subscales were conducted to examine associations between gender and aggression. Gender was significantly associated with proactive physical aggression (r = –.11, p < .05), indicating that boys reported higher levels than girls did. However, gender was not significantly associated with total physical (r = –.10) or reactive physical (r = –.08) aggression, although in both cases there was a trend for boys to show more physical aggression than girls. In contrast, the total relational (r = .08), proactive relational (r = .04), and reactive relational aggression (r = .10) subscales showed nonsignificant associations with gender, and in each case, girls showed slightly but not significantly more relational aggression than boys.
Profiles of reactive and proactive aggression
To test whether distinct profiles of reactive and proactive aggression emerge, a two-step cluster analysis procedure was performed in Statistical Package for the Social Sciences version 19 in order to classify the participants on the PCS reactive and proactive aggression subscales, which were standardized prior to analyses. The two-step method is an autocluster procedure that combines both Bayesian information criteria (BIC) and ratio of distance between clusters in order to determine the optimal number of clusters to retain (SPSS, 2004). The clustering procedure consists of two steps and is based on a probabilistic model where the distance between clusters is parallel to the decrease in log-likelihood function, which is a result of merging nearest neighbors (Chiu, Fang, Chen, Wang, & Jeris, Reference Chiu, Fang, Chen, Wang, Jeris, Lee, Schkolnick, Provost and Srikant2001). For the first step, preclusters are formed based on a sequential approach. A likelihood distance measure is used to determine each case's similarity to an existing precluster, and preclusters are formed when the log-likelihood is maximized. The second step uses a model-based hierarchical technique, similar to agglomerative hierarchical techniques. The optimal number of clusters is determined by the statistical program, which weighs both the ratio of distance between clusters and the change in BIC, such that a decrease in BIC from a previous model suggests better fit. In addition, the silhouette coefficient of cluster separation (distance of cases from the next closest cluster) and cohesion (distance of a case from the center of its own cluster) was examined as a fit indicator for the resulting clusters. This coefficient ranges from –1 (poor fit) to 1 (excellent fit; Kaufman & Rousseeuw, Reference Kaufman and Rousseeuw1990). Cluster analyses were conducted separately for the reactive and proactive physical aggression subscales and for the reactive and proactive relational subscales, as well as separately for boys and girls.
For physical aggression in the full sample, the two-step cluster analysis selected a three-cluster model as best fitting, which was a good fitting model according to the silhouette coefficient (0.6). The profile of the three clusters is provided in Figure 1a. Consistent with predictions, there was a low aggression cluster (n = 174; 57%), a cluster relatively high on reactive aggression (n = 101; 33%), and a group high on both reactive and proactive aggression (combined cluster; n = 32; 10%). As noted in Figure 1a, the combined cluster showed the highest rate of both reactive and proactive aggression. When boys and girls were analyzed separately, the three-cluster solution resulted in similar groups and was a good fitting solution for both boys (Figure 1b) and girls (Figure 1c) with silhouette coefficients of 0.6 and 0.7, respectively. For relational aggression, the results were not consistent with predictions. That is, in each case (full sample, Figure 1d; boys, Figure 1e; and girls, Figure 1f) only two clusters emerged that differed on their levels of reactive and proactive aggression. The silhouette coefficient was 0.7 for each cluster model, indicating good fit.
Profiles of reactive and proactive aggression resulting from two-step cluster analyses in the community sample. Within reactive and proactive columns, different superscripts indicate significantly different means.
Differences in physical aggression clusters
Given that the three-cluster solution for physical aggression was similar for boys and for girls, the solution for the full sample (Figure 1a) was used to test for differences across the physical aggression clusters in this community sample. The three clusters did not differ significantly by gender, age, or ethnicity. Four separate analyses of variance (ANOVAs) were conducted using the three clusters as independent variables and emotional dysregulation, behavioral dysregulation, CU traits, and delinquency as dependent variables. The results of these analyses are reported in Table 1. All four ANOVAs were significant, and Bonferroni-adjusted pairwise comparisons indicated the same pattern of differences for all four variables. Specifically, the low aggression cluster (n = 174) was significantly lower on emotional and behavioral dysregulation, CU traits, and delinquency than were both other clusters. Further, the high reactive cluster (n = 101) was significantly lower than the combined aggression cluster (n = 32). In addition, the three aggression clusters differed from each other on the percentage reporting being arrested, χ² (2) = 23.18, p < .001; ϕ = 0.28, and pairwise comparisons indicated that both the reactive (16.8%) and the combined cluster (34.4%), showed higher arrest rates than the nonaggressive cluster (5.7%), but the two aggression clusters did not differ from one another.Footnote 1
Differences in physical aggression clusters in the full community sample
Note: CU, Callous–unemotional. Cluster effects are from a one-way analysis of variance. Within rows, means with different subscripts are significantly different based on Bonferroni-adjusted pairwise comparisons.
***p < .001.
Overlap in physical and relational aggression clusters
Chi-square analyses were conducted to determine the overlap across the physical and relational aggression clusters for boys and for girls separately (see Table 2). Both chi-squares were significant, χ² (2) = 20.84, p < .001, ϕ = 0.40 and χ² (2) = 40.57, p < .001, ϕ = 0.49, respectively, indicating significant correspondence in the participants classified as aggressive using both forms of aggression. Not all of the girls and boys in the high relational aggression cluster fell into one of the high physical aggression clusters, and this was somewhat more common for girls (n =19; 11.1%) than for boys (n = 9; 6.8%). In contrast, boys were more likely to fall in one of the high physical aggression clusters and in the low relational aggression cluster (n = 29; 21.9%) relative to girls (n = 14; 8.1%).
Overlap in the physical and relational aggression clusters for boys and girls in the community sample
Note: Bold values designate participants who were low on one form of aggression but fell into one of the high aggression clusters of the other form.
***p < .001.
Method for Sample 2: Residential
Participants
There were 1,917 adolescent participants (1,582 boys, 327 girls) between the ages of 16 and 19 years (M = 16.94, SD = 0.85). Eight participants (0.4%) were missing gender information, and 16 participants (0.8%) were missing age information. The ethnic composition of the sample was Caucasian (42.2%) and African American (38.8%), with a small percentage of other ethnicities (12.3%); 6.7% of the sample did not report ethnicity. Youth were recruited from a nonsecure, voluntary, residential, military-style intervention program for youth who have dropped out of school. Like the community sample, this residential sample was located in the southeastern United States. This sample was considered to represent a moderate risk sample, because 44% had been arrested at least once, whereas only 12% of the community sample reported being arrested.
Measures
The measures for this sample were the same as those collected for the community sample, with the exception that the ADI was not collected in this sample and a 34-item SRD (Elliott et al., Reference Elliott, Huizinga and Ageton1985) was used to capture the more severe range of delinquent behaviors that have a high base rate in juvenile offender samples (items 11 and 12 related to sexual behavior were not administered). The PCS (Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011) was used to measure aggression, and the subscales showed good to excellent internal consistency (total physical α = 0.91, total relational α = 0.91, reactive physical α = 0.88, reactive relational α = 0.83, proactive physical α = 0.86, proactive relational α = 0.86). The subscales of the PCS were all significantly intercorrelated, ranging from r = .42 (proactive relational and reactive physical) to r = .80 (proactive physical and proactive relational; all p < .001). The ICU (Kimonis et al., Reference Kimonis, Frick, Skeem, Marsee, Cruise and Muñoz2008) was used to measure CU traits (α = 0.77), and the 34-item version of the SRD (Elliott et al., Reference Elliott, Huizinga and Ageton1985) was used to measure delinquency (α = 0.91).
Procedures
Institutional review board approval was obtained prior to data collection. The director of the intervention program, who serves as guardian ad litem for the youth in the program during their enrollment, was fully informed of the purpose and procedures of the study. The director gave consent for the youth to be informed of the study, with the adolescents being allowed to participate voluntarily after being fully informed of the study procedures. The PCS, ICU, and SRD (as part of a larger battery of self-report questionnaires) were administered orally in groups of approximately 12–18 participants.
Results for Sample 2: Residential
Gender differences
Bivariate correlations between gender (coded as 0 = boys, 1 = girls) and each PCS subscale were conducted to examine associations between gender and aggression. Gender was significantly negatively associated with total physical (r = –.11, p < .001), proactive physical (r = –.10, p < .001), and reactive physical (r = –.10, p < .001) aggression, indicating that boys reported higher levels than did girls. In contrast, gender was significantly positively associated with reactive relational aggression (r = .07, p < .01), indicating that girls reported higher levels of this type of aggression than did boys. Gender was not significantly associated with total relational (r = .03) or proactive relational (r = –.02) aggression.
Profiles of reactive and proactive aggression
As in Study 1, the two-step clustering procedure was used to classify participants on the standardized PCS reactive and proactive aggression subscales. Consistent with the findings from the community sample, the predicted three clusters emerged for the full sample and for boys and for girls when the physical aggression subscales were used. Specifically, in all three analyses (see Figure 2a–c) clusters low on aggression, relatively high on reactive aggression, and high on both reactive and proactive aggression (combined) emerged. In addition, consistent with the community sample, the combined cluster was higher than the other two clusters on both reactive and proactive aggression. These cluster models were all relatively good fitting according to the silhouette coefficient (0.6, 0.5, and 0.6, respectively). When these cluster analyses were repeated for relational aggression (see Figure 2d–f), the predicted expected three-cluster solution emerged only for girls (Figure 2f) with a silhouette coefficient of 0.7.
Profiles of reactive and proactive aggression resulting from two-step cluster analyses in the residential sample. Within reactive and proactive columns, different superscripts indicate significantly different means.
Differences in aggression clusters
Given that the three-cluster solution for physical aggression was similar for boys and for girls, the full sample was used to test for differences across the aggression clusters. The groups did not differ significantly by age or ethnicity, but they differed by gender χ² (2) = 19.98, p < .001, ϕ = 0.10, with the low aggression cluster having a lower percentage of boys (78.8%) than either the high reactive (85.2%) or the combined (88.5%) clusters. Thus, gender was included as a covariate in the analyses testing for differences across the physical aggression clusters. Two separate analyses of covariance were conducted using the three clusters as independent variables and CU traits and delinquency as dependent variables, and both were significant (see Table 3). Bonferroni-adjusted pairwise comparisons indicated the same pattern of differences for each dependent variable. Specifically, the low aggression cluster was significantly lower on CU traits and delinquency than was the high reactive cluster, which was significantly lower than the combined aggression cluster. The three aggression clusters also significantly differed from each other on the percentage reporting being arrested, χ² (2) = 61.92, p < .001; ϕ = 0.18; pairwise comparisons indicated that the combined cluster showed the highest arrest rates (59.4%), followed by the high reactive cluster (47.9%), and both aggression clusters were significantly higher than the low aggression cluster (35.2%).
Differences in physical aggression clusters in the full residential sample
Note: CU, Callous–unemotional. Cluster effects are from a one-way analysis of covariance covarying gender, and means are least-squared means adjusted for the covariate. Within rows, means with different subscripts are significantly different based on Bonferroni-adjusted pairwise comparisons.
***p < .001.
The three clusters that emerged for relational aggression in girls (Figure 2f) were also compared on CU traits, delinquency, and arrest history. The groups did not differ significantly by age or ethnicity. Two separate ANOVAs were conducted using the three clusters as independent variables and CU traits and delinquency as dependent variables, and both ANOVAs were statistically significant (see Table 4). Bonferroni-adjusted pairwise comparisons indicated that the combined aggression cluster was significantly higher on CU traits than were the low aggression and the high reactive clusters, but the latter two clusters did not differ. For delinquency, the combined aggression cluster was higher than the low aggression cluster, but neither cluster differed from the high reactive cluster. The three cluster groups showed similar rates of arrest (low aggression: 36.0%; high reactive: 36.4%; and combined: 44.4%) with a nonsignificant chi-square, χ² (2) = 1.17, p = ns; ϕ = 0.06.Footnote 2
Differences in relational aggression clusters for girls in the residential sample
Note: CU, Callous–unemotional. Cluster effects are from a one-way analysis of variance. Within rows, means with different subscripts are significantly different based on Bonferroni-adjusted pairwise comparisons.
*p < .05. ***p < .001.
Overlap across physical and relational aggression clusters
Chi-square analyses were conducted to examine the overlap across the physical and relational aggression clusters for boys and for girls separately (see Table 5). For boys, the three physical aggression clusters were compared across the two relational aggression clusters (low and high). For girls, the expected three-cluster solutions for both physical and relational aggression were compared. For both boys and for girls, chi-square analyses indicated significant overlap in the clusters formed by physical and relational aggression, χ² (2) = 581.92, p < .001, ϕ = 0.61 and χ² (4) = 180.31, p < .001, ϕ = 0.75, respectively, with the ϕ coefficient suggesting that the strength of the association was substantial. However, despite this high level of correspondence, not all of the girls and boys in the high relational aggression cluster(s) fell into one of the high physical aggression clusters, and this was more common for girls (n = 22; 6.7%) than for boys (n =12; 0.8%). In contrast, boys were more likely to fall in one of the high physical aggression clusters but not in the low relational aggression cluster (n = 720; 45.6%) relative to girls (n = 65; 20.0%).
Overlap across the physical and relational aggression clusters for the residential sample
Note: Bold values designate participants who were low on one form of aggression but fell into one of the high aggression clusters of the other form.
***p < .001.
Method for Sample 3: Detained
Participants
There were 659 adolescent participants (591 boys, 68 girls) between the ages of 12 and 19 years (M = 16.15, SD = 4.78) who had been detained in a secure facility following an arrest in the southeastern United States. The boys in this sample were collected from five separate facilities across two states. The girls were collected from four separate facilities in one state. Two participants (0.3%) were missing age information. The primary ethnic groups in the sample were Caucasian (31.8%) and African American (55.5%), with a small percentage of other ethnicities (10.8%). Approximately 1.7% of the sample did not report ethnicity.
Measures
The measures for this sample were the same as those collected for the community and voluntary residential samples, with the exception that self-report of arrest was not obtained because all of the youth were detained owing to arrest for a criminal offense; the full 36-item SRD (Elliott et al., Reference Elliott, Huizinga and Ageton1985) was collected; and the ADI was collected for girls only. Specifically, the PCS (Marsee et al., Reference Marsee, Barry, Childs, Frick, Kimonis and Muñoz2011) was used to measure aggression (total physical α = 0.90, total relational α = 0.87, reactive physical α = 0.88, reactive relational α = 0.80, proactive physical α = 0.80, proactive relational α = 0.78). The subscales were also all significantly intercorrelated, ranging from r = .42 (reactive physical and reactive relational) to r = .72 (proactive relational and reactive relational; all p < .001). The ICU (Kimonis et al., Reference Kimonis, Frick, Skeem, Marsee, Cruise and Muñoz2008) was used to measure CU traits (α = 0.81), the ADI (Mezzich et al., Reference Mezzich, Tarter, Giancola and Kirisci2001) was used to measure emotional (α = 0.79) and behavioral (α = 0.77) dysregulation for detained girls only, and the SRD (Elliott et al., Reference Elliott, Huizinga and Ageton1985) was used to measure delinquency (α = 0.88).
Procedures
Parental consent was obtained in one of two ways depending on the preference of the participating detention facilities. For the majority of the youth, a staff member from the detention center contacted the parents or legal guardians of youth currently residing at the facility and informed them of a study being conducted by researchers at a local university and asked permission to forward their phone number to the researchers. For two of the boys' facilities, an announcement was made explaining the details of the study in the facility, and boys assented to their own participation and provided parents' contact information. Those parents who agreed to be contacted by the researchers were telephoned and the study procedures were explained to them. Parents were informed that their child's participation in the project would in no way influence his or her treatment at the detention center or his or her legal standing in the adjudication process. As approved by the institutional review board, parents or legal guardians who agreed to have their child participate were asked to allow the consent process to be audiorecorded and were subsequently mailed a copy of the consent form for their records. On average across the sites, approximately 82% of parents contacted provided consent for their children to participate in the study. The researchers met with youth whose parents provided consent at the detention centers in order to explain the study and obtain assent. Questionnaires were administered orally in small groups (three to eight participants) at the detention centers. Participants received a snack (e.g., candy or pizza) or a small monetary incentive for their participation, depending on facility preference.
Results for Sample 3: Detained
Gender differences
Bivariate correlations between gender (coded as 0 = boys, 1 = girls) and each PCS subscale were conducted to examine associations between gender and aggression. In contrast to the previous samples, gender was significantly positively associated with total physical (r = .15, p < .001), reactive physical (r = .19, p < .001), total relational (r = .33, p < .001), proactive relational (r = .20, p < .001), and reactive relational (r = .39, p < .001) aggression, indicating that girls in this sample reported higher levels of aggression on these subscales relative to boys. Gender was not significantly associated with proactive physical aggression (r = .05).
Profiles of reactive and proactive aggression
Again, the two-step cluster analysis was used to select the optimal cluster model for physical aggression and relational aggression separately, for the full sample, for boys, and for girls. The results of these analyses are provided in Figure 3. For physical aggression, the only analysis in which the predicted three-cluster model emerged was for boys (Figure 3b), with a silhouette coefficient of 0.6. Further, the profiles across aggression functions for boys were in line with expectations, with a low aggression group (n = 214; 36%), a relatively high reactive group (n = 275; 47%), and a high combined aggression group (n = 102; 17%). For the full sample (Figure 3a) and for girls (Figure 3c) two cluster models were selected that differed only on severity of aggression, both with silhouette coefficients of. 06. For relational aggression, the expected three-cluster pattern only emerged for girls (Figure 3f), with low aggression (n = 29; 43%), high reactive (n = 31; 46%), and high combined (n = 8; 12%) clusters. The silhouette coefficient was 0.6, indicating good model fit. For the full sample, a two-cluster solution emerged (Figure 3d) with two aggression clusters differing on severity (silhouette coefficient = 0.7); for boys, a three-cluster solution was obtained (silhouette coefficient = 0.6), but these clusters only differed on relative levels of aggression (Figure 3e).
Profiles of reactive and proactive aggression resulting from two-step cluster analyses in the detained sample. Within reactive and proactive columns, different superscripts indicate significantly different means.
Differences in aggression clusters
The three physical aggression clusters that emerged for boys (Figure 3b) were compared and the groups differed significantly by age (high aggression cluster was significantly older than both other clusters), but not ethnicity; thus, age was included as a covariate in the following analyses. The three clusters were compared on CU traits and delinquency, and both analyses of covariance were significant (see Table 6). Bonferroni-corrected pairwise comparisons indicated the same pattern across both variables, with the low aggression cluster being significantly lower on CU traits and delinquency than was the high reactive cluster, which was significantly lower than the combined aggression cluster.
Differences in physical aggression clusters for detained boys
Note: CU, Callous-unemotional. Cluster effects are from a one-way analysis of covariance covarying age, and means are least-squared means adjusted for the covariate. Within rows, means with different subscripts are significantly different based on Bonferroni-adjusted pairwise comparisons.
***p < .001.
The three clusters for relational aggression for girls did not differ significantly by age or ethnicity. Four separate ANOVAs were conducted and three of the four ANOVAs were significant (see Table 7). Bonferroni-corrected pairwise comparisons indicated that, for emotional dysregulation, the combined aggression group was significantly higher than the low aggression cluster but not the high reactive cluster. For behavioral dysregulation, the combined group was significantly higher than both other clusters. In addition, the combined aggression cluster and the high reactive aggression cluster were both significantly higher on CU traits than was the low aggression cluster, but did not differ from each other. Finally, the three relational aggression clusters did not differ significantly on self-reported delinquency.Footnote 3
Differences in relational aggression clusters for detained girls
Note: CU, Callous–unemotional. Cluster effects are from a one-way analysis of variance. Within rows, means with different subscripts are significantly different based on Bonferroni-adjusted pairwise comparisons.
*p < .05. **p < .01. ***p < .001.
Overlap across physical and relational aggression clusters
Chi-square analyses were conducted to determine the overlap across the physical and relational aggression clusters for boys and for girls (see Table 8). For boys, the three physical aggression clusters (low, relatively high reactive, and combined) were compared across the three relational aggression clusters (low, moderate, and high). For girls, the two physical aggression clusters (low and high aggression) were compared across the three relational aggression clusters (low, relatively high reactive, and combined). For both boys and girls, chi-square analyses indicated significant overlap in the clusters formed by physical and relational aggression, χ² (4) = 116.94, p < .001, ϕ = 0.45 and χ² (2) = 18.65, p < .001, ϕ = 0.52, respectively. However, not all of the girls and boys in one of the elevated relational aggression clusters fell into one of the high physical aggression clusters, and this was relatively more common for girls (n = 17; 25%) than for boys (n = 33; 6.5%). In contrast, boys were more likely to fall in one of the high physical aggression clusters but in a low relational aggression cluster (n = 207; 35.0%) relative to girls (n = 5; 7.4%).
Overlap across the physical and relational aggression clusters for the residential sample
Note: Bold values designate participants who were low on one form of aggression but fell into one of the high aggression clusters of the other form.
***p < .001.
Discussion
The current study was designed to examine several important issues related to the forms and functions of aggression across three samples using a single measure of aggression. Considering a broader definition of aggression that includes both physical and relational aggression could have important implications for understanding gender differences found in prior studies. For example, past meta-analyses have documented a consistent association between gender and physical aggression, with boys showing more physical aggression than girls, but with a more equal gender ratio for relational aggression (Card et al., Reference Card, Stucky, Sawalani and Little2008). However, it not clear how consistent these findings are across different types of samples, especially when using a single measure developed to have similar levels of severity for both physical and relational aggression.
Our results were generally consistent with past findings in all but the detained sample. That is, in both a nonreferred community sample and a sample of at-risk children in a residential program, boys showed higher rates of physical aggression, whereas girls were more likely to show relational aggression, albeit nonsignificant in the community sample. However, the pattern of gender associations was different in the detained sample in which girls showed higher rates of both physical and relational aggression than did boys. Given that the same measure was used across samples, our findings cannot be attributed to measurement differences. Instead, these findings support previous research indicating that girls detained for delinquent behavior often have more serious behavioral and mental health problems than do detained boys (Gavazzi, Yarcheck, & Chesney-Lind, Reference Gavazzi, Yarcheck and Chesney-Lind2006; Silverthorn, Frick, & Reynolds, Reference Silverthorn, Frick and Reynolds2001).
Another important question addressed in the current study is whether broadening the construct of aggression to include relational aggression captures impaired children who may not be identified if only measures of physical aggression are used. In our study, this appears to be the case for girls but not for boys. That is, although boys showed a substantial rate of relational aggression, sometimes at a rate similar to girls (community sample), there were very few boys who fell into a high relational aggression cluster who did not also show high rates of physical aggression (6.8%, 0.8%, and 6.5% of boys in the community, residential, and detained samples, respectively). However, there were a substantial number of girls who fell into a high relational aggression cluster but who did not show high rates of physical aggression (11.1%, 6.7%, and 25% of girls in the community, residential, and detained samples, respectively). In short, it was relatively rare for boys to show high rates of relational aggression if they were not also physically aggressive, but this was not uncommon for girls, consistent with the findings reported by Crapanzano et al. (Reference Crapanzano, Frick and Terranova2010) in a nonreferred middle school sample. These findings are also consistent with past research indicating that relational aggression, despite showing similar rates in boys and in girls, is uniquely related to problems in adjustment when controlling for physical aggression in girls but not in boys (Card et al., Reference Card, Stucky, Sawalani and Little2008).
Girls high on relational aggression generally showed higher rates of CU traits and dysregulation than did nonaggressive girls in our samples. Past studies have found an association between relational aggression and problems in adjustment for both the perpetrators and the victims of this type of aggression (Marsee & Frick, Reference Marsee, Frick, Arsenio and Lemerise2010). Thus, a failure to consider relational aggression may lead to the underidentification of impaired girls in need of mental health treatment (Leff & Crick, Reference Leff and Crick2010). From our findings, this appears to be especially true for girls who are detained for committing delinquent acts. In our detained sample, one in four (25%) girls showed high rates of relational aggression but not elevated physical aggression. Further, our findings, as well as those of others, have identified factors that may contribute to girls' relational aggression and that may be important targets for intervention, such as problems of emotional regulation (Crapanzano et al., Reference Crapanzano, Frick and Terranova2010; Marsee & Frick, Reference Marsee and Frick2007; Marsee, Weems, & Taylor, Reference Marsee, Weems and Taylor2008; Mathieson & Crick, Reference Mathieson and Crick2010; Ostrov & Houston, Reference Ostrov and Houston2008) and problems in the development of empathy and guilt, as indicated by the presence of CU traits (Crapanzano et al., Reference Crapanzano, Frick and Terranova2010; Marsee & Frick, Reference Marsee and Frick2007).
We also examined which profiles of aggressive behavior emerged across samples in terms of the functions of aggression (either reactive or proactive). Research has consistently shown that the substantial correlation between reactive and proactive aggression appears to be because most children who show high levels of proactive aggression also show high rates of reactive aggression (Brown et al., Reference Brown, Atkins, Osborne and Milnamow1996; Crapanzano et al., Reference Crapanzano, Frick and Terranova2011; Dodge & Coie, Reference Dodge and Coie1987; Frick et al., Reference Frick, Cornell, Barry, Bodin and Dane2003; Muñoz et al., Reference Muñoz, Frick, Kimonis and Aucoin2008; Pitts, Reference Pitts, Raine, Brennan, Farrington and Mednick1997). Across the three very different samples in the current study, this pattern was found consistently for physical aggression; it was found for boys in all three samples and for girls in two of the three samples. For relational aggression, the pattern was not as consistent across gender and samples. These aggression clusters (high reactive and combined) did not emerge for boys in any of the samples but emerged for girls in two (residential and detained) of the three samples (in the two samples in which a relatively high rate of aggression would be expected). However, in none of the cluster analyses across samples and across gender did a group high on proactive aggression alone emerge. Thus, theories for explaining the different functions of aggression need to consider the fact that proactive aggression is often rare in the absence of reactive aggression (Marsee & Frick, Reference Marsee, Frick, Arsenio and Lemerise2010).
One possibility that has been proposed is that the presence of proactive aggression is simply a marker of a more severe pattern of aggression (Bushman & Anderson, Reference Bushman and Anderson2001; Walters, Reference Walters and Morgan2005). Our results were largely consistent with this possibility in that when cluster analyses identified a purely reactively aggressive cluster, the level of reactive aggression was much lower in this group than in the group showing both reactive and proactive aggression in every analysis. Further, on measures of the severity of antisocial behavior (i.e., self-reported delinquency and arrests), the reactive group was generally more severe than the nonaggressive group but not as severe as the combined group. Finally, and most important, this same pattern was generally found on potentially important causal variables (i.e., emotional and behavioral regulation and CU traits). The one notable exception to this pattern was for relational aggression in girls in the residential sample, where the combined reactive and proactive group was the only group to differ from the nonaggressive group on level of CU traits.
Thus, with this one exception, our results were not consistent with the possibility that individuals with both reactive and proactive aggression would show different emotional and cognitive correlates compared to those with reactive aggression alone (Crick & Dodge, Reference Crick and Dodge1996; Hubbard et al., Reference Hubbard, Smithmyer, Ramsden, Parker, Flanagan and Dearing2002; Marsee & Frick, Reference Marsee, Frick, Arsenio and Lemerise2010). Instead, our findings are more consistent with the contention that those high on both types of aggression are the most aggressive overall and show higher levels (but not differences in type) of most risk factors for aggression. Our failure to find evidence for distinct emotional processes in the combined group may have been because emotional regulation was only assessed via self-report. There is some evidence that individuals who show proactive aggression may report being angry, may look angry and emotionally dysregulated to others (Hubbard et al., Reference Hubbard, Smithmyer, Ramsden, Parker, Flanagan and Dearing2002), and may report reactive aggression and appear reactive to provocation (Muñoz et al., Reference Muñoz, Frick, Kimonis and Aucoin2008), but they may not show the physiological arousal that typically accompanies these emotional responses. Thus, differences between aggressive groups may have emerged if other indices of emotional reactivity had been used.
Relying on self-report for all study measures influences other interpretations as well. That is, there is evidence that self-report of aggression leads to smaller gender differences for both physical and relational aggression (Card et al., Reference Card, Stucky, Sawalani and Little2008). Thus, if other ratings of aggression, especially from peers, had been obtained, there may have been stronger correlations between aggression and gender. In addition, the largest sample (n = 1,917) was the one in which the expected profiles of reactive and proactive aggression were clearest (see Figure 2), at least for physical aggression in both boys and girls, and relational aggression in girls. Unfortunately, there was no measure of emotional dysregulation available in this sample, and thus, some of the potential differences in emotional characteristics across aggressive groups could not be tested in this large sample. Further, participants across all three samples were adolescents or young adults. This was important to ensure that any differences across samples could not be attributed to developmental differences. However, it also means that the findings may not generalize to younger samples.
Within the context of these limitations, our findings have two key implications for extending Nicki Crick's work on the forms and functions of aggression. The first implication is that theories on the different functions of aggressive behavior need to consider the fairly consistent findings that proactive aggression is relatively rare in the absence of significant levels of reactive aggression. As noted above, this may be because it is a marker of more severe aggression. If future research supports this possibility, proactive aggression may help to designate which aggressive adolescents are most in need of intensive interventions to reduce their risk for harming others. Alternatively, if other studies uncover emotional and/or cognitive differences related to the two types of aggression, then these theories also need to explain how reactive aggression is present in those with and without proactive aggression. As an example, Muñoz et al. (Reference Muñoz, Frick, Kimonis and Aucoin2008) provided data to suggest that reactive aggression, when it is present with proactive aggression, is not associated with increased emotional arousal to provocation. These authors suggest that such youth may only appear angry to others in order to intimidate and/or dominate them.
The second key implication of our results is related to the forms of aggression. Specifically, our results support those reported by Crapanzano et al. (Reference Crapanzano, Frick and Terranova2010) in suggesting that there are a substantial number of girls, but not boys, who show elevated levels of relational aggression without also showing elevated levels of physical aggression. These girls also show a number of indicators of impairment (e.g., high rates of delinquent behavior) and need for mental health treatment (e.g., problems in emotional and behavioral regulation, and elevated CU traits). Thus, one of the most important legacies of Nicki Crick's impressive body of work may be that it helped to identify a group of girls who are in need of mental health treatment who previously were not identified by traditional definitions of aggression (Leff & Crick, Reference Leff and Crick2010).
