In today’s organizations managers operate in contexts of increasing change and complexity, leading to more dynamic decision making (DDM) (Kozlowski, Gully, Brown, Salas, Smith, & Nason, Reference Kozlowski, Gully, Brown, Salas, Smith and Nason2001). DDM is a type of decision making categorized by scenarios with real-time continual change, novelty, ambiguity and time constraints (Brehmer, Reference Brehmer1992). Scholars suggest responding to these demands through individual adaptability (Hesketh, Reference Hesketh1997; Pulakos, Arad, Donovan, & Plamondon, Reference Pulakos, Arad, Donovan and Plamondon2000; Baird & Griffin, Reference Boyatzis, Goleman and Rhee2006). Adaptability is generally referred to as an ability to change when necessary. However, scholars who explicitly contribute to the study of adaptability do not often address the complexity inherent in the cognitive aspects of adaptive performance, as it pertains to such real-time dynamic tasks.
Understanding an individual’s capacity to be adaptive at the cognitive level is a vital starting point to successfully navigate dynamic real-time environments (Glynn, Reference Glynn1996). In order to be adaptive in a real-time dynamic context, one must create a new understanding of information in the environment (Zaccaro, Reference Zaccaro2001), allow it to alter the course of thinking when necessary (Mitroff, Simons, & Franconeri, Reference Mitroff, Simons and Franconeri2002) and remain focused on relevant information (Lustig, May, & Hasher, Reference Lustig, May and Hasher2001). The purpose of this paper is to propose and test a new construct: cognitive agility, a cognitive capacity that supports real-time adaptive performance within the scope of a single DDM task.
A context that has proved useful to researchers studying cognition is the DDM microworld. Microworlds (real-time interactive computer-based simulations) mirror the complexity in real-life (Gonzalez, Vanyukov, & Martin, Reference Gonzalez, Vanyukov and Martin2005), presenting real-time decisions in a changing environment (Brehmer, Reference Brehmer1992). Microworlds offer an experimental alternative to the ‘paper and pencil’ tests that are often used in the assessment of ‘dynamic, interactive, and time oriented phenomena’ (DiFonzo, Hantula, & Bordia, Reference DiFonzo, Hantula and Bordia1998: 280). This study predicts that cognitive agility will lead to better performance in a DDM microworld. The variables that form cognitive agility will be measured using multiple methods (performance measures, self-reports and external-rater reports).
LITERATURE REVIEW
Theoretical foundations of cognitive agility
Environmental influences on cognition (like novelty and complexity) are not fixed but instead more perceptual, suggesting individual differences in how information is attended to, filtered, encoded and interpreted (Neisser, Reference Neisser1967). Given that managers face more change and complexity, an investigation into cognitive-related constructs that support adaptability in real-time dynamic environments is needed.
Adaptability is often used as an overarching term to describe a set of individual behaviors, leading to adaptation (Briscoe & Hall, Reference Briscoe and Hall1999). Therefore, it is important to unbundle the constituent concepts and seek greater clarity in both the concepts and measures as they relate to particular contexts. Aspects of adaptability have been interpreted as parts of various aspects of a person, including personality traits (Morrison, Reference Morrison1977), competencies (Boyatzis, Goleman, & Rhee, Reference Boyatzis, Goleman and Rhee1999), learning style (Mainemelis, Boyatzis, & Kolb, Reference Mainemelis, Boyatzis and Kolb2002) and cognitive style (Sternberg, Reference Sternberg1997). Others admit that, regardless of their definition or approach, adaptability is in part a cognitive capability (Mumford, Zaccaro, Harding, Jacobs, & Fleishman, Reference Mumford, Zaccaro, Harding, Jacobs and Fleishman2000).
While adaptability is a general ability to support change, cognitive agility is a specific cognitive capability applied to contexts that require a series of individual adaptations. The primary distinction comes from the possibility of categorizing adaptations along the dimensions of time and task. For instance, one can adapt in the moment, to real-time tasks (Lerch & Harter, Reference Lerch and Harter2001), or over a longer period of time, as in adjusting to a new job (Ashford & Taylor, Reference Baltes and Staudinger1990). In addition, one can adapt within a particular task that is changing (Cañas, Quesada, Antolí, & Fajardo, Reference Cañas, Quesada, Antolí and Fajardo2003) or across various tasks that make up a dynamic context, like adapting well to the introduction of a new technology (Edmondson, Bohmer, & Pisano, Reference Edmondson, Bohmer and Pisano2001).
Many studies predict abilities or characteristics of individual adaptability (Pulakos et al., Reference Pulakos, Arad, Donovan and Plamondon2000); most of which, at least implicitly, track adaptations across a longer time horizon and across multiple tasks. Fewer studies have focused on the cognitive aspects of adapting within a single real-time DDM task (exceptions include LePine, Colquitt, & Erez, Reference LePine, Colquitt and Erez2000; Cañas et al., Reference Cañas, Quesada, Antolí and Fajardo2003). Therefore, further inquiry into capabilities that predict real-time adaptive performance is a useful addition to the adaptability literature.
Real-time adaptive performance
Real-time adaptive performance within a task (i.e., how well an individual performs within a changing task, Kozlowski et al., Reference Kozlowski, Gully, Brown, Salas, Smith and Nason2001) likely requires a range of skills. A real-time DDM task context has change, novelty, ambiguity and complexity (Brehmer, Reference Brehmer1992). Individuals must enact a series of continuous decisions with various task-related tradeoffs (Tverskey, Sattath, & Slovic, Reference Tversky, Sattath and Slovic1988). To adapt one must be able to shift thinking when necessary (Mitroff, Simons, & Franconeri, Reference Mitroff, Simons and Franconeri2002) and doing so requires flexibility to override a dominant or automatic response in favor of a more appropriate one (Clark, Reference Clark1996).
Flexibility is preceded by the ability to notice stimuli of consequence. Yet, noticing too much information may cause unwanted distractions (Kuhl & Kazen-Saad, Reference Kuhl and Kazen-Saad1988) that can result in missing other relevant data (Shapiro & Raymond, Reference Shapiro and Raymond1994) and ultimately a loss in decision-making effectiveness (Anderson, Reference Baddeley and Hitch1983). Therefore, while it is important to notice relevant stimuli for task adaptability, adaptive decisions are also supported by the capacity to avoid less relevant data (Shanteau, Reference Shanteau1988).
Cognitive agility seeks to synthesize and evolve simultaneously the current conceptualizations of adaptability, adaptive performance and flexibility. Specifically, cognitive agility represents an individual’s cognitive capacity to flexibly operate with cognitive openness and focused attention. The following section provides theoretical reasoning for the choice of agility as a construct name as opposed to variations of adaptability or flexibility (i.e., other terms more commonly used to suggest a range of individual appropriate and variable behavior).
There are multiple ways in the literature to describe the capacity to make an appropriate change in response to the environment. Adaptability as described throughout this paper is perhaps the most common naming convention (Pulakos et al., Reference Pulakos, Arad, Donovan and Plamondon2000). Yet, adaptability is often operationalized as performance in a task that is complex, novel or ambiguous (LePine, Colquitt, & Erez, Reference LePine, Colquitt and Erez2000), regularly referred to as adaptive performance (Kozlowski et al., Reference Kozlowski, Gully, Brown, Salas, Smith and Nason2001). Therefore, describing adaptability as an ability leading to adaptive performance becomes tautological and potentially limiting toward extending knowledge of the construct. Some have labeled such adjustments as ‘cognitive adaptability,’ which means an ability to change decision frameworks or knowledge to meet the environmental needs (Haynie, Shepherd, Mosakowski, & Earley, Reference Haynie, Shepherd, Mosakowski and Earley2010). What this conceptualization may be missing with regard to the current study are the cognitive needs of the particular context. In real-time dynamic tasks, the speed of change is a necessary component, along with trying to identify what the individual is changing to and from in terms of orientation and decision frameworks. The context of a real-time DDM task is different than adapting on a longer time horizon, and therefore may represent a completely distinct class of decision-making performance that is separate from single decisions, requiring a different set of abilities (Sitkin & Pablo, Reference Sitkin and Pablo1992).
Cognitive flexibility is another construct used to signify appropriate adjustment to situational needs. It is saddled by a heterogeneity of definitions and operationalizations ranging from set-shifting to being creative, making it difficult to carefully conceptualize (Luchins & Luchins, Reference Luchins and Luchins1959; Isen, Daubman, & Nowicki, Reference Isen, Daubman and Nowicki1987; Spiro & Jehng, Reference Spiro and Jehng1990; Martin & Anderson, Reference Martin and Anderson1998; Cañas et al., Reference Cañas, Quesada, Antolí and Fajardo2003). In addition, flexibility is often used synonymously with adaptability (Pulakos et al., Reference Pulakos, Arad, Donovan and Plamondon2000), further confusing the two constructs. Statements from leading scholars on the subject such as, ‘cognitive flexibility is defined as a person’s willingness to be flexible and adapt to the situation’ are not at all uncommon (Martin & Anderson, Reference Martin and Anderson1998: 1). Again by using the terms ‘adapt’ and ‘flexible’ together, in place of one another, or to describe flexibility as being flexible, makes the understanding of the constructs less clear and contextually unbound.
Taken together, it is evident that cognitive flexibility is a construct with a conceptualization that has not been fully agreed upon. Yet, across the diverse definitions is a theme of intelligently adjusting to one’s environment (Berg & Sternberg, Reference Berg and Sternberg1985), through various forms of shifting, restructuring or expanding cognition. It is suggested here that the ability to control one’s thinking and change the decision strategy is just one aspect of cognition that allows for more adaptive performance in dynamic contexts (Cañas et al., Reference Cañas, Quesada, Antolí and Fajardo2003). Cognitive flexibility alone does not adequately describe what an individual is actually changing about his or her cognition. The particular change is likely relevant to specific environmental contexts. The elements encountered in a real-time dynamic context likely require a particular integration of flexibility with other necessary dimensions.
There is a need for identifying unique clusters of capacity that support adaptive performance in specific contexts. Therefore the word agility was chosen as it represents integration, coordination and a balance of multiple capabilities amidst changing conditions. It suggests an ability to do so quickly, which aligns with adapting in a real-time DDM. At the organizational level, agility describes the capacities of the firm to respond quickly to continual and ongoing environmental changes. Therefore cognitive agility is a construct at the individual-cognitive level that predicts adaptive performance within the specific context of a real-time dynamic task.
As a formative construct, cognitive agility includes the variables of cognitive openness, focused attention and cognitive flexibility (Figure 1). These three variables operate in unison within a task that demands dynamic real-time updates. In the particular microworld used (the Networked Fire Chief [NFC]) as well as others, intelligence has shown to predict success (Ackerman, Reference Anderson1992; Brehmer & Dörner, Reference Brehmer and Dörner1993; Gonzalez, Vanyukov, & Martin, Reference Gonzalez, Vanyukov and Martin2005). This study attempts to support these findings and extend them by demonstrating that cognitive agility, measured by multiple methods, impacts adaptive performance beyond intelligence.
Formative construct of cognitive agility
Intelligence and real-time adaptive performance
Intelligence is the ability to balance the demands of the situation to adapt with success (Sternberg, Reference Sternberg1999). General intelligence (g) refers to the individual capability to ‘broadly’ comprehend one’s environment and effectively plan a response (Gottfredson, Reference Gottfredson1997). Higher levels of g are associated with greater results in tasks that are novel and complex (Hunter & Hunter, Reference Hunter and Hunter1984; Ackerman, Reference Ackerman1988). Intelligence has been shown to predict performance outcomes in DDM scenarios (Gonzalez, Vanyukov, & Martin, Reference Gonzalez, Vanyukov and Martin2005) and g has been shown to predict adaptive performance in particular (LePine, Colquitt, & Erez, Reference LePine, Colquitt and Erez2000; Zaccaro, Reference Zaccaro2001).
The DDM scenario used in this study requires one to adapt in a novel and complex environment. As past DDM studies have shown that intelligence predicts success in the changing context(s) (Ackerman, Reference Anderson1992; Gonzalez, Vanyukov, & Martin, Reference Gonzalez, Vanyukov and Martin2005; Rigas, Carling, & Brehmer, Reference Rigas, Carling and Brehmer2002), it should predict higher performance in the particular DDM – a changing scenario with conflicting goals and numerous options to choose among (Brehmer, Reference Brehmer1995).
Focused attention and real-time adaptive performance
In dynamic situations one must be able to manage the effects of incoming information within the current course of cognitive action. With ongoing changes to the environment, focusing attention can fail owing to a situational occurrence that disrupts attention (Yantis, Reference Yantis1993; Theeuwes, Reference Theeuwes1994). Adaptive performance within a task requires that focus be paid to relevant data, which provide ongoing cues to adjust activity accordingly. If focused attention were low then an individual may find his or her attentional resources diluted in relation to an intended task. In essence, he or she may become overwhelmed by information, and less able to follow a coherent decision path toward adapting well with the changes (Anderson, Reference Baddeley and Hitch1983).
Cognitive openness and real-time adaptive performance
Cognitive openness as a concept is associated with several streams of existing literature, which are each, linked to adaptability. There is not an existing performance instrument or questionnaire to measure, explicitly, the cognitive aspects of being open. Therefore, the links between openness, creativity and mindfulness are established here to support the choice of terminology and measurement.
Openness is most often cited as ‘openness to experience,’ as used in Big Five personality trait measurements (Costa & McCrae, Reference Costa and McCrae1985). Still, other constructs such as creativity (Gough, Reference Gough1979), curiosity (Littman, Reference Littman2005) and mindfulness (Langer, Reference Langer1989), while they do not always employ the term openness per se, also highlight the importance of ‘being open’ or employing related qualities of cognition with respect to dynamic contexts. The term cognitive openness describes noticing novelty and creating new associations; a skill-related creativity. In fact, openness as used in the Big Five is often conceptually related to creativity to the point that scholars use the term creativity to refer to openness (Digman, Reference Digman1990; Matthews & Deary, Reference Matthews and Deary1998).
In a similar vein, Langer’s (Reference Langer1989) work on mindfulness, suggests a propensity to cognitively take in more aspects of a situation, allowing individuals to be more adaptive. One area within mindfulness, where this is most salient, is novelty seeking. Novelty seeking from a mindfulness conceptualization and measurement perspective includes aspects of openness, curiosity and creativity (Bodner, Reference Bodner2000). In a real-time dynamic context this combined resource supports one in seeking understanding (curiosity), embracing change (novelty/openness) and searching for ways to categorize information (creativity) that could lead to adaptive performance.
Cognitive flexibility and real-time adaptive performance
As previously mentioned, cognitive flexibility is necessary for adaptive performance within DDM contexts (Cañas et al., Reference Cañas, Quesada, Antolí and Fajardo2003). Compared with the earlier exploration of the more general definitions of cognitive flexibility, this study focuses on cognitive flexibility as an executive function that supports successful adaptation by regulating cognition and overriding routinized responses (Clark, Reference Clark1996).
Cognitive flexibility as conceptualized here, is in part a metacognitive regulative capability. Metacognition, commonly referred to as ‘thinking about thinking’ (Flavell, Reference Flavell1979), includes both the knowledge and regulation of cognitive activity (Moses & Baird, Reference Moses and Baird1999). The regulation component of metacognition is synonymous to cognitive flexibility, as it encompasses bottom-up processes like monitoring (e.g., error detection, source monitoring) and the top-down processes of cognitive control that include error correction, inhibitory control, planning and resource allocation (Reder & Schunn, Reference Reder and Schunn1996; Fernandez-Duque, Baird, & Posner, Reference Fernandez-Duque, Baird and Posner2000).
Cognitive agility and real-time adaptive performance
Adapting within a real-time dynamic task requires that one flexibly operate, being both open and focused. An individual must be able to notice novelty and have the capacity for the creation and inclusion of new information (Wallach & Kogan, Reference Wallach and Kogan1965; Kozlowski et al., Reference Kozlowski, Gully, Brown, Salas, Smith and Nason2001). Cognitive openness supports noticing relevant stimuli including aspects of the situation that may be easily overlooked (Langer, Reference Langer1989; Chan & Schmitt, Reference Chan and Schmitt2000). Yet, an over inclusiveness of information can hamper individuals who see many associations between ideas but have trouble focusing on one thing. Focused attention supports the decision maker in being able to resist the handling of less relevant information (Lustig, May, & Hasher, Reference Lustig, May and Hasher2001). Therefore, the individual must be able to do both. Cognitive flexibility, the ability to cognitively control and shift mental set (Rende, Reference Rende2000), then is also necessary in real-time adaptive performance. A combination of these three capacities, as a formative construct, should support adaptive performance through a real-time changing task context. In this study we measure cognitive agility using performance tests, self-reports and external-rater reports.
Hypothesis 1: Intelligence will explain significant variance in the adaptive performance score in the DDM scenario.
Hypothesis 2: The formative construct of cognitive agility measured by performance scores will explain significant variance in the adaptive performance score in the DDM scenario beyond that explained by intelligence.
Hypothesis 3: The formative construct of cognitive agility measured by self-reports will explain significant variance in the adaptive performance score in the DDM scenario beyond that explained by intelligence.
Hypothesis 4: The formative construct of cognitive agility measured by external-raters will explain significant variance in the adaptive performance score in the DDM scenario beyond that explained by intelligence.
METHODS
Participants
Undergraduate business students volunteered to be included on a contact list to take part in university-supported research studies. The overall response rate from the research contact list was 44% (420 email invitations were sent, 195 originally agreed to participate) with an effective response rate of 43% (181/420). The total useable sample consisted of 101 males and 80 females with a mean age of 21 years.
Procedure
The study had two parts. In Part I, participants were assigned to individual computer terminals in a lab setting where they completed three questionnaires, three performance tests and multiple trials of the microworld simulation (NFC). Participants were required to provide three email addresses of people who ‘know them best’ to take a survey on their behalf. Once Part I was completed, Part II began with an automated email sent to the persons the participant provided. The email connected to a survey that included the same three questionnaires the participants completed in Part I, but were reworded to collect information about how the selected ratee views the behavior of the participant (i.e., external-rater reports). Other than a change to the pronouns (e.g., from ‘I’ to ‘Him/Her’) the items remained exactly the same.
MEASURES
Control variables
The control variables were selected based on past empirical support that have used the same microworld simulation as a performance measure.
Verbal intelligence
The basic word vocabulary test is a 40-item test that measures the number of basic words that one knows (Dupuy, Reference Dupuy1974). The measure has high internal consistency close to 0.96 (Dupuy, Reference Dupuy1974). It correlates highly with other nationally standardized measures of verbal ability to include a 0.76 median correlation with the verbal sections of the Sequential Tests of Educational Progress and the School and College Abilities Tests (Dupuy, Reference Dupuy1974).
Visual–spatial intelligence
In the Card Rotations Test subjects look at a two-dimensional shape and decide whether eight figures to the right can be rotated to match the original image or are instead a mirror image of it. It is a 224-item test with a 6 min time limit. The test has shown reliability at 0.80 for males and 0.83 for females (Ekstrom, French, & Harman, Reference Ekstrom, French and Harman1976). The test correlates significantly with the Raven’s Progressive Matrices (r=0.40) (Pallier, Roberts, & Stankov, Reference Pallier, Roberts and Stankov2000), and the Short Form Test of Academic Aptitude (r=0.41) (Burns & Gallini, Reference Burns and Gallini1983). A total of correct answers is used as a measure of visual–spatial intelligence (Ekstrom, French, & Harman, Reference Ekstrom, French and Harman1976).
Independent variables
Cognitive openness self- and external-rater reports
Cognitive openness was measured using the 6-item Novelty Seeking subscale from the Langer Mindfulness Inventory (Bodner, Reference Bodner2000). An example item from this questionnaire is ‘I seek out new information even in a familiar situation.’ Those scoring highly on this scale are more likely to look for new information. This scale correlates with the Big Five factor model of ‘openness to experience’ (r=0.50), the Multiple Perspectives Inventory (r=0.64) and is negatively correlated with the need for cognitive closure (r=−0.20) (Bodner, Reference Bodner2000). The standardized factor loadings are 0.53 to 0.62 for the novelty seeking subscale (Bodner, Reference Bodner2000).
Cognitive openness performance score
The Alternate Uses Test (AUT) challenges the participant to list as many possible uses for a common item in a timed setting (i.e., a brick and paperclip used here; given 4 min/object; Guilford, Christensen, Merryfield, & Wilson, Reference Guilford, Christensen, Merrifield and Wilson1978). The number of items generated was used as the cognitive openness performance score. Reliability for the AUT has been demonstrated from 0.62 to 0.85 (AUT manual). The AUT correlates significantly with openness on the NEO (r=0.46; Chamorro-Premuzic, Reference Chamorro-Premuzic2006), the Barron Symbolic Equivalence Test (r=0.49; Barron, Reference Barron1988) and with greater sensitivity in a habituation process (r=0.36; Martindale, Anderson, Moore, & West, Reference Martindale, Anderson, Moore and West1996).
Focused attention self- and external-rater reports
Focused attention was assessed using the 9-item Focus of Attention subscale from The Attentional Control Scale (Derryberry & Rothbart, Reference Derryberry and Rothbart1988). High scores on the scale indicate a perceived capacity to limit the influences of irrelevant information from the environment. An example item from this scale is ‘When concentrating, I can focus my attention so that I become unaware of what’s going on in the room around me.’ The total scale correlates with the Inhibitory Control Scale (r=0.25) and with Trait Anxiety (r=−0.50) (Derryberry & Rothbart, Reference Derryberry and Rothbart1988). The measure is internally consistent (α=0.88).
Focused attention performance
The go/no go is a well-known paradigm for measuring focused attention (Zimmermann & Fimm, Reference Zimmermann and Fimm2000). Participants are challenged to respond (by pressing the space bar) as fast as possible when a ‘go’ stimuli appears and to withhold from responding (by not pressing the space bar) when ‘no go’ stimuli appear. Participants were asked to memorize two 3×3 textured squares (go stimuli). Then squares appear that are the same (go stimuli) and slightly different (no go stimuli). Reaction time (at the level of milliseconds) was used as the performance score. Reliability for go/no go has been demonstrated with split half and odd even coefficients at 0.998 (Zimmerman & Fimm, Reference Zimmermann and Fimm2000). Performance scores on the go/no go correlate with the Barrett Impulsiveness Scale (r=0.40, p<.01) and perseverative error on the Wisconsin Card Sorting Task (r=−0.46) (Keilp, Sackeim, & Mann, Reference Keilp, Sackeim and Mann2005).
Cognitive flexibility self- and external-rater reports
The 35-item Regulation of Cognition subscale, from the Metacognitive Awareness Inventory, was used as a proxy measure of cognitive flexibility (Schraw & Dennison, Reference Schraw and Dennison1994). The scale commonly assesses the ability to monitor and control strategies in goal-driven situations. An example item is, ‘I find myself-analyzing the usefulness of strategies while I make decisions.’ While explicit cognitive flexibility scales exist (e.g., Martin & Rubin, Reference Martin and Rubin1995), they do not conceptually express the control and appropriate shifting of cognition as cognitive flexibility is defined in this study (the Metacognitive Awareness Inventory is the most well-aligned survey found). The cognitive regulation subscale correlates with the Motivated Strategies for Learning Questionnaire on the Individual Learning Strategies Scale (0.72; Pintrich, Smith, Garcia, & McKeachie, Reference Pintrich, Smith, Garcia and McKeachie1991). The original Metacognitive Awareness Inventory has shown strong reliability at 0.90 (Schraw & Dennison, Reference Schraw and Dennison1994).
Cognitive flexibility performance score
The Stroop Task (Stroop, Reference Stroop1935) was used to measure cognitive flexibility as a performance score. The word-color Stroop task challenges the participant to respond to a font color in which an incongruent word of a color is presented (e.g., the word green written in red font). The participant is meant to select the word of the font color (red) instead of the word itself (green). As responding to the word rather than color is the automatic reaction (MacLeod, Reference MacLeod1991), it serves as a measure of the participant’s ability to override and flexibly respond. Performance was measured for 60 trials at the level of milliseconds. The Stroop Task correlates significantly with the Self-Monitoring Scale at r=0.43 and with demonstrated reliability at 0.86 (Koch, Reference Koch2003).
Dependent variable
Adaptive performance
The NFC is a real-time DDM microworld that was used to measure adaptive performance (Omodei & Wearing, Reference Omodei and Wearing1995). The NFC was originally developed with funding from the Australian Defence Force and the Australian Research Council to study real-time decision making as it occurs in real-world fire fighting. The NFC allows participants to ‘play’ the role of a ‘fire chief’ who extinguishes fires by delivering water carried by helicopters and fire trucks (see Figure 2). The microworld is constantly changing depending on various elements (e.g., fire spreading, flame intensity and wind direction/intensity).
Networked Fire Chief
While the NFC uses the context of fire fighting it is not meant to provide direct external validity. As with other DDM microworlds, it intends to simulate conditions of dynamism presented to individuals in external working conditions (Funke, Reference Funke1991; Brehmer & Dorner, Reference Brehmer and Dörner1993). The reason for using the microworld is to ‘map the functional relationships between the variables studied in the microworld, and not necessarily the surface similarities between the microworld and a particular field setting’ (DiFonzo, Hantula, & Bordia, Reference DiFonzo, Hantula and Bordia1998: 282). Microworlds are considered especially optimal for studies investigating highly complex phenomena in a dynamic context (Brehmer, Leplat, & Rasmussen, Reference Brehmer, Leplat and Rasmussen1991; Brehmer & Dorner, Reference Brehmer and Dörner1993). The NFC has demonstrated good reliability at 0.70 (Omodei et al., Reference Omodei, Wearing, McLennan, Hansen, Clancy, Elliott, Ley, Taranto and Thorsteinsson2001). Performance scores on the NFC have shown significant correlation with the core perceptual-cognitive characteristics of naturalistic decision making; namely, speed, accuracy, efficiency and planning (ranging from r=0.75, p<.01 for proactive planning to r=0.36, p<.05 for the percentage of time the closest appliance was allocated first; Elliot, Welsh, Nettelbeck, & Mills, Reference Elliot, Welsh, Nettelbeck and Mills2007). A performance score is produced by the program, which is calculated adding every safe cell (i.e., unburned). Demonstrating gains over losses after an individual has interacted with a changing environment is a way of calculating adaptive performance (Baltes & Staudinger, Reference Brehmer1996). A composite for the three trials (15 min in total) was used as a measure of adaptive performance.
RESULTS
Descriptive statistics
Table 1 shows descriptive statistics, bivariate correlations and reliabilities (in parentheses). All variables demonstrate reliability scores that are above 0.70. All variables with the exception of the external-rater reports for openness and focused attention correlate significantly or reach near significance with the dependent variable (DV). Contrary to expectation, the external-rater and self-report measures of cognitive flexibility were negatively correlated with the DV. This will be discussed in detail below and has implications for further analyses.
Descriptive statistics
Note. N=181. Reliabilities are given in parentheses on the diagonal.
† p<.1; *p<.05; **p<.01.
Factor analysis
Given the proposed formative nature of the cognitive agility construct, one way to further assess construct validity before regression analysis is through factor analysis (Rossiter, Reference Rossiter2002). Results of this analysis appear in Table 2. The factor analysis shows three factors, loading by method, with the exception of focused attention self-report, which loads weekly (0.331) on factor 3. Owing to the formative nature of the proposed construct, this variable was left in the analysis, as its removal would violate structure (Jarvis, MacKenzie, & Podsakoff, Reference Jarvis, MacKenzie and Podsakoff2003). Results support grouping the variables for adequate regression analyses.
Pattern matrix
Note. N=181. Principal component analysis with a promax rotation with three factors.
Owing to the unpredicted negative correlation that the cognitive flexibility self-report and external-rater report had to the DV, it was not possible to create a proper formative or composite score for these methods. Therefore, reliability analysis was run to see the α score for factors 2 and 3 established through the factor analysis. The collective group of external-rater reports had a Cronbach’s α of 0.70, which suggests a strong grouping. The self-report measures had an α of 0.50, which is weak yet adequate. The performance measure did not require a reliability score as it will be combined into a composite score or formative construct (Jarvis, MacKenzie, & Podsakoff, Reference Jarvis, MacKenzie and Podsakoff2003).
Tests of hypotheses
Before analysis, the two intelligence scores were formed in a factor score as both are well-validated instruments (Hair, Anderson, Tatham, & Black, Reference Hair, Anderson, Tatham and Black1998). The verbal (crystallized) and spatial measures (fluid) of intelligence when combined, create a general cognitive intelligence factor (Cattell, Reference Cattell1963; Horn, Reference Horn1985).
Hypothesis 1:
Table 3 shows all the regression results. Model 1 introduced the intelligence factor. Results demonstrate a significant positive relationship of the intelligence factor with the adaptive performance outcome on the NFC, supporting Hypothesis 1. The intelligence factor was then used in each of the subsequent models as a control variable.
Regression table for cognitive agility measures
Note. N=181.
† p<.1; *p<.05; **p<.01.
Hypothesis 2
Hypothesis 2 predicted the formative construct of cognitive agility, as measured by the performance tests, would demonstrate significant variance beyond the intelligence factor. Model 2 shows regression results for cognitive agility, as measured by the performance test beyond intelligence, providing 11% unique variance on the NFC. Hypothesis 2 was supported. This hypothesis was tested in a prior study with an alternative theoretical framing (Good & Michel, Reference Good and Michel2013).
Hypothesis 3
Hypothesis 3 sought to demonstrate that the formative self-report construct (composite of all three self-report measures) of cognitive agility would explain significant variance in adaptive performance beyond intelligence. As mentioned earlier, given the negative correlations that both of the cognitive flexibility scales (self-report and external-rater report) had to the DV, the self-reports and external-rater reports could not be made into a composite or true formative measure; as the negative correlations would dilute the power of the measure. Therefore, as an alternative, linear regressions were used entering all three self-report measures into a single block listwise. This allowed each of the individual measures to interactively demonstrate impact on the DV rather than have the cognitive flexibility measure(s) dilute the results with a negative correlation. Model 3 shows that Hypothesis 3 was upheld as the addition of the self-reports accounted for 6% unique explained variance beyond intelligence in the adaptive performance score. Though a closer examination of the results demonstrate findings that are more nuanced. Focused attention does not add any unique variance (β=0.05, p=.49) and cognitive flexibility has an inverse relationship to the DV (β=−0.19, p=.01).
Hypothesis 4
Hypothesis 4 predicted that the formative construct of cognitive agility measured by external-raters will explain significant variance in the adaptive performance score in the DDM beyond that explained by intelligence. Like the self-reports, a linear regression was used entering all three external-rater measures into a single block listwise as an alternative to a composite score. Model 4 shows that Hypothesis 4 was upheld as the total model explained 6% unique variance on the DV. Like the self-reports, the external-reports have more nuanced results beyond the total change in R 2, including a significant negative association that cognitive flexibility has to the DV (β=−0.32, p<.01).
DISCUSSION
The purpose of this study was to better understand the cognitive capabilities and orientations that support successful adaptation within a real-time dynamic context. While adaptability has become a popular buzzword in management and has been subject to an increasing degree of investigation, individual differences leading to adaptive performance within real-time dynamic tasks remain unclear. Given the increase of DDM demands individual face in organizations, additional investigation is warranted.
As hypothesized, the formative construct of cognitive agility, as assessed by three methods (performance, self-reports and external-rater reports) demonstrated significant variance beyond intelligence in the DDM microworld. Most notable was the impact of cognitive agility as measured by the performance scores (ΔR 2=0.11, p<.001). This finding is meaningful given that previous research using microworlds seldom goes beyond general intelligence in assessing variance of additional variables. In a similarly designed simulation study, both conscientiousness and openness to experience (as measured by the NEO) accounted for 2% unique variance beyond intelligence (LePine, Colquitt, & Erez, Reference LePine, Colquitt and Erez2000). This gives a frame of reference for the potential significance of this finding.
Overall, the performance tests are a more powerful set of predictors in the DDM context than the self- or external-rater reports. This makes theoretical sense based on the implicit nature of the performance tests and the fire chief exercises (Dovidio, Kawakami, & Beach, Reference Dovidio, Kawakami and Beach2001). The self- and external-rater reports are missing this implicit nature, and may have a lower degree of consistent predictability with the DV. Furthermore, each of the performance tests and the DV have a time-intensive component to them; whether it be reaction time (Stroop and go/no go), product creation (in the case of the AUT) or decision speed (with the DV).
Contrary to expectation, negative relationship(s) was found between the questionnaire used for cognitive flexibility and the NFC (self-report β=−0.19, p<.01; external-rater β=−0.32, p<.01). These cognitive flexibility measures were assessed using a metacognitive regulation subscale. Metacognitive regulation accounts for control of cognitive operations leading to greater flexibility (Fernandez-Duque, Baird, & Posner, Reference Fernandez-Duque, Baird and Posner2000). Research shows that metacognitive capacities are linked to expert decision making (Chi, Feltovich, & Glaser, Reference Chi, Feltovich and Glaser1991), successful problem solving (Mayer, Reference Mayer1998) and increased adaptability in uncertain and dynamic contexts (Earley & Ang, Reference Earley and Ang2003). Therefore, it was predicted that metacognitive regulation (i.e., monitoring and control over one’s cognition) would lead to increases in adaptive performance. The results did not support this assertion.
DDM microworlds require a series of continual and real-time rapid decisions, which may represent a completely distinct class of performance, separate from single decisions (Sitkin & Pablo, Reference Sitkin and Pablo1992). Perhaps the ongoing and constant decisions that must be made require one to be less ‘regulated’ in how one thinks. In such a dynamic context, speed of operation compliments accuracy of decisions. Strong metacognitive regulatory tendencies may increase a participant’s likelihood to monitor and evaluate his/her ongoing perception of success and failure (Jacobs & Paris, Reference Jacobs and Paris1987; Schraw & Dennison, Reference Schraw and Dennison1994). Past studies indicate that as one detects error they tend to slow down speed of operation in order to increase accuracy (Rabbit, Reference Rabbit1966; Robertson, Manly, Andrade, Baddeley, & Yiend, Reference Robertson, Manly, Andrade, Baddeley and Yiend1997). Therefore, in this context, participants with strong metacognitive regulation may be more engaged in online monitoring and therefore more aware of possible errors. This awareness may slow down processing at a time when doing so may cause performance decrements. This performance deficit may also be linked to literature on intuitive thought, which suggests that such metacognitive activity creates interference in necessary unconscious material (Kuhn, Reference Kuhn1989; Baylor, Reference Baylor2001). Given the short time horizon of the microworld used here (15 min), slowing down and controlling cognitive operation may have led to a decrease in performance. This may suggest some practical implication for training in which individuals regularly engaged in DDM activity can learn to switch from conscious to automatic processing as real-time dynamism increases (Louis & Sutton, Reference Louis and Sutton1991).
Cognitive agility describes the flexibility between what can be thought of as opposing phenomena (openness and focus). Yet, managing and leading in organizations has been said to require the ability to employ the simultaneous use of opposites (Quinn, Spreitzer, & Hart, Reference Quinn, Spreitzer and Hart1992). Rather than consider them in opposition the goal is to enhance the ‘the ability to act out a cognitively complex strategy by playing multiple, even competing roles, in a highly integrated and complementary way’ (Hooijberg & Quinn, Reference Hooijberg and Quinn1992: 164). Adding to this, organizational ambidexterity (a firm’s ability to manage the dilemma between exploration and exploitation) has been increasingly conceptualized as multi-level phenomena to include an individual-level ambidexterity construct (Mom, van den Bosch, & Volberda, Reference Mom, van den Bosch and Volberda2007). It is likely that there is an individual difference in being able to manage this dilemma well, yet empirical evidence is sparse (Gupta, Smith, & Shalley, Reference Gupta, Smith and Shalley2006). Drawing from a small slice of the data in the current paper, the author has made such a theoretical connection to individual ambidexterity (Good & Michel, Reference Good and Michel2013), suggesting a similarity between being agile and ambidextrous. The more dynamic the context becomes, the more individual ambidexterity is thought to be important for adaptation (Davis, Eisenhardt, & Bingham, Reference Davis, Eisenhardt and Bingham2009). Such dynamism will likely make the management of tension(s) a more important future area of interest for individual difference studies. It may be important to investigate how a capacity like agility can inform other common tensions experienced in organizational life, such as converging/diverging, differentiation/integration and advocacy/inquiry.
Limitations and conclusion
This study has several major limitations that when addressed may provide future research opportunities. This was an initial step in proposing a new construct, and therefore a great deal of future work toward validation is necessary. This could include using a more robust measure of intelligence as a control variable (Salgado, Reference Salgado1999). Future research may consider other variables that could be related to DDM adaptive performance such as working memory (Baddeley & Hitch, Reference Berg and Sternberg1974), situational awareness (Endsely, Reference Endsley1995) and tolerance for ambiguity (Endres, Chowdhury, & Milner, Reference Endres, Chowdhury and Milner2009). The testing scenario of the microworld and the undergraduate sample population substantially limits generalizability of the results to DDM of individuals in organizations. In particular, the choice to use a microworld was as a way to capture the elements found throughout the dynamic experiences individual encounter often within organizational life (Funke, Reference Funke1991; Brehmer & Dorner, Reference Brehmer and Dörner1993; Omodei & Wearing, Reference Omodei and Wearing1995). While few would argue that experiences in organizational life have become more dynamic and complex, the results produced by this microworld-based laboratory study still need to be handled with caution. Results pertaining to cognitive agility and the individual variables which form it (cognitive openness, focused attention and cognitive flexibility) are promising in being able to predict aspects of adaptive performance, yet, this outcome is within the context of a computer-simulated game and laboratory-based decision-making studies do not fully capture real-life decision making (Dawes, Reference Dawes1988).
In conclusion, this study suggests that the formative construct of cognitive agility, as measured by the three performance tests, predicts adaptive performance in the DDM scenario beyond measures of general intelligence. This finding suggests that the unique combination of cognitive openness, focused attention and cognitive flexibility (as they are measured here), may be an important cluster of capabilities in managing real-time dynamic contexts. However, the inverse relationship of the cognitive flexibility questionnaire(s) to the DV raises important questions about the roles of cognitive regulation, cognitive control and cognitive flexibility in real-time DDM contexts. Overall, this study investigates specific capabilities targeted to a specific context. Studies of highly contextualized aspects of adaptive performance are vital to our development in meeting the demands for research and practice in dynamic times.
