INTRODUCTION

This paper addresses the issue of the provenance of clinical tests of executive function, and how best to set about developing the next generation of clinical measures. We will argue that whilst traditional tests of executive function have been remarkably useful, we are now at the stage in the development of the field where one could create bespoke tests specifically intended for clinical applications rather than adapting procedures emerging from purely experimental investigations, as has been almost exclusively the case until recently. We will further argue that whilst one should continue to consider as potentially useful these procedures which emerge almost coincidentally, a more planned and deliberate programme should consider a “function-led” approach as a particularly fruitful avenue. Such an approach may have many scientific and clinical advantages, and address recent criticisms of existing assessment procedures (e.g., Manchester et al., 2004). But to make these points, it will first be necessary to define our terms:

Argument 1. The distinction between construct, cognitive operation, and behavioural function: A taxonomy for discussion.

The term ecological validity, first coined by Brunswik (1956, pp. 48–50; see Hammond, 1966 for review), was originally used to refer to the degree of relation between a proximal (e.g., retinal) cue and the distal (e.g., object) variable in perceptual experiments (see Hammond, 1998, for discussion). Now, ironically, it is most often used in reference to quite separate concerns of Brunswik's, those of “generalisability” and “representativeness.” This misuse (from a historical perspective) is now so prevalent that we will, somewhat reluctantly, adopt the “incorrect” use of the term here. We will adapt the nomenclature of Kvavilashvili and Ellis (2004) for the current application by using “ecological validity” to refer both to the degree of “representativeness” of a task (the extent to which a clinical test corresponds in form and context to a situation encountered outside the laboratory), and the “generalisability” of test results (the degree to which poor performance on the test will be predictive of problems outside the laboratory; for recent examples of this use see Chaytor & Schmitter-Edgecombe, 2003; Clark et al., 2000; Coughlan et al., 2004; Farias et al., 2003; Gioia & Isquith, 2004; Higginson et al., 2000; Kibby et al., 1998; McDonald et al., 2004; Odhuba et al., 2005; Semkovska et al., 2004).

The foregoing argument will also require clear distinctions between the levels of explanation used in the cognitive neuroscience of executive function. So we will start with a very basic description of terms. We will make a distinction between Constructs, Operations, and Functions (see Figure 1). (Perhaps the most commonly used term, Processes, will not be used here since it is a term which has been used so flexibly in the literature.) Construct refers here (as is conventional) to a hypothetical cognitive resource whose existence is inferred from research findings (e.g., a correlation between two variables). Common examples of constructs are “Working Memory,” and “General intelligence.” These are ideas which motivate experiments, and are interpretations of data sets rather than phenomena that are part of them. In this respect, it is important to note, as Coltheart and Davies (2003) point out, that “The truth of an explanatory theory goes beyond the data and so is never logically guaranteed by the data” (p. 188).

Levels of explanation in the cognitive neuroscience of executive function.

Operations in the current context are the individual component steps of cognition which are not directly observable, but are inferred from a combination of task analysis and change in some dependent variable. In a different circumstance this could signify the involvement of some other operation. They may or may not be broken down into subcomponents. Many experimental observations in the cognitive neuroscience or neuropsychology of executive function would fall into this category. For instance, increases in reaction time or errors, changes in cerebral blood flow and so forth, have all been held to be indicators (i.e., observable correlates of) of cognitive operations. It is important to note here that operations can be understood at the level of the individual rather than the individual's interaction with the environment, and that this is another level at which they differ from functions. The operation of single-word reading can happen in the mind without any outward behavioural sign (e.g., speech), but be accompanied perhaps by changes in blood flow in the reading areas of the brain. It is also important to note that operations are behavioural changes whose utility (rather than existence) can be made only in reference to outcome in the world. In one context, the absence of a behaviour can be a useful thing—for example, withholding a behaviour in a motor inhibition task—but in another may be evidence of dysfunction—say, inability to produce a word in the Hayling Sentence Completion Task part 2 (Burgess & Shallice, 1996). Functions, by contrast, are the directly observable behavioural output which is the product of a series of operations. They are usually understood in terms of a goal, instruction or intention to act, and it is therefore usually clear what constitutes success or failure. Simple examples might be reading, naming objects, describing a previously heard story; more complex behavioural examples might be preparing a meal, mailing a letter at a previously intended time, and so forth.

An example of the use of this kind of taxonomy appears in the excellent study by Owen et al. (1999). In the abstract to the study, the purpose of the study is described thus: “[this] study used positron emission tomography (PET) to demonstrate that working memory processes within the human mid-dorsolateral and mid-ventrolateral frontal regions are organized according to the type of processing required…. Two spatial working memory tasks were used which varied in the extent to which they required different executive processes…. During a ‘spatial span’ task that required the subject to hold a sequence of five previously remembered locations in working memory a significant change in blood-flow was observed in the right mid-ventrolateral frontal cortex…. By contrast, during a ‘2-back’ task that required the subject to continually update and manipulate an ongoing sequence of locations within working memory, significant blood flow increases were observed in both mid-ventrolateral and mid-dorsolateral frontal regions (Owen et al., 1999, p. 567). It is clear from this description (and the rest of the paper) that the operations of “holding a sequence of numbers” in mind, and continually updating and manipulating an ongoing sequence are considered potentially fractionable, but both fall within the general cognitive domain of “working memory” (a construct).

Argument 2. The study of “central processes” makes special demands: From theory to practice in the cognitive neuroscience of executive function.

Burgess (1997) argues that the field of executive function (EF) differs from most others in cognitive neuroscience in the differing emphasis on these levels of explanation. Specifically, the use of constructs in the field of executive function is much more prevalent. This is largely because of the “low process-behaviour correspondence” (pp. 85–89) inherent in the study of “central processes” compared with other fields which deal with more “routine processing” or “encapsulated resources” (Shallice, 1988; both terms assuming a particular organisation of the cognitive system). The abilities supported by the frontal lobes are thought to operate largely on the products of processing of other systems, and also output through other systems. Thus the operation of the “central executive” (Baddeley, 1996) or “supervisory attentional system” (Shallice, 1988) that the frontal lobes are thought to support is bound up with the operation of other systems to a degree that the nonexecutive components of, say, vision or hearing are not. Anatomical support, for instance, comes from the absence of direct pathways between prefrontal cortex and motor cortex (Leichnetz, 1986; Passingham, 1993, p. 125). And the clearest supporting demonstration is that, unlike with other cerebral regions, destruction of prefrontal cortex is not associated with loss of basic sensory or motor capacities or with obvious impairment of speech (Benton, 1991, p. 3). Linked to this is that at least some of the processing supported by prefrontal structures has been shown to be material-nonspecific (Burgess et al., 2001; 2003; Gilbert et al., 2005). In other words, it operates on input from a variety of sources, with a corresponding lack of dedicated process-output pathways.

This contrasts sharply with conceptions of fields of study dealing with behaviour deriving from the operation of abilities which are supposed to be more dedicated to it. As an example of an “encapsulated resources” theory, Burgess (1997) considers Bruce and Young's (1986) model of face recognition (where the behaviour of seeing a face and identifying would be a function, and the theoretical individual contributory components to this behaviour—such as recognition—are the operations). And although more recently, the debate concerning how people identify faces has taken a more complex stance, contrasting these “domain-specific” accounts with “domain-general” ones, where visual recognition proceeds through the operation of resources which are specialised for a particular form of processing that can operate on a wide variety of classes of stimuli (Duchaine et al., 2003, 2004; Kanwisher, 2000), the processes are still assumed to be specialised for the processing of visual stimuli. This general principle could be extended to most other areas of enquiry within cognitive neuroscience, such as visual perception, reading, spoken language, motor control, audition, and so forth, where the predominant theories all hold that there are resources dedicated to that domain of cognition, if not the modality (see, e.g., MacSweeney et al., 2004, for an example within the auditory linguistic processing domain) or at least are agnostic as regards this possibility. By contrast, according to most conceptions, executive processes theoretically manifest themselves in a range of quite different situations, involving quite different stimuli, the only unifying feature of which might be the involvement of that process.

To illustrate the universality of this viewpoint, we will consider briefly the opinion of six leading theories of executive function. The most obvious example of a theory of the “central processing” kind is Duncan's (e.g., Duncan, 2005; Duncan et al., 1995) contention that “executive processes” are synonymous with Spearman's g, and his recent “adaptive coding” model is a clever theory extending this assertion by addressing the possible neural underpinning of g (e.g., Duncan & Miller, 2002). However, there are many other theories which share this domain-nonspecific aspect (see, e.g., Burgess & Alderman, 2004; Burgess & Robertson, 2002 for brief review). For instance the theory of Cohen and colleagues (e.g., Cohen & Servan-Schreiber, 1992; Cohen et al., 1990, 1998) holds that their single system may underpin both active memory and behavioural inhibition, determined by the current context, and they specifically state that their own theory is incomplete as regards the representation of many different information types, and this aspect is left as a lacuna in the model.

This aspect of “central processing” (i.e., domain- and material-insensitive) is also reflected in “working memory” theories of executive function. For example, Petrides (1994, 1998; Petrides & Pandya, 2002) maintains that the mid-ventrolateral PFC “subserves the expression within memory of various first-order executive processes, such as (the) active selection, comparison and judgement of stimuli,” (p. 107) without specifying further what those stimuli might be. Goldman-Rakic's (1995) working memory account does however propose some basic domain- or material-specificity since the key aspect of it is that the various different frontal lobe regions all perform a similar role in working memory, but that each processes a different type of information. However, the account still holds that different forms of observable behaviour may result from dysfunction in one central processing resource: Working memory is defined as the ability to “hold an item of information ‘in mind’ for a short period of time and to update information from moment to moment,” (Goldman-Rakic, 1998, p. 90) and it is argued that dysfunction of this system can cause a variety of deficits (see Goldman-Rakic & Leung, 2002, for further discussion.).

Even those theories that propose a series of potentially dissociable executive resources still maintain this centrality (e.g., Fuster 1997; 2002), but this characteristic is most obvious in those executive accounts which rest upon the concept of “attention.” For example, Stuss et al. (1995) propose seven different attentional functions: sustaining, concentrating, sharing, suppressing, switching, preparing, and setting. But since there is no specification of the specific situations or material which require the differential use of these resources, the reader is invited to assume that they operate regardless of the form or source of the input source. Even the most complex model in this domain, that of Shallice (e.g., Norman & Shallice, 1986; Shallice, 1988, 2002; Shallice & Burgess, 1991a, 1991b, 1993, 1996; Shallice et al., 1989) still utilises the concept of attention, where the possibility of domain- or material-specificity is either denied or left as a lacuna. Most recently, the putative organisation of the SAS has been articulated in more detail (Shallice & Burgess, 1996/1998). In this model, the SAS plays a part in at least eight different processes: working memory, monitoring, rejection of schema, spontaneous schema generation, adoption of processing mode, goal setting, delayed intention marker realisation, and episodic memory retrieval. However it is largely assumed that these central processes are independent of modality (of input or output). (For further evidence see also computational accounts, e.g., Dehaene et al., 1998, and representational accounts, e.g., Grafman, 2002.)

Argument 3. “Construct hegemony”: Historical hangover?

The field of executive function has been dominated by these construct-level, “macro” theories for the last 20 years in particular. By contrast there are relatively few accounts of the processing required to deal with specific situations which might tap these constructs. The extent to which this is the case may in fact in time become of interest to historians of science. For instance, one of the earliest and most compelling descriptions of cognitive deficit following frontal lobe damage was given by Penfield and Evans (1935; see also studies by Ackerly & Benton, 1947; Brickner, 1936). This seminal paper described a catastrophic impairment in the higher-order organisation of behaviour, manifesting itself for instance particularly sharply as an inability to coordinate the various activities involved in cooking a meal (see also Fortin et al., 2003). Given this observation, one might imagine that by now we would know quite a lot about the processes involved in cooking a meal, since Penfield and Evan's astute and influential observation in the context of a relative dearth (at that time) of empirical evidence would surely have suggested a “cardinal” situation for tapping the cognitive processes supported by the frontal lobes, and opened up a promising avenue for empirical studies. But this is not the case at all. We know virtually nothing about how the brain allows us to organise simple, everyday activities like cooking, and interacts with environmental factors in doing so (although there are a few honourable exceptions, see, e.g., Kirsh, 1995). Instead, EF research has spent several decades investigating the dynamics of (by comparison) esoteric activities such as performing the Wisconsin Card Sorting Test.

So how and why did EF research develop in this way? This may at least in part be a consequence of the history of research in the field of executive/frontal lobe function. The very existence of cognitive abilities supported by the frontal lobes was doubted for many years (Benton, 1991, p.19). Indeed, whilst early experimental physiologists were making sound progress in the fields of visual perception and language more than a hundred years ago, frontal lobe function was beset with null findings and failures to replicate effects across species. The scepticism this engendered was repeated in the observation of humans with frontal lobe damage: Even the facts of what was to become probably the most famous case of the “frontal lobe syndrome” (Harlow's [1848; 1868] Phineas Gage) were not believed by some at the time of its report. This attitude persisted well into the 1960s: For example, Teuber (1964), one of the leading authorities of the time, referred to the functions of the frontal lobes as a “riddle.” In retrospect this lack of development in the area seems odd: The reports of the effects of frontal lobe damage in humans were actually quite consistent, and there also many successful demonstrations in primates (e.g., Jacobsen, 1931; 1936). Moreover, the effects of damage to frontal lobe and related structures was considered predictable enough for frontal lobe psychosurgery to thrive (e.g., Malmo, 1948) during this period. The scepticism persisted, however, and one might argue that the tendency of investigators to assign ill-defined labels to the behaviours they saw (e.g., “perseveration”), rather than investigate the characteristics of the often myriad situations that provoked them, meant that research became quite conservative in its method (i.e., tending to use the same tasks and paradigms), whilst also—in the absence of constraining empirical evidence—rather ambitious in the scope of its theorising.

However, the past 25 years have seen rapid development in the cognitive neuroscience of executive functions, prompted in particular by technological advance (e.g., new methods of brain imaging, better methods of localising lesions). The repeated and sometimes unexpected findings of prefrontal involvement in even the simplest tasks has overthrown the old view of the frontal lobes as “functionally silent,” and now no leading theoretician of whom we are aware would maintain that the frontal lobes do not support “executive” (Pribram, 1973) abilities that are critical to human cognition.

Such a rapid development has however not been without cost, and it could be argued that the rate of innovation of method in the field of executive function has not matched the rate of technological advance. The majority of the early research papers using the new technologies (e.g., functional brain imaging) understandably leaned towards using the previously existing tools, principally (for example), because of the need to validate new technologies. This may have had the unintended effect of prolonging the use of these older tests and procedures both experimentally and clinically. Moreover, the rapid swing (in historical terms) from one extreme position (i.e., the frontal lobes are functionally silent, ergo “executive functions” don't exist) to the other (i.e., the frontal lobes support executive processing which is involved in a very wide range of situations [see, e.g., Burgess et al., 2005b, 2006; Grady, 1999 for reviews]) has invited explanation at the construct level. Moving beyond this level is a challenge: The empirical findings in the field of executive functions of course support the newer view at the broadest level, but are still beset with complications. For example, Stuss et al. (2000, 2002) have shown that relatively small changes in task format can have marked differences in terms of demands upon executive abilities (see also Burgess, 1997, for review of methodological difficulties), and the sheer volume of recent findings presents its own conceptual difficulties. For example, in trying to understand the functions of the largest single cytoarchitectonic PFC region (Brodmann's Area [BA] 10), findings of BA 10 PFC blood flow changes (see, e.g., Gilbert et al., in press; Grady, 1999) in a very wide range of types of tasks in imaging studies provides few constraints for theorising (see Burgess et al., 2005b for discussion).

The combination of these interrelated factors—historical inheritance, rapid conceptual change; numerous but often complex and unpredictable empirical findings—plus the practical and technical difficulties of investigations at the function level has, we would contend, invited accounts of executive function at the construct level, and for this reason basic scientific investigations (i.e., contrasted with applied or clinical investigations) have been, until very recently, overwhelmingly driven by this perspective.

This construct-level, or construct-to-operation-level theorising, combined with the experimental conservatism engendered by the historical research difficulties has led to research into executive abilities largely concentrating upon these two facets (i.e., examination of particular experimental, and clinical, paradigms on the one hand, and construct-level theorising of the type above on the other) whilst largely ignoring the issue of the translation of the hypothesised resources into the real world. The dynamic interplay between situational factors and the hypothesised resources (i.e., function-level analysis) has received very little attention. This has led to gaps in the links between the three explanatory levels which are required in order to link explanations at the basic science level with those required in clinical practice (i.e., my hypothetical resource can be termed Alpha; it supports cognitive operations X, Y and Z; and these resources are tapped by, or used to deal with, situations which have characteristics A, B, and C). Yet it is precisely at this level that clinical tests of executive function should in large part be targeted since in clinical practice the most important issue is usually performance outside the testing situation. How odd that one should not try to mimic (or at least understand) the demands of life outside the clinic when trying to determine the generalisability of test performance within it! Instead a common practice in clinical work and applied research has been to use performance on tests which emerged—often many years ago—from traditional experimental psychological investigations. The standpoint of these original studies has largely been to view the variability in contextual or environmental factors which naturally occur outside the lab as a confound (see Kvavilashvili & Ellis, 2004, for discussion of the long history of similar issues within the memory research domain). But it is exactly at this level (i.e., the interaction between the individual and their context: the “function” level) in which the clinician is interested. This dominance of the construct level investigation in neuroscience executive function research is reflected in (or probably reflects) the historical focus of the related field of cognitive science on the formation and manipulation of internal representations, rather than the ways in which agents interact with their environments in order to achieve goals (see, e.g., Clark, 1997; Hutchins, 1995).

Argument 4. Time to consider a new approach?

One can add to these criticisms that traditional tests were not developed to address the concerns of clinicians—that is, to measure the most clinically significant deficits and do so in a way that makes the implications of the test score quite plain.

Consider for example, the history of, and current application of the currently most used and most thoroughly investigated single measure in the field of human frontal lobe function: the Wisconsin Card Sorting Test (WCST). This test was developed (in almost its current form) by Esta Berg in 1948, and slightly modified by Grant and Berg (1948). Both these studies involved only healthy participants, and the procedure was merely a variant of those used much earlier. In fact, the origin of the association between sorting tasks and frontal lobe function can be traced to Gelb and Goldstein (1920), who first developed sorting procedures for assessment of brain damage following the First World War. They adapted for clinical use the methods that had been used by Ach and his colleagues to investigate concept formation in healthy individuals. Following Gelb and Goldstein's discoveries, there were many investigations of sorting impairments in various patients groups, including those with brain-damage, dementia, learning difficulties, and schizophrenia carried out by groups in Germany and Russia (see Weigl, 1927; Goldstein & Scheerer, 1941 for review). The translation into English of a number of these papers during the 1935–45 period was followed by a rapid increase of interest in these methods in the U.S.A. Unfortunately, the reluctance of the American authors of the time to cite the original papers rather than their translations gives the appearance that these later papers were more contemporary than they were.

It is an important historical irony that by the time the WCST was developed, there were already tried and tested (by the standards of the time) sorting procedures available for clinical use. Yet it is the WCST, which was not developed for clinical use, that has now become dominant. This is largely attributable to the effect of a study by Milner (1963) who, more than 40 years since the original demonstration of sorting impairments, administered the Grant & Berg procedure to a group of neurological patients, with virtually the only modification being to give the test twice in succession.

Milner's principal achievement was to show that patients with dorsolateral prefrontal lesions performed worse on the test than those with either orbitofrontal or lesions outside the frontal lobes. However this finding has never to our knowledge been entirely replicated, either by Milner or by any other investigator, although there are many failures to replicate (see, e.g., Anderson et al., 1995; Corcoran & Upton, 1993; van den Broek et al., 1993). Psychometric studies have now shown that the WCST is a multifactorial test (e.g., Greve et al., 1997; Miyake et al., 2000). These results are congruent with those from recent functional imaging studies that show a network of frontal and nonfrontal brain regions to be involved in WCST performance (e.g., Marenco et al., 1993; Mentzel et al., 1998), and similar results from careful human lesion work (Stuss et al., 2000). Both methodologies stress how relatively minor changes in task format can have marked effect upon results.

Now consider in this context the typical uses of the WCST in clinical assessment. It is possible to give a rationale for understanding the three levels of WCST based on current common conceptions thus: WCST is a measure of “working memory” (construct), which supports “concept shifting” (operation), which enables correct choice of card sorting (function). However, the explanation at the level of construct is not very helpful in clinical assessment at the individual level, since such explanations go beyond the data and thus belong in the field of scientific hypothesis-testing rather than patient diagnosis. And the outline of the empirical findings above suggests explanation at the level of operation is not well-established (e.g., there may be many different reasons for failure, especially in different populations). Finally—and this is a particularly critical problem—the situation that performance of the WCST presents is so unlike everyday situations, that knowledge of performance in it is of very little help for assessment since it is of uncertain predictive validity (or “generalisability”): We do not really know what situations in everyday life require the abilities that the WCST measures.