Introduction
Past research has shown that executive functions play an important role in predicting everyday functioning in psychiatric patients (Green, Reference Green1996; Green, Kern, Braff, & Mintz, Reference Green, Kern, Braff and Mintz2000; Jaeger & Douglas, Reference Jaeger and Douglas1992; Velligan, Bow-Thomas, Mahurin, Miller, & Halgunseth, Reference Velligan, Bow-Thomas, Mahurin, Miller and Halgunseth2000). From a diagnostic and therapeutic perspective, the ability to measure executive functioning reliably and in an ecologically valid manner, therefore, is a relevant issue. This article focuses on how the measurement of planning ability can be improved by using the computer-based test “Plan-a-Day,” which simulates day planning scenarios in a work setting. We present data from patients with schizophrenia, but propose that the test can also be used in other populations.
Deficits in executive functioning in general and planning difficulties in particular are commonly observed in patients with schizophrenia (e.g., Morris, Rushe, Woodruffe, & Murray, Reference Morris, Rushe, Woodruffe and Murray1995; Pantelis et al., Reference Pantelis, Barnes, Nelson, Tanner, Weatherley, Owen and Robbins1997). A possible explanation for this could be difficulties in translating plans into “willed intentions,” which may be related to impaired frontrostriatal connectivity (Frith, Reference Frith1987; Robbins, Reference Robbins1990). More recently, Burgess, Dumontheil, and Gilbert (Reference Burgess, Dumontheil and Gilbert2007) suggested a neuro-cognitive explanation that may underlie several executive deficits and seems particularly applicable to planning difficulties. According to their gateway hypothesis of rostral prefrontal cortex function, the ability to switch between external stimulus-oriented representations (e.g., the task at hand) and internal stimulus-independent thought (e.g., creative thoughts, plans) may be impaired in patients with executive deficits. As planning ability is highly relevant for many every-day activities such as shopping, cooking, or successfully participating in the job market, valid measurement of this ability seems desirable from a diagnostic point of view and may be useful for psychiatric rehabilitation.
Ecological Validity of Tests of Executive Functioning
The question of how to improve the ecological validity of neuropsychological tests of executive functioning is a matter of ongoing debate (e.g., Burgess et al., Reference Burgess, Alderman, Forbes, Costello, Coates, Dawson and Channon2006; Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003; Wilson, Evans, Emslie, Alderman, & Burgess, Reference Wilson, Evans, Emslie, Alderman and Burgess1998). In this context, ecological validity can be defined as “the degree to which test performance corresponds to real-world performance” (Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003, p. 182). Many existing tests of executive functioning were originally used as experimental laboratory tasks and only later repurposed as neuropsychological diagnostics, for example the Stroop task or the Tower of London and its variants (Burgess et al., Reference Burgess, Alderman, Forbes, Costello, Coates, Dawson and Channon2006). These tests are easy to administer and score, have reasonable psychometric reliability and show satisfactory convergent validity with respect to other standard tests of cognitive ability. However, their relation to real-life outcome measures, such as work performance or activities of daily living is often unclear or lacking (Burgess, Alderman, Volle, Benoit, & Gilbert, Reference Burgess, Alderman, Volle, Benoit and Gilbert2009; Manchester, Priestley, & Jackson, Reference Manchester, Priestley and Jackson2004; Shallice & Burgess, Reference Shallice and Burgess1991). This is not entirely surprising, considering how little many standard neuropsychological tests resemble the challenges and complexities of everyday life. When the main purpose of neuropsychological testing is to assist with the diagnosis of brain pathology, this is not necessarily a problem. However, with a rising interest in ecological validity and predicting functional outcome (Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003; Manchester et al., Reference Manchester, Priestley and Jackson2004), it becomes pertinent to ask to what extent existing neuropsychological test procedures possess ecological validity and how it can be increased (Chaytor, Schmitter-Edgecombe, & Burr, Reference Chaytor, Schmitter-Edgecombe and Burr2006). This has also been termed the “veridicality approach” to ecological validity (Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003; Chaytor et al., Reference Chaytor, Schmitter-Edgecombe and Burr2006). It can be contrasted with the “verisimilitude approach” that involves constructing new testing paradigms to capture the cognitive demands of real-life tasks more accurately. Burgess and colleagues (Reference Burgess, Alderman, Forbes, Costello, Coates, Dawson and Channon2006) have suggested several task demands that pose difficulties for patients with impaired executive functioning, yet are absent in many traditional neuropsychological tests. Among these are the ability to multi-task, work for comparatively long periods of time on a task without receiving feedback, and, more generally, to find a goal-oriented and structured approach to “ill structured” and complex tasks (Burgess, Reference Burgess2000; Goel, Grafman, Tajik, Gana, & Danto, Reference Goel, Grafman, Tajik, Gana and Danto1997).
The Behavioral Assessment of the Dysexecutive Syndrome (BADS; Wilson, Alderman, Burgess, Emslie, & Evans, Reference Wilson, Alderman, Burgess, Emslie and Evans1996), which compiles several tests of executive functioning into one standardized test battery, has been developed to meet these requirements using an office-based test format. Alternatively, there are real-life assessments of activities of daily living, such as shopping, cooking, or running errands (e.g., Knight, Alderman, & Burgess, Reference Knight, Alderman and Burgess2002; Rempfer, Hamera, Brown, & Cromwell, Reference Rempfer, Hamera, Brown and Cromwell2003; Semkovska, Bédard, Godbout, Limoge, & Stip, Reference Semkovska, Bédard, Godbout, Limoge and Stip2004; Shallice & Burgess, Reference Shallice and Burgess1991), which emphasize maximum realism at the expense of tying the test to local conditions, such as the availability of suitable shopping opportunities. In between classical office-based tests and real-life assessments lie computer-based simulation tests that provide a certain degree of realism and complexity while maintaining high control over the task. Some computer-based tests use realistic three-dimensional virtual reality environments (e.g., Kurtz, Baker, Pearlson, & Astur, Reference Kurtz, Baker, Pearlson and Astur2007; McGeorge et al., Reference McGeorge, Phillips, Crawford, Garden, Della Sala, Milne and Callender2001; Rand, Rukan, Weiss, & Katz, Reference Rand, Rukan, Weiss and Katz2009), whereas others consist of schematic representations of real-life situations, simplified with respect to visual presentation and possibilities for interaction (e.g., Craik & Bialystok, Reference Craik and Bialystok2006; Larøi, Canlaire, Mourad, & Van Der Linden, Reference Larøi, Canlaire, Mourad and Van Der Linden2010). While the former approach has an appeal in terms of approximating real life as closely as possible, the latter encourages a focus on task characteristics essential for assessment purposes as opposed to realistic but potentially incidental surface features. The Plan-a-Day test belongs to this category.
It is important to note that, while high realism usually affords face validity, it does not guarantee ecological validity. More important than surface resemblance to real-life situations is whether a test taps the ability (or combination of abilities) relevant for particular outcome situations. Therefore, the ecological validity of neuropsychological tests should be shown empirically, regardless of their apparent degree of realism (cf. Chaytor & Schmitter-Edgecombe, Reference Chaytor and Schmitter-Edgecombe2003). Furthermore, tests with a high degree of realism may also be less specific with respect to the cognitive constructs they measure, as most real-life activities require a combination of several cognitive abilities.
The Plan-A-Day Approach To Measuring Planning Ability
Plan-a-Day is a computer-based scheduling task framed in workplace semantics that requires participants to schedule a list of work-related activities (e.g., picking up mail at the post office or checking inventory at a warehouse) while considering various constraints about when, where, and for what duration the activities have to be carried out. The difficulty of each Plan-a-Day problem is determined by the number of tasks to be scheduled and the number and interaction of constraints that need to be considered. In its present form Plan-a-Day is a modified version of an initial design developed for a job assessment center, which had already been tested with healthy control samples (Funke & Krüger, Reference Funke and Krüger1995). Results indicated moderate levels of reliability and provided preliminary evidence for criterion validity in a context of personnel selection. The present version of the test was modified to increase measurement reliability and adapt task difficulty for patients with mild to moderate impairments of executive functioning. We decided to carry out the validation study with a sample of patients with schizophrenia that had a comparatively high level of functioning. This group optimally matched the test profile, both with respect to general executive functioning and the level of planning ability in particular. However, we assume that findings from this group are likely to generalize to other patient populations with a comparable level of executive impairment.
Many existing neuropsychological planning tests, such as the Tower of London or labyrinth tasks, are move planning tasks with a strong visuo-spatial component. Plan-a-Day in contrast is a scheduling task that requires planning in the temporal domain, drawing mostly on verbally encoded information. As such, it complements existing tests by covering another planning domain and a different mode of representation. Moreover, temporal scheduling problems are a common feature of everyday life, whereas most move planning problems do not have direct real-life equivalents.
The formal tractability of the type of constraint-satisfaction problem used in Plan-a-Day allows the systematic construction of items with varying degrees of complexity and hence difficulty. At the same time, the parameter space available for item construction is comparatively large, so that each item can pose its own slightly different challenges, requiring the flexible use and monitoring of various heuristics and strategies for finding the solution. Workplace semantics were chosen to increase the face validity of the task, which may be important for acceptance of the test and, therefore, for motivation and engagement of participants. Furthermore, the semantics also help to activate pre-existing knowledge about approaching scheduling problems (cf. Blessing & Ross, Reference Blessing and Ross1996), which is desirable from a perspective of ecologically valid testing. Finally, the computer implementation of Plan-a-Day allows an easy, economical, and standardized administration of the test with automatic data logging and scoring.
Aims and Hypotheses
The purpose of the present study was to validate Plan-a-Day in a sample of patients with schizophrenia entering a rehabilitation program in preparation for returning to a work environment. Our hypotheses were as follows: (1) Plan-a-Day shows good internal consistency and acceptable retest-reliability. (2) Plan-a-Day performance correlates with performance in other planning tests indicating good construct validity and specificity. (3) Compared to standard neuropsychological tests, Plan-a-Day can explain additional variance on a global level of functioning, thus demonstrating ecological and incremental validity.
Method
Design
This study was carried out as part of a project comparing the effects of planning and problem solving training with the training of basic cognitive functions (processing speed, attention, memory) in cognitive rehabilitation. A neuropsychological test battery including the Plan-a-Day test was administered twice with a 4-week interval during which participants received cognitive ability training and inpatient work therapy. For introducing the Plan-a-Day test, we will focus on data from the first measurement.
Participants
Eighty patients meeting Diagnostic and Statistical Manual of Mental Disorders—Fourth Edition (DSM-IV) criteria for schizophrenia or schizoaffective disorder participated in the study. The diagnosis was confirmed through the Mini-International Neuropsychiatric Interview (MINI; Ackenheil, Dietz-Bauer, & Vossen, Reference Ackenheil, Dietz-Bauer and Vossen1998). Participants were recruited from an inpatient unit at the mental health hospital SRH Klinik Karlsbad-Langensteinbach, Germany. Patients were living in the community before entering a treatment program aimed at facilitating return to work. This included patients with persistent problems after an acute illness episode as well as those with a longer illness course. In addition to a diagnosis of schizophrenia or schizoaffective disorder inclusion criteria were (1) age of 18 or older, (2) being in a post-acute phase of illness (score of all Positive and Negative Syndrome Scale [PANSS] positive items <5), and (3) having an estimated premorbid IQ of 80 or above. Exclusion criteria included (1) a primary diagnosis of a neurological disorder, (2) illicit substance use during the last month, and (3) having a current comorbid Axis I disorder. All participants gave written informed consent, and the study was approved by the ethics committee of the University of Heidelberg Medical Faculty. For demographic and clinical characteristics of participants see Table 1.
Demographic and clinical characteristics of patients included in the study (N = 80)
Note. DSM-IV = Diagnostic and Statistical Manual of Mental Disorders—Fourth Edition; MWT-B = Mehrfachwahl-Wortschatzintelligenz-Test Version B; GAF = Global Assessment of Functioning; PANSS = Positive and Negative Syndrome Scale.
aMWT-B raw scores: M = 27.17 (SD = 4.89); for this test N = 77.
Plan-a-Day
Plan-a-DayFootnote 1 is a computer-based scheduling task with workplace semantics. Participants are asked to imagine that they work for a small company where they have to plan their daily activities. Information about the tasks to be carried out each day is presented in a task information area on the right side of the screen (see Figure 1), while the left side of the screen displays the different locations, distances between locations, and current position. The constraints that need to be considered for solving Plan-a-Day problems include earliest start, latest finish, location, duration of tasks, and the distance between different locations. Participants are instructed to first plan their daily activities and then implement their plan by moving the symbol representing their current positions by clicking on the corresponding locations on the map. If they are too late to carry out a task at a location, a small information window with a corresponding message is displayed. There is only one correct solution for each Plan-a-Day problem. When participants notice a mistake in their plan, they can undo previous moves using the “back”-button, which is accompanied by an information window confirming the undo operation and an acoustic signal. Participants are instructed to avoid using the “back”-button by planning ahead appropriately.
Plan-a-Day screenshot at the start of a problem. The current location, indicated by a small manikin on the map, is at the “Printer.” Below each location the travel time from the current location is displayed. Tasks to be scheduled are displayed in the top right hand corner, the planned schedule for this day (bottom right) so far only contains the starting location and current time.
Assessment problems were designed by systematically varying two dimensions of working memory load and computational complexity: the number of errands, ranging from two to four per problem, and the number of information elements that need to be considered to solve the problem. The second dimension had three levels: problems that can be solved just by looking at start and end times (i.e., there is no overlap of the time frames for different errands), problems that also require considering errand durations, and problems that require errand durations and the time needed for reaching a particular location. Figure 2 displays the structure of a typical Plan-a-Day problem with four tasks and a medium difficulty level.
Schematic display of the structure of a Plan-a-Day item with four tasks and medium computational complexity. Locations to be visited are displayed on the vertical axis, descending in order of the correct solution. Virtual time is displayed on the horizontal axis. Large light gray bars indicate the time window during which a task may be carried out at each location. The black line at the top marks the starting point (“Conference”), thin dark gray bars indicate travel time, black bars execution of a task at a specific location. The tasks to be scheduled for this item were: “Between 14:00 and 15:30 you have to go to the administration building to sign a contract. This will take 15 minutes. Between 12:00 and 15:00 you want to meet a business partner at the cafe. The meeting will last for 45 minutes. Between 12:30 and 14:30 you have to dictate a letter to the secretariat. This is going to take 30 minutes. You have to pick up your mail between 13:30 and 17:30 at the post office. This will take 30 minutes.”
The test consisted of two practice problems and eight assessment problems of increasing difficulty, requiring 20 to 30 minutes for completion. The operation of the program was explained and interactively demonstrated with the first practice problem, using scripted instructions. Participants were then given the opportunity to further practice the operation of the program with the second practice problem. No time limit was set for the main assessment phase but the test program could be exited early after six of eight assessment problems if a participant seemed overchallenged.
Three scores were calculated for the Plan-a-Day test: total solution time, the planning ratio (percentage of total time spent planning), and the number of problems solved without corrections (i.e., not using the “back”-button during the execution phase). Plan-a-Day was designed in such a manner that solution time captures most of the performance-relevant information. To achieve this, the program only proceeded to a new problem when the previous problem had been correctly solved. As a side effect, this may have improved participant motivation as every task is ultimately successfully solved. This mechanism also means that planning mistakes incur a time penalty: Undoing an incorrect move requires clicking the “back”-button and confirming the corresponding information message with another click before the move is undone. The number of solutions involving no corrections was analyzed as it may be comparable to the accuracy score used in other tests, and the planning ratio seemed a promising measure for how well-planned and strategic participants proceeded.
Neuropsychological Tests
In addition to Plan-a-Day, planning ability was measured with a computer-based variant of the Tower of London (Kohler & Beck, Reference Kohler and Beck2004) using eight planning problems of increasing difficulty, and the Zoo-Map subtest from the German version of the BADS (Ufer, Reference Ufer2000). The Zoo-Map test from the BADS requires participants to plan how they would visit a range of locations at a zoo considering a given set of rules and draw their solution on a map using a pen. Working memory was assessed with three subtests from the German version of the Wechsler Adult Intelligence Scale—Third Edition (von Aster, Neubauer, & Horn, Reference von Aster, Neubauer and Horn2006) to assess verbal memory maintenance and manipulation: Digit Span Forward, Digit Span Backward and Letter-Number Sequencing. The Corsi Block-Tapping Task was used to assess spatial working memory maintenance and manipulation analogous to Digit Span Forward and Backward (Schelling, Reference Schelling1993). The Trail Making Test (TMT; Reitan, Reference Reitan1992) and a simple color Stroop test (Markela-Lerenc, Kaiser, Fiedler, Weisbrod, & Mundt, Reference Markela-Lerenc, Kaiser, Fiedler, Weisbrod and Mundt2006) were used to assess processing speed (TMT, Version A and reaction time in the Stroop neutral condition) and response inhibition (TMT Version B and difference between congruent an incongruent Stroop trials). Furthermore, a vocabulary-based test (Mehrfachwahl-Wortschatz-Intelligenztest Version B [MWT-B]; Lehrl, Triebig, & Fischer, Reference Lehrl, Triebig and Fischer1995) was used to obtain an estimate of premorbid crystallized intelligence and a test of arithmetic ability (Zahlenverarbeitungs- und Rechentest [ZRT]; Kalbe, Brand, & Kessler, Reference Kalbe, Brand and Kessler2002) was also included.
Clinical and Functional Assessment
Psychiatric symptoms were assessed by interview using the Positive and Negative Syndrome Scale (Kay, Fiszbein, & Opfer, Reference Kay, Fiszbein and Opfer1987), a 30-item symptom rating scale completed by clinically trained research staff.
The first measure of everyday functioning was the Global Assessment of Functioning (GAF) according to DSM-IV criteria, a single item rating on a scale from 0 to 100. The rating was given by a trained research psychologist as part of the intake-interview. The GAF rating is a commonly used measure for axis V of the DSM-IV to assess psychological, social and work-related functioning based on descriptions of ten broad levels of functioning. Despite being only a single-item scale, the GAF shows good inter-rater reliability and satisfactory concurrent validity with other instruments measuring levels of functioning and psychosocial impairments (Hilsenroth et al., Reference Hilsenroth, Ackerman, Blagys, Baumann, Baity, Smith and Daniel2000; Startup, Jackson, & Bendix, Reference Startup, Jackson and Bendix2002). Functional capacity was also assessed with the 30-item Osnabrück Work Capabilities Profile (Wiedl & Uhlhorn, Reference Wiedl and Uhlhorn2006), which is based on the widely used Work Personality Profile (Bolton & Roessler, Reference Bolton and Roessler1986). Using the O-AFP, a work therapist rated functional capacity based on the patients’ performance in work therapy.
Results
Consistency and Reliability
Average scores for the Plan-a-Day test are displayed in Table 2. The internal consistency (Cronbach's α) of Plan-a-Day was relatively high for total solution time with α = .78. The planning ratio also turned out to be a moderately consistent characteristic with α = .67, whereas the consistency of the accuracy score was rather low with α = .47.
Descriptive statistics for Plan-a-Day performance (N = 80), average over all eight problems
A retest was carried out 4 weeks after the initial assessment. In the interval, all patients participated in the regular treatment program consisting of occupational therapy, physiotherapy, social skills training, and—for half of the sample—computer-based training of basic cognitive abilities (memory, reaction speed, continuous attention) using the RehaCom training package (Hasomed GmbH, Germany). As part of another study, the other half received a dedicated planning training instead. Test–retest reliability was only analyzed for the first group (n = 40) to avoid effects of the planning-specific training intervention. Still, the reported test–retest reliabilities are likely to represent lower boundaries of the actual values due to varying treatment-related gains in cognitive ability. The test–retest reliability of Plan-a-Day with an interval of 4 weeks was r = .82 for the solution time measure. For comparison, the reliability of the solution time for the other planning tests in this study was r = .58 for the Tower task and r = .59 for the Zoo-Map. The test–retest reliability of the planning ratio, r = .39, and accuracy, r = .33, showed that these measures were not particularly stable over time.
Construct Validity
Table 3 illustrates the relation of Plan-a-Day performance measures to other cognitive ability tests.Footnote 2 With a sample size of 80 participants, the study provided sufficient statistical test power to reliably detect small to medium correlations between tests with 80% test power to detect r > .30 at an alpha-level of .05. As expected, Plan-a-Day solution time showed clear convergent validity with other planning tests, and a small but statistically significant correlation with Stroop neutral reaction time. Discriminant validity was shown with respect to both verbal and spatial working memory, as well as crystallized intelligence as estimated by the MWT-B. The planning ratio measure, however, behaved differently and was associated with spatial working memory, arithmetic ability, and crystallized intelligence. As for accuracy, there was a small correlation with Letter-Number Sequencing and also with crystallized intelligence. Accuracy also showed a moderate correlation with years of formal education, r = .36, p < .001. None of the Plan-a-Day measures was significantly correlated with psychiatric symptoms as measured by the PANSS positive, negative, and global scales.
Convergent and discriminant validity of Plan-a-Day expressed as correlation (Pearson's r) with other neuropsychological tests (N = 80)
Note. TMT = Trail Making Test; MWT-B = Mehrfachwahl-Wortschatz-Intelligenztest Version B; ZRT = Zahlenverarbeitungs- und Rechentest; *Statistically significant at p < .05.
**Statistically significant at p < .01.
Ecological Validity
The raw correlations between Plan-a-Day and the GAF were r = −.32, p < .01, and r = −.15, not significant, between Plan-a-Day and the OAF-P. This implies that 10.2% of variance in the GAF can be explained by Plan-a-Day alone. To establish the incremental validity of Plan-a-Day, we entered the main ability variables into a stepwise regression to find the best predictors of everyday functioning for this test battery. Of all tests other than Plan-a-Day, only Letter-Number Sequencing was able to predict functional capacity as measured by the GAF to a notable extent, R 2 = .21, β = .42, p < .001. Adding Plan-a-Day as a predictor showed that it was able to explain a significant amount of unique variance in the criterion beyond Letter-Number Sequencing, R 2 = .26, ΔR 2 = .05, β = −.24, p < .05, showing its incremental validity. No other cognitive test in this study was able to significantly enhance the prediction of the GAF any further.
Discussion
The present data support Plan-a-Day as a useful instrument for measuring planning ability at the level of executive functioning. The principal measure of Plan-a-Day (solution time) showed a satisfactory level of reliability, good convergence with other measures of planning ability and contributed to predicting global functioning. We will now discuss the properties of different measures derived from the Plan-a-Day test, as well as potential limitations and future extensions.
Consistency and Reliability
The internal consistency of the principal measure of Plan-a-Day (solution time) was satisfactory with an acceptable level of test–retest reliability, comparing favorably with the other planning tests used in this study. The planning ratio measure also turned out to be surprisingly consistent, although not particularly stable, while the accuracy score showed comparatively low values in both respects. Many participants apparently followed the instruction to take a longer time planning and keep the number of errors low. The planning ratio may reflect such strategic and intentional behavior, which is conceivably more variable across different testing sessions than measures of basic cognitive capacity. The stability of this variable was probably further affected by the cognitive training interventions between test and retest. The low internal consistency of the accuracy measure may be due to a related reason: When participants deliberately spend a longer time planning to avoid errors, ability-related systematic variance is shifted away from the accuracy score toward the time-based measures, as intended by design. Since the accuracy score still seemed to capture a facet of performance notably different from solution time, it may be worthwhile to reconsider this initial design goal and investigate possibilities to make this measure more reliable.
Construct Validity
Results seem to generally support the role of Plan-a-Day as a specific planning test. Solution time showed clear associations with other measures of planning, while discriminant validity was shown with respect to most other cognitive ability tests, except for a small correlation with reaction time in the Stroop task. This may indicate a processing speed component, which seems plausible for a time-based performance measure. As mental planning is a process that clearly draws on working memory, the weak link to working memory scores seems puzzling. However, all task information is permanently displayed on screen, which may render raw working memory capacity secondary to the ability to create and monitor a working strategy in the face of a comparatively complex and novel problem. This contrasts with the demands posed by simple and clearly defined tasks such as the working memory tests used in this study. The working memory test that comes closest to reaching a meaningful correlation with Plan-a-Day solution time is the one with the most complex task set and highest demand on mental manipulation (Letter-Number Sequencing), which incidentally was the only other task that predicted global functioning to a significant extent.
Considering other Plan-a-Day variables, the planning ratio emerged not only as a reasonably consistent overall measure, but also showed a characteristic pattern of relations to Tower task accuracy, spatial working memory, arithmetic ability, and—like the accuracy score—Letter-Number Sequencing and crystallized intelligence. Assuming that the planning ratio reflects strategic behavior, it seems plausible that it correlates with tasks benefiting from a well-planned approach and ad hoc strategies. This could also explain why comparatively simple, less strategy-prone tasks (e.g., TMT-A, Stroop, Digit Span Forward) did not show a strong relation to this indicator. Similar reasons may explain the results for the Plan-a-Day accuracy score, which also correlated with Letter-Number Sequencing and crystallized intelligence, as a high accuracy score can be achieved through strategic and well-planned behavior.
Several critical points can be raised with respect to construct validity. First, considering the amount of arithmetic involved in solving a Plan-a-Day problem one may be inclined to think that Plan-a-Day is just a semantically framed test of mental arithmetic. Although performing simple calculations is certainly part of the task, the near zero correlation of Plan-a-Day and the arithmetic test shows that this is unlikely to be the dominating element. Plan-a-Day may be less characterized by mathematical complexity (simple addition will do), but rather by the necessity to extract and integrate relevant information from the task environment.
Second, we have claimed that Plan-a-Day does not have a strong visuo-spatial component, but does the prominent map displayed on screen not directly contradict this statement? The map may be helpful for participants to more vividly visualize the simulated task setting; however, there are reasons to believe this does not constitute a strong visuo-spatial component. All information is presented verbally and numerically on screen and the spatial layout of the map alone is neither required nor sufficiently precise to solve the problems. This was supported empirically by the discriminant validity of Plan-a-Day performance and the Corsi Block Tapping test of spatial working memory.
One important omission in the present test battery is a test of fluid intelligence, which was not included for time reasons. It is possible that fluid intelligence may explain a significant amount of variance in the Plan-a-Day task, as it is correlated with problem solving ability in many domains. This gap may be filled by future studies.
Ecological Validity
Besides largely showing the expected pattern in terms of construct validity, Plan-a-Day also demonstrated incremental validity in predicting global functioning as measured by the GAF. This is an encouraging result, as one aim of the development of Plan-a-Day was to move closer to measuring real-life functioning, for which the GAF is a first approximation. These results should be viewed with some caution, as the O-AFP measure of functional capacity was not significantly correlated with Plan-a-Day variables. However, the O-AFP did not show systematic relations to any of the ability tests used in this study, although its basic psychometric properties have been shown to be adequate in a large norming study (Wiedl & Uhlhorn, Reference Wiedl and Uhlhorn2006). It seems plausible that the OAF-P captures an aspect of functional capacity not directly related to the cognitive functions measured by the test battery used in the present study, emphasizing higher-level social and organizational skills instead (e.g., learning from feedback, punctuality, adequate communication).
There may be several reasons why Plan-a-Day was able to deliver an increment in external validity above other neuropsychological tests. The realistic task setting may play a role in increasing the transferability of test results to everyday situations, or conversely, activate existing prior knowledge and skills relevant for approaching the task. Additionally, the apparent face validity may also motivate participants to take the task seriously as they see its relation to similar real-life situations. From a different perspective, following the reasoning of Burgess et al. (Reference Burgess, Alderman, Forbes, Costello, Coates, Dawson and Channon2006), what may lie at the heart of Plan-a-Day as an executive function test is a realistic degree of complexity beyond mere surface semantics as well as its comparative novelty as a task. It requires a considerable amount of strategic thinking, flexible adaptation of prior knowledge, and meta-cognition to develop and monitor a working strategy for solving Plan-a-Day problems. In this respect, it fulfills several of the requirements that have been put forth as desirable for ecologically valid measures of executive functioning (e.g., Burgess, Reference Burgess2000; Goel et al., Reference Goel, Grafman, Tajik, Gana and Danto1997): Participants have to develop a strategy to handle a comparatively complex and “ill structured” situation and do not receive direct feedback while working on the task for a relatively long period of time. While Plan-a-Day does not require multi-tasking in a narrow sense (participants only work on the scheduling task), it does require setting and balancing priorities of the tasks to be scheduled.
Plan-a-Day as an Instrument for Clinical Assessment and Research
The present results show that Plan-a-Day is a valid and reliable instrument for measuring planning ability, with some indication that it also possesses incremental validity with regard to everyday functioning. Although further studies are needed to verify and extend the present findings, these results render Plan-a-Day an interesting option for neuropsychological assessment. Additionally, Plan-a-Day is easy to administer and score, and we believe that the face validity afforded by the workplace semantics has a positive influence on motivation and acceptance of the test on part of the participants.
Further studies that use a wider range of functional outcome measures and investigate the ability of Plan-a-Day to distinguish between relevant criterion groups (e.g., patients with executive deficits versus healthy controls) are desirable. Another priority will be to exactly quantify the task parameters that determine difficulty and differential external validity of Plan-a-Day problems. Furthermore, a short version of the test and test norms for use in individual assessment are currently under development.
Plan-a-Day may also offer some interesting perspectives for basic research. In contrast to the largely visuo-spatial movement planning problems often used in planning research, it constitutes a largely verbally presented temporal planning problem. This could be an advantage for studying modality-independent aspects of planning by contrasting Plan-a-Day with other paradigms. At the same time the task can easily be formally described, which enables systematic analysis and manipulation of various task parameters for experimental studies.
In summary, it appears that the Plan-a-Day approach shows some promise to fill the middle ground between traditional neuropsychological tests and real-world tasks, while offering interesting perspectives for both clinical application and research.
Acknowledgments
This research was supported by the German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung), grant number 01GW0639. D.H. and J.F. have developed a commercially distributed computer-based training for planning ability, which is based on principles similar to the test described in this article. We thank two anonymous reviewers for their helpful comments.
