%20-%20https://sketch.com%20--%3e%3ctitle%3e3ECBBF50-FD08-412A-943D-2CFC5E025A16%3c/title%3e%3cdesc%3eCreated%20with%20sketchtool.%3c/desc%3e%3cg%20id='Stylesheet'%20stroke='none'%20stroke-width='1'%20fill='none'%20fill-rule='evenodd'%3e%3cg%20transform='translate(-926.000000,%20-668.000000)'%20id='BUTTONS'%3e%3cg%20transform='translate(100.000000,%20597.000000)'%3e%3cg%20id='BTN-/-RIGHTS'%20transform='translate(816.000000,%2065.000000)'%3e%3cg%20id='ICON-/-Rights'%20transform='translate(10.000000,%206.000000)'%3e%3cpath%20d='M13.5752841,0%20L1.34943182,0%20C0.59375,0%200,0.586666667%200,1.33333333%20L0,22.6666667%20C0,23.4133333%200.620738636,24%201.34943182,24%20L17.6505682,24%20C18.40625,24%2019,23.3866667%2019,22.6666667%20L19,5.36%20L13.5752841,0%20Z%20M13.5752841,1.89333333%20L17.0838068,5.36%20L13.5752841,5.36%20L13.5752841,1.89333333%20Z%20M1.34943182,22.6666667%20L1.34943182,1.33333333%20L12.1988636,1.33333333%20L12.1988636,5.36%20C12.1988636,6.10666667%2012.7926136,6.69333333%2013.5482955,6.69333333%20L17.6235795,6.69333333%20L17.6505682,22.6666667%20L1.34943182,22.6666667%20Z'%20id='Shape'%20fill='%23595959'%20fill-rule='nonzero'%3e%3c/path%3e%3ccircle%20id='Oval'%20stroke='%23595959'%20stroke-width='1.5'%20fill='%23595959'%20cx='9.5'%20cy='13.5355339'%20r='5.5'%3e%3c/circle%3e%3cpolyline%20id='Path-2'%20stroke='%23FFFFFF'%20stroke-width='1.5'%20transform='translate(9.500000,%2013.035534)%20rotate(45.000000)%20translate(-9.500000,%20-13.035534)%20'%20points='8%2016.0355339%2011%2016.0355339%2011%2010.0355339'%3e%3c/polyline%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/g%3e%3c/svg%3e)
Recent research on the use of Virtual Reality (VR) in motor rehabilitation (Brunner et al., Reference Brunner, Skouen, Hofstad, Strand, Becker, Sanders and Verheyden2014) and psychotherapy (Freeman et al., Reference Freeman, Antley, Ehlers, Dunn, Thompson, Vorontsova and Slater2014) has demonstrated evidence of therapeutic effects. Over the past decade, its applicability has been targeted to cognitive assessment and rehabilitation. VR may have superior ecological validity compared to traditional pencil-and-paper neuropsychological tests for cognitive assessment because VR can simulate environments and objects that resemble patients’ daily routines (Climent-Martinez et al., Reference Climent-Martínez, Luna-Lario, Bombín-González, Cifuentes-Rodríguez, Tirapu-Ustárroz and Díaz-Orueta2014). Despite the limitations (expensiveness, difficulties to interact with technological devices, and negative effects due to exposure, such as headaches and nausea), VR has a considerable potential for neuropsychological assessment (Canty et al., Reference Canty, Fleming, Patterson, Green, Man and Shum2014; Fajnerova et al., Reference Fajnerová, Rodriguez, Levčík, Konrádová, Mikoláš, Brom and Horáček2014). At first, few studies have addressed the viability of VR tasks for cognitive assessment and verified if the target population is be able to perform the activities (Mendez, Joshi, & Jimenez, Reference Mendez, Joshi and Jimenez2015).
In addition to demonstrating the applicability of the tasks in VR to evaluate cognitive functions, the investigation of its psychometric qualities, specifically validity and reliability, is required (Cooper et al., Reference Cooper, Camic, Long, Panter, Rindskopf and Sher2012). In the clinical context, there still is uncertainty about the ecological validity of the traditional tests used for cognitive assessment of older persons (Schultheis, Himelstein, & Rizzo, Reference Schultheis, Himelstein and Rizzo2002), as they are not often linked directly to the demands of daily life activities (Morganti, Reference Morganti, Riva, Botella, Légeron and Optale2004). Therefore, the VR allows the development of scenarios similar to the reality of patients with neurodegenerative diseases or neuronal damage, in which cognitive deficits can be detected (Climent-Martínez et al., Reference Climent-Martínez, Luna-Lario, Bombín-González, Cifuentes-Rodríguez, Tirapu-Ustárroz and Díaz-Orueta2014).
Previous research has demonstrated that patient performance on VR tasks is correlated with other measures of formal cognitive functioning and self-report scales of functional, behavioral and cognitive abilities (Grewe et al., Reference Grewe, Lahr, Kohsik, Dyck, Markowitsch, Bien and Piefke2014; Jovanovski et al., Reference Jovanovski, Zakzanis, Ruttan, Campbell, Erb and Nussbaum2012). Grewe et al. (Reference Grewe, Lahr, Kohsik, Dyck, Markowitsch, Bien and Piefke2014) asked patients with focal epilepsy to complete a VR task simulating a supermarket. The activity required patients to recall and find a list of previously learned items. The performance of the patients with focal epilepsy was significantly associated with performance on visual episodic memory measures such as the Rey–Osterrieth Complex Figure – ROCF (accuracy in Immediate and Delayed Recall), as well as a questionnaire of subjective cognitive complaints. Jovanovski et al. (Reference Jovanovski, Zakzanis, Ruttan, Campbell, Erb and Nussbaum2012) examined the performance of patients with stroke and traumatic brain injury on the Multitasking in the City Test (MCT), a VR task in which the participants must buy a range of items from the pharmacy, stores, and restaurants. Scores on this test were significantly correlated with the Trail Making Test, Wisconsin Card Sorting Test, and an informant questionnaire of executive deficits. Thus, a better performance in the classic neuropsychological tests corresponded to a better performance on in VR tasks, suggesting that VR can be a valid assessment of cognitive flexibility, planning, and reasoning skills.
The use of VR for cognitive and functional capacity assessment is particularly relevant for older adults, as memory abilities, executive functioning and functional capacity are relevant to the diagnosis of neurocognitive disorders including dementia (American Psychiatric Association, 2013). In a preliminary study, Flynn et al. (Reference Flynn, van Schaik, Blackman, Femcott, Hobbs and Calderon2003) examined the functional abilities of elders with Alzheimer’s disease using VR tasks. The activities in VR task included making a phone call, sending and receiving a letter, and finding a proper place to sit and relax. Despite the small number of participants in the sample and the absence of a control group, the results showed that the activities in VR are applicable to people with dementia. The applicability was described by the sense of presence reported by the participants (how they felt inserted in this virtual environment and how this was considered realistic), interaction capacity, and comfort using the joystick. The ecological validity was verified through comparison of performance in activities in the “real world” similar to VR. The Virtual Reality Day-Out-task (Tarmanas, Tsolaki, Nef, Muri, & Mosimann, Reference Tarnanas, Tsolaki, Nef, Müri and Mosimann2014), is another VR designed for use with older adults. In this task, participants must use their executive and memory abilities in order to find a safe way to escape a building on fire. Performance on this task demonstrated excellent sensitivity and specificity (97% and 100%, respectively) in discriminating between mild cognitive impairment and healthy controls. There have also been several studies designed to use VR to test for allocentric and egocentric orientation abilities with older adults because deficits in these domains are common in individuals with dementia (Jebara, Orriols, Zaoui, Berthoz, & Piolino, Reference Jebara, Orriols, Zaoui, Berthoz and Piolino2014; Plancher, Tirard, Gyselinck, Nicolas, & Piolino, Reference Plancher, Tirard, Gyselinck, Nicolas and Piolino2012). Zucchella et al. (Reference Zucchella, Sinforiani, Tassorelli, Cavallini, Tost-Pardell, Grau and Nappi2014) proposed tasks for cognitive assessment for older persons with VR including: 1) finding objects in a particular location (evaluating memory, attention and spatial orientation); 2) watering plants while listening to the radio (evaluating planning and divided attention); 3) making a phone call (assessing executive functioning, selective attention and prospective memory); and 4) identifying common objects (assessing recognition memory).
Although VR is used in the assessment of cognitive skills, few studies have reported the development stages of their stimuli and instructions, as well as any pilot studies that were performed. Evaluation of psychometric characteristics such as validity is necessary to ensure the quality of the instruments. These characteristics are essential if VR is going to be employed in clinical settings to assist with differential diagnosis. Therefore, this article aims to report the stages of the development process of an ecological task of cognitive assessment to older adults carried out in a virtual environment (ECO-VR), showing evidence of content validity, construct validity, and feasibility.
Material and Methods
The development stages of ECO-VR were based on the procedures proposed by Cooper et al. (Reference Cooper, Camic, Long, Panter, Rindskopf and Sher2012), which highlight the importance of developing cognitive tasks with theoretical and scientific basis and modalities of validity evidence. This study was approved by the ethic committee of the Pontifical Catholic University of Rio Grande do Sul (CAAE 15961113.0.0000.5336) and the participants volunteered to participate after signing a consent form.
Stages of development of the ECO-VR
Construction of tasks
We conducted a literature review for the most common cognitive complaints present in older adults. The ECO-VR tasks were formulated from the results found in this theoretical review and the authors’ clinical experiences. The authors then debated and decided on real-world activities that would assess these cognitive domains (watch television news, listen to a message on the answering machine, separate food for a meal, find a particular object in the living room, and recall the date of the medical consultation). The authors also discussed the optimal type of interaction (immersive or non-immersive) to be used with older adults. The immersive interaction occurs in a dynamic three-dimensional scenario projected by head mounted displays and cave automatic virtual environment (rooms where three-dimensional graphics are projected on its walls) systems, for example, while non-immersive interaction refers to three-dimensional objects projected on a screen of a computer. In addition, the authors created instructions for each of the five activities, as well as the scores that should be recorded (e.g., accomplishments, rule-breakings and strategies).
Expert and non-expert judges’analysis
The suitability of the three-dimensional visual stimuli used in ECO-VR was studied by 29 non-expert judges between the ages of 18 and 64 years old (M = 27.1, SD = 4.26). We asked these judges to attempt to name 39 visual stimuli (three-dimensional representations of the scenarios of the house, as well as objects such as a remote control, answering machine, household appliances, etc.), from which 20 of those figures were target stimuli used in the ECO-VR. The remaining figures were distractor stimuli that were not household objects (e.g., tire, cashier) and belonged to the software image database NeuroVR2 (Riva et al., Reference Riva, Gaggioli, Villani, Preziosa, Morganti, Corsi and Vezzadini2007). We chose to include these stimuli to prevent the judges from becoming biased towards recognizing only household objects during this naming task. The target stimuli were randomly organized and interspersed with the remaining distractor figures. We showed participants the target stimuli and distractors through Microsoft Office PowerPoint 2010 in computers or notebooks with 15-inch screens. Each participant was handed out a registration sheet where they attempted to write the names of the figures. The simple frequency of correct responses of participants was calculated for every item. We kept all target stimuli that had 70% or higher correct responses.
Afterwards, we consulted five independent expert judges (three neuropsychologists and two speech therapists with expertise in neuropsychology) about the suitability of the ECO-VR instructions as well as the main cognitive domains tested by each of the five tasks. The selection criteria of the expert judges are: 1) to have a significant number of published studies addressing the development and / or adaptation of neuropsychological assessment tools; 2) to possess a significant number of scientific publications about psychometric tests in the last five years; 3) to have participation in other content analysis processes of neuropsychological instruments; and 4) to be fluent in Brazilian Portuguese. The participation of speech therapists with expertise in neuropsychology was important because psycholinguistic characteristics and verbal/written stimuli of instructions could influence the performance of other neuropsychological functions, especially in older adults who might have cognitive impairment. When analyzing the cognitive domains (based on the instructions, stimuli and tasks of the ECO-VR), the experts judges identified from a list of various cognitive processes those that were required to perform each task. The expert judges’ suggested edits were debated among the authors and adopted when appropriate.
Pilot studies
We conducted three pilot studies with the aim of checking the comprehensibility of the instructions and feasibility of the tasks in the ECO-VR, as well as their practicality and administration time. Six older adults aged 60 years and above from different educational backgrounds participated in this stage. Exclusion criteria were: 1) scores indicating general cognitive impairment in the Mini-Mental State Examination – MMSE (adapted by Chaves & Izquierdo, Reference Chaves and Izquierdo1992), in which the cut-off points for the South-Brazilian population are: < 22 for individuals between 1–5 years of schooling, < 23 for 6–11 and < 24 for 12 or more (Kochhann, Varela, Lisboa, & Chaves, Reference Kochhann, Varela, Lisboa and Chaves2010); 2) scores ≥ 6 in the Geriatric Depression Scale, short version – GDS-15 (adapted by Almeida & Almeida, Reference Almeida and Almeida1999): 3) scores ≥ 2 in the CAGE Questionnaire (acronym that refers to the four questions: cut down, annoyed by criticism, guilty and eye-opener), adapted by Masur and Monteiro (Reference Masur and Monteiro1983), which investigates problems related to alcohol use; 4) presence of psychiatric and neurological disorders investigated by self-report in the sociodemographic questionnaire; and 5) visual or hearing impairments not corrected during the evaluation (glasses or hearing aids, for example), investigated by self-report in the sociodemographic questionnaire. Their previous computer experience was evaluated using a dichotomous question (“yes” or “no”): “Do you use, or have used, a computer such as a desktop computer, notebook or netbook?” At the end, the participants assigned a rating from 0 to 10 describing their level of anxiety while completing the ECO-VR (higher scores represent higher levels of anxiety). The authors discussed difficulties encountered in the performance of the ECO-VR tasks during the pilot studies and made changes wherever necessary.
Preliminary evidence of construct validity
Thirty-seven older persons aged 60 years and above responded to the ECO-VR and a battery of cognitive instruments to verify preliminary evidence of the ECO-VR construct validity. The cognitive instruments were selected by evaluating cognitive domains involved in ECO-VR, especially memory and executive functions. The exclusion criteria were the same of the pilot studies, whereas seven older persons were excluded for the GDS-15 score.
Instruments
The ECO-VR tasks were completed in a three-dimensional VR environment of a house adapted from an existing database available in the software NeuroVR2. The ECO-VR task was presented on a 17-inch notebook computer with a Pentium Dual Core processor and ATI Mobility Radeon HD 4650 (M96) video board. The interaction in the virtual environment was non-immersive and performed with a joystick. We instructed participants to complete five tasks involving simultaneous activation of multiple cognitive skills such as focused concentration, prospective memory, problem solving, spatial orientation, initiative, and inhibition.
Initially, we familiarized participants with the software and joystick controls in a simple 3D environment that simulates a plaza. Participants are encouraged to freely explore the environment and learn how to move their avatar (e.g., moving forward and backwards; from left to right; up and down; and to activate/turn off). The environment is experienced from a first-person perspective, and participants are able to interact with objects such as a water fountain, vending machines, people, and animals. The familiarization phase had no time limit. The ECO-VR was introduced after the participant reported feeling comfortable with the controls.
Instructions for activities of the ECO-VR were printed on a sheet of paper (18-point Arial font) and read aloud to the participants. Participants kept the sheet and were allowed to check them as often as necessary, and they were instructed to take notes if they wished. The instructions contain rules for accomplishing the tasks (e.g., perform the tasks in order, take notes, and ask for support in handling the controls as necessary). Afterwards, we handed participants another sheet of paper listing the tasks to be accomplished. We read to the participants the activities in the sequence that they must be performed in the house environment (Figure 1). The sequence is as follows: 1) watch television news, 2) listen to a message on the answering machine, 3) organize a meal (check on a schedule which guests would come for the meal and which types of food should be served, finding food in the kitchen, and arranging it on a table), and 4) fulfill the request that was mentioned on the answering machine in task 2. Finally, participants are asked to remember information from the television news in task 1 and the day and time of their scheduled visit to the doctor (from the message in task 2). The ECO-VR scores include the time (in seconds) required to complete all the tasks as well as the number of tasks successfully accomplished, with no time limit. Furthermore, there are rule-breaking scores (frequency of steps not performed, visiting rooms unrelated to the tasks, and task omissions, with a maximum score of 21) and strategies used (marking the tasks performed, taking notes, consulting the instruction sheet, and asking for help, with a maximum score of 24) based on the Raspaelli et al. (Reference Raspelli, Pallavicini, Carelli, Morganti, Poletti, Corra and Riva2011) protocol. Except for the time to completion and rule-breaking, higher scores represents better performance.
Home environment of the ECO-VR.
In addition to the ECO-VR, we administered the MMSE (adapted by Chaves & Izquierdo, Reference Chaves and Izquierdo1992) to all participants in order to assess general cognitive abilities. Depressive symptoms were assessed using the GDS-15 (adapted by Almeida & Almeida, Reference Almeida and Almeida1999). The occurrence of alcohol-related problems was investigated by the CAGE questionnaire (adapted by Masur & Monteiro, Reference Masur and Monteiro1983). These instruments were administered in order to exclude subjects with signs of cognitive impairment and depressive symptoms that could adversely impact performance on the ECO-VR. The older adults of the preliminary evidence study for the ECO-RV construct validity answered a battery of cognitive instruments to evaluate executive functions and memory that included: the Wisconsin Card Sorting Test, short form – WCST-64 (Kongs, Thompson, Iverson, & Heaton, Reference Kongs, Thompson, Iverson and Heaton2000), the Rey-Osterrieth Complex Figure Test – ROCF (Rey, Reference Rey1941), the Spatial Span subtest of the Wechsler Memory Scale, third edition – WMS-III (Wechsler, Reference Wechsler1997), and the Dysexecutive Questionnaire – DEX (Wilson, Alderman, Burgess, Emslie, & Evans, Reference Wilson, Alderman, Burgess, Emslie and Evans1996).
Data analysis
The concordance between judges for evaluating the cognitive domains assessed in the five tasks was evaluated by the method of Fagundes (Reference Fagundes1995). The simple concordance index was calculated using the formula [(number of agreements / sum of agreements and disagreements) x 100]. The results in the pilot studies were analyzed qualitatively, based on the questioning of the participants on the understanding of each task and the observations made during the ECO-VR tasks. Preliminary evidences of construct validity were investigated by partial correlation to identify the association between cognitive measures and scores of the ECO-VR, controlling the effect of previous computer experience. Data distribution was investigated by Kolmogorov-Smirnov test. Data were considered significant if p < .05, and the statistical package used was the SPSS version 22 for Windows.
Results
Analysis of expert and non-expert judges
The naming performance of the target stimuli by non-expert judges is shown in Table 1. All 20 three-dimensional visual stimuli were recognized by greater than 75% by the participants, which was above our specified level of 70% for including the stimuli in the study.
Analysis of Non-Expert Judges for the Representation of Target Stimuli of the ECO-VR
Note: AP = Answers provided; CA = Correct answers.
The percentage of agreement among the five expert judges in rating the cognitive processes predominantly evaluated by each of the ECO-VR tasks is presented in Table 2. The agreement was considered high among the judges (> 80%). Regarding the suitability of the instructions, a change in instruction of task 5 was suggested to make it more understandable. After a discussion between the authors, we decided to adopt the suggestion. The expert judges did not suggest any additional changes to the instructions.
Concordance analysis of the Expert Judges of Cognitive Components Predominantly Evaluated in the ECO-VR Tasks
Pilot studies and reformulations
Table 3 summarizes demographic and clinical information for the six participants in the pilot studies. In the first pilot study, the ECO-VR was administered to two older adults (participants A and B) who reported no previous experience with computers. The scores on MMSE and GDS-15 showed preserved general cognitive ability and absence of clinically significant depressive symptoms. In general, the instructions of the ECO-VR were well understood by participants as well as the joystick controls. However, the two participants had difficulties finding the target stimulus (a remote control) to complete the first activity, because it was too small. Thus, we increased the size of the stimulus and modified the instructions to make its location more explicit.
Socio-Demographic, Clinical and Cognitive Data of the Participants in Pilot Studies
Note: MMSE = Mini Mental State Examination; GDS-15 = Geriatric Depression Scale, short form.
The second pilot study involved four older adults (participants C, D, E and F) who had previous experience with computer use, performed within normal limits on the MMSE, and who did not report clinically significant symptoms of depression on the GDS. In this second version of ECO-VR, including the changes to the remote control stimulus and instructions for finding it, the participants properly understood the instructions and had no problems using the joystick. However, we observed difficulties in spatial localization and mobility. In this version of the pilot, participants started with ECO-VR avatar located outside the house and were required open the door to enter the house. We instructed participants to travel a route through the following rooms in order: 1) corridor; 2) living room; 3) kitchen; 4) corridor; 5) bedroom; 6) corridor; and 7) living room. As a result, the authors have chosen to reduce the rooms involved in the activities, because the participants were often lost and visited the same rooms several times, which increased the total time to complete the activities. The starting point of the activities was changed to the living room and the target stimulus (lamp) of the fourth task was also relocated to the living room instead of the bedroom. Thus, the revised activities in the ECO-VR included only the living room and the kitchen. The instructions were revised and a third pilot study was conducted.
The third pilot study had the same two older adults who participated in the first pilot and occurred 63 days after the first administration of the instrument. Both participants were able to adequately understand the instructions and we did not make any further revisions. Table 4 shows the scores obtained by the participants in the ECO-VR in pilot studies. Furthermore, the subjective anxiety level was small for all participants, ranging between 0 and 1. The total completion times for the second and third pilot studies ranged from 8 to 16 minutes, rule-breaking and strategy scores ranged from 2 to 8 and from 6 to 13, respectively. The most common rule breaks included completing the tasks out of order and omissions in tasks 3 and 4. With regard to strategy scores, participants frequently consulted the instructions and asked the experimenters for information before beginning a task. Although they were allowed to take notes, no participant used this strategy.
Scores and Total Time to respond to the ECO-VR Tasks of Pilot Studies and Construct Validity Study
Standardization of instructions and scoring guidelines
After the pilot studies, the instructions of the final version of the ECO-VR were standardized, as were detailed guidelines for the recording of participant responses. We adapted rule-breaking and strategy scores from Raspaelli et al. (Reference Raspelli, Pallavicini, Carelli, Morganti, Poletti, Corra and Riva2011) based on the ECO-VR tasks.
Preliminary evidences of construct validity
The average age and education of the sample was, respectively, 68.67 years (SD = 6.59) and 12.73 years (SD = 5.04). Regarding the sample, 97% were women and 60% had previous computer experience. Table 5 shows the results of a correlation analysis between cognitive measures and the ECO-VR scores.
Partial Correlations between Cognitive Measures and ECO-VR Scores
Note: MMSE = Mini Mental State Examination; WCST-64 = Wisconsin Card Sorting Test, short form; ROCF = Rey–Osterrieth Complex Figure Test; WMS-III = Wechsler Memory Scale, third edition; DEX = Dysexecutive questionnaire; * = p < .05; ** = p ≤ .01; *** = p ≤ .001.
There were moderate correlations between the total score of the ECO-VR and the MMSE, WCST-64 (achieved categories), ROCF (score), Spatial Span (forward and backward), and DEX questionnaire instruments. The frequency of strategies correlated moderately with WCST-64 (perseverative errors and trials to complete the first category) and DEX questionnaire. The completion time of the ECO-VR also obtained moderate correlations with WCST-64 (achieved categories and trials to complete first category) and ROCF (time and score) and strong association with perseverative errors of the WCST-64.
Discussion
This study aimed to present the development, feasibility, content validity, pilot testing and preliminary evidence of construct validity of an ecological task in VR designed for cognitive assessment in older adults, the ECO-VR. We developed tasks based on theoretical and clinical assumptions of common domains of cognitive impairment in older adults and ecologically valid household activities. The representation of three-dimensional visual stimuli was analyzed by non-expert judges and the adequacy of the instructions, as well as the cognitive skills involved in each task, were evaluated by expert judges (neuropsychologists and speech therapists with expertise in neuropsychology). After these steps, we conducted three pilot studies with elders. After some adjustments, two older adults with no previous familiarity with computers were able to complete the task in a reasonable amount of time (between 8 and 16 minutes), demonstrating the applicability of the ECO-VR for this population.
The initial step included literature review and theoretical discussions about common domains of cognitive deficits due to aging. In addition, the clinical knowledge of the authors contributed to the formulation of the tasks of ECO-VR, prioritizing activities requiring temporal and spatial orientation, executive functioning, attention, verbal memory, and prospective memory. Most VR tasks used for cognitive assessment for older persons include complex tasks requiring multiple cognitive processes. For example, in the VR task from Tarnanas et al. (Reference Tarnanas, Tsolaki, Nef, Müri and Mosimann2014), participants were required to perform a series of sub-tasks to respond to an emergency, including using the phone to call the fire department, sound the fire alarm, and reach the lower floors to the ground to escape safely from a fire.
In addition to the literature review, in the initial stage it was important to choose the type of interaction with the virtual environment. The immersive interaction may not be suitable for older adults because of the possibility of adverse symptoms such as nausea and headache (Flynn et al., Reference Flynn, van Schaik, Blackman, Femcott, Hobbs and Calderon2003). Thus, the non-immersive interaction was chosen for the ECO-VR. We also chose to allow for a familiarization period to help participants become comfortable with the virtual environment. This training phase is reported in several studies regardless of the population investigated (Plancher et al., Reference Plancher, Tirard, Gyselinck, Nicolas and Piolino2012; Tarnanas et al., Reference Tarnanas, Tsolaki, Nef, Müri and Mosimann2014; Weniger, Ruhleder, Lange, Wolf, & Irle, Reference Weniger, Ruhleder, Lange, Wolf and Irle2011).
We discussed the formulation of ECO-VR instructions at this stage and attempted to make them as clear and direct as possible. We chose a fixed order of carrying out the activities because the execution of certain tasks depended on successful completion of previous tasks, encompassing aspects of planning and memory. The instructions for the activities were printed in a large font to facilitate easier reading for the participants. We also read the instructions aloud to the participants to further ensure comprehension.
The stage of analysis of non-expert judges was essential in choosing the three-dimensional stimuli and scenarios in the ECO-VR. These visual and three-dimensional stimuli were chosen from those more familiar and found in daily routine of older persons, such as a remote control or a telephone. The accuracy of the activities depends in large part on the participant’s ability to recognize the target stimuli. Thus, we chose to include judges with varying ages, from young adults to older adults. The judges’ recognition rates for the target stimuli were all very high, ranging between 75% and 100%. Although the ECO-VR has been developed initially for older persons, this will allow the expansion for the normative sample hereafter. The scenarios and target stimuli were recognized properly by the judges, with no needed additional modifications.
The expert judges evaluated the content validity of the ECO-VR tasks for cognitive domains relevant to aging. The level of inter-rater agreement with regard to the main cognitive functions assessed by tasks of the ECO-VR was high, confirming the theoretical assumptions that supported the development of the tasks. Instrumental activities of daily living involve various cognitive skills such as memory, attention, and executive functioning, so it is difficult to consider that one specific cognitive function could contribute solely to any particular task. Other VR studies support this notion, highlighting the roles of various cognitive abilities, especially components of executive functioning (Tarnanas et al., Reference Tarnanas, Schlee, Tsolaki, Müri, Mosimann and Nef2013). In addition, the analysis of expert judges contributed to the construction of comprehensible task instructions. They analyzed the extent and comprehensibility of the instructions and verbal stimuli and suggested only one modification, which was adopted by the authors. The language was suggested by the judges as one of the cognitive functions also evaluated by ECO-VR, as the participants could read (and comprehend) the instructions to perform the tasks.
The pilot studies highlighted the need for changes in some of the activities of the ECO-VR. In the first pilot study, there was a difficulty in the first task related to finding the stimulus target, resulting in an increase in total performance time to an infeasible length (21 and 47 minutes). Thus, we modified the instructions for this task to better explain the location of the target stimulus. In the second pilot study, there were issues with location and mobility between the various rooms of the house that were initially planned for the tasks, resulting in interventions by the examiners to help the participants. However, the time to complete the task ranged between 8 and 16 minutes. Thus, the number of rooms was reduced and the locations for activities were altered. Finally, a third pilot study was conducted that suggested that no further changes were necessary because participants were able to adequately complete the tasks between 8 and 12 minutes.
The pilot studies also contributed to estimate the administration time of the ECO-VR. Participants assigned low scores on the level of subjective anxiety generated due to the activities (all at 0 or 1 out of 10), including the participants who were not familiar with computer games or virtual environments. Adverse symptoms from exposure to VR (e.g., headache or dizziness) were not reported. In the study by Flynn et al. (Reference Flynn, van Schaik, Blackman, Femcott, Hobbs and Calderon2003), a non-immersive exposure to a virtual reality also did not cause oculomotor problems, disorientation, or nausea in older adults participants. Participants reported no difficulties in using the joystick, showed little problems to perform the activities, and reported that the VR experience was fun. The authors affirm that the joystick seems to be the best way to interact with virtual reality for older persons with dementia.
Finally, scores of the ECO-VR (total, strategies and completion time) were moderately associated with most executive functions measures and the MMSE, suggesting that the planning skills, working memory, inhibition, processing speed, and general cognitive ability are present for the execution of activities in the virtual environment. It is important to emphasize that previous computer experience was controlled in this study, suggesting that the associations encountered are not due to the familiarity of the older adults with computers. Although these are preliminary results, other studies with VR tasks have demonstrated that executive functions are essential for tasks involving multiple demands (Burgess, Veitch, de Lacy Costello, & Shallice, Reference Burgess, Veitch, de Lacy Costello and Shallice2000), whereas the activities consist of goals and sub-goals that require planning and problem solving, among other cognitive processes, for its adequate completion (Canty et al., Reference Canty, Fleming, Patterson, Green, Man and Shum2014).
However, it is necessary to highlight limitations in this study, as the absence of a measure to verify the usability of the ECO-VR interaction system. Furthermore, the sample size was small and the exclusion of a clinical sample (mild cognitive decline or Alzheimer’s disease) limits the applicability of the ECO-RV for people with cognitive impairments. Although the MMSE was used to identify the presence of cognitive impairment, it is necessary to consider other neuropsychological formal instruments to monitor more precisely the exclusion criteria adopted (such as the occurrence of neurocognitive disorder) and to identify associations between other cognitive functions.
The ECO-VR showed evidence of content validity, initial construct validity and feasibility with older adults. However, further studies are needed to investigate other psychometric characteristics, as criterion validity and reliability. Future research should involve the establishment of normative data for older persons in order to assist in the diagnostic process of mild cognitive impairment and other neurocognitive disorders, investigating the sensitivity, specificity and accuracy of ECO-VR for those populations.
The inclusion of technological tools in the clinical neuropsychology aims to enhance the assessment methods, especially in cognitive disorders associated with the aging process. VR would assist in the construction of more naturalistic environments and activities to the patients’ daily lives, not necessarily replacing the traditional evaluation, but complementing the results of formal tests widely recognized. The ECO-VR aims to add to the neuropsychological assessment, presenting itself as a technique that enhances the ability to predict functional capacity through tasks that are more similar to older adults’ daily life.
The authors thank the Coordination for the Improvement of Higher Education Personnel (CAPES) and for the Research and Editing Consulting Program (RECP).
