INTRODUCTION
In France, as in many countries, the population is aging. The French National Institute of Statistics and Economic Studies (INSEE) estimates that there are more than 6 million people over the age of 75, or 9% of the French population (INSEE, 2014). As the number of older adults increases, so does the incidence of pathologies that cause major neurocognitive impairment interfering with autonomy and requiring the intervention of a caregiver. And as a consequence, the number of people with Alzheimer’s disease is expected to surpass 1.3 million in 2020.
Facilitating access to the diagnosis of dementia is, therefore, an obvious need. However, the capacity for making these diagnoses is limited in medical deserts with insufficient healthcare coverage where, for lack of an expert opinion, patients are often underdiagnosed (Chodosh et al., Reference Chodosh, Petitti, Elliott, Hays, Crooks, Reuben and Wenger2004). With the rapid advances in telemedicine in recent years, new digital and communication technologies can offer a solution to the growing exodus of doctors from certain regions in France, facilitating access to neuropsychological assessments and providing clinicians with new tools in line with evolving practices (Jollivet, Fortier, Besnard, Le Gall, & Allain, Reference Jollivet, Fortier, Besnard, Le Gall and Allain2018).
Teleneuropsychology (TeleNP) (neuropsychological assessment by videoconference) now allows clinicians to administer neuropsychological tests remotely (Grosch, Weiner, Hynan, Shore, & Cullum, Reference Grosch, Weiner, Hynan, Shore and Cullum2015) and thereby help with diagnosis (Harrell, Wilkins, Connor, &Chodosh, Reference Harrell, Wilkins, Connor and Chodosh2014). The feasibility and reliability of this new method of administering neuropsychological tests have been demonstrated in adults with and without cognitive impairment in an urban setting (Castanho et al., Reference Castanho, Amorim, Moreira, Mariz, Palha, Sousa and Santos2016; Cullum, Hynan, Grosch, Parikh, & Weiner, Reference Cullum, Hynan, Grosch, Parikh and Weiner2014; Cullum, Weiner, Gehrmann, & Hynan, Reference Cullum, Weiner, Gehrmann and Hynan2006; Galusha-Glasscock, Horton, Weiner, & Cullum, Reference Galusha-Glasscock, Horton, Weiner and Cullum2016; Hildebrand, Chow, Williams, Nelson, & Wass, Reference Hildebrand, Chow, Williams, Nelson and Wass2004; Jacobsen, Sprenger, Andersson, & Krogstad, Reference Jacobsen, Sprenger, Andersson and Krogstad2003; Vestal, Smith-Olinde, Hicks, Hutton, & Hart, Reference Vestal, Smith-Olinde, Hicks, Hutton and Hart2006) as well as in a rural population (Adjorlolo, Reference Adjorlolo2015; Wadsworth et al., Reference Wadsworth, Galusha-Glasscock, Womack, Quiceno, Weiner, Hynan and Cullum2016). The videoconference-based neuropsychological assessment procedure was also shown to distinguish between cognitively impaired and non-impaired subjects similar to a face-to-face (F-F) assessment (Wadsworth et al., Reference Wadsworth, Dhima, Womack, Hart, Weiner, Hynan and Cullum2018). This new assessment method seems to be well accepted by a large part of healthy older adults and patients with cognitive impairment, but a higher percentage of subjects with mild cognitive impairment or early stage Alzheimer’s disease indicated that they preferred to be tested by traditional F-F administration (Parikh et al., Reference Parikh, Grosch, Graham, Hynan, Weiner, Shore and Cullum2013).
In France, no studies evaluating the reliability of neuropsychological assessment by TeleNP versus a standard F-F assessment have been conducted to date. Our primary objective was to evaluate the feasibility and the reliability of a videoconference-based neuropsychological assessment procedure performed in accordance with the guidelines of the Joint Task Force for the Development of Telepsychology Guidelines for Psychologists (2013). To this end, we compared the neuropsychological scores obtained with a battery of tests in two conditions: administered by F-F and by TeleNP, in a large sample of urban adults with cognitive complaints. This study includes a battery of tests widely used in France during neuropsychological assessment for adults. We hypothesized that there would be no difference between the neuropsychological test scores obtained by TeleNP and those by traditional F-F administration.
We will also assess the level of satisfaction of participants with the TeleNP assessment procedure.
METHODS
Population
A total of 150 participants were included. The participants all consulted in memory clinics and the course of the assessment was based on usual clinical practice. The study was conducted at the Pitié-Salpêtrière hospital memory clinic and in the Aloïs Memory Network in the greater Paris area. Inclusion criteria were: (1) men or women aged 60–80; (2) consulting at the memory clinic for a cognitive complaint; (3) Mini-Mental State Examination (MMSE; Folstein, Folstein & McHugh, Reference Folstein, Folstein and McHugh1975) score greater than 20/30; (4) having at least primary school education; (5) fluent in French; (6) able to have a second neuropsychological test 4 months later; (7) having received and signed the informed consent form. All subjects were tested by a single experienced examiner under both conditions (TeleNP and traditional F-F).
At the inclusion visit, all participants underwent a traditional F-F neuropsychological assessment as part of their standard care. The medical history (neurological, psychiatric, cardiovascular, hematological, respiratory, digestive, hepatic, renal, dermatological, endocrine, and metabolic) of the participants were collected during this visit.
Four months after study inclusion, they were randomized to have an additional neuropsychological assessment either by F-F administration or by TeleNP (see Figure 1). By the end of the second assessment, participants in the videoconference arm were asked to rate the method of assessment via a five-level Likert scale (very satisfied/moderately satisfied/rather unsatisfied/very unsatisfied/unknown), as well as a series of statements about their satisfaction regarding this new mode of testing, their anxiety during the tele-evaluation, their preference between the two methods of testing, and how they felt regarding the presence of the assistant (see below for the role of the assistant).
Flowchart of the patients analyzed in the study.
The same neuropsychological tests were used at the inclusion (assessment 1) and 4 months after inclusion (assessment 2) and the scores were compared test by test under both conditions (F-F vs. TeleNP). Whenever alternative versions of tests existed, they were used during assessment 2.
This multicenter, prospective, randomized, comparative study was conducted in accordance with the Helsinki Declaration and the rules governing telemedicine (telemedicine decree 2010-1229 of 19 October 2010) and was approved by an Ethics Committee (CPP Ile-de-France X – Hôpital Robert Ballanger – 93602 Aulnay-sous-Bois Cedex – N° ID-RCB : 2014-A01971-45). Participants signed an informed consent for this study on forms approved by this Ethics Committee.
Material
The F-F assessment took place at the Aloïs Memory Network centers and at Pitié-Salpêtrière Hospital. The assessment lasted 2 h. Independently of the administration condition (F-F or TeleNP), the questionnaire and the tests were administered in a specific order: State-Trait Anxiety Inventory (STAI-Y; Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, Reference Spielberger, Gorsuch, Lushene, Vagg and Jacobs1983), MMSE, Digit Span forward and backward (Wechsler, Reference Wechsler1997), learning phase of the Free and Cued Selective Reminding Test (FCSRT; French version of the FCSRT; Van der Linden et al., Reference Van der Linden, Coyette, Poitrenaud, Kalafat, Calacis, Wyns, Van der Linden and Adam Agniel2004), Mahieux gestural praxis battery (Mahieux et al., Reference Mahieux-Laurent, Fabre, Galbrun, Dubrulle and Moroni2009), Frontal Assessment Battery (FAB; Dubois, Slachevsky, Litvan, & Pillon, Reference Dubois, Slachevsky, Litvan and Pillon2000), Trail Making Tests (TMT) A and B (Godefroy et al., Reference Godefroy, Azouvi, Robert, Roussel, Le Gall and Meulemans2010), delayed recall phase of the RL/RI-16 items (Van der Linden et al., Reference Van der Linden, Coyette, Poitrenaud, Kalafat, Calacis, Wyns, Van der Linden and Adam Agniel2004), Rey complex figure test (Meyers & Meyers, Reference Meyers and Meyers1995), category (animals) and letter (letter “P”) verbal fluency tests (Godefroy et al., Reference Godefroy, Azouvi, Robert, Roussel, Le Gall and Meulemans2010), Rey complex figure recall (Meyers & Meyers, Reference Meyers and Meyers1995), 80-picture naming test (DO 80, Deloche & Hannequin, Reference Deloche and Hannequin1997). The neuropsychological tests used in this study are the tests most commonly used in France for the cognitive assessment of dementia. Some are encountered very frequently in the American and international literature (TMT-A and -B, Digit Span forward and backward, FAB), while others are specific to cognitive assessments in France and are validated in a French population in F-F mode of testing (the French version of the FCSRT, Mahieux gestural praxis battery).
The TeleNP assessment used a MacBook pro system with an Internet connection (download speed of 13.14 Mbit/s and an upload speed of 0.90 Mbit/s), a microphone, and speaker. Participants were seated in a chair 2 m away from a 40-inch LCD monitor. A mobile pivoting camera was placed above the screen and could film the entire room, while another camera (handled by the assistant) could be used when needed to film anything the subjects wrote or drew. Through a videoconference interface on a 14-inch LCD monitor, the neuropsychologist examiner could simultaneously see the entire room and the writings and drawings of the subjects. In the F-F situation, the assistant was not present during the assessment.
In the TeleNP condition, the subjects did not handle any of the equipment. The assistant is a secretary who has undergone 1 h of training in computer equipment management (switch on the computer, connect and open the videoconference interface, check that the loudspeakers are working properly). The assistant had no training in neuropsychology. She/he intervened at the request of the neuropsychologist to hand over to the subject the parts of the testing material that we did not seem desirable for the participants to handle themselves (especially in the case of a participant with a cognitive impairment that would not allow him to fully understand when and how to handle the testing material). The assistant welcomed the participants, positioned them, and explained to them that they would not have to handle the computer equipment. The assistant stayed at their side during all the tests. Above all, he/she provided practical help with computer videoconferencing equipment. For the neuropsychological assessment itself, the assistant intervened for the following tests: (1) the learning phase of RL/RI-16 by showing and hiding successively the four boards of the four words to learn; (2) the Rey’s complex figure test in which the assistant provided a sheet of paper for the participant to draw a copy of the complex drawing on, and one to recall it; (3) the subtests 'Prehension behavior’ (the assistant touches the participant’s hands) of the FAB; and (4) the TMT A and B for which the assistant provided the sheet on which the participant must perform the test. When errors were observed in the Trail-Making Test A and B, the assistant was not allowed to intervene; only the examiner intervened.
After each evaluation, the data were recorded in an anonymous database. Tests that were not completed due to cognitive impairment were coded as missing data. In the event of hardware malfunctions (e.g., important problems with Internet speed, with image and sound out of sync, etc.), the data from these tests were not used. Any incident likely to disturb the conduct of the videoconference assessment was recorded: delay, out of sync, image or sound frozen, echo, stress, anxiety, poor comprehension, or behavioral problem.
Statistical Analysis
Data management and statistical analyses were performed with SAS software (version 9.4). For the descriptive analysis, qualitative variables were expressed as number and frequency for each modality, and quantitative variables as the mean, standard deviation (SD), median, interquartile range (IQR), and minimum/maximum (Min-Max). In order to measure the effect between neuropsychological test scores at assessments 1 and 2 according to the administration method (F-F vs. TeleNP), the Cohen’s d was calculated.
The Student’s t test and the Pearson’s chi-squared test (used when all expected numbers were greater than 5, otherwise the Fisher’s exact test was applied when relevant) were used to compare means and proportions, respectively, between the F-F and TeleNP arms (characteristics at baseline and neuropsychological results at each assessment). The Student’s t test for paired data and the McNemar-Bowker test were used to compare distributions of neuropsychological test scores and symmetry for categorical endpoints, between assessments 1 and 2, for each administration method (F-F and TeleNP). In addition, weighted kappa coefficients were provided to measure the agreement (very low: 0–0.21; low: 0.21–0.4; moderate: 0.41–0.60; high to almost perfect: 0.61–0.80; almost perfect: 0.81–1; Landis & Koch, Reference Landis and Koch1977) between ordinal endpoints as STAI-Y scores (anxiety state and trait). Moreover, ANCOVA models were performed to compare neuropsychological test scores between the F-F and TeleNP arms at assessment 2 controlling results at assessment 1. In order to consider the multiplicity of tests, the Benjamini–Hochberg method (Benjamini, & Hochberg, Reference Benjamini and Hochberg2000) was used to control the false discovery rate (FDR) and to provide adjusted p-values. The level of significance was set at 5% for all statistical tests.
RESULTS
Sample Characteristics
Across the whole sample, 6.7% (n = 10) of participants left the study before the second assessment. For the 140 participants who had assessments 1 and 2 (regardless of administration condition), the mean MMSE score was 27.4 ± 2 at assessment 1. At assessment 2, there were 69 participants (49.2%) in the F-F arm and 71 (50.7%) in the TeleNP arm. Demographic characteristics are shown in Table 1 and were similar between both arms.
General characteristics of patients according to administration condition
Both arms were also comparable in terms of medical history and more specifically neurological (around 30% of patients, p = 0.53), psychiatric (around 37% of patients, p = 0.9), and vascular history (around 50% of patients, p = 0.53), with the exception of a significant difference in terms of the presence of hypertension (44.9% vs. 28.2%, p = 0.04), between the F-F and TeleNP arms, respectively, (p-values from chi-square tests).
Neuropsychological Test Scores at Assessments 1 and 2 in Each Arm
The results indicate that the cognitive test scores from assessments 1 and 2 were stable in both arm (F-F and TeleNP) for almost all of the assessment tests, respectively : MMSE (p = 0.50 vs. 0.37), learning phase of the RL/RI-16 items (p = 0.21 vs. 0.61), total of three free recalls of the RL/RI-16 items (p = 0.98 vs. 0.42), total of three recalls of the RL/RI-16 items (p = 0.18 vs. 0.62), Mahieux gestural praxis battery (p = 0.61 vs. 0.49), TMT A (time only, p = 0.26 vs. 0.10) and B (time p = 0.92 vs. 0.26, and number of errors p = 0.70 vs. 0.72), Rey complex figure test (p = 0.75 vs. 0.10), fluency task category “animals” (p = 0.54 vs. 0.08) and letter “P”( p = 0.48 vs. 0.18). A few significant difference was initially noted in the F-F arm (DO 80 picture naming p = 0.05 vs. 0.22, and FAB p = 0.01 vs. 1.00) and in TeleNP arm (Digit Span forward and backward p = 0.61 vs. 0.02, and TMT A number of errors p = 1.00 vs. 0.01) between assessments 1 and 2 with the use of tests for paired data (Student’s or McNemar-Bowker’s) (Tables 2 and 3). No significant difference exists when controlling the FDR with the Benjamini–Hochberg method (Table 3). Scores were not significantly different between assessments 1 and 2 for the DO 80 picture naming test (initially p = 0.05 vs. adjusted p = 0.43) and the FAB (initially p = 0.01 vs. adjusted p = 0.17) in the F-F arm, and for the Digit Span forward and backward (initially p = 0.01 vs. adjusted p = 0.17) and TMT A number of errors (initially p = 0.01 vs. adjusted p = 0.15) in the TeleNP arm (Tables 2 and 3). Moreover, the Cohen’s d values showed small-to-medium size effects with a maximum for the FAB (Cohen’s d = 0.33) in the F-F arm and for the TMT A number of errors (Cohen’s d = 0.35) in the situation of videoconference after first F-F assessment (Table 2).
Comparison of neuropsychological test scores at assessments 1 and 2, according to administration method (face to face and video) and between both methods.
*Student’s t test for paired data; **Adjustment according to the Benjamini–Hochberg method; ***Student’s t test; ****ANCOVA model: comparison of assessment 2 value between F-F and video arms controlling assessment 1 value (see Table 3 for adjustment according to the Benjamini–Hochberg method for adjusted p-value ANCOVA).
P-values adjustments (FDR) for neuropsychological tests scores
* Student’s t test for paired data except for all scores except for anxiety state and trait tested with Bowker’s tests of symmetry.
ANCOVA models showed that score results were similar between the two administration conditions at time 2, controlling assessments results at time 1 (confirmed controlling FDR, Tables 2 and 3). All assessment results were not significantly different between both administration modes (at time 2).
Level of Anxiety during Test Administration
Fisher’s tests used indicated an absence of significant difference between the two types groups in terms of anxiety level felt by patients, both in assessment 1 (F-F assessments) and assessment 2 (F-F vs. TeleNP) (Table 4). A total of 63.1% and 70.4% of patients judged their anxiety state (STAI-Y-A) low or very low at assessment 2 in, respectively, F-F and TeleNP conditions, higher proportion in TeleNP, but not significantly (p = 0.26). These proportions of low anxiety state were 74.2% and 79.7% at the first F-F assessment in F-F and TeleNP arm, indicating in both groups a trend of a lower level of anxiety at the second assessment, but not significantly (Bowker’s test for paired categorical data in each arm, p-values ≥0.27). Low-to-moderate agreements are measured with Kappa’s coefficients K between assessments 1 and 2 in each arm (K≤0.61). These trends are also observed with regards to level of Anxiety Trait (STAI-Y-B) (Table 4).
Comparison of the State-Trait Anxiety Inventory (STAI-Y) at assessments 1 and 2, according to administration method (face to face and video)
*Bowker’s tests of symmetry: Assessment 1 versus Assessment 2, **Fisher’s tests: F-F versus video; ***Adjustment according to the Benjamini–Hochberg method.
Satisfaction with Videoconference Administration
Overall satisfaction with assessment by videoconference was 87.1% (participants “very satisfied”), and, respectively, 82.9% and 74.2% of participants assessed the F-F testing as effective and tiring as TeleNP testing, while 70% reported not feeling anxiety with this new administration method. Only 2 participants indicated that they were rather unsatisfied and 0 very unsatisfied.
Administration or Hardware Malfunctions Across the Study
In total, eight incidents occurred in our study. Six involved minor problems that were rapidly resolved (slow connection, sound/image out of sync) and two were “participant” events (one subject was disturbed by the presence of the camera and another by the noise of the computer), leading one participant to stop the assessment. Details of administration or hardware malfunctions across the study are presented in Table 5.
Frequency of administration or hardware malfunctions across the study (n = 8)
DISCUSSION
To date, this is the first French study of neuropsychological assessments administered by videoconference. Our data indicate highly similar results across both administration conditions (F-F and TeleNP). These results are consistent with most of the findings from previous studies in TeleNP (Cullum & Grosch, Reference Cullum, Grosch, Myers and Turvey2012; Munro Cullum et al., Reference Munro Cullum, Hynan, Grosch, Parikh and Weiner2014; Wadsworth et al., Reference Wadsworth, Dhima, Womack, Hart, Weiner, Hynan and Cullum2018). In addition, we also investigated the feasibility and reliability in routine clinical practice of using neuropsychological tests commonly employed in dementia evaluations and including complex tasks evaluating processing speed or motor and visual abilities. In our study, the TeleNP assessment procedure was well accepted by all subjects. Most participants (87.1%) appeared to be very satisfied with the videoconference testing, and, respectively, 82.9% and 74.2% of participants assessed the F-F testing as effective and tiring as TeleNP testing, which is more than in the study of Parikh et al. (Reference Parikh, Grosch, Graham, Hynan, Weiner, Shore and Cullum2013) who noted that 57% of subjects with no cognitive impairment and 63% with cognitive impairments showed no preference for either testing mode. However, our study did not involve videoconference testing from the get-go but only during the second evaluation. Therefore, evaluating satisfaction in this second evaluation may have resulted in a positive bias due to the fact that the patients have already started their follow-up and care. In addition, it would have been interesting if the survey was also provided to participants who received the F-F component in order to allow a comparison of the two modalities on the question of satisfaction.
All participants in the TeleNP arm were very satisfied with the assistant’s support. Yet, the influence that an assistant may have and the need for his/her presence during the testing have been debated in the literature. The meta-analysis of Brearly et al. (Reference Brearly, Shura, Martindale, Lazowski, Luxton, Shenal and Rowland2017) did not include studies involving an assistant because of considerable heterogeneity in the results of “motor-dependent” tasks between the two administration methods. On the other hand, other studies have administered motor-dependent tasks with staff to assist with the presentation of stimuli and test versions, while a qualified examiner provided verbal instructions from a distance and did not find any significant differences (Temple et al., Reference Temple, Drummond, Valiquette and Jozsvai2010; Turkstra et al., Reference Turkstra, Quinn-Padron, Johnson, Workinger and Antoniotti2012). In our study, all the subjects gave positive feedback regarding the assistant’s support whose presence minimized active interactions between the participant and the computer hardware; and provided help with possible communication problems related to computer malfunction and with the presentation of certain stimuli and understanding of test phases. In summary, the assistant made test administration proceed more smoothly (facilitating the complex order of the rules for test administration). In our experience, the assistant’s contribution seems even more important if the participants are likely to present cognitive disorders as in the case of apraxia or executive disorders.
Loss of information due to audio or video stuttering or poor signal quality may, depending on the task, have a greater or lesser impact on subject performance. However, one of the limitations of TeleNP is related to current imperfections in signal transmission techniques. Although in our study, there were no significant audio or visual transmission problems, or inability to understand and follow the test instructions for all participants in the TeleNP arm, it is important to consider (especially at a time of expansion of this technique) that eight incidents occurred in our study (Table 5). Otherwise, some tasks such as the MMSE copy task or the Rey complex figure test required the use of additional equipment (mobile tripod camera manipulated by the assistant when requested by the test). On two occasions, this caused a delay between the sound and the image on timed tests (TMT and Rey complex figure). This disturbance could have been avoided if the examiner had been able to manipulate the camera remotely. Further investigations with tests more complex to administer, and development of more ergonomic equipment (designed and specific to this type of assessment), as well as standardized methods for presenting visual stimuli, are therefore needed to improve the sensitivity, homogeneity, and fluidity in test administration via TeleNP.
Tests used in routine clinical practice have been standardized and validated through an F-F testing mode, which raises the question of their clinical interpretation and the use of norms during TeleNP testing. Validation tests specifically designed for TeleNP will be an absolute necessity in the future. In our study, changes in the testing procedure, even minor ones (e.g., administering RL/RI-16 boards over the computer instead of in-person, although boards can also be shown on the screen by the examiner), must also be considered in our protocol.
Generally speaking, the conclusions of our study cannot be applied to all population groups. Indeed, our participants do not have major cognitive impairments, have a fairly good education level and they shared the same language and culture than the examiner.
Increased access to this clinical resource would offer multiple opportunities for underserved areas, where these diagnostic tools are not currently available to help isolated individuals living in medical deserts (Adjorlolo, Reference Adjorlolo2015), institutionalized dependent or disabled peoples, and even emigrants. TeleNP can become an instrument of choice in this case. On the other hand, our procedure does not allow for social distancing between the assistant and the patients, and therefore cannot be recommended to identify cognitive disorders in patients who are on lockdown, like in the current coronavirus pandemic (COVID-19). After a short video training, a member of the nursing staff could become the TeleNP assistant. But this model is not applicable for “Directly Into the Home” testing, as it is imperative that the assistants do not personally know the patient being tested, in order to allow them to be as neutral as possible, and to prevent them from helping or distracting the subject. Moreover, the presence of an assistant also makes it possible not to leave the tests available to the patients after their evaluation, in particular to prevent the dissemination of tests in the public domain and the possibility for patients to practice their administration.
Incorporated as part of an interdisciplinary plan for the care and management of dementia, TeleNP would facilitate early diagnosis (delaying the loss of autonomy, promoting home care), allow better care for patients and their caregivers (better targeting of pharmacological and non-pharmacological interventions), and of course, lead to a reduction in treatment costs (savings for patients and for public funds).
In conclusion, the results of our study support the use of TeleNP in medical deserts, but this method needs to be tested in subjects with more severe cognitive impairment and with subjects with major cognitive disorders and significant sensory limitations to ensure generalization. With technological developments, TeleNP may in the future be enriched by new digital communication tools and thus become a precious aid in case of geographical isolation.
ACKNOWLEDGMENTS
The authors wish to thank Professor Bruno Dubois, Director of the “Institut de la Mémoire et de la Maladie d’Alzheimer” (IM2A), neurologist at IM2A, for sending participants who were evaluated for the study in the institute’s premises.
CONFLICT OF INTEREST
The author(s) have nothing to disclose.
FINANCIAL SUPPORT
This work was supported by KLESIA pension funds N°20150227.
ETHICAL STANDARDS
This study was conducted in accordance with the Helsinki Declaration and the rules governing telemedicine (telemedicine decree 2010–1229 of 19 October 2010) and was approved by an Ethics Committee (CPP Ile-de-France X – Hôpital Robert Ballanger – 93602 Aulnay-sous-Bois cedex – N° ID-RCB : 2014-A01971-45).
