Introduction
Concussion, also known as mild traumatic brain injury, is a very common injury, with an estimated 42 million people worldwide affected every year (Gardner & Yaffe, Reference Gardner and Yaffe2015), including 30,000 per year in New Zealand (Feigin et al., Reference Feigin, Theadom, Barker-Collo, Starkey, McPherson, Kahan, Dowell, Brown, Parag, Kydd, Jones, Jones and Ameratunga2013). Monitoring recovery of cognitive impairment following concussion is important for clinical care and research (Silverberg et al., Reference Silverberg, Crane, Dams-O’Connor, Holdnack, Ivins, Lange, Manley, McCrea and Iverson2017).
Cognitive assessment after concussion should evaluate domains of attention, memory, processing speed and executive functioning (Silverberg et al., Reference Silverberg, Crane, Dams-O’Connor, Holdnack, Ivins, Lange, Manley, McCrea and Iverson2017). In contrast with traditional pen-and-paper cognitive assessments, brief and reliable computerised cognitive assessments may enable efficient coverage of all these domains (Holdnack, Iverson, Silverberg, Tulsky & Heinemann, Reference Holdnack, Iverson, Silverberg, Tulsky and Heinemann2017). One such tool is the National Institute of Health (NIH) Toolbox for Assessment of Neurological and Behavioural Function (Gershon et al., Reference Gershon, Cella, Fox, Havlik, Hendrie and Wagster2010, Reference Gershon, Wagster, Hendrie, Fox, Cook and Nowinski2013). The Cognition Battery of the NIH Toolbox (NIHTB-CB) is a computerised screening battery for assessing cognitive functioning and was developed for use with individuals between 7 and 85 years. It appears sensitive to cognitive impairment after traumatic brain injury of all severities (Holdnack et al., Reference Holdnack, Iverson, Silverberg, Tulsky and Heinemann2017) and has demonstrated excellent psychometric properties including significant test–retest correlations and minimal practice effects (Heaton et al., Reference Heaton, Akshoomoff, Tulsky, Mungas, Weintraub, Dikmen, Beaumont, Casaletto, Conway, Slotkin and Gershon2014; Scott, Sorrell & Benitez, Reference Scott, Sorrell and Benitez2019). It is comprised of seven validated cognitive tests that measure executive function, attention, episodic memory, language, processing speed and working memory (see Table 1) and has been developed and validated for use in both clinical and research contexts (Holdnack et al., Reference Holdnack, Iverson, Silverberg, Tulsky and Heinemann2017; Weintraub et al., Reference Weintraub, Dikmen, Heaton, Tulsky, Zelazo, Bauer, Cariozzi, Slotkin, Wallner-Allen, Fox, Beaumont, Mungars, Nowinski, Richler, Deocampo, Manly, Borosh, Havlik, Conway and Gershon2013). In an ideal world all neuropsychological tests, including the NIHTB-CB, would be validated across populations and cultures, however for many countries such as New Zealand such work has not been comprehensively completed. Consequently clinicians have to decide on the best tools for their clinical practice based on international evidence. The NIHTB-CB has not previously been used within a New Zealand clinical or research setting, therefore its validity and utility as a screening tool for cognitive impairment after concussion in New Zealand is unknown.
The continuous advancements in technology and its increasing accessibility, has led to greater opportunities for the development and adoption of computerised cognitive testing by practitioners (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012). This mode of assessing has the potential to enhance clinical practice and computerised testing has many advantages over traditional pen-and-paper methods. These include efficiency, the ability to measure performance on time-sensitive tasks more precisely, automated scoring and better standardisation for test administration (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012; Parsons, McMahan & Kane, Reference Parsons, McMahan and Kane2018). Computerised testing can save clinical administration time, be more engaging for participants and can be cost saving (Parsons et al., Reference Parsons, McMahan and Kane2018; Robillard, Lai, Wu, Feng & Hayden, Reference Robillard, Lai, Wu, Feng and Hayden2018).
In spite of these advantages, neuropsychologists have underutilised computer technologies for cognitive assessment and have been slow in adopting advances in the technology available (Parsons et al., Reference Parsons, McMahan and Kane2018; Rabin et al., Reference Rabin, Spadaccini, Brodale, Grant, Elbulok-Charcape and Barr2014). This may reflect clinician confidence in using newer technologies and concerns about the lack of appropriate normative standards. It may also relate to the conflicting evidence regarding risks for integrating technology in cognitive assessment (Rabin et al., Reference Rabin, Spadaccini, Brodale, Grant, Elbulok-Charcape and Barr2014). Such risks include the concern for error to arise in test administration due to problematic hardware and software interactions and lack of validation across different patient populations. A recent study conducted by Porrselvi and Shanker (Reference Porrselvi and Shankar2018) demonstrated that the method of test administration can impact performances. Using a test of verbal fluency, they compared traditional pen-and-paper methods with the integration of technology and showed that older adults performed significantly better on the pen-and-paper version, underscoring the importance of developing normative standards for computerised versions of cognitive tests.
From the patient’s perspective, there may be other factors. An individual’s familiarity and competence with technology may influence the results and attitudes toward computer-based cognitive testing (Feldstein et al., Reference Feldstein, Keller, Portman, Durham, Klebe and Davis1999; Iverson, Brooks, Ashton, Johnson & Gualtieri, Reference Iverson, Brooks, Ashton, Johnson and Gualtieri2009; Rabin et al., Reference Rabin, Spadaccini, Brodale, Grant, Elbulok-Charcape and Barr2014). Age may be associated with technology familiarity and apprehension for utilising such technologies (Wild, Howieson, Webbe, Seelye & Kaye, Reference Wild, Howieson, Webbe, Seelye and Kaye2008). The cultural relevance of computerised cognitive tests is also important. Cognitive testing, regardless of mode of administration, can be culturally biased with potential to impact on performance and the accuracy of results (Lim et al., Reference Lim, Prang, Cysique, Pietrzak, Snyder and Maruff2009; Malda et al., Reference Malda, van de Vijver, Srinivasan, Transler, Sukumar and Rao2008).
In addition to validation concerns, patient and clinician experiences of computerised cognitive testing are helpful to understand in terms of test utility. Utility is generally considered to broadly reflect how practical a measure is, for example in terms of time to administer, ease of administration and language used (Bannigan & Watson, Reference Bannigan and Watson2009). There is limited research though exploring user experiences of the utility of computerised cognitive testing, with research instead tending to focus on other important psychometric properties such as validity and reliability. In one study evaluating experiences of computerised cognitive testing in persons with Alzheimer’s disease, Robillard and colleagues (Reference Robillard, Lai, Wu, Feng and Hayden2018) showed a growing acceptance of the use of technology in healthcare, but the need for balance between technology and human interaction. General attitudes to the computerised tests were mixed, however these attitudes were largely focussed on the nature of cognitive testing generally rather than the computerised format per se. There is no research that we are aware of to date exploring experiences of computerised cognitive testing in the concussion population, including experiences of the computerised NIHTB-CB.
Study objectives
This study was designed to find out more about the experiences of people with concussion who had been assessed in a research setting with a computerised cognitive battery, the NIHTB-CB. Clinician perspectives of NIHTB-CB’s potential for assessing people with concussion were also explored. The NIHTB-CB is one of a number of computerised batteries available for cognitive assessment, none of which have been specifically validated for the New Zealand population. Our objective was to gather perspectives of the NIHTB-CB’s potential advantages and drawbacks for use with the New Zealand concussion population. This is an initial exploratory evaluation of utility, bearing in mind this is just one aspect of evaluating psychometric properties of cognitive measures.
Method
Study design
This study used qualitative description (Sandelowski, Reference Sandelowski2000) to investigate participant views and experiences of a computerised cognitive battery, the NIHTB-CB, in a context of assessment following concussion. Ethical approval to carry out this study was obtained from the University of Otago Human Ethics Committee (Health) (approval number H19/131).
Participants and setting
This is a qualitative sub-study of a larger study (hereafter called the ‘main study’) currently underway in New Zealand called ‘Optimising Recovery after Concussion’. The main study is a prospective observational study following people for 6 months after concussion, with a focus on understanding psychological factors associated with recovery. In the main study adults aged 16 or over, with a confirmed concussion diagnosis who were consecutively referred to outpatient concussion services in Christchurch, New Zealand within 1 month of injury, were eligible to take part. The diagnosis of concussion was made by a medical specialist and was based on the definition described by Menon, Schwab, Wright & Maas (Reference Menon, Schwab, Wright and Maas2010) for mild traumatic brain injury. This is the definition endorsed by the Accident Compensation Corporation, New Zealand’s primary injury rehabilitation funder, for eligibility to receive concussion services.
Using convenience sampling methods, five patient participants from the main study were invited to take part in a semi-structured interview to explore their experiences of the NIHTB-CB. We sought people from different backgrounds in order to maximise the range of perspectives and experiences of using the NIHTB-CB and to achieve variation in ages, genders and ethnicity. In the main study, participants completed the computerised NIHTB-CB twice, approximately 4–6 weeks after their concussion and again 6 months later. We targeted participants who had recently completed their testing with the NIHTB-CB as part of the main study at either their first or second study visit. Accordingly, participants who were enrolled in the present study were interviewed within 1 month of their main study assessment with the NIHTB-CB, that is either within 1 month of their first visit early after injury or within 1 month following the second visit 6 months later (see Table 2).
Patient participant characteristics
a Traditional cognitive assessment = pencil and paper cognitive testing administered during healthcare alongside study participation.
b Zoom videoconferencing platform; P = patient.
We sought clinician participants with a mix of familiarity with the administration or nature of the NIHTB-CB and experience in assessing and treating people with concussion. We also sought to recruit cultural advisors. In New Zealand health services and research facilities include cultural advisors who ensure that the needs and perspectives of non-European cultural groups, especially New Zealand Māori, are considered.
Data collection
Participants were recruited and data were collected between November and December 2019. During the recruitment period, a research assistant from the main study asked potential patient participants at their study visit if they were interested in being interviewed about their experiences of the NIHTB-CB. Potential participants were then contacted by phone by the first author (CM) and invited to participate in the study. Clinician participants were identified purposively through networks available to the study team. No potential participant approached by either the research assistant from the main study or at later phone follow-up declined to participate in an interview. Semi-structured interviews were conducted with participants (n = 10), with three interviews being conducted electronically using the videoconferencing platform Zoom (Zoom Video Communications Inc., 2016), which has been shown to be a viable tool for collection of qualitative data (Archibald, Ambagtsheer, Casey & Lawless, Reference Archibald, Ambagtsheer, Casey and Lawless2019). The two cultural advisors requested they be interviewed together. Face to face interviews were conducted at Burwood Hospital, the main rehabilitation hospital based in Christchurch New Zealand. All interviews were conducted by a female psychology honours graduate student (CM). Participants were given an information sheet and consent form prior to taking part in the study and provided written consent prior to data collection.
The semi-structured interview guide for patient participants comprised questions about their views of the NIHTB-CB, what they liked and disliked, the iPad format, how long the assessment took, how they felt after the assessment and how they compared the computerised testing to traditional pen-and-paper approaches. The clinician and cultural advisor interview guide was modified slightly, for example we did not ask these participants about how they felt after completing cognitive testing with the NIHTB-CB but we did ask how they thought a person with a concussion might experience the NIHTB-CB. Questions were deliberately open-ended consistent with the qualitative descriptive approach (Sandelowski, Reference Sandelowski2000). Interviews ranged in duration from 10 to 60 min (average 20 min), with the longest interview being the joint interview with the two cultural advisors. We stopped recruitment after 10 interviews because no new information was being offered by participants. Interviews were audio-recorded and then transcribed verbatim, with all participants given a reference code to retain anonymity.
Data analyses
Thematic analyses were based on the general inductive approach described by Thomas (Reference Thomas2006). This is a straight forward approach that allows identification of general themes within raw data without any prior theory or methodological constraints directing the analyses. We followed the six steps to general thematic analysis described by Braun and Clarke (Reference Braun and Clarke2006) and all coding was undertaken by hand. All transcripts were read by two independent coders (CM: honours student and data collector with some previous qualitative research experience and the study lead, DS: neuropsychologist and experienced qualitative researcher), to ensure both coders were familiar with the data. In this way, we aimed to use both investigator and data source triangulation (Carter, Bryant-Lukusius, DiCenso, Blythe & Neville, Reference Carter, Bryant-Lukusius, DiCenso, Blythe and Neville2014). We generated preliminary codes by line by line coding of each segment of the data that was relevant to the research questions. This was followed by round table discussion to verify themes, agree on features, altering, adding and modifying codes using highlighters and mind maps to diagram the connections between them. During the final stage the two coders (CM, DS) worked back and forth between the data (transcripts), comparing and combining results and initial themes until a near-final thematic structure emerged. The findings were reviewed by members of the wider study team to ensure each theme was substantial, for example that there were multiple representative participant quotes to illustrate the theme and data from all participants were considered. This verification process resulted in some relabelling and merging of themes and subthemes before the final structure was agreed upon.
Results
Characteristics of the study sample
Of the 10 participants who agreed to take part, 7 were female, with age ranging from 21 to 68 years (mean age 39.4 years). One participant identified their main ethnicity as British, six identified as New Zealand European, one as Australian and two as New Zealand Māori. Of the clinician participants, one was an experienced test administrator working as a research assistant in the main study, two were neuropsychologists with more than 5 years’ experience working with people with concussion. The two cultural advisors included one experienced health professional. Tables 2 and 3 summarise the characteristics of the participants.
Clinician and cultural advisor characteristics
a Zoom videoconferencing platform.
b Experiential = has participated in testing with the NIHTB-CB as a testee; C = clinician; CA = cultural advisor.
Overview of qualitative findings
There were three broad themes that emerged from the participant interviews. These were (1) using technology for cognitive testing made sense, (2) there were some cultural relevance questions and (3) cognitive testing after concussion could have challenges. Fig. 1 provides an overview of the findings from the study.
Overview of participant themes.
Using technology for cognitive testing made sense
Participants said they thought technology had advantages for cognitive testing. Two main ideas were captured: (1) the iPad was a useful tool and (2) cognitive testing using technology had advantages.
The iPad was a useful tool
The use of the iPad to administer the tests was positively received by participants. The iPad format was considered easy to use and follow. The visual representation of the tests on the iPad was also described as an advantage.
The tests themselves were pretty easy to understand uh because it’s all quite visual, easy to follow. [Patient Participant (P) 2]
Engaging in the tasks and visual stimuli it uses, it seems to have a bit more of real world type of stimuli that are used in the tasks so I think that’s…some really good advantages. [Clinician Participant (C) 3]
The flexibility of the iPad for people with concussion was also identified as an advantage. The ability to change the lighting of the iPad screen was seen as helpful to ensure symptoms were not exacerbated, especially in the early phases.
Yeah the fact that the screen is quite bright it does need to be toned down um with anyone with a concussion otherwise it will exacerbate the headaches even further, exacerbates the fatigue further. [P2]
Differing levels of use and familiarity with iPad technology was not expressed as a concern in relation to the NIHTB-CB. Most participants in this study said they felt they had a high level of familiarity with technology.
Yup, yeah no I use it [technology] all the time so for me it wasn’t really an issue. [P2]
The oldest participant revealed that they had ‘no experience’ using an iPad, but said she was still fine with the use of technology.
I could be the upper ceiling of age though that would be fine with that [technology], but maybe that’s me doing disservice to 70 and 80 year olds. [P4]
One clinician participant reflected that some people could be constrained by testing being technology-based, with familiarity with technology a potential issue for the NIHTB-CB’s acceptability. Yet their opinion of the tool was that it seemed constructive and suitable for most people.
I think for most its good, I think for some um you do run into people who will refuse to use technology and so there’s just no way around that. [C2]
Overall, participants’ attitudes were positive regarding the use of the iPad to portray the tests. Most participants commented that the use of technology was relevant in their everyday lives, as they use technology every day. It felt natural for the NIHTB-CB to be administered on an iPad.
I felt it seemed quite natural to me that it was on an iPad, really … it’s more in the context of my life [P4]
Cognitive testing using technology had advantages
Participants liked the NIHTB-CB tests themselves. Patient participants commented that they found the experience of completing the NIHTB-CB interesting in that they were curious to see what type of tests would be used during cognitive assessment. Participants said they thought the NIHTB-CB was efficient and relevant.
I thought the tests were pretty good to be honest ... I thought they did a pretty good job, really. [P4]
When participants (who had been assessed using more traditional pencil-and-paper tests in a clinical setting) were asked to compare the NIHTB-CB with traditional pen-and-paper cognitive testing, all four who had experienced both said they preferred the NIHTB-CB. Participants related this to technology being in context with their lives, but also the efficiency of computerised testing likely benefiting the care received:
I guess I have an appreciation for how much additional work there actually is when you do something on paper and actually doing something on paper and that then has to get processed by somebody afterwards … If anything it’s a reduction in the level of care you potentially receive because they are more focusing on getting the paper translated. [P2]
Clinician participants also felt the efficiency of the NIHTB-CB was an advantage, with its ability to obtain required information in a brief and accurate manner.
It cuts down on clinician administration time but also if you’re dealing with people after concussion or any brain injury really, if you can keep the testing shorter it’s much more pleasant for them and so I thought there were quite a few advantages in keeping that to a minimum. [C2]
Comparing the NIHTB-CB to traditional pen-and-paper testing, clinician participants said they felt the computerisation of the NIHTB-CB facilitated the accuracy of scoring and also allowed for an interpretive report with corrected scores. The instructions related to each sub-test are automatically given by the iPad, therefore allowing the administration of the NIHTB-CB to be standardised.
The advantages of it being computerised are you’re ensuring the administration is standardised and that errors in scoring are mitigated, given that it is automated. [C3]
I think it cuts out a lot of the problems that we have with concussion screening at the moment. Particularly with uh… the start points and end points vary as you go through so you have a lot less assessment potentially so you are getting similar quality information to what we are getting at the moment but faster. [C2]
Participants rated the acceptability of the computerised NIHTB-CB for its intended use as being high and commented on the good range of cognitive material the NIHTB-CB has, targeting key areas of interest.
It appears to capture key cognitive domains … I like the Toolbox [NIHTB-CB]. I think it’s a really really good tool. [C3]
There were some cultural relevance questions
This theme captured participants’ views on the cultural relevance of the NIHTB-CB for use in New Zealand and included three main ideas: (1) some content in places didn’t feel relevant, (2) the unfamiliar accent made it difficult at times and (3) some of the demographic options were limited for our context.
Some content in places didn’t feel relevant
Participants questioned the relevance of the content of one of the subtests, the Picture Sequence Memory Test, specifically the ‘typical day in the park’ component of this test. Some participants viewed the depiction and accompanied imagery as not necessarily reflective of New Zealander experiences and therefore this felt somewhat difficult to connect with.
Some of those didn’t seem particularly kind of genuine or didn’t feel like everyday life in a New Zealand context necessarily. [C2]
It’s like in New Zealand you don’t really typically go to the park and play in a sandbox … kick a soccer ball around, you know what I mean … it’s not really what kids in New Zealand do. [P1]
One participant suggested that the test could be modified to be more relevant within the New Zealand setting.
You could change to make it a bit more kiwi [New Zealander]-like … pictures, yeah, and words. Even if you did the one at the park as a rugby thing or yeah … something a bit more kiwi. [Cultural Advisor (CA) 2]
One participant said the way some of the pictures were labelled and displayed was not necessarily familiar to them or how they would describe them as a New Zealander. Consequently, this made some components of the Picture Sequence Memory Test stand out more than others.
Either that then makes the word stick out more or, because it’s a word you’re not normally used to so you’re more likely to get it correct, or um distracts you from the other ones. [P5]
The unfamiliar accent made it difficult at times
Participants said they found the North American accent distracting at times and said they thought it would be beneficial to have the NIHTB-CB administered with a New Zealand speaker.
I think it would be better getting it done by like a kiwi [New Zealander] voice … for some reason I keep thinking of the word ‘garage’ [American pronunciation] and ‘garage’ [New Zealand pronunciation]. [P5]
Two clinician participants said they believed the unfamiliar accent had potential to impact on patient performance and hinder some patients, especially for the Picture Vocabulary Test.
It took him [patient] a few times getting familiar with the American accent to then be able to kind of know how to take on the word and what it actually was referring to, so I think that might be one of the concerns that I have in terms of just using within a New Zealand population. [C3]
One participant expressed that for some patients they might need to verbalise the word themselves in the Picture Vocabulary Test, rather than entrusting the NIHTB-CB.
I felt they [the patient] would know and hence by saying the word they just went ‘oh I get it now’. [C3]
There were a couple of things where it was quite American so I couldn’t kind of translate it to what I was thinking … you had four images and you had to relate the word to the image … So I think there might have been some language issues with that. [P2]
It is possible that this is associated with age because the youngest participant in the study said she felt comfortable with the accent due to constant exposure to the American accent on television.
Umm no it didn’t bother me, I mean every time you turn on the TV … you’re watching an American TV program so it’s yeah, OK. [P1]
Some of the demographic options were limited for our context
Clinician and cultural advisor participants felt the demographic options provided by the NIHTB-CB were limiting. They noted that the dropdown list in the demographics section did not include ethnicities beyond the North American context such as Māori and Pasifika. Consequently, when using the NIHTB-CB they said Māori and Pasifika ethnicities would have to be identified as ‘other’, possibly impacting on available norms.
The data, the demographic collection um they don’t have a lot of options in terms of Māori and Pasifika and other races you can identify with, apart from Hispanic or Latino ... [C1]
The inclusion of these ethnicities was considered important for using the NIHTB-CB with the New Zealand population. One participant said it may be culturally relevant to modify the demographic selection by including an IwiFootnote 1 selection.
A lot of them now you can select what Iwi … Instead of typing it in, just have it already there where you can pick and choose … or if you had the option so you could type it in, as you know not all Māori know their iwi but yeah … use the drop-down list and find your iwi, so … [CA1].
Cognitive testing after concussion could have challenges
Participants reflected on potential impacts of computerised cognitive testing after concussion. Three main ideas were captured: (1) it took a long time at the beginning but got easier as I recovered, (2) I thought there might be potential for symptom exacerbation but then that’s the concussion and (3) interaction with screens needs to be considered.
It took a long time at the beginning but got easier as I recovered
Participants said that the NIHTB-CB took longer to complete in the early and more symptomatic phases of recovering from a concussion, versus the last phases of recovery. For example, time to complete the battery varied depending on time since concussion and level of symptoms, ranging between 50 min and 2 h. Some participants discussed their need to take breaks between the sub-tests due to the impact on symptoms such as fatigue and felt the ability to pause between subtests was helpful.
I feel like I was there for a couple of hours the first time um because I had to have a lot of breaks, regularly in session, and it took me longer to actually complete the tests … whereas this time [6 months post-concussion] I didn’t really need any breaks um and I got through it a lot quicker. [P2]
Hence participants felt the timing of administering the NIHTB-CB made a difference, in that it was more difficult and they needed more rest breaks in the early versus the later stages of their concussion recovery. In addition, having the opportunity to be retested after 6 months was expressed as helpful for participants in terms of monitoring their recovery.
Measuring the difference between … So I liked having something at the beginning and having something six months later as well. [P2]
There might be potential for symptom exacerbation but then that’s the concussion
Participants expressed views that computerised cognitive testing might be experienced as challenging for people after concussion and could exacerbate concussion-related symptoms, although they felt this was associated with the concussion rather than the technology per se. Headaches and fatigue after testing were noted in all participants’ narratives, yet they expressed that these symptoms were normal for them at their stage of recovery in terms of their everyday functioning and not specific to the computerised cognitive testing format.
I felt okay [after the testing], apart from a bit of a headache but that’s kind of normal with me at the moment. [P3]
It is challenging, really does have repercussions, yeah definitely fatigue and the headaches are the worst. [P2]
I was quite frustrated and like pretty fatigued [after the session] but then that’s like the concussion. [P5]
Light sensitivity was another symptom related to concussion that was raised as potential issue when using the NIHTB-CB, as it relies on screen interaction, as reflected by this participant’s response.
So, being somebody with concussion it’s quite challenging to use a screen … especially for a prolonged period of time. [P2]
The advantage of the NIHTB-CB being brief in relation to potential for exacerbation of symptoms was emphasised by one participant, stating:
For most people it’s really good and it will be increasingly be fine when the administration is being kept nice and short you’re less likely to run into issues with screen time affecting fatigue, or any problems with that. [C2]
Overall, many patient participants discussed the impact of the computerised NIHTB-CB on their concussion-related symptoms, but did not necessarily isolate this to the NIHTB-CB, rather cognitive testing more generally. In regards to the NIHTB-CB for use after concussion, one participant stated:
I just think it’s a really really valuable tool for anyone who’s had a concussion. [P2]
Interaction with screens needs to be considered
Participants raised the screen interaction of the computerised NIHTB-CB as a potential issue, worrying that the use of a screen for between 30 min to an hour could exacerbate concussion-related symptoms. Clinicians reported they typically advise people with concussion to limit screen use, as screens may worsen symptoms.
And one of my concerns would be its strong focus on a computer and the implications that could have for people with um visual sensitivity or light sensitivity and um just a constant focus on a visual screen for 30 minutes may exacerbate some symptoms. [C3]
In addition, the possible exacerbation of concussion symptoms, especially for people with visual issues, may be connected to the timing of administering the NIHTB-CB.
Straight after [the concussion], possibly. I guess it depends when the screening is happening … if it is by around that month mark most people should be fine … yeah again it’s going to vary a little bit with the presentation of what people are demonstrating, for some people it probably is a challenge. For most, my guess is it would be fine. [C2]
Being mindful of their light sensitivity, most participants expressed that they required light alterations to be made on the iPad to complete cognitive testing.
I have sort of learnt how to bring the warmth down, colours and things for people who are affected by the brightness and um the light of the iPad. [C1]
One participant said that completing the testing was challenging whether being on a screen or not, highlighting their belief that their exacerbation of symptoms was not necessarily a direct consequence of the screen interaction itself.
Irrespective of being on screen or being on paper each meeting was as challenging as the other. [P2]
Discussion
This study explored experiences of a computerised cognitive battery, the NIHTB-CB in a small sample of participants familiar with the tool. Participants described positive and negative experiences using the NIHTB-CB and there were marked similarities in the themes that emerged from patient and clinician participant narratives. Participants liked the computerised format of the NIHTB-CB and the cognitive domains included in the battery. Negative perceptions were related to computerized testing in general, requirements for sustained mental exertion after concussion regardless of mode of assessment, and aspects of the NIHTB-CB in particular.
First, narratives captured concerns regarding risk for concussion symptom exacerbation associated with computerised assessment formats, for example in terms of the interaction with computer screens and devices and testing early following concussion. Concerns about symptom exacerbation following cognitive exertion and impacts of using screens are not specific to cognitive testing but reported across many contexts by people recovering from concussion (Mansur et al., Reference Mansur, Hauer, Hussain, Alatwi, Tarazi, Khodadadi and Tator2018). However in our small study, the lighting flexibility of the iPad with its constructive function of adjusting the screen and changing the lighting when appropriate or necessary and the ability to pause between subtests, were identified as advantages. Interestingly, our study participants preferred computerised cognitive testing compared with more traditional, pencil-and-paper methods of test administration.
Second, participants described cognitive exertion required for sustained cognitive performance in any context, as an issue. Patient participant narratives in particular reflected that their experiences during cognitive testing were not specific to the NIHTB-CB or its computerisation format. Cognitive fatigue and headaches are common symptoms experienced early following concussion with potential to mutually interact with and amplify other post-concussion symptoms (Iverson, Reference Iverson2019). There were similarities between our participant experiences and those described by Robillard and colleagues in older adults attending a dementia clinic. For example, their participants’ experiences of cognitive testing were less focussed on the battery used or computerisation but rather effort associated with cognitive testing more generally (Robillard et al., Reference Robillard, Lai, Wu, Feng and Hayden2018).
Participants though raised specific concerns about the NIHTB-CB in relation to cultural relevance especially for two of the subtests, the Picture Vocabulary subtest and the items making up the Picture Sequence Memory Test. Clinician and cultural advisor participants also raised the lack of diversity provided by NIHTB-CB demographic options. The direct translation of cognitive tests regardless of mode of administration, to new populations can be affected by differences in understanding and interpretation, cultural appropriateness and comparable test score interpretations (Malda et al., Reference Malda, van de Vijver, Srinivasan, Transler, Sukumar and Rao2008; Vijver & Tanzer, Reference Van de Vijver and Tanzer2004). Dudley, Barker-Collo, Wilson & Garrett (Reference Dudley, Barker-Collo, Wilson and Garrett2019) highlighted related cross-cultural concerns in regard to the Wechsler Adult Intelligence Scale (WAIS-IV: Wechsler, Reference Wechsler2008) in a non-European New Zealand population sample, finding that variables such as income and education impacted the WAIS-IV performance of Māori. Whilst it was out of the scope of the present study to examine impacts on performances, participants never the less questioned whether cognitive performance might be impacted in places by cultural factors, especially in regard to the Picture Vocabulary and the Picture Memory Sequence subtests of the NIHTB-CB.
Our findings are consistent with research identifying that a broad range of factors need to be considered in determining the value of computerised neuropsychological test batteries. Many of the advantages of computerised testing discussed by Bauer and colleagues (Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012) and Parsons and colleagues (Reference Parsons, McMahan and Kane2018), were also identified by our study participants. These included the efficiency, ease of administration, standardisation, visual engagement and automated scoring of the NIHTB-CB. The limitations of integrating technology in cognitive testing raised by Iverson and colleagues (Reference Iverson, Brooks, Ashton, Johnson and Gualtieri2009) and Porrselvi and Shanker (Reference Porrselvi and Shankar2018), such as technology familiarity were not raised by our participants. While none of our patient participants raised their individual technology familiarity as in issue, one clinician participant discussed the possibility some patients may refuse to use such technology. Our participants, like those of other studies (such as Robillard et al., Reference Robillard, Lai, Wu, Feng and Hayden2018) noted benefits including increased accuracy, efficiency and the freeing up of clinician time.
In spite of noted advantages, utilisation rates of computerised testing amongst neuropsychologists are still reported to be low, with Rabin and colleagues (Reference Rabin, Spadaccini, Brodale, Grant, Elbulok-Charcape and Barr2014) demonstrating that in a sample of 495 neuropsychologists surveyed, 59.6% reported either ‘never’ or ‘rarely’ using computerised test batteries in clinical practice. The benefits for neuropsychologists incorporating advances in technology in their practice continue to be discussed in relevant literature (Parsons et al., Reference Parsons, McMahan and Kane2018). One aspect related to the low utilisation rates could be the concern about lack of appropriate normative standards for computerised cognitive testing batteries (Bauer et al., Reference Bauer, Iverson, Cernich, Binder, Ruff and Naugle2012). The participants in our study, in raising concerns about aspects of the cultural relevance of the NIHTB-CB and associated impacts on cognitive performances, were alluding to the applicability of normative standards provided by the NIHTB-CB.
Clinical implications
Negative feedback from participants appeared to be predominantly associated with cultural considerations and concern about post-concussion fatigue and symptom exacerbation. These are not specific or limited to the NIHTB-CB but are general concerns across any assessment with the concussion population and when using measures developed in different cultural contexts. Clinicians should consider the usual factors when they evaluate NIHTB-CB performance as they would in any context when they apply impairment definitions (Guilmette et al., Reference Guilmette, Sweet, Hebben, Koltai, Mahone, Spiegler, Stucky and Westerveld2020). Regarding computerised cognitive testing more generally in the concussion population our participants were positive overall about this, albeit cautious about risks for symptom exacerbation from prolonged screen interaction. Some of these concerns were allayed by the adaptations and flexibility available with the iPad settings that are not necessarily specific to the NIHTB-CB but any battery mediated by iPad technology. To promote computerised cognitive testing in a clinical setting, clinicians may need to be aware of timing post-injury and use breaks and pace testing appropriately. This was a positive aspect of the NIHTB-CB, that is, the option of pausing between subtests to enable patients to manage fatigue and other post-concussion symptoms likely to be evident especially early after injury.
Limitations
This was a qualitative study with a small sample size, limiting the generalisability of the findings. The lack of ethnicity, age and gender diversity within the patient participant sample was a further limitation, although this was a consequence of the main study recruitment progress at the time of the present study. In contrast with Iverson and colleagues (Reference Iverson, Brooks, Ashton, Johnson and Gualtieri2009), this study found no concerns related to computerised testing and technology familiarity, yet it is unknown whether this finding may be a consequence of the relatively limited age and gender diversity of our sample. Hence, it would be informative for future studies to include older and more male participants, to gather further information on the impact of such factors on computerised testing and technology familiarity. Finally, clinician participants had varying degrees of experience with the NIHTB-CB from having experienced this as a testee only through to having administered the battery many times. Although all clinician participants were familiar with the impacts of concussion, such diverse levels of experience with the NIHTB-CB are likely to have influenced participant ability to respond to interview questions.
Conclusions
The present study provides information about some aspects of utility of the NIHTB-CB for New Zealanders as well as participant experiences of computerised cognitive assessment more generally after concussion. Our participants described both positive and negative experiences with the NIHTB-CB. They were optimistic about its computerisation format however the cultural relevance of the NIHTB-CB was raised as an area of concern. Given this was a small qualitative study with a convenience sample, more robust evaluation of the current form of the computerised NIHTB-CB is needed to assure the validity and utility of the instrument for use with New Zealanders.
Financial Support
This study was completed as part of the University of Otago’s summer studentship programme and was funded by the Canterbury Medical Research Foundation.
The main study is funded by the Health Research Council of New Zealand (ref 18/046).
Conflicts of Interest
The authors have no conflicts of interest to declare.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
