Current Concepts and Status of Assessment in Neurorehabilitation

There are at least a dozen different roles of assessment in neurorehabilitation (see Tate, Reference Tate, Levin, Shum and Chan2014). Some of these roles involve direct assessment (e.g., making a differential diagnosis, documenting improvement, evaluating treatment effectiveness), and others are the translation of the results of assessment (e.g., planning treatments, helping people make practical decisions, educating people with brain injury and their families).

It stands to reason that different types of assessment instruments are required for different purposes. The type of assessment that is suitable for differential diagnosis will not necessarily be appropriate to monitor progress during treatment. Moreover, there are many types of instruments that vary along different dimensions (quantitative vs. qualitative), modes of administration (observational, performance-based vs. proxy/self-report) and the measurement parameter (e.g., frequency, duration, intensity). In the context of assessment, the International Classification of Functioning, Disability and Health (ICF; WHO, 2001) has a lot to offer.

The ICF is a framework that describes all types of health conditions – neurological, immunological, musculoskeletal and so forth. The consequences of any health condition can be considered from the individual’s perspective in terms of impairments, activities and participation, environmental and personal factors. The ICF consists of two parts (Function and Disability, and Contextual Factors) and five components (Body Functions, Body Structures, Activities and Participation, Environmental Factors and Personal Factors), as shown in Fig. 1. The structure of the ICF is hierarchical and nested, depicted in Fig. 2, with each of the five components being subdivided into domains, totalling n = 30. Each domain is further subdivided into categories, altogether there being n = 1424 categories, each of which is identified by an alphanumeric code.

Figure 1.

Overview of the International Classification of Functioning, Disability and Health (ICF).

Source: World Health Organization (2001, p. 18).

Figure 2.

The hierarchical and nested structure of the ICF.

Source: Tate and Perdices (2008, p. 285).

Figure 3 further describes the nested structure of the ICF, by showing the second-level categories for the Mental Functions domain of the Body Functions component. These categories themselves can be further subdivided into third- and fourth-level categories. For example, the Orientation category (b114) can be broken down into Orientation to person (third-level category, b1142), which itself can be further subdivided to the fourth-level category, Orientation to self (b11420) versus Orientation to other people (b11421).

Figure 3.

The ICF at the category level (for the Mental Functions domain).

Source: Tate and Perdices (2008, p. 287).

We used the ICF structure in a systematic review of outcome instruments used for adults with TBI (Tate, Godbee, & Sigmundsdottir, Reference Tate, Godbee and Sigmundsdottir2013). Our initial searches yielded 5735 articles of which 2593 met selection criteria, in which they described 728 unique instruments. We classified each of the 728 instruments into the ICF framework in terms of the construct being measured, rather than the usual linking procedure which occurs at the item level (see Cieza et al., Reference Cieza, Geyh, Chatterji, Kostanjsek, Üstün and Stucki2005).

The lion’s share of instruments for adults with TBI, two-thirds (n = 466), addressed the Body Functions component (i.e., function at the impairment level), and a much smaller proportion, 15% (n = 109) addressed the overlapping constructs involved in Activities/Participation. A very small percentage addressed Environmental (3%, n = 22) and Personal (5%, n = 36) Factors. A substantial number (8%, n = 60) were multidimensional instruments, sampling from multiple components and domains, and 5% (n = 35) involved concepts that are not covered by the ICF, such as quality of life instruments.

The 2013 article comes with a handy 10-page appendix, which enumerates each of the 728 instruments and classifies them into their respective ICF category at the detail of the second level. This study, and the mapping exercise, also provides good information about areas of glut versus need for instrument development. In the area of TBI, arguably it would be advantageous to focus more on instruments that are functionally based at the Activities/Participation level, as well as having a greater choice at the Environmental/Personal Factors levels.

The overwhelming number of 1424 ICF categories, together with the huge number of assessment instruments for acquired brain impairment (n = 728 for TBI adults alone), are unwieldy to work with, both for research and for clinical practice. To answer this problem, various working groups of the ICF research branch have developed ‘ICF Core Sets’ for various health conditions. Of relevance to neurorehabilitation are ICF core sets for multiple sclerosis (Coenen et al., Reference Coenen, Cieza, Freeman, Khan, Miller, Weise and Kesselring2011), stroke (Geyh et al., Reference Geyh, Cieza, Schouten, Dickson, Frommelt, Omar and Stucki2004), TBI (Laxe et al., Reference Laxe, Zasler, Selb, Tate, Tormos and Bernabeu2013), older adults with and without dementia (Scherer et al., Reference Scherer, Federici, Tiberio, Pigliautile, Corradi and Meloni2012; Spoorenberg et al., Reference Spoorenberg, Reijneveld, Middel, Uittenbroek, Kremer and Wynia2015, respectively), post-acute rehabilitation (Grill et al., Reference Grill, Strobi, Müller, Quittan, Kostanjsek and Stucki2011) and vocational rehabilitation (Finger et al., Reference Finger, Escorpizo, Glässel, Gmünder, Lückenkemper, Chan and Cieza2012).

The core sets are usually developed at the category level of detail and represent those categories that are necessary and sufficient to describe a health condition. Usually two core sets are developed per condition: a comprehensive core set and a brief core set. The comprehensive core set for TBI, for example, contains 139 codes; the brief core set contains 23 codes. Table 1 shows the brief ICF core set for TBI, which comprises eight categories for Body Functions, one for Body Structures, seven (plus one domain) for Activities/Participation and six for Environmental Factors (Personal Factors of the ICF has not yet been classified and so that component is not considered as part of ICF core sets).

Table 1:

TBI brief core set (n = 23 codes) and the number of behavioural assessment instruments identified in the Tate et al. (Reference Tate, Godbee and Sigmundsdottir2013) systematic review

Table 1 also depicts the number of instruments from the Tate et al. (Reference Tate, Godbee and Sigmundsdottir2013) systematic review that evaluate constructs contained in the ICF categories. The numbers adjacent to the categories refer to an instrument that exclusively addresses only that category. For example, in the category of Consciousness within the Body Functions component, we identified 15 instruments such as the Glasgow Coma Scale, Coma Recovery Scale – Revised, Wessex Head Injury Matrix, Confusion Assessment Protocol, which exclusively address Consciousness and no other category. In total, we identified 239 instruments that focussed on a specific ICF category in the Body Functions component. But other instruments (see bottom row of Table 1) addressed multiple categories both within and across ICF domains. For example, instruments such as the Glasgow Outcome Scale and the Mayo-Portland Adaptability Inventory sample items from both the Body Functions and Activity/Participation components. Accordingly, we identified 65 instruments that were multidimensional across categories within the Mental Functions domain, 32 instruments that were multidimensional across domains within the Body Functions component and 60 instruments that were multidimensional across components of the ICF, encompassing both Functioning and Activity/Participation.

At the Activity/Participation level, there were far fewer instruments that specifically targeted a single Activity/Participation category (n = 40). Apart from the category of Walking, in general, Activity/Participation instruments were multidimensional within the Activity/Participation component, with 48 additional multidimensional instruments. Environmental Factors is an area of instrument development that is still in its infancy, as reflected by the very small number of both specific instruments (n = 8) and even multidimensional scales (n = 7). In total then, these 22 items from the ICF brief core set for TBI (excluding the single Body Structure item) mapped to 499 of the 728 instruments identified in the Tate et al. (Reference Tate, Godbee and Sigmundsdottir2013) systematic review.

The ICF core sets provide valued direction about what should be assessed in a particular health condition. This, however, does not speak to how the construct should be assessed; more specifically, which instruments should be used. The aforementioned systematic review demonstrated that there is a large selection of instruments from which to choose; just in TBI adults alone. As with the ICF itself, there is a need for core sets of assessment instruments, which are generally referred to as ‘common data sets’. There have been quite a number of groups that have developed common datasets of instruments for application in different health conditions and different types of research.

A common dataset was developed by our Australian team (Honan et al., Reference Honan, McDonald, Tate, Ownsworth, Togher, Fleming and Ponsford2019), within the auspice of the NHMRC Centre for Research Excellence in psychosocial recovery after TBI (McDonald et al., Reference McDonald, Anderson, Ponsford, Tate, Togher, Morgan and Murdoch2012). The group made recommendations for the use of 56 instruments for assessing psychosocial consequences of TBI. The ICF was loosely used to classify the instruments and, similar to the Tate et al. (Reference Tate, Godbee and Sigmundsdottir2013) review, took the perspective of classifying the construct that the instrument was intending to measure. Instruments were classified into 11 groups: Communication (n = 8 instruments); Social Cognition (n = 2); Behavioural and Executive Function (n = 3); Other Neuropsychological (n = 7); Psychological Status (n = 7, mainly focusing on emotional function); TBI-related Symptoms (n = 4); Activities/Participation (n = 6); Environmental Factors, specifically Support and Relationships (n = 5); Personal Factors (n = 7); Global Outcomes (n = 3) and Concepts Not Covered by the ICF (n = 4). Another paper by the team recommends 34 instruments for research in psychosocial recovery after paediatric TBI (Wearne et al., Reference Wearne, Anderson, Catroppa, Morgan, Ponsford, Tate and McDonald2018).

Evidence-Based Assessment and the Scientific Quality of Assessment Instruments

The preceding lines of research point to the main current concepts in assessment in TBI – the role of the ICF, ICF core sets and common datasets. The next question in this era of evidence-based clinical practice is, how good are the instruments that are available? What is their scientific quality? This is referred to as evidence-based assessment. Hunsley and Mash (Reference Hunsley and Mash2007, p. 29) describe evidence-based assessment as emphasising ‘the use of research and theory to inform selection of assessment targets, the methods and measures used in assessment, and the assessment process itself’. Hunsley and Mash observe that while evidence-based practice has traditionally been applied to interventions, there is equally a need to ensure that assessment practices are evidence-based so that they are ‘clinically relevant, culturally sensitive, and scientifically sound’ (p. 30). Yet, they emphasise that evidence-based assessment involves more than the administration of psychological tests with strong psychometric properties; it is ‘inherently a decision-making task in which the clinician must iteratively formulate and test hypotheses by integrating data that are often incomplete or inconsistent’ (p. 30). While not diminishing the importance of clinically relevant and culturally sensitive measures, nor the broader role of the decision-making process, the themes that I would like to pursue in this address are areas where there is scope for improvement in neurorehabilitation assessment.

Diagnostic assessment is one of the roles of assessment in neurorehabilitation. Yet in neurorehabilitation the neurological diagnosis is often already known. But what diagnosis is pertinent to in neurorehabilitation is differential diagnosis for determining the type of aphasia, the type of dementia, the type of memory disorder and so forth.

Prognostic assessment is another role. This should be a fertile area of research activity in neurorehabilitation, but it is presently underdeveloped. Prognosis takes many forms. The role of post-traumatic amnesia (PTA), for example, is probably the most widely known variable used as a predictor of outcome after TBI. Consider this as an example of the potential application of prognostic instruments in neurorehabilitation practice.

Our aim in this study was to develop a simple and clinically useful method to predict PTA duration from data collected soon after admission to rehabilitation (Tate, Perdices, Pfaff, & Jurjevic, Reference Tate, Perdices, Pfaff and Jurjevic2001). We used a PTA instrument, the Modified Oxford PTA Scale (Pfaff & Tate, Reference Pfaff and Tate2004), to estimate the duration of PTA within 5 days of PTA testing using three predictor variables. The regression equation has impressive predictive accuracy, with the R ²_(adj) = 0.89, indicating it accounts for 89% of the variance. Correlation between the predicted and observed duration of PTA was high (ICC = 0.95). We developed a second model, which was simpler than the first model, and which we recommended for use in clinical practice. Its predictive accuracy was comparable to the first model (R ²_(adj) = 0.88, ICC = 0.95) and the regression equation was as follows:

• Log₁₀ predicted duration of PTA = 1.22 + (0.14 × √DAYSTART) – (0.009 × PTA1–5)

where DAYSTART = the day post-trauma on which PTA testing commenced, and PTA1-5 = the sum of the first 5 days of PTA scores on the MOPTAS.

The regression equation is easily applied, as shown in our worked example (Tate et al., Reference Tate, Perdices, Pfaff and Jurjevic2001): a 28-year-old male had documented PTA duration of 26 days. PTA testing commenced at Day 14 post-trauma (therefore DAYSTART was 14, and √14 = 3.74). The sum of the PTA scores for the first 5 days of testing was 31 (5.5 + 3.5 + 5.5 + 7.5 + 9). Entering the data into the equation:

• Log₁₀ predicted duration of PTA = 1.22 + (0.14 × √14) – (0.009 × 31) = 1.22 + 0.52 – 0.28 = 1.46

Thus 1.46 is the logarithm (to base 10) of the estimated duration of PTA. This can be converted to a real number using a scientific calculator readily available on computers: enter 10, press the x^y key, enter 1.46, press x^y again. The last operation yields the number 28.8, which represents the estimated duration of PTA (i.e., 29 days). Thus, in this case we predicted that PTA duration would be 29 days, and in reality it was 26 days. Our team has subsequently completed further work in refining the regression model in order to predict duration of PTA in a broader range of injury severity (mild to severe) and at a point in time more proximal to the trauma, using the first 3 days of PTA data. The results are equally impressive.

Having such prognostic information available is valuable for many reasons: Early prediction of PTA duration will facilitate family education and decision-making; clinical resource allocation for planning nursing resources needed in the post-acute phase and determining acute bed availability; forward planning of post-acute clinical management, including the timing and the instigation of formal assessments, discharge planning, liaison with community services; and so forth. It would be desirable to see more use of prognostic studies in neurorehabilitation and application/translation of the results to clinical practice.

Over the past decade, a lot of activity has occurred in developing standards regarding the quality of assessment instruments. Specific meta-instruments, if you like, are available to evaluate the scientific quality of a particular instrument. We reviewed six such instruments (Rosenkoetter & Tate, Reference Rosenkoetter and Tate2018) in last year’s Special Issue of Brain Impairment on quantitative data analysis.

Our review highlighted the COSMIN (COnsensus-based Standards for the selection of health status Measurement INstruments; Mokkink et al., Reference Mokkink, Terwee, Patrick, Alonso, Stratford, Knol and de Vet2010), as being particularly carefully developed using Delphi procedures. It is used in conjunction with the Terwee standards (Terwee et al., Reference Terwee, Mokkink, Knol, Ostelo, Bouter and de Vet2012). The particular advantage of using these two instruments is that they allow levels of evidence of assessment instruments to be established, albeit using a rather different model to the NHMRC levels of evidence. Although the COSMIN was developed for so-called patient-reported outcome measures (PROMs), the criteria are also applicable to other outcome instruments. More recently, Mokkink and colleagues (Reference Mokkink, de Vet, Prinsen, Patrick, Alonso, Bouter and Terwee2018) have revised the item content of the COSMIN, which is also now supplemented with a ‘risk of bias’ checklist.

The COSMIN is a standardised tool focusing on assessment of the methodological design of a psychometric study (e.g., was a sufficiently large sample size used in the psychometric study?). In the original version (Mokkink et al., Reference Mokkink, Terwee, Patrick, Alonso, Stratford, Knol and de Vet2010), the instrument comprised 114 items, in 9 ‘boxes’ (with 5–18 items/‘box’) addressing nine measurement properties: internal consistency, reliability, measurement error, content validity, construct validity (structural validity, hypothesis testing, cross-cultural validity), criterion validity, responsiveness. Different scoring systems can be applied to score the items: (a) yes/no/na; OR (b) a 4-point scale (excellent, good, fair, poor; e.g., a sample size of ≥100 is excellent, 50–99 good, 30–49 fair).

Ratings of the items within each ‘box’ are synthesised using a ‘worst-score-counts’ algorithm – this makes the scoring very stringent. For example, if a responsiveness study (Box I) obtains a rating of ‘good’ for 15/16 items in that box, but the sample size for the responsiveness study is, say n = 20, then it will be scored as ‘poor’ (viz. ‘poor’ corresponds to a sample size of n < 30), irrespective of the statistical results regarding responsiveness. The Terwee criteria can be applied to the COSMIN items and focuses on assessment of statistical outcomes of measurement properties. A 3-point rating system is used: positive (+), indeterminate (?), negative (–). For example, for reliability (Box B):

• + = ICC/K _w ≥ 0.7 OR Pearson’s r ≥ 0.8

• ? = neither ICC/K _w OR Pearson’s determined

• – = ICC/K _w < 0.7 OR Pearson’s r < 0.8

Schellingerhout and colleagues (Reference Schellingerhout, Verhagen, Heymans, Koes, de Vet and Terwee2012) have developed a method of synthesising the COSMIN design ratings and the Terwee statistical outcome ratings into five levels of evidence: strong, moderate, limited, conflicting and unknown. The Rosenkoetter and Tate (Reference Rosenkoetter and Tate2018) article provides a worked example of applying the COSMIN/Terwee/Schellingerhout criteria.

In the past, the focus of scientific adequacy of assessment instruments has been on the results (statistical outcomes) of psychometric studies. Virtually no attention was paid to the design adequacy of a study. However, the COSMIN group disagrees with this lack of attention to method quality. In their view, if the design of a psychometric study of a particular instrument is appropriate, then the results can be used to assess the quality of the instrument. But when the method quality is inadequate, the results cannot be trusted and the quality of the instrument remains unclear (Terwee et al., Reference Terwee, Mokkink, Knol, Ostelo, Bouter and de Vet2012). As noted, the COSMIN criteria are stringent. This has partly to do with their emphasis on the design of psychometric studies, together with the ‘worst score counts’ algorithm. The authors have justified their emphasis, and it certainly raises the bar in terms of standards of psychometric investigative rigour for assessment instruments.

In relation to COSMIN/Terwee criteria, notwithstanding their stringent standards, in general there is room for improvement in health status measures used in neurorehabilitation. Very few of the 728 instruments identified in our systematic review (Tate et al., Reference Tate, Godbee and Sigmundsdottir2013) would meet evidence-based standards using COSMIN/Terwee criteria. Indeed, even the most respected and widely used instruments in acquired brain impairment have psychometric gaps (see Tate, Reference Tate2010).

Moving Beyond Outcome Assessment

The foregoing has focused on standardised outcome instruments, which allow examination of diagnosis, prognosis, treatment effect and many other applications. But in terms of assessment in neurorehabilitation, use of standardised instruments only tells part of the story. In particular, standardised outcome instruments make a limited contribution to monitoring progress.

There are two interrelated reasons for the limited applicability of standardised outcome instruments to monitoring progress in neurorehabilitation. First, standardised instruments do not necessarily map to the behaviour targeted by the intervention (e.g., an impairment-based instrument measuring memory is not appropriate to evaluate a functionally based memory intervention to determine whether the memory goals have been achieved). In neurorehabilitation, it is frequently the case that everyday function in a particular area can improve, in the absence of any improvement in the underlying impairment. Second, standardised instruments are generic, in the sense that total or subscale scores are not necessarily specific to the behaviour being targeted by the intervention. Both these features mean that standardised instruments may be insensitive to and not capture real changes that occur in the patient/client.

We can contrast the standardised outcome instrument approach with a functional, individualised approach. Consider the timeline in Fig. 4: a patient sustains a brain injury and is admitted to hospital and subsequently the inpatient rehabilitation unit. Early on in the rehabilitation process, at point A, the patient has a lot of assessments using standardised instruments for diagnosis, prognosis, treatment planning and individualised assessments for goal setting. And these assessment instruments can be re-administered later on, at point B, to determine the following: what is the outcome? has there been any treatment effect? have the goals been attained and so forth? Here the focus is clearly on change: Is the behaviour targeted by the intervention any different – has it changed?

Figure 4.

Assessment in neurorehabilitation.

Note. tx = treatment

This is good, but it could be better. Another perspective could be to look in more detail at the rehabilitation process to monitor progress throughout the intervention, while it is taking place, that is, between points A and B. In other words, to measure progress as an evolving process towards an improved state or more advanced condition.

In order to monitor progress, the type of standardised instrument identified in the aforementioned systematic review is generally not used (although there are exceptions). Rather, as with SMART goal principles (Specific, Measureable, Achievable, Relevant/Realistic, Time-limited; Evans & Krasny-Pacini, Reference Evans, Krasny-Pacini, Wilson, Winegardener, van Heugten and Ownsworth2017), it is necessary to use a measure of the behaviour that is highly specific, observable (or at least quantifiable) and repeatable. In single-case methodology, this highly specific dependent variable, that is proximal to the behaviour which the intervention aims to change, is referred to as the target behaviour (see Tate & Perdices, Reference Tate and Perdices2019; Tate, Perdices & Wakim, in press). However, it is not always an easy task to identify a specific, observable and repeatable measure of the target behaviour. The clinician/researcher needs to think creatively, particularly in the situation where abstract constructs need to be operationalised and measured in specific and quantitative terms. For example,

• inertia – consider using time (latency of response; duration of an activity) as the operationally defined measure

• disorganisation – perhaps it is possible to take photographic evidence of the mess

• social isolation – one could track the number of social contacts, time spent out of the house

In order to adopt this approach, the clinician/researcher may need to construct a tailor-made instrument specifically for the individual. Figure 5 depicts data from a participant from one of our studies (Tate, Wakim, Sigmundsdottir, & Longley, Reference Tate, Wakim, Sigmundsdottir and Longley2018), who wanted to engage in a more active lifestyle. This happened to be in the context of a single-case experimental design, but equally it could have been observational data. One of the measures we used was the number of steps he walked each day, as measured by a pedometer. Figure 5 depicts his progress, both before the intervention commenced and also during the 15 weeks it was in progress. Monitoring the behaviour in this way has the advantage of being able to immediately see the response to therapy (e.g., what is happening between sessions 2 and 3, 5 and 6, and 11 to 13 that his performance decreases?). Advantages of continuous monitoring of the target behaviour throughout the intervention are that (i) immediate steps can be taken if necessary to address adverse responses during therapy, and (ii) it provides immediate and useful feedback for the patient during therapy.

Figure 5.

Monitoring behaviour during an intervention.

Note. TB1 = target behaviour 1

Source: Adapted from Tate et al. (2018).

From an evidence-based assessment perspective, however, there are challenges in adopting this approach because essentially I am advocating use of an unstandardised, single-item measure. A number of issues (and their solutions) are involved:

1. (i) First, data from a single individual (the patient/client) are collected, and second there is naturally occurring variability of behaviour. For these reasons, multiple datapoints need to be taken to document the stability (or otherwise) of a behaviour, which in turn assists in identifying the pattern of performance and whether there is a treatment effect

2. (ii) Use of unstandardised, single-item measures is restricted to simple documentation of performance; there is generally no external standard (e.g., normative data) against which progress can be compared. In answer to this problem, our team has used goal attainment scaling (GAS; see Evans & Krasny-Pacini, Reference Evans, Krasny-Pacini, Wilson, Winegardener, van Heugten and Ownsworth2017) in conjunction with raw data from the target behaviour. Figure 5 derived from Tate et al. (Reference Tate, Wakim, Sigmundsdottir and Longley2018) shows how we plotted continuous data points on the left-side Y-axis, against GAS performance on the right-side Y-axis. This approach allows for both continuous measurement (which from a measurement perspective is better than the ordinal level of measurement used in GAS), as well as maps that continuous performance against an external standard (in this case, the GAS level of zero, 5,000 steps, represents the expected level of performance).

3. (iii) To quantify the trends and change in performance in any rigorous way, data analytic techniques need to be applied. Fortunately, an increasing number of techniques are available to analyse time-series data at the individual level, and the website developed by Manolov, Moeyaert, and Evans (Reference Manolov, Moeyaert and Evans2015) is a valuable resource: www.ub.edu/gcai/soft/Software_resources_for_applied_researchers_April2015.pdf

4. (iv) A final evidence-based assessment concern about the use of unstandardised measures relates to the reliability of observations. This is an important concern that needs to be addressed to meet standards of assessment. If a permanent product record can be made, then another person can be used to establish inter-rater agreement, which will assist in documenting the reliability of the measure used. Arguably, using machine-generated data, such as a calibrated pedometer, largely overcomes the problem of unreliable measurement. But many of the behaviours targeted in neurorehabilitation interventions rely on human judgement and therefore require demonstration of their reliability in terms of agreement between different observers.

In conclusion, assessment is important, because assessment provides the data which in turn provide the evidence. However, there is a need to carefully select instruments and be mindful of their scientific quality. In neurorehabilitation, both measures of outcome and measures that monitor progress are needed. Measurement should not be restricted to evaluating outcomes at the end of treatment; rather it is critically important to use measures during treatment to monitor progress. That way, unsuccessful treatments can be changed, barriers to slow progress can be identified and required dosage can be estimated. And most excitingly, while we hear a lot about evidence-based practice, using data-driven interventions clinicians can be empowered to engage in practice-based evidence, thereby strengthening neurorehabilitation.