Psycho-educational assessment is often complex and challenging. The process generally involves gathering information from multiple sources such as parent interviews and previous records; conducting observations within relevant settings; administering appropriate standardised tests; gathering data with the use of informal measures; and integrating the data in psychometrically robust and clinically meaningful ways. As Sattler (Reference Sattler2008, p. 5) has eloquently stated, data from complementary sources must be ‘woven together so that the final tapestry is integrated, understandable, meaningful and consistent’.
At times though, it can be difficult to integrate assessment data because of low levels of cross-informant agreement, discrepancies across measures, or inconsistencies between test results and clinical observations. Relatively poor concordance among mothers, fathers and teachers is often reported for ratings of children's internalising and externalising behaviours (e.g., Gross, Fogg, Garvey, & Julion, Reference Gross, Fogg, Garvey and Julion2004; Sollie, Larsson, & Morch, 2013; Stone, Speltz, Collett, & Werler, Reference Stone, Speltz, Collett and Werler2013). Although response bias can account for some inconsistencies across informants, discrepancies may also reflect the situational specificity of some behaviours, thus providing valuable information (Edwards, Reference Edwards2005).
Less well documented in the literature is the issue about how to reconcile test scores that are inconsistent. It is reasonable to expect that two tests that purport to measure similar constructs, such as general intelligence, will produce similar results. Recent research has shown, however, that we should be cautious about the presumption of interchangeability. Grondhuis and Mulick (Reference Grondhuis and Mulick2013) compared test results for children (aged 3–12 years) with Autistic Disorder and Pervasive Developmental Disorder — not otherwise specified, using the Leiter International Performance Scale-Revised (Leiter-R; Roid & Miller, Reference Roid and Miller1997) a non-verbal measure of intellectual ability, and the Stanford-Binet Intelligence Scales, Fifth Edition (SB5; Roid, Reference Roid2003). Leiter-R scores were significantly higher, and younger children had the largest score differences. These results suggest that children with disorders that affect communication, such as autism spectrum disorder (ASD), may be disadvantaged on intelligence tests like the Wechsler and Stanford Binet.
In samples of typically developing children and adolescents, two studies have demonstrated inconsistencies in scores between the Wechsler Scales and the Stanford Binet. In a study of 36 4-year-old children, Garred and Gilmore (Reference Garred and Gilmore2009) found that, despite moderate to strong correlations among Wechsler Preschool and Primary Scale of Intelligence — Third Edition (WPPSI-III; Wechsler, Reference Wechsler2004) and SB5 composite scores, a notable proportion of the sample achieved substantially different scores on the two instruments. More than 20% had differences of 10 or more points on the two Full Scale IQs and Verbal IQs, and almost half displayed a discrepancy of 10+ points between the non-verbal scales. The discrepancies were as great as 22 points on the two verbal composites and 35 on the non-verbal scales, but there were no identifiable patterns with respect to the higher scoring instrument, or any specific characteristics of individual children or the testing environment.
Although it might be reasonable to assume that test performance is more variable in younger children, a subsequent comparative study of the Wechsler Intelligence Scale for Children — Fourth Edition (WISC-IV; Wechsler, Reference Wechsler2005) and SB5 with adolescents showed even greater score discrepancies. Wilson and Gilmore (Reference Wilson and Gilmore2012) reported significant Full Scale and Verbal IQ differences in a sample of 30 12- to 14-year-old students, with WISC-IV scores being higher. Examination of individual scores showed that half of the students would be placed in different descriptive ranges on the basis of their WISC-IV and SB5 results. The most extreme differences were seen for three students whose results showed a differences of two descriptive categories (Average on the SB5 but Superior on the WISC-IV; High Average on the SB5 and Very Superior on the WISC-IV).
In combination, these results suggest that it cannot be presumed that different intelligence tests necessarily provide similar results and are thus interchangeable. In addition, the possibility that retesting even with the same instrument may produce markedly different scores for a small number of children cannot be discounted. The results of these studies reinforce the potential dangers of making important decisions on the basis of a single test that is administered by one person on a particular day.
Inconsistencies between test results and clinical observations present another potential dilemma for psychologists. Considered to be an important part of the assessment process (Sattler, Reference Sattler2008), observations range from those that are structured with the use of established rating scales (e.g., in classrooms), to those that are more informal and less structured, such as the observed behaviours that are routinely noted during administration of standardised tests. Observations provide potentially important insights that may have direct relevance for interpretation of test performance, as well as clues about aspects of functioning that need to be followed up in further testing, and pointers to guide future interventions. However, observations may create confusion if they are inconsistent with the results of standardised testing, and clinical judgment may be required to make sense of discrepancies.
In this article, we present the case of a young boy who was referred for psycho-educational assessment. We describe the unfolding of this case through standardised testing and clinical observations, and the way in which we collaborated to make sense of the various inconsistencies that emerged.
Case Report
Developmental History and Previous Assessment
Steven was born at 37 weeks gestation, following an uncomplicated pregnancy. Developmental milestones were achieved at typical ages, and there were no concerns during the early childhood years apart from reflux and sleep problems in infancy. Steven experienced no significant childhood events such as injuries, major illnesses or seizures, and the family history was unremarkable.
In the first year of formal schooling, Steven's teacher commented on his difficulties with writing (especially initiation) and speaking in front of the class. He was referred to the school counsellor who administered the WISC-IV. The results indicated verbal (VCI) and non-verbal (PRI) skills in the very superior and superior ranges, respectively. Steven's working memory (WMI) and processing speed (PSI) were average for his age. The counsellor concluded that Steven's difficulties were likely the result of performance anxiety, and suggested that a gifted and talented program might provide him with an appropriate level of challenge.
Over the following 2 years, Steven saw an optometrist, who identified problems with eye convergence and prescribed glasses for close work. An occupational therapist noted difficulties with visual motor integration, handwriting and body awareness. Following a 10-week block of occupational therapy, little improvement was demonstrated on reassessment.
Psycho-Educational Assessment
At the time of referral to the first author, Steven was 8 years old and in the third year of primary school. He was experiencing considerable difficulties with writing, and his mother reported that he did not always speak coherently and fluently. Steven had recently changed schools following his parents’ separation and relocation to another city with his mother and sibling.
Initial hypotheses and observations
Previous assessment had shown that Steven was an intellectually gifted child. The hypothesis at this point was that Steven, as an intellectually gifted child, may have been struggling with writing because of perfectionistic tendencies and associated performance anxiety.
In order to understand Steven's profile of academic strengths and weaknesses, the Wechsler Individual Achievement Test — Second Edition (WIAT-II; Wechsler, Reference Wechsler2002) was administered. During the testing session, Steven's behaviour varied according to the type of task he was asked to complete. He was enthusiastic and interested in many of the activities, particularly reading, which he undertook very competently. However, he was very much less engaged with writing tasks. He fidgeted and squirmed, talked about unrelated topics, or simply did not respond at all. At times, it was clear that he was becoming upset by the requirement to write, although he tried to hide his distress. Nevertheless, Steven was never openly resistant or uncooperative. He interacted comfortably, responding appropriately and initiating conversation about a range of topics. His conversation was sometimes confusing though, and he had a habit of starting and restarting a conversation many times, sometimes stopping mid-sentence for a silent interval that lasted from several seconds to a couple of minutes. Generally, his attention was good and he was not distracted by occasional external noises.
Academic achievement
On the WIAT-II, Steven's ability to read individual words was average (73rd percentile) and his reading comprehension was well above average, falling just into the superior range at the 91st percentile. His performance in the mathematics area was average. Not surprisingly, given the referral concerns, Steven's writing was in the low average range for his grade level. He wrote very slowly and reluctantly, making numerous errors involving omissions of capital letters, missing words, poorly formed letters and spelling mistakes.
In the Oral Expression subtest, Steven was asked to tell stories using picture prompts, and to provide instructions on how to perform two common tasks. He appeared to have considerable difficulty with these activities. His descriptions were hesitant and disjointed. At times he stopped and started again from the beginning, becoming more confused as he progressed. His result for this subtest was in the low average range, at the 12th percentile.
The WIAT-II assessment confirmed the difficulties with writing that had been noted by Steven's mother and teachers, in marked contrast to his stronger reading skills. The problems with oral expression were consistent with his mother's reports, and warranted referral to a speech and language pathologist for further evaluation. Although Steven's reading skills were quite strong, his academic profile was not entirely consistent with intellectual giftedness. There were no indicators of a phonologically based specific learning disability on the WIAT-II or on a specific measure of phonological skills (the Queensland University Inventory of Literacy; QUIL; Dodd, Holm, Oerlemans, & McCormack, Reference Dodd, Holm, Oerlemans and McCormack1996).
It was decided to do further, more targeted assessment of writing and, in addition, to administer the SB5 to confirm conclusions from the previous WISC-IV assessment. At the same time, referral for speech and language assessment was made to the second author.
Writing skills
Several subtests from the Test of Written Language — Third Edition (TOWL-3; Hammill & Larsen, Reference Hammill and Larsen1996) provided standard scores ranging from 11 for Vocabulary to 13 for Logical Sentences. Despite achieving results that were average or above average, Steven's behaviours during testing were of note: he worked very slowly, fidgeting and stopping frequently to sigh or chat about unrelated topics. He had numerous delay tactics and often needed prompting to continue. When asked to write a story using a picture cue, he picked up his pencil but did not write. After a few minutes, when asked ‘Are you thinking of ideas, Steven?’ he replied ‘Maybe’, and then managed to write a few words. After that, he insisted on drawing a diagram instead of writing any more.
Intellectual ability
The SB5 provided lower results than those obtained on the previous WISC-IV assessment. Steven's overall level of ability was within the average range (FSIQ = 108; NVIQ = 103; VIQ = 112). Composite scores on the five indexes ranged from the 58th to 84th percentiles, with his highest result being in the area of working memory.
Speech and language
On the Clinical Evaluation of Language Fundamentals — Fourth Edition (CELF-4; Semel, Wiig, & Secord, Reference Semel, Wiig and Secord2003), Steven's scores on overall language ability, as well as receptive, expressive, content and structure of language, were all in the average to above average range. On the Recall of Narrative Index (ERRNI; Bishop, Reference Bishop2004), a measure of expressive language and story comprehension, his results were similarly average to above average.
Despite these results, when Steven had no external structure for expressive language (i.e., target words/pictures on the CELF-4 or picture sequences on the ERRNI), he displayed some significant difficulties. He started a story but lost the thread of his narrative so that he ended up talking around the main ideas. His difficulties appeared to be in two main areas: planning/organisation and word retrieval. When telling a story, he began with the detail, rather than with the main ideas. Although his vocabulary was assessed as being more than adequate for his age, he had great difficulty retrieving words during spontaneous verbal expression, frequently pausing for 5 or 6 seconds when relating a story.
Additional testing
Given Steven's apparent difficulties with initiation, planning and organisation when he was writing and speaking without the benefit of substantial external structure, we decided an investigation of executive functioning was warranted. Results from the Behavior Rating Inventory of Executive Function (BRIEF; Gioia, Isquith, Guy, & Kenworthy, Reference Gioia, Isquith, Guy and Kenworthy2000), completed independently by Steven's mother and teacher, indicated clinically significant difficulties with many executive functions, including initiation, working memory, organisation and behaviour regulation.
Neuropsychological testing by the third author indicated that Steven had significant difficulties with visuomotor precision (assessed on the NEPSY; Korkman, Kirk, & Kemp, Reference Korkman, Kirk and Kemp1998) and below average visual-spatial ability on the Rey-Osterrieth Complex Figure Test (Meyers & Meyers, Reference Meyers and Meyers1995). However, verbal fluency, assessed on the Delis-Kaplan Executive Function System (D-KEFS; Delis, Kaplan, & Kramer, Reference Delis, Kaplan and Kramer2001), was average to above average.
Dilemmas and Decisions
Our first dilemma was how to reconcile the discrepancies between the WISC-IV and SB5 results. Careful checking of the SB5 protocol revealed no examiner errors, but it was not possible to access the WISC-IV protocol. We noted that Cancellation had been administered and used as a substitute for Coding, on which Steven had obtained a very low score; however, we could find no explanation to justify the substitution. Data from interviews, observations and informal measures appeared more consistent with the SB5 findings, and we tentatively placed more reliance on those results. At the same time, we were aware of potential bias involved in discounting assessment data from other sources that was inconsistent with our own findings (Bell & Mellor, Reference Bell and Mellor2009; Edwards, Reference Edwards2005).
More than 2 years after the original WISC-IV administration, Steven's intellectual ability was reassessed by the third author. Interestingly, two of his WISC-IV composite scores were identical (Perceptual Reasoning = 123) or very similar (Working Memory = 110 and 107) at the two time points, but the Verbal Comprehension standard score dropped from 138 to 104, and a composite for Processing Speed could not be calculated because the two subtest results were very discrepant (Coding = 6, Symbol Search = 14). Steven had obtained a subtest score of 5 on Coding at the earlier assessment but, as mentioned above, the psychologist substituted Cancellation for Coding when calculating Processing Speed at that time.
The difference of 34 points in the VCI at the two administrations was perplexing. It is possible that Steven was on a slightly different developmental trajectory from the majority of his peers, with more advanced verbal abilities at age 6 that had levelled out by age 8. Alternatively, the score discrepancy may reflect some loss of previous skills or the increasing impact of subtle but pervasive difficulties in executive functioning, language and motor planning, and the emergence of higher level deficits that were not fully expressed at the younger age. It is also possible that the drop in VCI was related, at least in part, to Steven's self-awareness of his difficulties with speech and language over the previous 2 years. It would not be surprising if an accumulation of experiences of failure, negative evaluation and performance anxiety had reduced his confidence with spoken language and had led him to produce less adequate verbal responses at retesting.
Our second dilemma was how to resolve discrepancies between test results and clinical observations. We noted that the speech and language tests consistently provided a context or framework for verbal responses, and it was when this structure was not provided that we observed Steven's struggles. In spontaneous speech, there is no pre-existing framework and so, for individuals with planning weaknesses, the flow of ideas is harder to initiate and sustain. We concluded that, in this case, the observational data provided important insights into Steven's verbal difficulties that were not apparent on the standardised measures, with the exception of WIAT-II Oral Expression.
Our third dilemma involved decisions around diagnosis. A number of different diagnoses could potentially have been made on the basis of Steven's profile of strengths and weaknesses, including dysgraphia (a disorder of written expression), dyspraxia (motor planning disorder), and executive function disorder. No single label seemed sufficient, and we believed it was more appropriate at the time to describe the areas of weakness that were contributing to his difficulties with written and oral expression so that intervention could target those aspects, specifically:
1. planning/organisation of language and word retrieval;
2. fine motor and visual motor planning skills; and
3. executive functioning skills of initiation, working memory, planning and organisation.
We were particularly concerned about Steven's social-emotional functioning, given the distress he displayed during written tasks, and recommended support in this area as well. Monitoring his ongoing progress and response to intervention were considered essential, with further neuropsychological investigation of his executive function weaknesses to be considered in the future.
Discussion
Steven's case illustrates a number of challenging aspects of the process of psycho-educational assessment. First, it provides an example of the way in which a case unfolds as further psycho-educational assessment data become available, with the consequent need to revise hypotheses. It was initially presumed that Steven was an intellectually gifted boy whose writing was being affected by perfectionistic tendencies, often reported to be characteristics of giftedness (Maksic & Iwasaki, Reference Maksic and Iwasaki2009; Roedel, Reference Roedel1984; Schuler, Reference Schuler, Neihart, Reis, Robinson and Moon2002). Writing is a complex skill that takes time to develop (Adi-Japha et al., Reference Adi-Japha, Landau, Frenkel, Teicher, Gross-Tzur and Shalev2007), so it would not be surprising if writing skills lagged behind the development of other academic skills for gifted children in the early primary years. However, assessment highlighted Steven's difficulties with speech and language, motor skills and executive functioning, and subsequent intellectual assessment suggested ability within the average, rather than superior, range.
Of particular interest, this case shows the dilemmas that arise when the results of testing are inconsistent at two different time points, or when results are inconsistent with clinical observations. Reconciling these differences is challenging, but it is clearly essential for psychologists to attempt to provide possible explanations for discrepancies. The most obvious possibility is examiner error. If this is ruled out, child factors (e.g., fatigue, illness) that could account for better or worse performance on a particular day need to be explored. Ultimately though, there are times when no reasonable explanation can be found for variable levels of performance on different days (Garred & Gilmore, Reference Garred and Gilmore2009; Wilson & Gilmore, Reference Wilson and Gilmore2012).
Clearly, practitioners need to be cautious about making important decisions on the basis of test scores obtained from a single assessment. In the absence of literature to guide practitioners in resolving such dilemmas, best practice involves drawing together other sources of information, searching for possible explanations for discrepant results, and using clinical judgment to decide on the most appropriate interpretations. As Edwards (Reference Edwards2005, p. 445) has argued, the way in which we ‘integrate discrepant data will ultimately affect the accuracy of our clinical decision making’. Given the likelihood that clinical judgment is potentially influenced by confirmatory biases (i.e., the tendency to seek and use information that supports one's own hypotheses, and to minimise or discount data that fail to support, or disconfirm, hypotheses; Bell & Mellor, Reference Bell and Mellor2009), a high level of reflective practice is essential, with peer supervision or mentoring likely to be of value in this process.
The discrepancies between test scores and clinical observations highlight not only the limitations of standardised measures, but also the value of observations. Standardised tests may not always be tapping into a problem, so the finding of average to above average results does not necessarily mean that no problem exists. It is essential for practitioners to understand exactly what is actually being measured in each test they use. Gathering good observational data, both during formal testing sessions and in less formal contexts, is crucial. Although observational ratings seem to be used mainly to supplement standardised testing through descriptions of test behaviours, Steven's case illustrates the way in which careful observations can provide important insights that may inform diagnosis and intervention.
Finally, Steven's case provides an example of the value of collaboration among colleagues, each of whom can contribute a slightly different perspective or set of skills that, in combination, assist to better understand a child's difficulties. There is growing recognition of the importance of collaborative approaches across educational psychology, neuropsychology and speech-language pathology (Dunsmuir, Clifford, & Took, Reference Dunsmuir, Clifford and Took2006; Law & Durkin, Reference Law and Durkin2000; McConnellogue, Reference McConnellogue2011; Sander, Raymer, Wertheimer, & Paul, 2009). Finello (Reference Finello2011) has pointed out the particular value of cross-discipline collaborations in psycho-educational assessment and diagnosis. While acknowledging that such relationships require time, coordination, and a belief in the benefits of a team approach, she argues that an ‘interweaving of expertise’ (p. 444) has the potential to enrich the school psychologist's understanding of a child.
Effective collaborations between school psychologists, neuropsychologists and speech and language pathologists are facilitated by shared professional practices and frameworks, such as hypothesis testing and data collection across multiple sources and methods (Dunsmuir et al., Reference Dunsmuir, Clifford and Took2006). More commonly, however, these professionals work separately with a child, usually exchanging reports, but not weaving together the evidence from their respective assessments, resolving contradictions, and reaching joint decisions on the basis of all the available data. In adopting a transdisciplinary model, we were able to provide a more holistic and integrated assessment of Steven.
