Introduction
Cognitive behavioural therapy (CBT) is an empirically based psychological therapy that seeks to reduce psychological distress (Wenzel et al., Reference Wenzel, Dobson and Hays2016) through recognition of connections between thoughts, feelings, and behaviours, alongside strategies to elicit cognitive and behavioural change (Kennerley et al., Reference Kennerley, Kirk and Westbrook2017). CBT has an extensive evidence base, spanning various psychological difficulties and patient groups (for a review, see Fordham et al., Reference Fordham, Sugavanam, Edwards, Stallard, Howard, Das Nair and Lamb2021); however, adaptations are often recommended, including for children, neurodivergent individuals with conditions including attention deficit hyperactivity disorder (ADHD) and autism, and people with intellectual disabilities (e.g. National Institute for Health and Care, 2016; National Institute for Health and Care, 2019; National Institute for Health and Care, 2021).
To engage in CBT, clients are ‘socialised to the model’ – a preparatory phase that enables clients to comprehend and accept the core principles and practice of the therapeutic model, which they can then apply to themselves with support. Socialisation involves receiving information about the rationale for the intervention and ensuing process, alongside developing a shared understanding of the problem with the therapist (Mahoney-Davies et al., Reference Mahoney-Davies, Roberts-Collins, Russell and Loades2017; Roos and Wearden, Reference Roos and Wearden2009). Several factors can significantly influence the success of socialisation, and research has suggested that unsuccessful socialisation to the model may lead to early termination of therapy (Barrett et al., Reference Barrett, Chua, Crits-Christoph, Gibbons and Thompson2008).
One construct that plays a crucial role in the successful implementation of CBT is ‘psychological mindedness’, which broadly involves the ability to understand psychological processes (Beitel et al., Reference Beitel, Ferrer and Cecero2005). Psychological mindedness captures a range of foundational skills for CBT, such as emotional awareness, metacognition, and introspection (Applegate and Shapiro, Reference Applegate and Shapiro2005). Emotional awareness is the central aspect of psychological mindedness, facilitating one’s understanding of one’s feelings and how these feelings affect behaviour (Barrett et al., Reference Barrett, Gross, Christensen and Benvenuto2001). Additionally, metacognition – the ability to think about one’s thinking, being aware, and having a deeper understanding of one’s thinking process – is considered an important factor in psychological therapy (Flavell, Reference Flavell1979). Metacognitive skills can vary between individuals as metacognition focuses on how the individual receives, processes, and reciprocates information (Ayazgök and Yalçin, Reference Ayazgök and Yalçin2014).
Metacognitive skills have been found to improve learning across a broad range of domains (Cromley and Kunze, Reference Cromley and Kunze2020). Research into metacognitive abilities and CBT is limited. Metamemory, which is awareness of the content of memory and use of strategies to manage memory, has been found to be associated with performance on a CBT skills task in younger children (Jones et al., Reference Jones, Souchay, Moulin, Reynolds and Adlam2019). Introspective abilities such as self-reflection and insight have been found to be associated with cognitive flexibility and self-regulation (Grant et al., Reference Grant, Franklin and Langford2002), which may be relevant to understanding and working within a cognitive behavioural framework.
Baseline CBT skills
Developing an understanding of the client’s knowledge of the CBT model and their psychological mindedness may help clinicians to discern when adaptations to CBT are required. Seminal work from Safran et al. (Reference Safran, Segal, Vallis, Shaw and Samstag1993) introduced key criteria for determining a client’s suitability for CBT. Criteria included the ability to access automatic thoughts; identify and distinguish between different emotions; and view the self as active in the change process. Subsequent work has suggested baseline skills for engagement in CBT, involving the client’s capability to distinguish between thoughts, feelings, and behaviours (TFB) and to recognise their connections, including that cognition mediates situations and emotions (Dagnan et al., Reference Dagnan, Chadwick and Proudlove2000; Hatton, Reference Hatton2002; Willner, Reference Willner2005).
For some people, utilising these skills may be difficult. For example, children, adolescents, and individuals with intellectual disabilities may struggle to engage in CBT depending on their developmental level and cognitive, emotional, and verbal abilities (Hatton, Reference Hatton2002; Stallard, Reference Stallard2014). Autistic people may find CBT challenging due to various characteristics associated with autism, co-existing conditions, and neuropsychological differences (Spain and Happé, Reference Spain and Happé2020). For example, some autistic people experience alexithymia, that is, challenges in recognising emotions and distinguishing between them (Kinnaird et al., Reference Kinnaird, Stewart and Tchanturia2020), which may impede discrimination between TFB and identification of connections between them (Anderson and Morris, Reference Anderson and Morris2006; Spain et al., Reference Spain, Sin, Chalder, Murphy and Happé2015). Autism has been associated with executive function differences (Demetriou et al., Reference Demetriou, Lampit, Quintana, Naismith, Song, Pye and Guastella2018) and metacognition is thought to reflect executive function skills. Executive function differences may impact an individual’s mental health recovery through challenges with flexibly generating thoughts, beliefs, and plans necessary for affecting positive change and evaluating the possible utility of different strategies (Anderson and Morris, Reference Anderson and Morris2006). Executive function differences are not specific to autism and are present in other neurodivergent conditions, including ADHD (for a review, see Willcutt et al., Reference Willcutt, Doyle, Nigg, Faraone and Pennington2005). Little is known about whether individuals with ADHD are likely to experience difficulties understanding the concepts required to engage in CBT, but various adaptations to CBT have been reported for this group (Ramsay, Reference Ramsay2010).
Importantly, there is considerable individual variation in cognitive and behavioural patterns, affected by one’s upbringing, culture, experiences, and personality traits (Beck, Reference Beck2020), meaning that baseline CBT skill competencies may vary greatly within the general population.
Measuring baseline CBT skills
Despite the potential clinical utility of measuring constructs relevant to ‘psychological mindedness’, there are a number of important practical and methodological issues. Whilst there are validated measures of abilities such as emotion recognition in the self and others, no measures assessing how these skills are integrated and applied to different practice-based problems exist for the adult population. Semi-structured interviews have been devised to assess such skills (e.g. Safran et al., Reference Safran, Segal, Vallis, Shaw and Samstag1993); however, these are not well suited to routine clinical practice as they may interfere with the process of assessment and formulation at a time of help-seeking, and are time-consuming. Other approaches have included rating therapy tapes to capture socialisation to the model (Daniels and Wearden, Reference Daniels and Wearden2010); however, this method is time-consuming, relatively subjective, and occurs after the fact (i.e. producing data that is useful once the client is some way through the CBT process). Therefore, an objective task is required to assess indices of psychological mindedness directly relevant to CBT, prior to starting any intervention.
Such tasks, hereonin termed ‘CBT skills’ tasks, have been developed for autistic children (Lickel et al., Reference Lickel, MacLean, Blakeley-Smith and Hepburn2012) and people with intellectual disabilities (Dagnan et al., Reference Dagnan, Mellor and Jefferson2009; Oathamshaw and Haddock, Reference Oathamshaw and Haddock2006; Parent et al., Reference Parent, Birtwell, Lambright and DuBard2016; Tsimopoulou et al., Reference Tsimopoulou, Kroese, Unwin, Azmi and Jones2018), including cognitive mediation and TFB discrimination tasks (Lickel et al., Reference Lickel, MacLean, Blakeley-Smith and Hepburn2012). However, there has been limited investigation into the psychometric properties of such measures, and standardised measures for gauging baseline CBT skills are absent from the literature (Morey, Reference Morey2020). Moreover, these tasks have not been adapted and tested for use with an adult population without intellectual disability, highlighting the need for a valid baseline CBT skills task that can be used in clinical practice across a range of client groups who may have neurocognitive differences.
Present research
The present work aimed to draw on paradigms from research with children and adults with intellectual disability to develop a vignette-based task suitable for young adults to assess baseline CBT skills. Following earlier pilot work (unpublished theses: Mahoney-Davies, Reference Mahoney-Davies2016; Roberts-Collins, Reference Roberts-Collins2016), Study 1 sought to investigate the internal consistency and convergent validity of a paper-based, four-vignette task with an adolescent population. Study 2 intended to extend the task to the young adult population, investigating the psychometric properties (internal consistency, convergent validity, and factor structure) of an online, two-vignette task. Young people aged 18–30 participated, including those diagnosed with autism and/or ADHD, as well as those without a neurodevelopmental condition.
Method
Task development
The Cognitive Behavioural Skills Task (CBT-SKI) was designed to assess the ability to: (1) distinguish between thoughts, feelings, and behaviours (TFB); (2) identify connections between them; and (3) identify what needs to change at an individual level to reduce emotional distress/the impact of an emotional problem. These three elements provide information about an individual’s propensity to ‘psychological mindedness’, as well as an indirect index of socialisation to the CBT model.
The task has been developed through multiple rounds of refinement and consists of vignettes of individuals experiencing different mental health problems, followed by a series of questions. The questions have remained consistent throughout each iteration of the task and involve distinguishing TFB, identifying connections between these phenomena, and generating potential avenues to alleviate distress for each character. Scoring responses to the questions has been refined as part of each subsequent study, with the final version detailed in the present materials.
Initial development of a single vignette version of the task (Mahoney-Davies et al., Reference Mahoney-Davies, Roberts-Collins, Russell and Loades2017) was in collaboration with an adolescent participation group and was for use with young people aged 12–17. Recommendations to expand the number of vignettes resulted in a four-vignette (Generalised Anxiety Disorder [GAD], Depression, Social Anxiety, and Behavioural Distress) paper-and-pencil version, which was piloted with adolescents aged 12–16 (Study 1; Morey, Reference Morey2020, unpublished DClinPsy thesis). The vignettes were developed to represent the most commonly occurring mental health problems amongst young people, using compelling language derived from social media sites (unpublished MSc thesis: Allen, Reference Allen2019). The adolescent pilot study included careful development and validation of scoring guidance, with findings suggesting that the behavioural distress vignette had reduced reliability. Following this work, the focus of the research group moved towards exploring the utility of an online version of the task with the young adult population, ensuring that the measure is valid for young adults who have ADHD and are autistic (Study 2).
The most recent and final version of the task used in Study 2 consists of two vignettes, chosen to cover the most common mental health problems: depression and anxiety (see Appendix A in the Supplementary material), with language amended to reflect adult concerns, for example, replacing ‘school’ with ‘workplace’. A social anxiety vignette was retained over a GAD vignette given epidemiological work indicating that social anxiety is more prevalent in both the general adult population and neurodivergent populations (Bandelow and Michaelis, Reference Bandelow and Michaelis2015; Nimmo-Smith et al., Reference Nimmo-Smith, Heuvelman, Dalman, Lundberg, Idring, Carpenter and Rai2020). The structure of the task remained the same across Study 1 and Study 2. The vignettes consist of a short paragraph describing the difficulties currently being experienced by an individual character. Participants were required to identify the character’s TFB and indicate if and how these could be connected. Participants were also asked what would need to change for each vignette character to get better, and to design an activity to help with the mental health problem, explaining how this might help them. The mode of administration differs, with Study 1 administered via paper and pencil and Study 2 administered online.
Study 1: An investigation of the internal consistency and convergent validity of a paper-based, four-vignette CBT skills task with an adolescent population (CBT-SKI-A)
Data for this study were collected in 2020.
Power analysis
A power calculation indicated that a minimum sample size of 77 was required to detect a medium effect (R 2=.13, based on a power of .80). Previous research has stated that a medium effect is clinically relevant (Reynolds et al., Reference Reynolds, Girling, Coker and Eastwood2006).
Participants
Participants (n=104) aged 12–16 years were recruited from a local secondary school, across four year groups. Four pupils declined to participate, leaving a sample of 100 participants. Data were collected in the classroom and no remuneration was offered.
Eligibility criteria required participants to have no diagnosis of autism or ADHD, and no previous or current access to psychological therapy. Any participants who did not meet the criteria were still invited to participate, with their data removed retrospectively, to avoid any distress at being excluded in the classroom.
Following exclusions due to therapy experience (n=21) or diagnoses (n=3 diagnosed with autism; n=2 diagnosed with ADHD), the final sample consisted of 72 participants aged 12–16 (M=13.58; SD=1.17).
Measures
Revised Child and Adolescent Anxiety and Depression Scale (RCADS; Chorpita et al., Reference Chorpita, Yim, Moffitt, Umemoto and Francis2000)
The RCADS is a self-report measure of depression and anxiety that was administered as part of the study protocol but not presented in this paper (DClinPsy thesis: Morey, Reference Morey2020).
Emotional Awareness Questionnaire (EAQ-30; Rieffe et al., Reference Rieffe, Oosterveld, Miers, Terwogt and Ly2008)
The EAQ-30 is a 30-item questionnaire using a 3-point Likert scale (1=not true, 3=often true). There are six subscales, with higher scores in each subscale reflective of better performance. Three subscales (‘analyses of emotions’, ‘attending to other’s emotions’, and ‘differentiating emotions’) were used to assess the convergent validity of the ‘feelings’ item in the CBT-SKI-A. The three subscales were selected as they measure one’s ability to identify precursors to emotions and a readiness to confront one’s own emotions and those of others – skills that are required to answer the feelings question in the CBT-SKI-A. The EAQ-30 has good internal consistency, concurrent validity, and criterion validity (Rieffe et al., Reference Rieffe, Oosterveld, Miers, Terwogt and Ly2008). For the current data, the internal consistency of the ‘differentiating emotions’ subscale was excellent (ordinal α=.90). For the ‘analyses of emotions’ and ‘attending to others’ emotions’ subscales, internal consistency was good (ordinal α=.80 and .88), respectively.
The CBT-SKI Adolescent (CBT-SKI-A)
Four vignettes (Depression, Social Anxiety, GAD, and Behavioural Distress) with accompanying questions were administered via paper and pencil. Each vignette had four questions (see Table 1). The first two questions aimed to assess one’s ability to distinguish TFB and recognise the connections between them, whilst the later questions sought to examine one’s ability to determine what needs to change and make active plans to alleviate distress.
CBT-SKI-A questions and possible score ranges
Table 1. Long description
A table titled ‘CBT-SKI-A questions and possible score ranges’ with four rows and three columns. The columns are labeled ‘Item’, ‘Question’, and ‘Score range’. The rows are as follows: Row 1: Item, 1; Question, What thoughts, feelings, and behaviours might X have?; Score range, 0-9. Row 2: Item, 2; Question, Could X’s thoughts, feelings and behaviours be connected? If you answered yes, show how by drawing arrows above.; Score range, 0-7. Row 3: Item, 3; Question, What needs to change for X to get better?; Score range, 0-4. Row 4: Item, 4; Question, Design an activity to help with X’s distress and tell us why it might help her/him.; Score range, 0-3.
Scoring criteria were developed on the basis of 10% (n=10) of responses. The possible range of scores for each question is shown in Table 1.
Inter-rater reliability
An independent rater scored 10% of the data (40 vignettes), with a total score reliability of 80% for the depression, social anxiety, and GAD vignettes, and 90% for the behavioural vignette.
Procedure
Informed consent and parental consent was obtained. Participants first completed the CBT-SKI-A, before moving on to the RCADS and EAQ-30. For participants who needed additional instructions or who did not wish to participate, support was available. Participation lasted approximately 40–50 minutes. The research was given ethical approval by the Psychology Research Ethics Committee at the University of Bath (reference: 19-026).
Analysis plan
Missing data for the EAQ-30 were handled using case mean substitution. Two participants did not provide data for any items of the EAQ-30 so their data was treated as missing. For the CBT-SKI-A, missing data were replaced with a score of zero as it was assumed that no response reflected a lack of knowledge about the answer.
Data were analysed using IBM SPSS Statistics (version 25) and the SPSS R-Menu, imported into SPSS (Basto and Pereira, Reference Basto and Pereira2012). Internal consistency for the CBT-SKI-A was calculated in RStudio (v2024.4.2.764) using the haven (v2.5.4; Wickham et al., Reference Wickham, Miller and Smith2025), psych (v2.4.12; Revelle, Reference Revelle2024), and polycor (v.0.8-1; Fox, Reference Fox2022) packages. Descriptive statistics were first inspected, before the internal consistency and convergent validity of the CBT-SKI-A were examined. Internal consistency was analysed using Cronbach’s alpha for total scores, and ordinal coefficient alpha for subscale scores. Convergent validity between the total feelings score on the CBT-SKI-A and three EAQ-30 subscales was assessed using Kendall’s tau.
Results
Descriptive statistics
The majority (82%) of participants completed all four vignettes and the response rate across the vignettes appeared consistent (66–68 responses for each vignette). Mean scores across the vignettes were similar (see Table 2). The distribution of scores for individual questions in each vignette were relatively symmetric across the range of possible values, except for the ‘Behavioural Distress’ vignette where the majority of responses to question three (identifying change) were 0.
Descriptive statistics
Table 2. Long description
A table titled ‘Total vignette score’ with five rows and three columns. The columns are labeled ‘Vignette’, ‘Total vignette score (max = 23)’, and ‘Range’. The ‘Total vignette score (max = 23)’ column is further divided into ‘Mean (SD)’. The rows list different conditions: Depression, Social Anxiety, Generalised Anxiety Disorder, Behavioural Distress, and Total CBT-SKI-A score (max = 92). Row 1: Depression, Mean (SD) 9.90 (5.31), Range 0-20. Row 2: Social Anxiety, Mean (SD) 9.13 (4.59), Range 0-17. Row 3: Generalised Anxiety Disorder, Mean (SD) 8.57 (4.70), Range 0-18. Row 4: Behavioural Distress, Mean (SD) 7.56 (4.17), Range 0-17. Row 5: Total CBT-SKI-A score (max = 92), Mean (SD) 35.15 (15.95), Range 4-64.
Internal consistency
Internal consistency for total scores across the vignettes was good (α=.87), and for the ‘TFB identification’ and ‘connecting TFB’ items (ordinal α=.91 and .92, respectively). For the ‘identifying change’ and ‘making change’ items, internal consistency was weaker (ordinal α=.55 and .68, respectively).
Convergent validity
To assess the convergent validity of the total feelings score on the CBT-SKI-A, three subscales of the EAQ-30 were used (‘differentiating emotions’, ‘attending to other’s emotions’, ‘analyses of own emotions’).
Of the three EAQ-30 subscales, ‘analyses of own emotions’ (τb=.18, p=.017) and ‘attending to others’ emotions’ (τb=.14, p=.058) were significantly associated with total CBT-SKI feelings score. The ‘differentiating emotions’ scores were not associated with total feelings score on the CBT-SKI (τb=–.08, p=.165).
Summary
Study 1 provided preliminary evidence about the reliability and convergent validity of a novel task to assess CBT skills in adolescents. Internal consistency for identifying and connecting TFB was good, but task items designed to assess implementation of this knowledge were less reliable. Furthermore, the pattern of responses to the ‘identifying change’ question for the Behavioural Distress vignette was concentrated at the lower end of the scale and deviated significantly from the pattern of responding on the other vignettes.
Study 2: What are the psychometric properties (internal consistency, convergent validity, and factor structure) of an online two-vignette task assessing CBT skills in young adults with and without neurodivergence?
Data for this study were collected in 2023.
Power analysis
For factor analyses, debate exists regarding sample sizes, and adequacy is only ascertainable after data acquisition (Bandalos and Finney, Reference Bandalos, Finney, Hancock and Mueller2010). Therefore, a total a sample size of n=300 was deemed initially suitable, based on guidance from DeVellis (Reference DeVellis2017).
Participants
Participants (n=300) were recruited from Prolific to obtain a sample of 100 participants with a diagnosis of autism, 100 participants with a diagnosis of ADHD, and 100 participants without autism or ADHD. Participants were paid £5.25 for their time.
Prolific enables the filtering of eligible participants based on set criteria. Eligibility criteria required participants from the United Kingdom, aged 18–30.
For the autism group, users who reported a diagnosis of autism as a child or an adult were selected for inclusion. Co-occurring ADHD was accepted to ensure a representative sample, given the high prevalence rate of co-occurring ADHD among autistic people (Rong et al., Reference Rong, Yang, Jin and Wang2021). For the ADHD group, users who considered themselves to have ADHD (but not co-occurring self-identified or diagnosed autism) were selected for inclusion. Filtering out autistic participants was necessary to ensure that the ADHD group was sufficiently distinct from the autism group. For the neurotypical group, users who did not consider themselves to have ADHD or self-identified or diagnosed autism were selected for inclusion.
Measures
Emotional Awareness Questionnaire (EAQ-30; Rieffe et al., Reference Rieffe, Oosterveld, Miers, Terwogt and Ly2008)
For the current data, the internal consistency of the ‘differentiating emotions’ subscale was excellent (ordinal α=.93). For the ‘analyses of emotions’ and ‘attending to others’ emotions’ subscales, internal consistency was good and acceptable (ordinal α=.84 and .78), respectively.
The CBT-SKI
The two-vignette (Depression and Social Anxiety) online version of the CBT-SKI was administered. Each vignette had seven questions (see Table 3), with the wording of the questions identical to that of the CBT-SKI-A, apart from the instructions needed for online administration.
CBT-SKI questions and possible score ranges
Table 3. Long description
A table listing CBT-SKI questions and their score ranges. The table has two columns: Item and Score range. It contains seven rows, each corresponding to a different question or activity related to Jo/Alex’s thoughts, feelings, behaviors, and potential interventions. Row 1: Item, What thoughts does Jo/Alex have? Score range, 0-3. Row 2: Item, What feelings does Jo/Alex have? Score range, 0-3. Row 3: Item, What behaviors does Jo/Alex show? Score range, 0-3. Row 4: Item, Could Jo/Alex’s thoughts, feelings and behaviours be connected? Score range, Not scored. Row 5: Item, Using your cursor, draw lines with arrowheads to show where you think Jo/Alex’s thoughts, feelings, and behaviors might be connected. You can draw as many arrows and arrowheads as you wish. Score range, Scored as three parts, each 0-2. Row 6: Item, What needs to change for Jo/Alex to get better? Score range, 0-4. Row 7: Item, Design an activity to help with Jo/Alex’s distress and tell us why it might help her/him. Score range, 0-3.
After answering the questions for the vignette about Jo, participants then repeated the questions for the vignette about Alex. aMaximum possible scores: 0–22 per vignette; 0–44 total. bQuestion 5 was only displayed to participants if they selected ‘Yes’ to the previous question.
Raters (n=3) were trained using responses from an earlier dataset of the two-vignette version administered online to adults. Calibration across raters on the present data was achieved by independently coding six responses from each group (6% of the total dataset), followed by a discussion to resolve disagreements, which led to scoring criteria refinements.
The raters coded the remaining data independently, with a third of the data from each sample randomly assigned to each rater (E.E., J.K., E.H.) using a random number generator. An independent fourth rater (M.P.) scored 13% of the data with equitable data distribution from the three raters (κ=.71).
Procedure
The study was published on Prolific and interested participants were directed to Qualtrics, where they read an information sheet and gave informed consent to participate. Demographic data was subsequently collected before information was obtained regarding participants’ self-identified and/or diagnosed neurodivergent conditions, alongside any current mental health conditions. Two measures not discussed in this project, the AQ-10 (Allison et al., Reference Allison, Auyeung and Baron-Cohen2012) and the ASRS-V1.1 (Kessler et al., Reference Kessler, Adler, Ames, Demler, Faraone, Hiripi and Walters2005) were subsequently presented, followed by the CBT-SKI. Participants were then asked about their experience of the task and whether they had previously attended or were currently attending CBT. Following the CBT-SKI, participants completed the EAQ-30 alongside additional measures for use in other projects not discussed here. Upon finishing the survey, a debrief form was displayed on screen, and participants were subsequently directed back to Prolific. The research was given ethical approval by the Psychology Research Ethics Committee at the University of Bath (reference: 23-021).
Analysis plan
Data were analysed using IBM SPSS Statistics (version 28). Internal consistency for the CBT-SKI-A was calculated in RStudio (v2024.4.2.764) using the haven (v2.5.4; (Wickham et al., Reference Wickham, Miller and Smith2025), psych (v2.4.12; (Revelle, Reference Revelle2024), and polycor (v.0.8-1; (Fox, Reference Fox2022) packages. For the EAQ-30, 13 participants had missing data for items 10–30 due to a technical issue, so their data for the EAQ-30 were treated as missing. Normality and outlier checks were also omitted, given the ordinal nature of the data and the maximum score range of 0–22 for each vignette. Descriptive statistics are presented in Table 4. Internal consistency of the CBT-SKI was calculated using Cronbach’s alpha for total scores, and ordinal coefficient alpha for subscale scores. Convergent validity between three subscales of the EAQ-30 and the CBT-SKI was calculated using Kendall’s tau. Exploratory factor analysis (EFA) was selected to identify a latent factor structure of the CBT-SKI.
Descriptive statistics
Table 4. Long description
A table titled ‘Descriptive statistics’ with three rows and four columns. The columns are labeled ‘Vignette’, ‘Completed vignettes’, ‘Total CBT-SKI score (max = 44)’, and ‘Range’. The ‘Completed vignettes’ column has the sub-column ‘n (%)’, and the ‘Total CBT-SKI score (max = 44)’ column has the sub-column ‘Mean (SD)’. The ‘Vignette’ row labels are ‘Depression’, ‘Social Anxiety’, and ‘Total vignette score (max = 22)’. Row 1: Depression, 300 (100 percent), 12.05 (4.14), 1-21. Row 2: Social Anxiety, 300 (100 percent), 12.25 (3.81), 2-20. Row 3: Total vignette score (max = 22), 24.30 (7.36), 5-41.
For the EFA, the SPSS R-Menu was imported into SPSS (Basto and Pereira, Reference Basto and Pereira2012), and O’Connor’s (Reference O’Connor2000) syntax was used for parallel analysis calculations. Of the recruited 300 participants, 150 were included in the EFA (50 from each participant group: autism, ADHD and neurotypical). The dataset was first checked for missing values, and the Kaiser-Meyer-Olkin measure was used to assess sampling adequacy (KMO; Kaiser and Rice, Reference Kaiser and Rice1974). Bartlett’s test of sphericity (Bartlett, Reference Bartlett1950) was used to examine the suitability of the data for factor analysis. The EFA used polychoric correlations because the data were ordinal, with fewer than 5–7 categories (Watkins, Reference Watkins2018). A principal axis factoring extraction method was selected because it makes no distributional assumptions and is appropriate for smaller sample sizes (Grieder and Steiner, Reference Grieder and Steiner2022). Oblique rotation (oblimin quartimin) was used, assuming any identified factors would be correlated. Eigenvalues and the scree plot were examined to determine the optimal number of factors to retain. Using Kaiser’s criterion, eigenvalues greater than one were retained (Kaiser, Reference Kaiser1960). The scree plot was inspected for breakpoints before the inflexion (Field, Reference Field2018). Parallel analysis and Velicer’s minimum average partial (MAP) were also calculated as simulation studies have indicated their accuracy (Bandalos and Finney, Reference Bandalos, Finney, Hancock and Mueller2010).
For the CFA, the remaining 150 participants’ data were used. Given the ordinal nature of the data, normality checks were omitted. Outlier checks were omitted, considering the maximum score range of 0–6. Analyses were carried out in RStudio (v2024.4.2.764) (v0.6-16; Rosseel, Reference Rosseel2012) and semTools (v0.5-6; Jorgensen et al., Reference Jorgensen, Pornprasertmanit, Schoemann and Rosseel2022) packages. Correlations were polychoric and the estimation method was weighted least squares with adjustments for mean and variance (WLSMV), as this is appropriate for ordinal data (Brown, Reference Brown2015). Absolute (chi-square [χ2], standardised root mean square residual [SRMR]), parsimony-adjusted (root mean square error of approximation [RMSEA]), and relative (comparative fix index [CFI], Tucker-Lewis index [TLI]) fit indices were used to determine model fit. A non-significant χ2 indicated model fit (Brown, Reference Brown2015). Cut-off criteria for the remaining indices followed Hu and Bentler’s (Reference Hu and Bentler1999) approximate guidelines: ≤.08 for SRMR, ≤.06 for RMSEA, and ≥.95 for TLI and CFI.
Composite reliability was used to assess internal consistency as this measure is thought to be more accurate than Cronbach’s alpha for latent variable modelling.
Results
Descriptive statistics
See Table 4 for a summary of the descriptive statistics.
Internal consistency
Results revealed internal consistency for total scores between the two vignettes was good (α=.83), and for the ‘TFB identification’ and ‘connecting TFB’ items (ordinal α=.79 and .94, respectively). For the ‘identifying change’ and ‘making change’ items, internal consistency was weaker (ordinal α=.57 and .53, respectively).
Convergent validity
To assess the convergent validity of the total feelings score on the CBT-SKI, three subscales of the EAQ-30 were used (‘differentiating emotions’, ‘attending to other’s emotions’, and ‘analyses of own emotions’).
Of the three EAQ-30 subscales, the ‘analyses of own emotions’ (τb=.13, p=.002) and ‘attending to other’s emotions’ (τb=.09, p=.021) subscales were associated with total feelings score. The ‘differentiating emotions’ subscale was not correlated with the total feelings score (τb=.06, p=.095).
Exploratory factor analysis
No data were missing. For participant demographics, see Appendix B in the Supplementary material. The polychoric correlation matrix is presented in Table 5. The mean total CBT-SKI score was 24.72, with a standard deviation of 7.35 (see Appendix C in the Supplementary material for descriptive statistics for each observed variable, hereafter called variables). The KMO test statistic was .76, and Bartlett’s test was significant at p<.001, highlighting the suitability of the data for factor analysis. Moreover, the determinant (0.0001) satisfied the >0.00001 heuristic (Field, Reference Field2018).
Polychoric Correlation Matrix
Table 5. Long description
A table showing polychoric correlation matrix values. The table has 16 rows and 16 columns. The rows are labeled with variables such as Jo Thoughts, Jo Feelings, Jo Behaviours, Jo Thoughts Feelings Arrows, Jo Thoughts Behaviours Arrows, Jo Feelings Behaviours Arrows, Jo Change, Jo Activity Design, Alex Thoughts, Alex Feelings, Alex Behaviours, Alex Thoughts Feelings Arrows, Alex Thoughts Behaviours Arrows, Alex Feelings Behaviours Arrows, Alex Change, and Alex Activity Design. The columns are numbered from 1 to 7. Each cell in the table contains a correlation value.
Correlations above 0.30 are displayed in bold.
Velicer’s MAP test indicated a two-factor model (MAP2=0.032); Kaiser’s criterion and the scree plot indicated a three-factor solution; and parallel analysis indicated four factors to be retained. Therefore, two-, three- and four-factor solutions were sequentially evaluated using EFA, considering variance explained and reliability statistics.
The two-factor solution was first examined, with rotated sum of squared loadings of 4.40 and 3.35, and ordinal coefficient alphas of .88 and .78. However, the activity design variables for both vignettes, and the change variable for the social anxiety vignette did not load onto either factor with absolute values greater than 0.4. Subsequently, the three-factor solution was analysed. This solution had rotated sum of squared loadings of 4.39, 3.20 and 1.92, with ordinal coefficient alphas of .92, .80 and .66. Finally, the four-factor solution was examined, with rotated sum of squared loadings of 4.38, 2.67, 1.95 and 2.35, and coefficient alphas of .92, .78, .66 and .66. However, the feelings variable for the social anxiety vignette did not load onto any factor with an absolute value greater than 0.4.
Given these indicators and the underlying research, the three-factor solution was further analysed.
Relevant metrics were applied to the table of Mundfrom et al. (Reference Mundfrom, Shaw and Ke2005), which indicated a good sample size. Subsequently, pattern and structure coefficients were examined (see Table 6). Pattern coefficients control for other factors, whereas structure coefficients do not (Bandalos and Finney, Reference Bandalos, Finney, Hancock and Mueller2010). All variables had pattern coefficients at 0.5 or greater for their respective factors, with values above 0.5 usually ‘fully satisfactory’ (Morin et al., Reference Morin, Myers, Lee, Tenenbaum, Eklund and Boiangin2020). All structure coefficient values were similarly above 0.5. The factor intercorrelations were low, at .25 for factors 1 and 2; .15 for factors 1 and 3; and .29 for factors 2 and 3. This indicated factors shared some variance, but largely represented unique dimensions of the CBT-SKI.
Pattern and structure coefficients
Table 6. Long description
A table with 14 rows and 6 columns comparing pattern and structure coefficients for various variables. The columns are labeled ‘Variable’, ‘Pattern coefficients F1’, ‘Pattern coefficients F2’, ‘Pattern coefficients F3’, ‘Structure coefficients F1’, ‘Structure coefficients F2’, and ‘Structure coefficients F3’. The rows list different variables and their corresponding coefficients. Row 1: Jo Thoughts Behaviours Arrows, 0.88, 0.84. Row 2: Jo Feeling Behaviours Arrows, 0.88, 0.90. Row 3: Alex Thoughts Behaviours Arrows, 0.85, 0.83. Row 4: Alex Feelings Behaviours Arrows, 0.77, 0.75. Row 5: Jo Thoughts Feelings Arrows, 0.75, 0.81. Row 6: Alex Thoughts Feelings Arrows, 0.73, 0.78, 0.40. Row 7: Jo Behaviours, 0.78, 0.79. Row 8: Alex Behaviours, 0.67, 0.65. Row 9: Alex Thoughts, 0.65, 0.67. Row 10: Jo Thoughts, 0.63, 0.66. Row 11: Jo Feelings, 0.55, 0.59. Row 12: Alex Feelings, 0.47, 0.47, 0.62. Row 13: Alex Activity Design, 0.63, 0.62. Row 14: Jo Change, 0.63, 0.66. Row 15: Jo Activity Design, 0.50, 0.49. Row 16: Alex Change, 0.50, 0.52. The table provides a detailed comparison of the coefficients for each variable across different factors.
Variables are sorted by pattern coefficient in descending order. Values <0.4 are omitted. Bold values signify those identified by the factor analysis as contributors to each factor.
Using the theoretical framework underpinning the research, the first factor was named ‘TFB Connection’ because the factor captured information about participants’ abilities to recognise connections between TFB. The second factor contained information about identifying TFB and was therefore named ‘TFB Identification’. The third factor was named ‘Applying CBT Skills’ because it contained information about participants’ abilities to use CBT skills to affect change.
Confirmatory factor analysis
No data were missing. For participant demographics, see Appendix D in the Supplementary material. The mean total CBT-SKI score was 23.93, with a standard deviation of 7.39 (see Appendix E in the Supplementary material for descriptive statistics for each variable).
The path diagram for the confirmatory factor analysis is shown in Figure 1. Model fit indices indicated an acceptable fit. Although the chi-square value was significant, χ2(101)=98.62, p=.015, the remaining indices satisfied cut-off criteria (CFI=.99, TLI=.99, RMSEA=.05 (90% CI [.02, .07]), SRMR=.07). See Table 7 for item factor loadings. Factor covariances were 0.21, 0.14 and 0.21. Composite reliability was calculated at .75 for TFB identification, .92 for TFB Connection, and .58 for Applying CBT Skills.
A path diagram depicting the factor structure of the CBT-SKI. Ovals represent latent factors, rectangles represent observed variables, and circles represent error terms. Latent factors were allowed to correlate, but error terms were not.
Factor loadings
Table 7. Long description
A table with three columns and fourteen rows. The columns are labeled TFB Identification, TFB Connection, and Applying CBT Skills. Each column is further divided into Unstandardised and Standardised sub-columns. The rows are labeled with different variables such as Jo Behaviours, Alex Behaviours, Alex Thoughts, Jo Thoughts, Jo Feelings, Alex Feelings, Alex Feelings Behaviours Arrows, Jo Feelings Behaviours Arrows, Alex Thoughts Behaviours Arrows, Jo Thoughts Behaviours Arrows, Alex Thoughts Feelings Arrows, Jo Thoughts Feelings Arrows, Alex Activity Design, Alex Change, Jo Change, and Jo Activity Design. Each cell contains numerical values representing factor loadings. The table provides a detailed comparison of these loadings across different variables and sub-columns.
a The factor was scaled by setting the variable loading to 1.00.
Discussion
We describe the development of a novel task to assess baseline CBT skills and report on preliminary investigations of task reliability and validity. Our aim was to develop a task that could have clinical and research utility with ecological validity. In particular, we wanted a task to support clinicians and patients to identify when adaptations to CBT protocols in the form of early, foundational work might be of benefit to aid socialisation to the model.
We present the findings of two studies, involving adolescents and young adults. The results of these studies indicate that task items relating to identifying and making connections between TFB are consistently reliable. Reliability for task items relating to applying CBT skills was less convincing, indicating that further refinement may be required. However, it is possible that the lower reliability for task items related to applying CBT skills may be simply explained by there being fewer items with which to capture such information. We excluded participants with prior experience of CBT. It is also possible that in the absence of formal socialisation to the CBT model, there may be differences within individuals about intuitive ideas about making change when someone is depressed versus socially anxious.
Task items about identifying the feelings of characters in the vignettes converged well with a validated self-report measure of analysing own emotions for adolescents and adults. These findings improve the potential utility of the task in clinic settings. Awareness of own emotions, an important facet of psychological mindedness and highly implicated in CBT, is a difference reported in approximately 50% of autistic young people and adults (for a review, see Huggins et al., Reference Huggins, Donnan, Cameron and Williams2021). A tool to assess this rapidly at the start point of a talking therapy such as CBT, without the need to administer more lengthy self-report questionnaires, is highly attractive.
These items also converged well with a measure of attending to other’s emotions, although they did not converge with a measure of differentiating emotions. This may be because the feelings questions in the task related to other people and just two clinical presentations, whereas the comparative measure concerned the differentiation of one’s own emotions more generally – two abilities that may utilise distinct psychological skills.
The exploratory factor analysis identified a three-factor structure. Each factor was thought to reflect a distinct dimension for determining a client’s affiliation with the principles of CBT. The first factor captured the ability to distinguish between thoughts, feelings, and behaviours; the second, the connections between them; and the third, abilities to use CBT skills to affect change. Each factor was loaded by more than three variables, satisfying the minimum recommended by (Yong and Pearce (Reference Yong and Pearce2013). The third factor had lower internal consistency than the first two, which is perhaps consistent with the other issues identified in the applying and making change variables. Confirmatory factor analysis supported the three-factor structure, although reliability was similarly highlighted as an issue for the third factor. This indicates that the items in this factor may not be reliably measuring the same underlying construct, precluding clinicians from using this factor to gauge a client’s baseline ability to identify what needs to change at an individual level to reduce emotional distress. Given that this is one of the aims of the tool, further refinement to the factor structure may be required, as the tool currently cannot be used to reliably inform clinicians about this skill at baseline. However, it is also possible that the lower internal consistency of the third factor may be the result of the factor being loaded by only four variables. Additional variables may provide more information about the underlying construct and accordingly improve the robustness of the factor. It is possible that this knowledge and ability in itself is multi-factorial, with differences between an understanding of the mediating role of behaviour distinct from an understanding about the mediating role of cognitions. Expanding the items tapping this ability and further analysis may support development of this area of the measure.
It is important to consider the notion that the identified factor structure could be an artefact, with each factor reflecting variables with a similar question format. Variables loading onto the first factor had question types that required participants to identify information within the text. Those loading onto the second factor required participants to draw arrows on a diagram, and those loading onto the third utilised free text responses.
Teasing these two explanations apart is difficult; however, the identified factor structure does align with the psychological theory underpinning the work. Therefore, it is possible that the factor structure is reflective of the theoretical constructs it is purported to capture, not simply the different question styles.
Strengths
This work is the first to present a task-based baseline CBT skills measure and examine its psychometric properties with adolescent and adult populations without intellectual disability. This measure could have clinical utility, supporting clinicians and individuals accessing therapy to rapidly assess baseline CBT skills using a standardised measure.
Limitations and future directions
Whilst this paper assesses the psychometric properties of the CBT skills task among adolescents and adults, it is noteworthy that the format of the task was different for these populations. Task development was iterative, such that the adolescent version was paper-based and featured four vignettes, whereas the refined version administered to young adults was delivered online and included only two vignettes. These differences, along with additional refinements to the scoring of the task, may have led to descriptive differences in performance on the task between adolescents and young adults. However, these differences may simply reflect the developmental stage of participants. Nonetheless, future work should seek to validate the CBT skills task among adolescents and adults with the same scoring criteria, scored by the same team.
Additionally, the adult participants were aged 18–30, and it is unclear whether the same findings would apply across a wider age range. Future studies should include a more diverse sample in terms of age. The ethnic composition of the sample was broadly matched to census data, which may enhance the representativeness of the findings; however, it is of note that the vignettes in the CBT Skills Task may not reflect depression and social anxiety symptoms in all cultures. Future work could include formally evaluating the vignettes for their cultural relevance and, where indicated, producing culturally adapted versions of the task, drawing on culturally adapted frameworks for CBT (e.g. Naeem et al., Reference Naeem, Sajid, Naz and Phiri2023).
Although the scoring guidance has been highly developed, it requires orientation and time on the part of the rater, which may reduce clinical utility. Moreover, there may be nuances in patient responses that are not captured in the scoring manual, creating uncertainty for the clinician and potentially introducing biases into scoring. Further development of the scoring criteria to ensure scoring consistency and practicability should be considered.
Conclusion
This paper presented the development and validation of the CBT-SKI, a tool designed to assess baseline CBT skills in young adults. An adolescent version of the task was also subject to preliminary evaluation. The evidence presented indicates that the tool has promise, although further refinements may be required, as items related to applying CBT skills were less consistent. Future studies are required to establish whether additional items would improve reliability, or whether the structure may not fully capture the construct. If used in a clinical context, clinicians should take care to use the results in the context of an individualised formulation, with an understanding of the individual’s overall cognitive profile linked to baseline CBT skills.
Key practice points
-
(1) Developing an understanding of the client’s baseline knowledge of the CBT model and their psychological mindedness may help clinicians to discern when adaptations to CBT are required.
-
(2) A standardised task-based measure of baseline CBT skills could support clinicians to efficiently identify adaptation needs prior to treatment.
-
(3) The CBT Skills Task is a novel, vignette-based tool, with preliminary evidence of reliability and validity, although further refinements are required.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1754470X2610083X
Data availability statement
We did not gain consent from participants to share their data in a public repository, so the datasets for the studies are not available.
Acknowledgements
Many thanks to the Mathematics Resource Centre (MASH) team, Emily Taylor, and Nathan Taylor for their support with study design and implementation, and to Maria Loades, Cara Roberts-Collins, and Gerwyn Mahoney-Davies for their earlier work, which started the process.
Author contributions
Emily Elson: Conceptualization-Supporting, Data curation-Equal, Formal analysis-Lead, Investigation-Equal, Methodology-Equal, Project administration-Equal, Writing - original draft-Lead, Writing - review & editing-Lead; Alice Morey: Conceptualization-Supporting, Data curation-Equal, Formal analysis-Supporting, Investigation-Supporting, Methodology-Equal, Project administration-Supporting, Writing - original draft-Supporting; Jasmine Hiu Tung Kwong: Conceptualization-Supporting, Data curation-Supporting, Investigation-Equal, Methodology-Supporting, Writing - original draft-Supporting, Writing - review & editing-Supporting; Luiza Walo: Methodology-Supporting, Writing - review & editing-Supporting; Megan Payne: Conceptualization-Supporting, Investigation-Supporting, Writing - review & editing-Supporting; Ethan Hare: Conceptualization-Supporting, Data curation-Supporting, Investigation-Supporting, Writing - review & editing-Supporting; Kate Cooper: Conceptualization-Lead, Formal analysis-Supporting, Investigation-Supporting, Methodology-Supporting, Project administration-Equal, Supervision-Lead, Writing - review & editing-Equal; Ailsa Russell: Conceptualization-Lead, Formal analysis-Supporting, Investigation-Supporting, Methodology-Supporting, Project administration-Equal, Supervision-Lead, Writing - review & editing-Equal.
Financial support
This work was funded in part by Avon & Wiltshire Mental Health Partnership NHS Trust Research Capability Funding (2023-24-017). K.C. was supported by the National Institute for Health Research fellowship award (NIHR303189).
Competing interests
The authors declare none.
Ethical standards
Throughout the research, the authors have abided by the Ethical Principles of Psychologists and Code of Conduct as set out by the BABCP and BPS. Both studies received ethical approval by the Psychology Research Ethics Committee at the University of Bath (Study 1: 19-026; Study 2: 23-021). Informed consent was obtained from participants in both studies.
Open access
Comments
No Comments have been published for this article.