Introduction
Changing the format of an assessment leads to major changes in the experiences of the learners and may change what is being assessed. Whilst the learners are the major stakeholders in any assessment, sometimes their views are considered last despite being an important part of learning/assessment. COVID-19 pandemic has caused a pivot to online clinical education and training (Choi et al. Reference Choi, Jegatheeswaran, Minocha, Alhilani, Nakhoul and Mutengesa2020), shifting examinations to an online format in line with social distancing requirements (Exams Royal College of Psychiatrists, 2020; Objective Structured Clinical Examination, 2020). The College of Psychiatrists of Ireland (CPsychI) also changed its sitting of postgraduate clinical professional examination called Basic Specialist Training (BST) examination to online in September and November 2020. Effective doctor–patient communication is a core competency for healthcare professionals and shifting to an online format can influence assessment. Our research question aims to explore the perspectives and opinion of key stakeholders i.e., the trainees and the examiners on the assessment of communication skills during online examinations in psychiatry.
Objective Structured Clinical Examination (OSCE) has been widely adopted as a means of assessing communication skills of students and trainees in health profession fields for both formative and summative assessments (McKnight et al. Reference McKnight, Rideout, Brown, Ciliska, Patton, Rankin and Woodward1987; Edwards & Martin, Reference Edwards and Martin1989; Townsend et al. Reference Townsend, McLlvenny, Miller and Dunn2001; O’Sullivan et al. Reference O’Sullivan, Chao, Russell, Levine and Fabiny2008; Talwalkar et al. Reference Talwalkar, Murtha, Prozora, Fortin, Morrison and Ellman2020). It is a structured and objective approach to overcome the issues of reliability (Harden et al. Reference Harden, Stevenson, Downie and Wilson1975; Harden & Gleeson, Reference Harden and Gleeson1979). There are reports in the literature of adoption of OSCE in psychiatry examination in different parts of the world (Kamatchi et al. Reference Kamatchi, Bandyopadhyay, Jainer, Somashekar, Marzanski and Marwaha2012; Hodges et al. Reference Hodges, Hollenberg, McNaughton, Hanson and Regehr2014). The Royal College of Psychiatrists (RCPsych) adopted OSCE in 2003 in the membership examination which was subsequently modified into the Clinical Assessment of Skills and Competence (CASC) in 2008. However, a critical appraisal showed that the CASC prompted the trainees to practice skills that could be accomplished in 10 minutes rather than focusing on developing interview skills, clinical formulation and management that was expected in a long case traditionally (Marwaha, Reference Marwaha2011). This issue was addressed by the CPsychI in 2014 when they devised their BST Clinical Examination (2020) which ensures a national standard in the assessment of clinical skills and competencies for psychiatric trainees. They used a long case for Clinical Formulation and Management Examination (CFME) along with OSCE to assess the judgement and reasoning skills of a senior psychiatric clinician. A brief overview of BST examination is given in Appendix 1.
Communication skills for healthcare professionals are an important tool that involve an interplay of verbal and nonverbal cues and are a first step towards developing a good therapeutic alliance (Thomson, Reference Thomson1992; van der Vleuten et al. Reference van der Vleuten, van den Eertwegh and Giroldi2019). Their assessment can be challenging as the score can vary depending on the stringency or leniency of the examiners (Harasym et al. Reference Harasym, Woloschuk and Cunning2008). Examiners tend to focus on the communication skills and professional behavior at the beginning of the OSCE station, before moving on to other skills i.e., history taking, examination, and management (Chahine et al. Reference Chahine, Holmes and Kowalewski2016). Several studies (Roberts et al. Reference Roberts, Wass, Jones, Sarangi and Gillett2003; Wilby et al. Reference Wilby, Govaerts, Austin and Dolmans2019a , Reference Wilby, Govaerts, Dolmans, Austin and van der Vleuten2019b ) considered patient-centeredness and attentive responding as the prime construct for good communication that encompassed verbal and non-verbal behaviors.
With the transition to an online examination format, nonverbal communication is more challenging to assess. Few studies to date have demonstrated an acceptable level of satisfaction with the online clinical assessment by all stakeholders (Chan et al. Reference Chan, Humphrey-Murto, Pugh, Su and Wood2014; Khalaf et al. Reference Khalaf, El-Kishawi, Moufti and Al Kawas2020) but technical glitches were the main issues reported. Langenau et al. Reference Langenau, Kachur and Horber(2014) showed that majority of the respondents deemed web based OSCE as feasible and easy to use overall but 80% preferred face-to-face encounters as they expressed dissatisfaction about not being able to touch the person or do a physical examination. The positioning of the video equipment and microphone has an impact on the adequacy of information to make a judgement especially for procedural skills (Yeates et al. Reference Yeates, Moult, Lefroy, Walsh-House, Clews, McKinley and Fuller2020). Hopwood et al. Reference Hopwood, Myers and Sturrock(2020) also mentioned about careful selection of the OSCE stations and positioning of camera in the “Twelve tips for conducting a virtual OSCE”.
The COVID-19 pandemic has opened new avenues for telemedicine which has emerged as a promising means of delivering timely, cost effective and efficacious care to the patients (Sartori et al. Reference Sartori, Olsen, Weinshel and Zabar2019). Many services are planning to use remote clinical assessment as an adjunct to their routine care even when the current COVID-19 crisis eventually passes (Usman & Fahy, Reference Usman and Fahy2020). Remote consultations require unique technical and communication skills that are distinct from face-to-face encounters (Sartori et al. Reference Sartori, Olsen, Weinshel and Zabar2019). If we are planning to use telepsychiatry as a new norm for patient care, we need to understand the specific components of communication that can be impacted via remote consultation and to assess these in the competency-based examinations.
Based on the literature review, we could not find any study that explored the opinion of trainees and examiners on the assessment of communication skills in summative online clinical examinations in psychiatry. The present study aims to explore how examiners view online assessment of communication skills and what trainees think of this phenomenon.
Methods
Research paradigm
This study was designed as descriptive qualitative research using a narrative approach and 1:1 interview for data collection. The underpinning philosophical paradigm was naturalistic inquiry which argues that reality is subjective and depends on the lens through which an individual sees the universe (DePoy & Gitlin, Reference DePoy and Gitlin1998). The research lens that was designed based on the qualitative wheel of research choices (Savin-Baden & Howell-Major, Reference Savin-Baden and Howell-Major2013 ) is given in Fig. 1.
Research lens.
Recruitment and sampling
The sampling frame consisted of approximately 30 candidates and 25 examiners per sitting of the exam (60 and 50 in total, respectively). The study proposal was approved by the management committee of the CPsychI that acted as a gatekeeper and circulated an email to all the trainees and examiners of the online BST examination in September and November 2020. The email contained an information leaflet and a link to a short survey form asking questions about demographics and consent to achieve a purposeful sample (Marshall, Reference Marshall1996). However, the number of respondents just matched the estimated sample size, so all of them were invited for interviews.
Data collection
A pilot interview was conducted first to assess the quality of data collected by the interview guide. Interviews were recorded by Zoom (Zoom Video Communications, 2011) and transcribed verbatim. The questions focused on differences in assessing communication skills (facial expressions, empathy, tone of voice or hand gestures) and encountering any technology issues (full guide attached in Appendix 2). Confidentiality was assured by storing the data on National University of Ireland (NUIG) student one drive. The transcripts were checked with the audio recordings to screen for any errors and to become familiarized with the data, which is the first step for analysis (Braun & Clarke, Reference Braun and Clarke2006). All participants received a copy of their transcript to make any amendments and only the revised transcripts were included in final analysis.
Data analysis
The Braun and Clarke Thematic analysis (TA) (Reference Braun and Clarke2006) was used for data analysis. It produces a more detailed and nuanced account by analyzing and reporting various patterns within the data. The raw data was open coded using NVivo software v20 Pro. The codes were redefined, and new codes created as the coding process progressed. The codes were then arranged into various themes. A thematic map was drawn for a visual representation of different themes to allow for more revision and refinement and to draw subthemes before embarking on results.
Rigor
Rigor was ensured based on four criteria as proposed by Lincoln & Guba (Reference Lincoln and Guba1985). Credibility (Houghton et al. Reference Houghton, Casey, Shaw and Murphy2013) refers to how believable the findings are. Interviewing two important stakeholders in the online examination triangulated the findings. Member checking was achieved by asking participants to check their transcripts after the interview. Dependability (Houghton et al. Reference Houghton, Casey, Shaw and Murphy2013) refers to stability of data and was ensured by audit trail using NVivo to outline the decisions made during analysis to reach the results. Reflexivity was maintained by keeping a reflective diary (Darawsheh, Reference Darawsheh2014) of past judgements and discussion with the supervisor. Transferability (Houghton et al. Reference Houghton, Casey, Shaw and Murphy2013) was achieved by describing the context of this setting in adequate detail and reporting the findings with quotations. Confirmability (Houghton et al. Reference Houghton, Casey, Shaw and Murphy2013) is the accuracy of data and was believed to be achieved by fulfilling above criteria.
Results
The number of respondents who gave consent was 15. One participant did not respond for scheduling interview, so a total of 14 participants were interviewed (seven candidates and seven examiners). The average length of interview was 30 minutes for candidates and 25 minutes for the examiners. Ireland is divided into nine regions for postgraduate psychiatry training known as deaneries. There was a good mix of participants from different deaneries having experience of one or both components of the online BST examination 2020. The examiners cohort comprised consultants from a variety of psychiatric specialties. The demographic details of participants are given in Table 1.
Demographics of participants
Figure 2 shows the themes and subthemes that emerged from data analysis.
Thematic map.
Communication
This was the main theme emerged as per objectives of the study. The candidates found that virtual communication was nearly as good as face-to-face, but latter would enable to build a better therapeutic rapport by picking more cues and reciprocate with empathy.
“I think was a bit more difficult online, establishing the initial rapport because of the artificial nature of it” (Candidate 6).
The following subthemes were drawn:
(a) Eye contact
Most of the candidates looked at the patient’s image on the screen to pick facial expressions. However, they occasionally looked at the camera to make eye contact and deemed both essential to develop good communication. Few candidates were very conscious of this before the exam and looked straight at the camera throughout performing the task to maintain eye contact but found this cumbersome.
“I was looking at the camera the whole time, rather than the images and I found that really tough because you're not getting any feedback from the patient.” (Candidate 1)
The candidates positioned their laptops in a way that the camera was at the eye level to minimize disruption to eye contact. This was also reinforced by the examiners who recommended to either use a laptop with built-in camera or position the camera on top of the screen to minimize distance between the patient’s image and the camera. The examiners were not particularly concerned about the real eye contact as they acknowledged that it was not possible through online platform. They however placed more importance on the connectedness between the candidate and the patient and if the interview was progressing meaningfully.
“I think you look at the camera to signal to the person that you're attending to them and you're listening to them and then you look at the face, because you're actually then attending to them and trying to pick up on subtle clues.” (Examiner 7)
(b) Hand gestures
Some candidates adjusted their computer to position themselves in the middle of the screen, so that their arms were visible for hand gestures. While others did not deem it necessary as they felt that they developed reasonable rapport with their face only. The candidates reported that it would be ideal to see the whole body of the patient as in face-to-face exam.
“It was hard not to be able to see someone’s entire body language” (Candidate 2)
All the examiners deemed preferable to see upper half of the candidates’ image to see the hand gestures (face, shoulders and arms). Some considered it even more important than eye contact, while others believed that the candidates were good enough with their facial expressions only.
(c) Tone of voice
The candidates spoke clearly and in small chunks to deliver their message.
The examiners expressed some difficulty when empathy was conveyed through the screen.
“I felt on some occasions that came across a little bit colder than maybe it would have been in face-to-face interview” (Examiner 3)
(d) Professionalism
The examiners stressed the importance of maintaining overall professional attitude more than mere eye contact or hand gestures.
“If you’re too casual with patients, that’s something that needs to be fixed” (Examiner 2)
Screen optimization
This theme relates more to candidates and refers to how the computer screen appeared during the exam. It consists of the following subthemes:
(a) View
Most of the candidates preferred to use the speaker view that makes the image of the speaking person bigger. Some were too afraid to touch anything in case “everything would go off”(Candidate 6) and stuck to whatever computer settings were on.
(b) Invigilator window
The invigilators shared screen to show the task and turned their cameras off which was appreciated by the candidates to minimize any distractions.
(c) Examiner window
Some candidates were “not distracted” by the presence of examiners on the screen as their mikes were muted and they appeared in the background of the speaker view. Others preferred for the examiners to turn off cameras as they felt conscious and distracted by their slightest expressions.
(d) Task
The candidates mentioned that the task was shown as a screen share by the invigilators before starting the OSCE station, but they did not have access to it during the performance. They found it particularly difficult for physical health stations in which the task was very lengthy with numerous examination findings and lab investigations. Some suggested to share the task separately by using email or chat function so that it would be accessible throughout, as it was in face-to-face exam. Others suggested to give more time for reading or giving more direct and concise information.
One examiner (3) commented that a candidate’s mike was on while reading the task and all “muttering” was obvious. The examiners felt that task instructions should be accessible throughout the station to replicate the conditions in face-to-face OSCE.
Overall experience
This theme encapsulates the overall experience with the online exam and a consistent expression by all participants was “better than expected”. The candidates were comfortable in their familiar surroundings that helped to alleviate the exam anxiety. They found the initial part a bit challenging to establish rapport but settled well once it started. They felt that the three days of exam were a bit tiring and suggested to make it two days or having a gap between OSCE and CFME components.
(a) Technical issues
The use of Zoom software was appreciated by the participants for ease of navigation and smoothness. Overall, the candidates reported minimal technical issues during the exam. However, a few candidates mentioned a delay of an hour before starting the exam which was very anxiety provoking. Some candidates also recalled connection glitches during performing their task but appreciated College support in terms of rectifying or giving extra time. The examiners also mentioned minor glitches and were mindful of them while marking the stations. The examiners felt that the exam was very well organized and ran smoothly overall.
“I actually think it was pretty successful” (Examiner 6)
“I really thought it was excellent” (Examiner 4)
(b) Time pressure
The candidates felt pressured to read the task, especially with lengthy instructions but had enough time to cover the task.
(c) Types of psychopathologies
Overall, the candidates felt that the actors performed well, and they were able to communicate effectively with them in various stations. A particular challenge came up when examining a manic patient as they were restless, and candidates couldn't see all the movements and use nonverbal gestures accordingly. The examiners stated that the instructions for actors were quite detailed and the only element missing was the “personal space-invading stuff” (Examiner 7).
(d) Anxiety
The candidates were anxious in the beginning about the whole online process or if anything went wrong but later found it to be a lot less anxiety provoking than face-to-face.
“the entire day in person was, I suppose a bit more stressful and kind of anxiety provoking” (Candidate 1)
This was also reiterated by the examiners.
“I imagine that on balance, its overall less anxiogenic scenario” (Examiner 2).
(e) Comparison with UK CASC
Some candidates had the experience of sitting CASC and commented on having more time to perform station with relatively calmer actors/patients and minimal technical issues in BST exam. On the contrary, the task in the CASC was focused and available throughout the station, but it was as the expense of patient’s image which was quite small and posed challenge to communicate effectively.
Continuation of online format post pandemic
This theme encompasses the views whether this exam should continue to be taken online after the COVID pandemic or moved to traditional in-person format.
(a) Practical reasons
All candidates favored online version of exam for practical reasons as it avoided the time and cost of traveling and overnight stay before and after the exam.
“I would find the online version preferable” (Candidate 5)
All examiners preferred to do face-to-face exam. One reason was the social aspect, the lack of which at home made the whole experience quite tiring while others alluded to better assessment of communication skills face-to-face.
“There is kind of collegiality and support throughout the Center” (Examiner 1).
“There’s something more authentic about seeing somebody face to face” (Examiner 3).
They however acknowledged the practical advantage of assessing from home as it can avoid the expenses for travel, overnight stays, and paperwork for the College. One examiner (Examiner 4) commented on a full day being quite long and suggested to have a larger pool of examiners with half a day for online exam.
(b) Competence
The candidates expressed that face-to-face exam allowed more fluidity in communication as they were able to pick more nonverbal cues. Online exam needed a lot of preparation as “if you are not experienced in doing video consultations, then you often interrupt each other” (Candidate 3). Some concern was expressed by the participants that it may not be effective for physical and cognitive assessment as well as assessing communication in more complex stations like manic patient. The examiners also stated to continue to have an online component of the exam to assess learning outcomes related to telepsychiatry as this is going to be used more often going forwards.
(c) OSCE vs CFME
All participants agreed that CFME is an oral exam and does not involve observation of practical skills, like the OSCE, perhaps making it more suitable for an online format. Most candidates favored online OSCE for above reasons but did express concerns that it may not be as good as face-to-face. The examiners also preferred to “go back” to traditional OSCE.
(d) Recording of exam
Some candidates felt recording would be good idea for appeals or marking on a separate day. Other candidates believed that it would undermine the trust and professionalism of the examiners and the College. They also mentioned about GDPR and confidentiality issues of storing and handling recorded videos.
Discussion
The results from this study show a preference to continue online CFME. Interestingly, some people argue that psychiatric assessments are largely communication, so they can be done remotely successfully. However, the findings from our study clearly demonstrate that an in-person consultation allows more fluidity in communication by picking nonverbal cues and reciprocate with empathy. The examiners remarked that the conveyance of empathy was less obvious than in face-to-face; this was another reason in favor of latter as empathy has been considered as vital for effective doctor–patient interaction (Kirmayer, Reference Kirmayer2008). The participants were largely satisfied with online OSCE but did mention some concerns about not being able to communicate as well as in face-to-face assessments. This observation was in line with a study on osteopathic residents (Langenau et al. Reference Langenau, Kachur and Horber2014) who preferred face-to-face OCSE due to limitation in physical examination. Another limitation for online OSCE was the physical and cognitive examination as the task was modified. These findings also highlight the importance of physical examination in psychiatry and the difficulty in carrying out physical examinations online.
An interesting finding was that the candidates were very concerned about making eye contact through the screen, but examiners placed more importance on their overall professional attitude and patient engagement. Due to the COVID-19 pandemic people are more used to virtual communication, so there was a consensus that not having an eye contact for 100% of the station was acceptable if patient was well engaged. Patient centeredness is the cornerstone of good communication (Wilby et al., Reference Wilby, Govaerts, Austin and Dolmans2019a) and is a main determinant of good consultation along with professional behavior (Chahine et al. Reference Chahine, Holmes and Kowalewski2016). The candidates may benefit from explicit instructions from the College about camera positioning and eye contact before the exam.
The candidates found it difficult to read the lengthy tasks as they couldn't access it during the station. They recommended either more reading time or smaller amount of information. It was given as feedback to the College after first online sitting and the reading time for task was increased in the subsequent cohorts. The comparison with UK membership exam CASC is out of scope for this study but a consistent positive remark about the CASC was the availability of task throughout the station. A potential option for future could be to share it via chat function or email so that candidates won't be dependent on invigilator sharing and moving their screens. However, security of email, as well as confidentiality of exam material may be part of the issue in using other ways to disseminate exam material.
Khalaf et al. Reference Khalaf, El-Kishawi, Moufti and Al Kawas(2020) found reduced anxiety level in oral online exam in dental students. Our candidates also preferred to continue online exam as it was less anxiety provoking due to the comfort of being in their homes and avoidance of travel and night stay close to exam venue. They however commented on 3 days in a row quite tiring and suggested to do in either 2 days or having a gap between a day of CFME and two days of OSCE. This raises the question of how long anyone can concentrate online which seems to be more difficult than in face-to-face settings. Again, the College took it into account and gave a week gap between OSCE and CFME in 2021 as the CFME and OSCE are two independent components that require different types of preparation. Another study (Stowell & Bennett, Reference Stowell and Bennett2010) found reduced test anxiety in online exam for students who normally experience high anxiety in classroom while the reverse was true for those low in classroom anxiety. The latter group in this study was more preoccupied if any technical issues would disrupt the exam and we observed same reason for anxiety in online setting in our cohort. This can be minimized by checking the audio and video equipment beforehand and a test call may help to alleviate. Lesser degrees of anxiety with online formats may not be universal, with some reports of being self-conscious on camera. In real life, this is a significant issue for a small proportion of exam candidates and may interfere with their performance as some candidates in our study preferred for examiners to turn off their cameras.
Strengths and limitations
To the best of our knowledge, this is the first study to explore the views of candidates and examiners on communication skills assessment in a high stakes’ postgraduate psychiatry examination. The perspectives of two important stakeholders triangulated the findings and can help to design similar exam in other specialties. A major limitation was low response rate. We aimed to achieve a purposeful sample and do ongoing analysis with data collection until it was saturated, and no new themes emerged. However, based on the number of respondents we could and did include them all. Even though they represented different deaneries, specialties, and age groups, we could not ascertain that data were saturated and it may be a possibility that we would have heard other perspectives should more participants have signed up for interview. The interviews were conducted after the candidates received their exam results. Although they were not asked about the results, the results may influence their perception of the online assessment. Another limitation was that the actors who participated as patients were not interviewed and could have provided another valuable perspective on how communication skills were assessed. This can potentially be an area for future research.
Conclusion
The participants were largely satisfied with the online examination but did not consider it equal to face-to-face for picking nonverbal cues. The candidates were very conscious of eye contact while examiners placed more emphasis on overall professional behavior and patient engagement. Candidates deemed online exam as less anxiety provoking and participants reported minimal technical issues overall. All candidates preferred to continue the online format post pandemic while all examiners preferred to go back to the in-person OSCE. However, continuation of online CFME was agreed to be possible by both groups. These findings may be helpful to modify the current Irish psychiatry membership examination or similar assessments in other countries and specialties.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/ipm.2023.19
Acknowledgements
The authors are grateful for Prof Aoibhinn Lynch, Dean of Education, CPsychI, for refining the study proposal and providing valuable feedback on the final draft. The authors thank the College for acting as a gatekeeper and all participant trainees and consultant psychiatrists for their valuable input and unique insights about the research question.
Authors contribution
All authors made substantial contributions to the conception of this project. MU did data collection and created the first draft, which was revised and edited by all authors. All authors approved the final version of the paper for publication and agree to be accountable for all aspects of the work.
Financial support
This research received no specific grant from any funding agency, commercial or not-for-profit sectors.
Competing interest
The authors declare no conflict of interest.
Ethical standard
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committee on human experimentation with the Helsinki declaration of 1975 as revised in 2008. The authors assert that ethical approval for this article was obtained from ethics committee of Sligo Research Education Foundation and National University of Ireland Galway.
