Case study
Open Access

Managing the COVID-19 risk: the practicalities of delivering high stakes OSCEs during a pandemic

Claire Ann Canning[1], Kirsty Jane Freeman[1], Ian Curran[1], Katharine Boursicot[1]

Institution: 1. Duke-NUS Medical School
Corresponding Author: Dr Claire Ann Canning ([email protected])
Categories: Assessment, Education Management and Leadership, Learning Outcomes/Competency, Technology, Clinical Skills
Published Date: 19/08/2020

Abstract

The impact of the COVID-19 pandemic on the teaching and assessment of clinical skills continues to pose significant challenges for healthcare education providers worldwide.  In March 2020 Duke-National University of Singapore (Duke-NUS) Medical School demonstrated how to design and implement clinical skills examinations (OSCEs) in the early phase of COVID-19.  As governing bodies continue to revise restrictions to help ‘flatten the curve’, educational institutions have to undertake a rapid review of assessment practices and adapt to this ever-changing environment.  This case study describes the risk-assessments and challenges faced when delivering high stakes OSCEs during the COVID-19 pandemic. We also describe successful mitigation strategies implemented to combat these risks, and how we also embraced and leveraged technology in a very creative way despite the restrictions and constrained environment.

We describe and share practical guidance that may be of help or interest to healthcare education providers across all disciplines on how to effectively deliver clinical and procedural skills examinations that are authentic, valid and comply with the strict national COVID-19 restrictions implemented during this time.

Keywords: OSCE; Clinical Examination; Pandemic; COVID-19; Simulation; Clinical Skills Assessment; Performance Skills Assessment; Quality Assurance; Assessment

Background

Duke-NUS Medical School is a US-style graduate-entry medical school of the National University of Singapore offering a Doctor of Medicine (MD) programme. As per the assessment strategy at Duke-NUS Medical School any final year medical student who is unable to achieve a passing grade in the final OSCE is required to undertake a supplementary examination to demonstrate competence and progress to graduation.  For the class of 2020 the main OSCEs were held in March.  Boursicot and colleagues describe how these final level OSCEs were conducted in their paper “Conducting a high-stakes OSCE in a COVID-19 environment” (Boursicot et al., 2020).  However, with the evolving pandemic and further control measures or restrictions due to COVID-19, the practices applied in March 2020 no longer reflected the risk mitigation strategies we were required to adopt eight weeks later.  This case study describes the enhancements, considerations and approaches we put in place when running the supplementary high stakes OSCEs. Specifically, we discuss the mitigation strategies implemented to combat these risks, and how we embraced and leveraged technology in a very constrained environment.

Since the outset of the COVID-19 outbreak, the Singapore government has implemented a comprehensive range of necessary, national control measures to reduce community spread of the SARS-coronavirus 2. These control measures are mandatory, legally enforced and have evolved over time to reflect emerging risk and understanding of the SARS-coronavirus 2. Key principles include a focus on strict infection control and personal protection along with an ‘elevated set of safe distancing measures’. The most rigorous, referred to as a ‘Circuit Breaker’ from 7 April 2020 until 4 May 2020 (Ministry of Health, 2020) amounting to a national lockdown.  On 21st April 2020 the government announced that the Circuit Breaker (CB) would be extended by another four weeks until 1st June 2020 (inclusive).  CB measures required the population to stay at home, avoid non-essential travel and limit face to face interactions. The supplementary OSCEs were scheduled to occur on 5th May 2020 so we had to rapidly review and enhance our planning to address these further challenges and restrictions. 

The final year OSCEs are essential to allow students to graduate and commence their post graduate training (PGY1) in June 2020.  Faculty buy-in, support from regulatory bodies and utilizing available resources are essential for the successful conduct of high stakes assessments (Sabzwari, 2020). This paper provides practical guidance on how to deliver clinical and procedural skills examinations (in an OSCE format) that are authentic, valid, and comply with the strict national COVID-19 restrictions implemented during these unprecedented times.

Risk mitigation

The safety of all involved in this OSCE was our number one priority.  The School engaged with the Medical School Safety Team, the National University of Singapore Provost, the Ministry of Health (MOH) and Ministry of Education (MOE), to identify, assess and mitigate risk when planning and running these high stakes OSCEs. We were required to seek formal approval for our plans from the University and the Ministries of Health and Education. All stakeholders supported the running of these examinations to give our students an attempt to graduate and crucially join the frontline healthcare workforce.

As part of the COVID-19 control measures the following risk mitigation strategies were implemented:

  • All attendees were required to wear a mask during commuting, and whilst on campus
  • All attendees were encouraged to travel using point to point transport service (e.g. taxi or private vehicle)
  • Staggered arrival time of attendees to ensure group cohorting, smooth arrival at the front entrance, and avoidance of congregation
  • All examiners were recruited from a single institution
  • Temperature was checked on arrival
  • Travel & Health Declarations completed by all parties coming to campus
  • Movement of attendees confined to the examination venue only
  • Hand sanitiser stations located at simulation centre entrance, outside and inside each clinical examination room
  • Use the Trace Together APP, a Singapore government smart phone app designed to use Bluetooth tracking to facilitate and support contact tracing of individuals if required
  • Practice safe social distancing and strict personal hygiene measures
  • Disinfection of surfaces and equipment in-between each student

Manpower

Key to the success in delivering a high stakes OSCE in a pandemic is to empower/engage an agile nimble team. Facilitating high stakes OSCEs is notoriously labour intensive and gregarious activity, personnel on-site can include examiners, real and/or simulated patients (SPs), assessment faculty and support staff.  In view of the national restrictions, the decision was made to eliminate face to face interactions wherever possible, by supporting the clinical examiners to assess students from off-site locations, thus enabling an overall reduction in the total number of individuals physically on-site. 

Within a 10-day window the assessment and clinical performance teams overhauled and revised a robust operational strategy to safely and effectively run high stakes OSCEs, adhering to five key principles:

  1. Minimise in person student:examiner interactions
  2. Minimise the number of SPs on-site
  3. Minimise the number of faculty/staff on-site
  4. Test the technological solutions robustly
  5. Familiarise all participants with the new arrangements

The OSCE included stations where students had to take a clinical history and summarise their findings, physically examine a patient, and a range of professional encounters such as explaining diagnoses/management to patients and demonstrate practical skills using manikins with SPs.  The latter hybrid stations comprised the SP who provided the professional context of patient interaction as well as the manikin on which the practical skill was performed.

It was deemed that 10 out of a total of 12 stations were conducive to having examiners score offsite, decreasing the total number of on-site examiners from 12 to 2.  Four of the stations were conducive to simulated patients participating remotely (from their homes), such as clinical history taking and explanation stations, which resulted in reducing the total number of SPs onsite from 12 to 8.  It was decided that the two physical examination stations required the student to be present on-site to perform the clinical examination, with the clinical examiner in the room. To decrease the number of staff on-site, we reviewed the tasks (not individuals) required to successfully, and safely and effectively run these examinations.  With the task list identified, each task was then categorized as on-site or off-site.  It was decided that 4 staff would be required to facilitate the on-site tasks, with 6 staff supporting off-site.

With minimal staff on-site to undertake the numerous tasks required to operate the practicalities of the OSCE, there was a requirement to upskill staff in several key areas.  For example, staff from the assessment team were trained to swap over the clinical skills stations consumables and reset the manikins in the break between students.  All members of the team were required to upskill on the Zoom video-conferencing platform to be able to troubleshoot any issues that may arise during the examination.

Embracing technology

Planning for OSCEs usually occurs over many months, however the unprecedented arrival of COVID-19 required the team to be dynamic and agile as the level of restrictions constantly changed often at short notice. Alexander and colleagues stated the “The pandemic should not be a time of educational stasis, it is a chance to harness the power of technology and novel educational tools”, and the same should be said for assessment (Alexander et al., 2020). Ten days prior to the examination our challenge was to embrace technology and adapt rapidly to the new requirements whilst maintaining the authenticity and validity of these high stakes supplementary OSCEs. 

The first practical challenge was to enable observation of students performing clinical procedural skills stations via video conferencing software, providing the maximal level of detail to enable an off-site examiner to successfully score the interaction.  The video conferencing platform adopted for this examination was Zoom Video Communications Inc.  We chose Zoom because staff and students were familiar with it due to recent e-learning practices. 

During COVID-19 restrictions, and due to the short timelines, we were not able to procure extra audiovisual devices to enable live streaming of patient/student interaction required for examiner scoring. We adopted the ‘under the stairs approach’ making use of the resources we had at hand.  Resources at hand were iPads, mobile phones, and laptops with webcam. The challenge was where and how to place devices in each room to enable them to capture a global view and a close-up view of the skill being performed. With duct tape, a selfie stick, two tripods, three IV poles, plastic sheet protectors and countless cable ties, we positioned the cameras in each room to ensure the examiner was able to view the students’ performance adequately (Figure 1).

Figure 1. Configuration of Simulated Patient, candidate and cameras to facilitate off-site examination.

Final thoughts

This paper provides practical guidance on how to deliver high stakes OSCEs during a pandemic. Whilst we highlight opportunities where stations could be rapidly converted to remote stations, so having examiners and SPs participate off-site, it is also important to note that for some stations, particularly the physical examination stations, face to face interactions were deemed essential for validity and authenticity. The clinical examiners also had to examine the patients to calibrate and confirm the clinical findings so they could fairly assess the performance and confirm the findings of the candidate.

Faculty buy-in is key to the successful implementation of innovative on-line OSCEs. Paramount in our activities was ensuring staff, students and simulated patients were safe. By engaging with all our key stakeholders, including students, clinical faculty, administrative staff and University and Ministry leaders we were able to ensure the safety of all, whilst maintaining the authenticity and validity of these high stakes examinations. Over 8 weeks after the conduct of the OSCE we have had no reported episodes of student, examiner or staff infection with COVID-19.

Take Home Messages

  • Conduct a thorough risk assessment and tailor the risk mitigation strategies to address the prevailing risks and control measures.
  • Expand the scope of practice of team members, be agile in your approach and enable transfer of skills across teams
  • Embracing technology does not have to be expensive – review resources at hand and use cost effective solutions such as duct tape and cable ties
  • Engage all stakeholders early and often - government agencies, university executives, examiners, simulated patients and students
  • Ensure adequate personal protective equipment for stations that require the students to interact with SPs

Notes On Contributors

Claire Ann Canning, Assistant Professor for Assessment and Progression, Duke-National University of Singapore Medical School, Singapore.  Main author, and along with Kirsty Jane Freeman conceptualized, wrote and revised manuscript based on comments and suggestions from the other authors.

Kirsty Jane Freeman, Lead Associate for Simulation, Duke-National University of Singapore Medical School, Singapore. Along with Claire Ann Canning conceptualized, wrote and revised manuscript based on comments and suggestions from the other authors. 

Ian Curran, Vice-Dean for Education, Duke-National University of Singapore Medical School, Singapore.  Reviewed draft, advised on national level and institutional directives.

Katharine Boursicot, Associate Dean for Assessment and Progression, Duke-National University of Singapore Medical School, Singapore.  Contributed to the conceptualization of the paper, reviewed and revised drafts.

Acknowledgements

We would like to thank everyone who was involved in the enabling, planning and delivery of these high stakes OSCEs, especially:

Shenna Chong; Abegail Fernandez; Goh Sok Hong; Tan Poh Huat Calvin; Janeshtha Vaswani; and Xin Weiren Wilson. 

All the simulated patients who took part in our OSCE.

All the examiners who gave up their time to engage in this important event.

All the students who participated in the examination.

Written permission was obtained from each of the individuals present within Figure 1 for the images to be used in this manuscript for publication. The individual images are the property of author Kirsty Jane Freeman.

Bibliography/References

Alexander, L., Ashcroft, J., Byrne, M. and Wan, J. (2020) 'All hands on deck: early graduation of senior medical students in the COVID-19 pandemic', MedEdPublish, 9(1). https://doi.org/10.15694/mep.2020.000096.1

Boursicot, K., Kemp, S., Ong, T., Wijaya, L., et al. (2020) 'Conducting a high-stakes OSCE in a COVID-19 environment', MedEdPublish, 9(54). https://doi.org/10.15694/mep.2020.000054.1

Ministry of Health (2020) 'Circuit breaker to minimise further spread of covid-19'. Available at https://www.moh.gov.sg/news-highlights/details/circuit-breaker-to-minimise-further-spread-of-covid-19, (Accessed: 10/04/2020).

Sabzwari, S. (2020) 'Rethinking Assessment in Medical Education in the time of COVID-19 ', MedEdPublish, 9(1), p. 80. https://doi.org/10.15694/mep.2020.000080.1

Appendices

None.

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

Ethics Approval was provided by the Vice Dean for Education, Duke-NUS Medical School.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Ronald M Harden - (30/08/2020) Panel Member Icon
/
I have argued that one reason for the OSCE's popularity is that the approach can be adapted to suit different needs. This useful paper demonstrates what can be achieved with imagination, creativity, attention to detail and the cooperation of the stakeholders in response to the present Covid constraints.
A highly recommended read for all interested in education in the health professions.
Richard Fuller - (28/08/2020)
/
This is a welcome follow up to the initial paper by the Duke-NUS group, and provides practical (and road tested) advice to those needing to run an OSCE in a variety of lock down situations.
Some really important take home messages for me
1) A radical rethink about some station formats. Does the examiner need to be there in person? Can we simulate a video consultation?
2) Rethinking ''Technology' - simpler solutions are more cost effective, and more palatable to stake holders

Some interesting questions for the future...!
1) With examiners remote from the interaction, how does this affect candidates behaviour. Less 'performance' and more authenticity - vs the difficulty of presentation to a screen to summarise? Really interesting research possibilities
2) The use of video scoring in OSCE - for readers who want to explore more, a series of paper from Keele in the UK give some interesting perspective - e.g. https://dx.doi.org/10.1080/0142159X.2020.1801997
Sandra Kemp - (25/08/2020) Panel Member Icon
/
This is an interesting paper. It clearly describes a way of approaching risk assessment and mitigation strategies for an OSCE. I think the practical guidance the paper provides will be useful to readers who are trying to conduct performance assessments in these Covid constrained times. I thought the description of the process for decision making around what could be tested onsite versus offsite/remote was also very helpful. I would recommend this paper to others who are involved in high-stakes OSCEs.
Possible Conflict of Interest:

I am an external examiner at the Duke-NUS School of Medicine and have knowledge of this performance assessment. I was also a co-author of the paper describing the final OSCE referred to in this paper: “Conducting a high-stakes OSCE in a COVID-19 environment” (Boursicot et al., 2020).

Trevor Gibbs - (21/08/2020) Panel Member Icon
/
A very interesting and practical paper at a time of almost "too-many " Covid papers. A slightly different approach when compared to the previous OSCE paper by Boursicot et al. I specifically appreciated the risk assessment and the tailoring the risk mitigation suggestions, something that we all need to think about these days. Certainly a paper for everyone involved in high-stakes OSCE examinations.
Possible Conflict of Interest:

For transparency I am one of the Associate Editor of MedEdPublish

Trudie Roberts - (20/08/2020) Panel Member Icon
/
Although like many others I'm feeling a little 'all CoVided out' by the number of papers purporting to give a new perspective on "Education in a Time of CoVid". I do think that practical guidance drawn from actual experience is very useful. This is what the authors of this paper provide. So I recommend this paper to others who are running or trying to run performance based assessments in the current climate are a clearly written and practical submission.
Possible Conflict of Interest:

I am an external examiner at the Duke-NUS School of Medicine and have inside knowledge of this performance assessment