New education method or tool
Open Access

Implementing a training program for resident-led virtual clinics during the COVID-19 pandemic

Lysa Boissé Lomax[1], Lindsay Crawford[1], Andy Wong[1], Gayathiri Balasubramaniam[1], Ramana Appireddy[1]

Institution: 1. Queen's University, Kingston, Ontario, Canada
Corresponding Author: Dr Ramana Appireddy ([email protected])
Categories: Assessment, Educational Strategies, Learning Outcomes/Competency, Postgraduate (including Speciality Training), Continuing Professional Development
Published Date: 02/06/2021


Given the impact of the COVID-19 pandemic on healthcare, physicians in the USA, Canada, and globally have begun to utilize virtual care more widely to provide safe, timely, and accessible ambulatory care. While the COVID-19 pandemic prompted its use more widely, it is very likely that this mode of healthcare delivery will continue after the pandemic as it is an effective consultation tool and allows physicians to better serve their patient care populations, particularly those in rural and remote communities. But physicians are not alone in providing virtual care. In medical teaching institutions, residents provide much of the frontline, ambulatory care and, being a newer mode of service, residents require education and training on this method of healthcare delivery; however, no standardized guidelines for training, supervision, or assessment exist. While there is overlap in the process used to conduct in-person and virtual consults, residents require training in specific areas that are unique to the online setting, particularly in conducting the physical exam. This paper outlines a structured process for developing and implementing a training program for resident-led virtual clinics, including the supervision, performance, and assessment expectations of residents. The recommendations provided are intentionally generic to facilitate use by other programs and institutions.

Keywords: Competence By Design (CBD); Competency Based Medical Education (CBME); Entrustable Professional Activity (EPA); Virtual Care


The applicability and use of virtual care has increased since the onset of the COVID-19 pandemic in North America as a way to continue to offer safe, timely, and accessible care (Centers for Disease Control and Prevention, 2020; Public Health Agency of Canada, 2020). While residents provide much of the frontline ambulatory care in teaching centres, very little research exists on how to properly educate and train residents on providing virtual care (Majersik and Reddy, 2020; Waldman et al., 2020). The Division of Neurology at Queen's University, Kingston, Ontario, Canada has been a pioneer in using virtual care with an active clinical and research program in virtual care since 2016 (Appireddy et al., 2019; Ontario Telemedicine Network, 2019). In this paper, we share a practical guide on how to implement a virtual care training program including its design and guidelines for resident supervision, performance, and assessment. The details are intended to be generic and should be modified to suit the specific needs of your program or institution.

The Use of Virtual Care

Traditionally, telemedicine involved remote delivery of healthcare services and the sharing of clinical information between two hospitals/healthcare facilities (American Telemedicine Association, 2020). More recently, however, advancements such as improved utility and availability of video conferencing devices (e.g. smartphones and tablets), patients and their family member(s) can participate in virtual care from their home or location of choice (Appireddy et al., 2019). The purpose of this mode of healthcare delivery is to facilitate high quality and efficient patient care, especially for those patients who cannot access in-person health care (Shaw et al., 2018; Lougheed, 2019; Canadian Medical Association, 2019a; Canadian Medical Association, 2019b). One of the well-known limitations of virtual care is the inability to do a full physical examination; however, this limitation can be overcome through the use of a modified physical exam (Supplementary File 1) (Wechsler, 2015). Virtual physical exams can be just as informative and directive for general examinations and many neurological conditions as an in-person visit in select clinical scenarios (Wechsler, 2015). Further, the use of remote monitoring devices can help with monitoring of vital signs and other physical examination findings (Andres et al., 2018; Volterrani and Sposato, 2019; Steth IO, 2020).

Development and Implementation

There is a lack of residency training and accreditation requirements for virtual care globally, including in the USA, Canada, and Australia with only one Accreditation Council for Graduate Medical Education (ACGME) accredited residency training program that includes virtual health in their assessment milestones (Canadian Medical Association, 2019a; Pourmand et al., 2020). While there are e-health milestones in CanMEDS 2015 (e.g. ME 1.4 “Use technology to facilitate consultation for patients who may have limited or delayed access to care”), this milestone and others are not included in the milestone requirements for Canadian Neurology training programs (Canadian Medical Association, 2019a; Royal College of Physicians and Surgeons of Canada, 2020c). Moreover, there is a gap in research guiding the assessment of residency training in the area of virtual care (Canadian Medical Association, 2019a). As such, we sought to develop our own virtual care training program and assessments. Drawing from the stages of learning proposed by Dreyfus and clinical competence ranking based on Miller's pyramid, we have developed a resident virtual care training program (Dreyfus and Dreyfus, 1980; Miller, 1990). This program was led by the residency program director (LBL), virtual care physician lead (RA), resident lead (AW), and the educational consultant (LC). Much like the clinical skills learned in medical schools, such as history taking and the physical exam, we found that there is a process to becoming an expert in virtual care that follows a similar pattern. When we began implementing this program, all of our residents were new to virtual care but, as Miller's pyramid suggests, with clear objectives (Competencies/Entrustable Professional Activities (EPAs)/tasks), direct observation and ongoing measurement (assessment and coaching) residents will gain the knowledge and competence needed to integrate virtual care into their clinical practice (Miller, 1990). Our approach can be broken down into 4 phases.

Phase 1 - Introduction to Virtual Care

During phase 1, the "knows" phase in Miller’s Pyramid, residents are introduced to virtual care (Miller, 1990). This phase is conducted in the form of didactic lectures with numerous opportunities for discussion. The initial discussions focus on changes in communication styles, virtual care etiquette, optimal use of technology, and virtual care clinical skills. Residents are provided a communication script, outlining communication, and etiquette considerations, as well as the modified physical exam that includes prompts to guide patient positioning and proximity to the camera. Discussions focus on the performance of a modified physical exam, utility, and scope of virtual care, patient selection, triaging, clinical conditions for virtual care, and advantages of virtual care as well as limitations. Future cohorts of residents entering the Neurology training program at Queen’s University will go through the virtual care training program, starting with Phase 1 during their orientation. 

Phase 2 - Exposing Residents to Virtual Care

During this phase, the "knowns how" phase, residents participate in simulated virtual care encounters with a standardized patient in a controlled environment where experienced observers provide feedback (more on this in the “Assessment Considerations” section) (Miller, 1990). The simulations are done virtually to best simulate a real-life virtual visit and to easily facilitate direct observation (via remote login), a key component in competency-based medical education (Hawkins et al., 2015; Holmboe, 2015). The domains that require feedback and coaching most are virtual care etiquette, creating the ideal environment, and the physical exam. We found that all the residents, including the most junior, have many transferrable skills that they intuitively integrate into their first standardized visits, namely verbal communication and history taking. These simulated sessions provide the resident with a safe environment to practice skills and consolidate knowledge, with the understanding that their performance will be monitored and assessed. These sessions are also an excellent opportunity for residents to receive feedback, not only from their supervising attending but also from the simulated patients who can identify communication strengths as well as areas for improvement. Residents are expected to attain 3 “achieving” assessments (Table 1) in the simulated sessions before they can participate in virtual care encounters with actual patients.

Table 1: Sample assessment for performing a history and physical examination for patients with common presentations 

 Milestones (Royal College of Physicians and Surgeons of Canada, 2020c)

 Not Observed

 Needs attention



 Elicit an accurate relevant history





 Perform a physical examination relevant to the presentation and context (e.g. virtual care)






 Conduct the interview in a patient-centred manner





 Exhibit appropriate professional behaviours





 Additions for eVisits

 Provide a clear structure for and manage the flow of a virtual visit (e.g. introductions and an orientation to virtual care)






 Optimized the virtual environment appropriate for the type of visit





 Encourage respect for collaborators and their roles (e.g. those coordinating meeting or assisting the patient).





 Communicate effectively with patients/family/caregivers using digital technology






Phase 3 - Integrating Residents into Virtual Care

As residents become comfortable running virtual visits, the focus should move toward teaching safe and effective use of virtual visits for patient care. At this stage, the resident may "show how" to conduct a virtual visit with real patients but not have reached the "does" phase, yet. (Miller, 1990). Depending on the degree of entrustment (the level of independence entrusted to the resident for a specific Competency/EPA/task), the attending may be conferenced in to 1. directly observe the entire visit, 2. review and confirm with the patient at the end of each visit, or 3. review the case with the resident at the end of the clinic day (see section on “Guidelines for Resident Supervision, Performance, and Assessment”). This is a critical phase as patient safety is dependent on the geographical location of the supervisor and their ability to intervene in patient care decisions if needed (Canadian Medical Association, 2019a). Compared to traditional models for in-person care, virtual care lends itself well to a variety of remote supervision scenarios ((Canadian Medical Association, 2019a). Decisions regarding the level of resident supervision should centre on ensuring patient safety and resident learning (Canadian Medical Association, 2019a). Through practice and working with different attendings, the most important skill the resident develops at this stage is their ability to judge whether a patient is safe to be assessed virtually. Ultimately, through experience and ongoing coaching from their supervisor, the resident will adjust their risk tolerance to what they will ultimately use for deciding when to apply a virtual over an in person visit.

Phase 4 - Mastery

The final stage is the mastery of residents’ virtual visit skills, the "does" phase (Miller, 1990). This stage will ultimately be obtained over multiple months to years of experience and practice where residents will learn the nuances that allow them to most effectively, and intuitively apply this valuable skill. This stage will continue on past graduation from residency and involves a commitment to lifelong learning and professional development.

Guidelines for Resident Supervision, Performance, and Assessment

Resident Supervision

Given the novelty of virtual care for residents, it is important to have appropriate levels of supervision to ensure patient needs are met, and that any gaps in communication, etiquette, and clinical assessment are addressed (Table 2) (Canadian Medical Association, 2019a; Wolfram, 2020). A significant advantage offered by the virtual care platform is the ability to remotely supervise residents which works well with the busy schedules of physicians and also ensures safe physical distancing during the COVID-19 pandemic.

Table 2: Supervision recommendations for virtual care 

  1. The supervisor must be available to residents promptly and must review all cases as they would if the care was provided in-person
  1. Direct observation is always recommended but the level of supervision should be based on a resident's experience and level of training (all initial virtual visits should be directly supervised)
  1. The supervisor must be readily available in instances of remote supervision (e.g., available by phone)
  1. Rounds, pre-clinic meetings, or other routine standard-setting and checkpoints should be maintained or initiated due to the limitations imposed by virtual care. These meetings may be accomplished through conference calls, video chats, or messaging apps
  1. Charts should be reviewed and signed off during or after each clinic or shift
  1. Residents should document all interactions regardless of the means of communication used


Performance Expectations of Residents

A key element when supervising residents is establishing performance expectations that align with their experience and level of training (Table 3) (UNC School of Medicine, 2020; Wolfram, 2020). While we cannot replace a clinical educator's ability to make judgments about a resident's readiness to participate in a task, we have developed a general guideline on resident performance expectations by stage of training to aid in the observation and assessment of residents. The guideline was based on the RIME Framework which uses a developmental approach to describe resident performance along four stages of development (UNC School of Medicine, 2020). While this framework was originally developed for undergraduate medical training programs, it aligns well with post-graduate developmental expectations. We chose to base the guideline on the staged progression implicit in CBME because all programs at Queen's University implemented CBME in July 2017, and many programs in Canada, the USA, and Australia have already transitioned to or are about to launch CBME (Dagnone et al., 2017; Royal College of Physicians and Surgeons of Canada, 2019b).

Table 3: Resident performance expectations by level/stage of training


Expectations (UNC School of Medicine, 2020)

Red Flags, including but not limited to (Wolfram, 2020)

Transition to Discipline

The resident can…

  • accurately and reliably gather clinical information on each of their patients
  • communicate clearly (both verbally and in writing) the clinical information they have obtained
  • distinguish important information from unimportant information and can focus data collection and presentation on central issues

1. Inability to accept ambiguity or avoidance of thinking beyond rigid or concrete frameworks leading to inaccurate diagnosis and management

2. Avoidance or lack of self-reflection

3. Clinical knowledge or skills clearly below what is expected for the stage of training

4. Avoidance of feedback, including not collecting assessments

Foundation of Discipline

The resident can…

  • identify problems independently and to prioritize problems, including new problems, as they arise
  • to develop a differential diagnosis independently and to make a case for and against each of the important diagnoses under consideration for a patient's central problem(s)
  • help/guide other residents in clinical and administrative tasks

1. Inability to interpret multiple data sources to make a reasonable diagnosis and management plan

2. Lack of demonstrable reflective practice and self-directed learning

3. Clinical knowledge or skill clearly below what is expected for the stage of training

4. Inappropriate application of clinical decision tools or guidelines


Core of Discipline

The resident can…

  • develop and defend a diagnostic and a therapeutic plan for each of their patients' central problem(s)
  • utilize their growing clinical judgment to decide when action needs to be taken
  • analyze the risk/benefit balance of specific diagnostic and therapeutic measures based on an individual patient's circumstances
  • function as a manager for some of their patients in the latter part of this stage

1. Clear lack of intuitive problem solving or pattern recognition

2. Inability to manage ambiguity or new problems

3. Inability to see patients promptly

4. Greater than usual number of patients and/or staff dissatisfied with care delivered by the resident

5. Trouble delegating tasks, leading care team, avoids teaching opportunities

Transition to Practice

The resident can…

  • identify important questions to research in more depth and has the drive to seek out evidence, and the skills to scrutinize the quality of evidence
  • share in educating the rest of the team
  • self-direct and reflect on areas to enhance knowledge and skills


1. Clinical reasoning that is not at the level of an independently practicing physician

2. Lack of a clear learning plan for ongoing professional development

3. Inability to manage the clinical and administrative duties of a physician within a reasonable time frame (for example, clinics, dictations, etc.)

4. Difficulty leading the care team, assigning responsibilities, resolving conflict

5. Inappropriate staff or patient interaction


Assessment Considerations

Whether patient care is delivered in person or virtually, the goal is the same and, thus, we modified existing assessments for various Competencies/EPAs/tasks in our program to suit the virtual context (Table 1). The Neurology training program at Queen’s University is currently using the national Competence By Design (CBD) EPAs (competencies/tasks), milestones, and assessments outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC) so we had existing templates to work from (Royal College of Physicians and Surgeons of Canada, 2020d). The milestones required by the RCPSC were not changed but, instead, additional milestones were added, as needed, to better assess the nuances of virtual care.

As suggested in Phase 4 of our resident virtual clinic training program, intuitive integration of knowledge and skills into professional practice comes after months to years of experience; thus, we included a self-reflection component to our training program where residents assess their perceived successes and potential areas of growth (Miller, 1990). This not only permits them to consider their growing competencies, but it also provides active training in the cognitive processes of assessing one’s performance, a vital skill in a self-regulated profession (Duffy and Holmboe, 2006).

As a supervisor/attending providing feedback or completing workplace-based assessments on residents, consider the guidelines laid out in Table 3. These guidelines are intended to inform the assessor of the knowledge and skills that are appropriate for the resident's current stage of training. They are also framed in such a way as to facilitate feedback that coaches for improvement toward the desired performance and expectations of the stage. This feedback will not only inform the resident where they performed well but will also generate clear next steps for further improvement. The "red flags" were adopted from the key indicators established by the Department of Family Medicine at Queen's University, Kingston, Ontario, Canada and are meant to aid assessors in identifying the behaviours and/or knowledge gaps in need of review by the Program Director, Academic Advisor/Primary Reviewer, and/or Competence Committee. Red flags indicate behaviours that deviate greatly from expectations such as patient safety incidents or significant vulnerabilities in knowledge or clinical skills. Red flags should be addressed promptly to ensure future behaviour or knowledge vulnerabilities can be managed appropriately. We recommend that red flags and the actions taken to address them are documented clearly to track recurrence or support decisions such as implementing a modified learning plan or remediation, if needed.

Resident Experiences to Date

There are three Neurology residents (including GB) currently completing virtual epilepsy clinics. Review of clinical encounters (by LBL, RA, AW) revealed that all relevant aspects of clinical care during an epilepsy clinic such as the physical exam, seizure frequency monitoring, antiepileptic drug (AED) compliance, side effects, dose titration, review of results of EEG and imaging could be done successfully by residents in the virtual environment. Virtual visits by residents are also being completed in General Neurology, Multiple Sclerosis, Neuromuscular, and Movement Disorder clinics. Resident assessment data is still being compiled but has been organized in such a way that we will be able to establish if there are performance variations between in-person and virtual visits among residents.


While the COVID-19 pandemic prompted an immediate need for residents to participate in virtual clinics, the need has always been present and will continue to play an important role in healthcare delivery. Due to the novel nature of virtual care, carefully planned measures need to be put in place to support residents while ensuring patient safety and effective health care delivery. By providing clear objectives (competencies/EPAs/tasks), direct observation, and ongoing measurement (assessments), residents can be positive contributors in this much-needed service while ensuring they develop the virtual care skills needed for future independent practice. A future version of this paper will highlight our ongoing quality improvement efforts as well as compare resident performance in in-person and virtual clinics based on the assessment data collected.

Take Home Messages

  • There will be an ongoing need/demand for virtual care after the COVID-19 pandemic
  • A structured approach to education and training is needed to ensure patient safety and that residents receive timely and effective feedback
  • A modified physical exam protocol is necessary to support virtual care
  • Coaching and assessment practices need to align with the competencies outlined by certifying colleges (e.g. the Royal College of Physicians and Surgeons of Canada)

Notes On Contributors

Lysa Lomax is an Assistant professor of Neurology at Queen’s University, Kingston, Ontario, Canada. She is also the Neurology Program Director at Queen’s University. Her clinical areas of expertise include Epilepsy and Sleep Medicine. She has been practicing Telemedicine and Virtual care for many years and has extensive experience in this area.

Lindsay Crawford is the Educational Consultant for the Department of Medicine at Queen’s University, Kingston, Ontario, Canada, supporting 14 post-graduate medical training programs in the development, implementation, and ongoing refinement of competency-based medical education. Her research interests include instructional design and assessment, in particular, how to provide meaningful learning experiences coupled with effective feedback. 

Andy Wong is a PGY-4 Neurology resident at Queen’s University, Kingston, Ontario, Canada within the Division of Neurology. He was also the chief resident during the implementation of virtual neurology clinics for residency. He is also the Neurology resident lead in the Queen’s University Competency Based Medical Education resident subcommittee.

Gayathiri Balasubramaniam is a PGY-3 Neurology resident at Queen’s University, Kingston, Ontario, Canada within the Division of Neurology. She has been an active participant in the development and implementation of a Neurology resident-led virtual clinic.

Ramana Appireddy is an Assistant professor of Neurology at Queen’s University, Kingston, Ontario, Canada and a Stroke neurologist. He has extensive clinical experience in Telestroke and Virtual Care. He leads a multidisciplinary virtual care research group and is funded by multiple external and internal peer-reviewed funding and has publications in this area.


Table 2 was created by the authors and was based on information available from Canadian Medical Association and Department of Family Medicine, Queen’s University (Canadian Medical Association 2019a; Wolfram, 2020).


American Telemedicine Association (2020) About Telemedicine. Available at: (Accessed: 04/04/2020).

Andres, E., Talha, S., Zulfiqar, A. A., Hajjam, M., et al. (2018) 'Current Research and New Perspectives of Telemedicine in Chronic Heart Failure: Narrative Review and Points of Interest for the Clinician', J Clin Med, 7(12, 544.

Appireddy, R., Khan, S., Leaver, C., Martin, C., et al. (2019) 'Home Virtual Visits for Outpatient Follow-Up Stroke Care: Cross-Sectional Study', J Med Internet Res, 21(10), p. e13734.

Canadian Medical Association (2019a) Virtual Care: Recommendations for scaling up virtual medical services: Report of the Virtual Care Task Force. Canadian Medical Association. Available at: (Accessed: 16/04/2020).

Canadian Medical Association (2019b) Virtual Care in Canada: Discussion paper, CMA Health Summit. Available at: (Accessed: 16/04/2020).

Centers for Disease Control and Prevention (2020) Guidance for Outpatient and Ambulatory Care Settings | CDC. Available at: (Accessed: 16/04/2020).

Dagnone, D., Stockley, D., Truelove, A. H., McEwen, L., et al. (2017) 'Building Capacity for CBME Implementation at Queen’s University', MedEdPublish, 6(1).

Dreyfus, S. E. and Dreyfus, H. L. (1980) in A five-stage model of the mental activities involved in directed skill acquisition. Berkley: University of California (Berkley).

Duffy, F. D. and Holmboe, E. S. (2006) 'Self-assessment in lifelong learning and improving performance in practice: physician know thyself', JAMA, 296(9), pp. 1137-9.

Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., et al. (2010) 'Competency-based medical education: theory to practice', Medical Teacher, 32(8), p. 8.

Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., et al. (2015) 'Implementation of competency-based medical education: are we addressing the concerns and challenges?', Med Educ, 49(11), pp. 1086-102.

Holmboe, E. S. (2015) 'Realizing the promise of competency-based medical education', Acad Med, 90(4), pp. 411-3.

Lougheed, T. (2019) 'Time to embrace the promise of virtual health care', CMAJ, 191(11), pp. E320-E321.

Majersik, J. J. and Reddy, V. K. (2020) 'Acute neurology during the COVID-19 pandemic: Supporting the front line', Neurology. Jun 2020, 94 (24) pp 1055-1057.

Miller, G. E. (1990) 'The assessment of clinical skills/competence/performance', Academic Medicine, 65(9 Suppl), pp. S63-7.

Ontario Telemedicine Network (2019) Video visits enhance access to neurology follow up care. Available at: (Accessed: 01/09/2019).

Pourmand, A., Ghassemi, M., Sumon, K., Amini, S. B., et al. (2020) 'Lack of Telemedicine Training in Academic Medicine: Are We Preparing the Next Generation?', Telemed J E Health. 27(1), pp 62-67.

Public Health Agency of Canada (2020) COVID-19 Pandemic Guidance for the Health Care Sector - Available at: (Accessed: 28/04/2020).

Royal College of Physicians and Surgeons of Canada (2016a) Entrustable Professional Activities (EPA) Fast Facts, (Accessed: 13/04/2020).

Royal College of Physicians and Surgeons of Canada (2019b) CBD Launch Schedule. Available at: (Accessed: 13/04/2020).

Royal College of Physicians and Surgeons of Canada (2020c) CanMEDS 2015 Guide. Available at: (Accessed: 17/04/2020).

Royal College of Physicians and Surgeons of Canada (2020d) Entrustable Professional Activities Guide for Adult Neurology. Available at: (Accessed: 07/04/2020).

Royal College of Physicians and Surgeons of Canada (2020e) What is competence by design? Available at: (Accessed: 22/07/2020).

Shaw, J., Jamieson, T., Agarwal, P., Griffin, B., et al. (2018) 'Virtual care policy recommendations for patient-centred primary care: findings of a consensus policy dialogue using a nominal group technique', J Telemed Telecare, 24(9), pp. 608-615.

Steth IO (2020) The Smartphone Stethoscope. Available at: (Accessed: 16/04/2020).

UNC School of Medicine (2020) RIME Framework | Medicine Clerkship (Inpatient). Available at: (Accessed: 14/04/2020).

Volterrani, M. and Sposato, B. (2019) 'Remote monitoring and telemedicine', Eur Heart J Suppl, 21(Suppl M), pp. M54-M56.

Waldman, G., Mayeux, R., Claassen, J., Agarwal, S., et al. (2020) 'Preparing a neurology department for SARS-CoV-2 (COVID-19): Early experiences at Columbia University Irving Medical Center and the New York Presbyterian Hospital in New York City', Neurology. May 2020, 94 (20) 886-891.

Wechsler, L. R. (2015) 'Advantages and limitations of teleneurology', JAMA Neurol, 72(3), pp. 349-54.

Wolfram, B. (2020) 'Remote Resident Supervision Guidelines for Community Preceptors', Queen's University. Kingston, Ontario, Canada. Available at: (Accessed: 10/04/2020).




There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (

Ethics Statement

Ethics approval was not needed for the design of this program but an ethics application will be submitted to the Queen’s University (Kingston, Ontario, Canada) and Affiliated Hospitals Health Sciences Research Ethics Board (HSREB) for the continuing, part 2 of this paper that examines resident performance data.

External Funding

This work is supported by PSI Foundation, Ontario, Canada. Dr. R. Appireddy reports grants from CIHR, PSI Foundation, Ontario, Canada Health Infoway, Department of Medicine, Queen's University, SEAMO, KGHRI during the conduct of the study.


Please Login or Register an Account before submitting a Review