Undergraduate medical education (UME) has traditionally relied on a time-based model with multiple summative assessments resulting in a letter grade after the completion of a time requirement (Holmboe et al., 2010). The movement to competency-based medical education (CBME) represents a significant shift towards a more holistic focus on learner progression based on readiness rather than time. This provides more complete support for the spectrum of learners, including early identification of struggling learners so that individualized academic plans can be developed, to early graduation of learners progressing at a rapid pace (Ross et al., 2018; Carraccio et al., 2016). CBME necessitates that educators commit to extensive assessments. One framework for CBME in UME is the AAMCs 13 core Entrustable Professional Activities (EPAs), a series of physician tasks which students are expected to be proficient at by graduation.

Workplace-based assessments (WBAs), obtained during the routine care of patients (Swanwick and Chana, 2009; Miller, 1990) is one way to accomplish CBME evaluation.  This approach allows for both context-based assessment and real-time feedback allowing learners to reflect upon and make immediate change to their practice (Sabey and Harris, 2012; Hicks et al., 2018; Hast Prins, Gjedde Brønd, Malling, 2019; Norcini and Burch, 2007). Challenges to WBA include a lack of programmatic and workplace integration of assessment, and hesitancy of students to interrupt patient flow and request faculty time to perform assessments (Saedon et al., 2012). Most studies of WBA have been conducted during residency training (Hicks et al., 2018; Kogan et al., 2017) or outside the United States (Hast Prins, Gjedde Brønd, Malling, 2019; Wilkinson et al., 2008; Mathew et al., 2018; Barrett et al., 2016). Furthermore, the majority of prior WBA efforts focus on a single discipline rather than programmatic wide implementation.

In our efforts to move toward CBME, we implemented a health system-wide WBA online data collection application across the entire undergraduate medical education program. We utilized the EPA framework with the goal of providing additional assessments for graduation. The purpose of this paper is to describe the experience of WBA development and implementation in an academic health center.

Educational Setting

Oregon Health & Science University (OHSU) is a 522 bed teaching hospital, biomedical research facility and level 1 trauma center that serves as the safety net hospital and academic health center for the state of Oregon and southwest Washington. The YourMD competency-based integrated medical school curriculum at OHSU launched in 2014 and replaced the previous traditional two years pre-clinical and two years clinical-discipline based curriculum. Woven throughout the curriculum are eighteen “threads” ranging from ethics, informatics, communications, and safety to professionalism and clinical reasoning. The curriculum is divided into two phases: Foundations of Medicine (FOM) and the Clinical Experience Phase (CEP). 

Throughout the FOM curricular phase, students undergo multiple low and high stakes assessments consistent with CBME. Assessment data are available to students, their coaches, and the assessment team using a secure password-protected portfolio system with a viewing portal. Once students complete the first phase of the curriculum and pass Step 1, they advance to the CEP.  Similar to the FOM, multiple points of assessment are undertaken, including midterm formative comments, knowledge-based quizzes, history and physical evaluations, presentations, self-reflections, and procedure logs, all of which are included in the portfolio system. Direct, real-time assessment data were not systematically collected before WBA implementation.

Implementation Team

WBAs were based on the 13 AAMC EPAs, and two groups were tasked with overseeing EPA development and implementation: the Core EPA Pilot Committee and the Entrustment Group. Members of the Core EPA Pilot Committee were responsible for representing OHSU in the AAMC national EPA pilot and developing recommendations from their concept groups, which included leaders in faculty development, curriculum and assessment, and entrustment as well as student and resident liaisons. The Entrustment Group was charged with developing processes and procedures to review student data to make decisions about EPA competency attainment. It was comprised of UME and graduate medical education (GME) leaders selected by application. Key UME staff were integrated into both groups. All faculty members of the Core EPA Pilot and Entrustment Committee were tasked with faculty development efforts and received funding (0.1 to 0.15 FTE) to support their work.

Development and Use of the WBA Online Tool

The WBA tool was designed as a student-driven system that would be easy to use and minimally interrupt clinical care. This resulted in a smart phone or tablet-based application (app) that could be handed to an evaluator for completion (Appendix 1). The app was built using Qualtrics^XM survey software (Seattle, WA). The Qualtrics app had the added benefit of being functional while offline, so WBAs could be collected with or without access to internet, and available on all user platforms. We were further able to program custom functions (such as a signature line) into the survey. This software was also licensed to OHSU for IT and Qualtrics support.

Students initiate assessments using the WBA app on their phone or tablet and are prompted to choose the desired EPA for assessment, followed by the clinical discipline, and the evaluation setting (e.g., inpatient). Students then select assessor type (faculty member, resident, fellow, other). The students hand their device to the clinical assessor who is prompted to select their involvement in the activity was using a modified version of the Ottawa Co-Activity Scale (Gofton et al., 2012). Finally, the evaluator is prompted to type or dictate one or two specific comments about how the student could improve, then the WBA is digitally signed, submitted, and uploaded to the server. Once submitted, students cannot change the feedback or score; a signature line and confirmatory email help verify legitimacy.

Submitted WBAs can be viewed via the same secure password-protected portfolio described earlier. The Research and Evaluation Data for Educational Improvement (REDEI) system was developed by a team of educational researchers and leaders in conjunction with a software programmer at OHSU. REDEI is a relational database that uses Structured Query Language (SQL) to display data. Data from multiple assessment systems, including Qualtrics^XM, is exported into REDEI using Application Programming Interfaces (API). Students, coaches, advisors, and administrative leaders can see learner performance metrics relative to the cohort mean for all assessments using the personalized REDEI viewing portal.  The REDEI system and all data collection processes have been reviewed and approved by OHSU’s Institutional Review Board (IRB#10873).  In addition, a Certificate of Confidentiality protects data in the REDEI System from forced disclosure.

Curriculum Integration

WBA requirements were implemented step-wise into the medical school curriculum to increase acceptability and to ease the transition for current students and faculty. As new students matriculate, EPAs and WBAs are emphasized as larger components of their pre-clinical and clinical education. The number of WBAs required to graduate increases depending on the year of matriculation. For the class described in this report, they were required to complete 50 WBAs before graduation.

To further ensure students gain competency in all EPAs, students are required to complete at least two WBAs for each EPA, while encouraged to complete as many WBAs in as many of the EPAs as possible to demonstrate competency. Each core clerkship also adopted two EPAs of focus. Students target WBA completion to specific clerkship-emphasized EPAs. WBAs can be completed by any supervisor and in any clinical environment, including both clinical cores and electives. Requirements starting with the class of 2022 include that at least 50% of the WBAs must come from attendings as opposed to residents, fellows, or other clinical staff.

Pre-clinical students are first introduced to WBAs in a course titled “Introduction to Preceptorship.” Students receive a two-hour, large group lecture facilitated by faculty members of the EPA committee. EPAs and WBAs are introduced conceptually, students practice using the Qualtrics app, troubleshoot technical challenges, and simulate WBAs with fellow students using pre-defined scenarios (Appendix 2). WBAs are reintroduced prior to the start of clinical rotations in a large group, two-hour long lecture that is similar in scope and content to the lecture provided in the pre-clinical years.

Faculty Development

Faculty development efforts followed recommendations published from the Core EPA Pilot (Favreau et al., 2017), and focused on creating a shared mental model for entrustment for UME educators. Didactic presentations and interactive workshops were held at University and Departmental levels detailing the Core EPAs and principles of trust. Within these presentations, WBAs and directions about how to use the smart phone application were described. To extend our message, we utilized a “train the trainer” approach. Core clinical experience directors (clerkship directors), curriculum committee leaders, and coaches were trained and encouraged to underscore the importance of WBAs at departmental faculty meetings and meetings with trainees. These efforts were tracked to ensure all departments were reached.

Core EPA and Entrustment Committee members also produced educational materials, which were distributed electronically throughout the University including our off-site practice locations. These included a narrated presentation, didactic presentations, workshops and a one-page tip sheet. The workshops and videos allowed for deliberate practice of WBAs and encouraged discussions about entrustment. Core EPA Pilot Members provided presentations at every GME orientation to capture the residents and fellows.

We also relied on students to assist with faculty development, recognizing it would be difficult to reach every clinical supervisor. Student workshops were held at orientation, and prior to preceptorship and formal clinical experiences. Included were practical tips to improve reciprocal feedback, and pocket cards describing how to solicit and receive feedback developed by students at Vanderbilt University, another Core EPA Pilot site, were distributed.

Data Analyses

A total of 3,599 WBAs were submitted by 148 students of the 2021 graduating class during the first 12 months of their clinical experiences. Of these, 31 (<1%) did not include either an EPA or level of assessment and were excluded, leaving 3,568 assessments for inclusion in analyses.  Though this cohort officially includes 155 students, upon which the demographic characteristics are based, seven students (4.5%) had left the cohort to work on masters or other doctorate degrees as part of their program.  Descriptive statistics, including frequencies, percentiles, means, standard deviations, and ranges (minimum, maximum counts), were calculated.

The average age of medical students included in this study was 29 (24-41 years). The majority were female (54.8%), white (69.7%), not of Hispanic origin (92.9%), single (81.9%) and were not parents (96.8%) (Table 1).

3,568 WBAs were completed. The mean number of EPAs recorded per student (Table 2) was lowest for EPA 10 (Recognizing a Patient Requiring Urgent or Emergent Care and Initiate Evaluation and Management) (mean=0.36; range 0-4; n=53) and was highest for EPA 6 (Provide an Oral Presentation for a Clinical Encounter) (mean=5.46; range 1-17; n=803).

The mean number of total EPA assessments per student was 24.27 (range 1-103). 100% of students in this cohort completed at least one WBA. In addition, the mean number of minutes it took to complete the assessments was 2.7 minutes with a standard deviation of 1.2 minutes (n=2,803) (data not shown). Figure 1 shows the distribution of the EPAs according to frequency.

Figure 1: Frequency of Assessments According to EPA

Figure 1, 2, and Table 4 EPA Legend: 

1. Gather a Hx and Perform PE 
2. Prioritize a Differential Dx Following a Clinical Encounter
3. Recommend and Interpret Common Diagnostic and Screening Tests
4. Enter and Discuss Orders and Prescriptions 
5. Document a Clinical Encounter in the Patient Record
6. Provide an Oral Presentation of a Clinical Encounter 
7. Form Clinical Questions and Retrieve Evidence to Advance Patient Care
8. Give or Receive a Patient Handover to Transition Care Responsibility
9. Collaborate as a Member of an Interprofessional Team
10. Recognize a Patient Requiring Urgent or Emergent Care and Initiate Evaluation and Management
11. Obtain Informed Consent for Tests and/or Procedures
12. Perform General Procedures of a Physician 
13. Identify System Failure and Contribute to a Culture of Safety and Improvement

EPA 6 (Provide an Oral Presentation of a Clinical Encounter) was most frequently assessed and EPA 10 (Recognize a Patient Requiring Urgent or Emergent Care and Initiate Evaluation and Management) was least frequently assessed. Figure 2 illustrates the level of supervisor involvement according to EPA. 

Figure 2: Level of Supervisor Involvement According to EPA

Supervision Level 4 (I was available just in case) was most commonly recorded for all EPAs with a range of 44.6% for EPA 12 (Perform General Procedures of a Physician) to 83.9% for EPA 13 (Identify System Failure and Contribute to a Culture of Safety and Improvement). Level 3 supervision (I directed them from time to time) was second most common with a range of 11.5% for EPA 13 (Identify System Failure and Contribute to a Culture of Safety and Improvement) to 40.6% for EPA 2 (Prioritize a Differential Dx Following a Clinical Encounter). Level 2 supervision (I talked them through it) ranged from 1.4% for EPA 6 (Provide an Oral Presentation of a Clinical Encounter) to 16.5% for EPA 12 (Perform General Procedures of a Physician), and Level 1 (I did it myself) ranged from 0.3% for EPA 2 (Prioritize a Differential Diagnosis Following a Clinical Encounter) to 25.9% for EPA 11 (Obtain Informed Consent for Tests and/or Procedures). Table 3 shows students’ WBAs according to clinical discipline and setting. 

Assessments were submitted by 19 distinct clinical disciplines. Family medicine assessments predominated in the outpatient clinic (n=583 or 58.8% of all assessments completed in the outpatient setting). Internal medicine assessments were common in inpatient setting (n=626 or 31.7% of all assessments done in inpatient settings). Neurology (n=95) and pediatrics (n=54) predominated in ICU, while surgery (n=33) was more common in the emergency department. Table 4 shows the EPAs according to core clinical disciplines with the grey shading indicating EPAs planned to be a focus of that clerkship.

Planned assessments ranged from 1.1% for psychiatry and EPA 11 (Obtain informed consent for tests and procedures) to 26.3% for OB/GYN and EPA 6 (Provide oral presentation of a clinical encounter). WBAs were obtained nearly exclusively in the core clinical experiences (93.5%). Figure 3 shows EPA submission dates, starting in March of 2019, peaking in October of 2019, and ending in March of 2020 when medical students were removed from clinical sites due to the COVID-19 pandemic. 

Figure 3: Dot Plot of EPA Assessment Submission Date

In this report, we describe the implementation of WBA for all 13 AAMC EPAs across an institution. Utilizing an online smartphone app, we developed, integrated, and disseminated use of WBAs according to an EPA framework across a wide variety disciplines and settings. With sufficient faculty and student development, WBAs can be collected at scale. This adds to what has been previously demonstrated on a smaller scale in individual clerkships (Fazio et al., 2018). 

Our students were able to complete a large variety of EPAs during their clinical experience. Not surprisingly, learners more often solicited gathering a history and physical exam, documentation in the electronic medical record, and giving an oral presentation as these skills are highly emphasized in clinical training. The least commonly assessed WBAs focused on handoff, informed consent, quality/safety, and initiating urgent/emergent care treatment. These are less likely to be emphasized or tracked by assessors in core rotations in our curriculum and the lack of WBA attainment in these areas may indicate curricular and/or assessment gaps. EPA 10 (initiating urgent/emergent care treatment), was the only EPA not tied a core clerkship requirement, potentially lowering its assessment frequency. Furthermore, this cohort of students had not completed their ICU/ED rotations where this task would be more commonly encountered.

We noted fewer reports of WBAs scored at level 1, 2 compared to level 3, 4. We believe this occurs because students are more likely to request an assessment when they feel well prepared. However, we plan to further investigate this trend as it could also represent other entities, such as characteristics of the assessor.

Nineteen different clinical disciplines assessed students through this inaugural WBA implementation, demonstrating breadth of uptake in varied settings, potentially aided by faculty and student development. This was likely further supported by linking specific EPAs, and therefore WBAs, to specific clerkships.  This focused students and assessors on specific skills during a given rotation and may have increased uptake of some of the less common EPAs. This will need to be elucidated in future research.

After an initial period of student and faculty education, we noted that the rate of WBA completion was relatively stable week-by-week over the subsequent 6 months. In the early fall, the requirements for WBAs changed and we saw a decrease in completion rates. This suggests student-initiated assessments are associated with medical school expectations.

We did note several issues related to the technical implementation of this tool, which warrant further discussion. To accommodate supervisors who did not want to use a mobile phone to complete the evaluation, access to the survey was also provided via web link. Students voiced concerns about privacy on their personal mobile phones, such as if notifications appeared when a supervisor was filling out an evaluation. To address this, we recommended they use either ‘Airplane’ or ‘Do not disturb’ mode to prevent notifications from appearing. We also considered liability issues if a personal phone was dropped or broken while used for this purpose, and the SOM developed a fund to replace phones broken during a WBA and not covered by insurance.

Strengths of this study include our ability to successfully implement a WBA app across clinical training in an entire medical school and the resultant large numbers of WBAs for analysis.  However, this study reflects the experience of a single institution and may not be generalizable to other institutions. In addition, results of a single year from a single cohort of students were included, and so lacks benefits of more years of data collection as students progress. There may be bias in the timing of student collection of WBAs and choice of assessors leading to a “good job” grade rather than a true reflection of competency. We have very little data for several EPAs (particularly 10, 11 and 13). The WBAs may showcase existing parts of our curriculum and should be considered within context of the entire spectrum of competency-based education. Further research is still needed to more fully characterize the utility and timing of WBAs in medical education.

Implementation of WBA according to the EPA framework is possible within a cohort of students in a medical school-based health system. We plan to report on the contributions of WBA to decisions around entrustability in EPA performance, academic advancement, and graduation.