Assessment of the Intrinsic CanMEDS Roles in Diagnostic Radiology Residents using an Objective Structured Clinical Assessment ( OSCE )

Purpose: Non-Medical Expert or Intrinsic CanMEDS Roles, as outlined by the Royal College of Physicians and Surgeons of Canada, can be difficult to evaluate in an objective and specialty-specific manner. This study investigates an Objective Structured Clinical Examination (OSCE) evaluation tool to assess these competencies for Diagnostic Radiology Residents. Methods: A five station CanMEDS OSCE was developed for postgraduate year 3 and 4 Residents to evaluate the Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional Roles. Performance was assessed by postgraduate year 5 Residents using standardized scoring rubrics. CanMEDS OSCE scores were correlated with American College of Radiology (ACR) scores and Medical Expert OSCEs. Results: Seventy Residents in three separate cohorts (n=21, 26, 23) participated in the CanMEDS OSCE. Mean station scores were consistent across cohorts. In general, one-way ANOVAs showed no effect of postgraduate year on station scores. There were no significant correlations between CanMEDS OSCE scores and ACR exam scores or CanMEDS OSCE scores and Medical Expert OSCE scores, demonstrating divergent construct validity. In turn, this indicates construct validity for the CanMEDS OSCE by demonstrating that unique competencies are being measured. In contrast, there was a correlation between ACR exam scores and Medical Expert OSCE scores, confirming that these both assess the same construct. Conclusions: An OSCE can be a useful assessment tool to assess Intrinsic CanMEDS Roles in a specialty-specific manner. Correlational analyses indicate that unique competencies are being evaluated that are not captured by other traditional assessment means. Probyn L, Lang C, Finlay K, Herold J, Bartlett E, Glover Takahashi S MedEdPublish https://doi.org/10.15694/mep.2016.000068 Page | 2


Introduction
Residency programs are required to evaluate their Residents in the Intrinsic CanMEDS competencies as outlined in the CanMEDS 2005 Physician Competency Framework by the Royal College of Physicians and Surgeons of Canada (RCPSC) (Frank, 2005).The Intrinsic CanMEDS Roles include Communicator, Collaborator, Manager, Health Advocate, Scholar and Professional.The Intrinsic Roles are inherent to the successful practice of medicine, and together with the Medical Expert Role, encapsulate global physician competence (Sherbino, et al., 2011).Residency program objectives for each CanMEDS Role have also been outlined in a specialty-specific manner, including for Diagnostic Radiology ("Objectives in Training", 2000).Similarly, in the United States, the Accreditation Council of Graduate Medical Education (ACGME) has outlined core specialty-specific clinical competencies that need to be integrated into the Diagnostic Radiology residency curriculum along with medical knowledge ("ACGME Program Requirement", 2016).These competencies are interpersonal and communication skills, patient care, systems-based practice, practice-based learning and improvement, and professionalism ("ACGME Program Requirement", 2016).
The establishment of the CanMEDS Roles and ACGME competencies follows the larger global shift from timebased to competency-based medical education (Nasca, et al., 2012).This shift aims to establish defined milestones that integrate various competencies together to support holistic professional development at every stage of a trainee's medical education (Swing, 2010).This has resulted in the subsequent need for the development of reliable assessment methods to determine when a candidate has reached these milestones for each of the Roles or competency domains.
Currently, the traditional assessment tools used by residency and medical education programs, such as multiplechoice tests, can be adequate to assess knowledge-based competencies, such as those encompassed by the Medical Expert Role.However, Intrinsic Roles can be much more difficult to evaluate, especially in a specialty-specific manner.The current in-training evaluation reports (ITERs) used to make these assessments are subjective and can be based exclusively on the discretion of rotation supervisors.Other potential disadvantages of ITERs include small sample size of observations, variation in evaluator leniency, time between observation and assessment leading to recall bias, and assessment of clinical skills being influenced by the evaluator's own clinical skills (Kassam, et al., 2014;Kogan, et al., 2010).
The objective structured clinical examination (OSCE) provides a supplementary method of assessment by simulating clinical encounters with standardized cases and coworkers (Harden, et al., 1975).The basic OSCE format involves candidates rotating around a series of stations that have predetermined marking schemes and time allotment.This design allows for direct observation and highly specific feedback of Intrinsic Roles, which is not necessarily facilitated by other, more indirect evaluation methods, such as ITERs or multi-source feedback (MSF) assessment tools.
The development of formal assessment methods for the Intrinsic Roles is mandatory but can prove challenging for many specialties, therefore this article will describe an OSCE assessment tool that was developed to formally evaluate the Intrinsic CanMEDS Roles for Diagnostic Radiology Residents and provide steps to implement OSCE stations for different specialties or subspecialties.As well, CanMEDS OSCE scores from three iterations of the biennial exam were examined.The analysis focus was twofold: to determine the relationship between the CanMEDS OSCE scores and the postgraduate year (PGY) of the Residents who achieved them, and to determine the relationship between CanMEDS OSCE scores and assessment methods used to assess Residents in the Medical Expert Role, specifically the American College of Radiology (ACR) exam and a Medical Expert OSCE.

Methods
In order to address the challenge of assessing the Intrinsic CanMEDS Roles, a five station OSCE was developed for Diagnostic Radiology Residents with the following objectives: to provide an assessment method that would engage learners as both examinee and examiner, to provide a formal assessment tool to assess Resident performance in a specialty-specific manner and to provide formative feedback for Residents.

CanMEDS Examination Development:
A small group of academic radiologists assembled to create clinical scenarios that occur in day-to-day practice in radiology (Table 1).These scenarios were selected to cover a broad range of Intrinsic CanMEDS Roles as defined in the source document from the RCPSC (Frank, 2005;"Objectives in Training", 2000).

Table 1. CanMEDS OSCE Station Design for Diagnostic Radiology Residents
The examination was designed for PGY-3 and PGY-4 Residents.PGY-5 Residents were involved in the assessment and feedback process of the OSCE.PGY-5 Residents took part in a briefing session prior to the OSCE implementation to provide them with the necessary information, training, and allow them an opportunity to ask any questions.By facilitating the learning of their peers, these senior Residents were enhancing their skill set for the Scholar Role.This design also assisted with the feasibility of the OSCE, and created an opportunity for formal peer assessment using standardized scoring rubrics.
Feedback and Assessment: Following the completion of the OSCE, a debriefing session was held to take up the exam.This provided immediate feedback and highlighted the important educational content for each station and reinforced the specialty-specific content.Each PGY-3 and PGY-4 Resident was also provided with an individual performance summary.The PGY-5 Residents assigned to each station participated in the debriefing process.
Steps to implement in your program: Evaluating the Intrinsic CanMEDS Roles can be challenging for many specialties, including radiology subspecialties, such as nuclear medicine, neuroradiology, and pediatric radiology.In order to implement a CanMEDs OSCE in your program, you must first select which Roles you are interested in evaluating.Then, you must design station content specific for your specialty.Finally, in order to maximize the objectivity of the assessment, you must develop standardized templates for each station, including instructions to Residents, scripts for standardized patients and healthcare professionals, answer sheets, and assessment criteria and forms.Where possible, senior Residents should participate in the exam to close the loop on the Scholar Role, which also assists in the logistics of delivering the exam.

Medical Expert Examinations:
The American College of Radiology (ACR) exam is a stand-alone test that aims to help Residents, faculty, and program directors to monitor trainee progress throughout their residency training (Monticciolo, 2010).The ACR exam covers 11 clinical subspecialty areas (breast, chest, cardiac, gastrointestinal, genitourinary, interventional, musculoskeletal, neuroradiology, nuclear medicine, pediatrics, ultrasound), and has a physics and general competencies section (Monticciolo, 2010).The general competencies section aims to add focus to the 6 general competencies outlined by the ACGME that mirror the CanMEDS Roles and are common to all medical specialties ("ACGME Program Requirements", 2016; Monticciolo, 2010).The ACR exam format consists of image-based questions, non-image based questions and single-best-answer multiple-choice questions (Monticciolo, 2010).ACR performance is reported as within-PGY percentile rank, with 1 being the lowest and 100 being the highest score.
The Medical Expert OSCE, which is administered by the Diagnostic Radiology residency program approximately 4 times a year, is similar to the ACR exam in that is aims to monitor trainee progress throughout their training.The Medical Expert OSCE usually consists of 11 clinical subspecialty sections (breast, cardiac, chest, emergency radiology, gastrointestinal/genitourinary, head and neck, vascular interventional radiology, musculoskeletal, nuclear medicine, obstetric ultrasound, pediatrics).The Medical Expert OSCE format consists of mostly image-based questions with some non-image based questions, often asking for the most likely diagnosis for an image and/or clinical history or testing anatomy knowledge.

Data Analysis:
The CanMEDS OSCE data was analyzed to determine (1) the relationship between the Resident's PGY and their CanMEDS OSCE scores, and (2) the relationship between the CanMEDS OSCE scores and the scores from exams assessing the Medical Expert Role, namely the ACR exam and a Medical Expert OSCE.
A one-way Analysis of Variance (ANOVA) was used to determine the effect of postgraduate year on station scores.Correlational analyses were used to determine the relationship between the CanMEDS OSCE scores, the ACR exam scores and the Medical Expert OSCE scores.Specifically, Pearson correlation coefficients were computed to examine the relationship between the CanMEDS OSCE and the Medical Expert OSCE scores, and Spearman's rank correlation coefficients were used for CanMEDS OSCE and ACR exam scores.P-values of less than 0.05 were taken to indicate statistical significance.All analyses were carried out in IBM SPSS Statistics for Windows (SPSS, IBM Corporation, Version 20).
This study was approved by the Research Ethics Board at the University of Toronto.

Results
A total of 70 Resident exam scores were analyzed from the 2009 (n=21), 2011 (n=26) and 2013 (n=23) administrations of the CanMEDS OSCE.Exam scores for each station were fairly consistent between exam administrations (Table 2).

Table 2: Comparison of CanMEDS OSCE scores across years
There was no effect of postgraduate year on CanMEDS station scores.The exceptions to this were the 2009 Manager station (ANOVA, p=0.046), the 2011 Safety station (ANOVA, p=0.029) and the 2011 Professional station (ANOVA, p=0.030).
There was no correlation between the CanMEDS OSCE scores and the ACR scores (Spearman correlation, -0.279 ≤ ρ ≤ 0.373).As well, there was no correlation between the CanMEDS OSCE scores and the Medical Expert OSCE scores (Pearson correlation, -0.114 ≤ r ≤ 0.362).Interestingly, there was a correlation between the ACR scores and Medical Expert OSCE scores (Spearman correlation, 0.471 ≤ ρ ≤ 0.508).

Discussion
It has been notoriously challenging for residency programs to meaningfully evaluate the Intrinsic CanMEDS Roles in radiology.The purpose of this article was to describe the development of a formal assessment method for a Diagnostic Radiology residency program to evaluate the Intrinsic CanMEDS Roles in a specialty-specific manner.Also, CanMEDS OSCE scores were analyzed to determine if there was an effect of PGY on exam score and if there was a correlation between CanMEDS OSCE scores and traditional assessment methods, specifically the ACR exam and a Medical Expert OSCE.
Our analysis indicated that there was no effect of Resident postgraduate year on CanMEDS OSCE scores.Intrinsic CanMEDS Roles are often taught in an indirect, non-linear way, and it is expected that adjacent postgraduate years will exhibit similar Intrinsic CanMEDS competency levels (Neira, et al., 2013).As a result of this, it may be advantageous to assess the Intrinsic CanMEDS Roles between a larger span of postgraduate year training, for example between PGY-2 and PGY-4 Residents.
As well, correlational analysis indicated that there was no significant relationship between CanMEDS OSCE scores and ACR scores or CanMEDS OSCE scores and Medical Expert OSCE scores.This divergent validity between the CanMEDS OSCE and the traditional assessment methods demonstrates that the CanMEDS OSCE is evaluating a unique construct, thereby confirming its validity.In contrast, there was a correlation between ACR scores and Medical Expert OSCE scores, confirming that these both assess the same construct.These findings indicate that Medical Expert competencies and Intrinsic CanMEDS competencies are not necessarily acquired together.It also highlights the importance of the development of specific assessment tools for the assessment of Intrinsic CanMEDS Roles separate from the assessment tools for the Medical Expert Role.
The OSCE format has been shown to be both reliable and valid (Harden, et al., 1975;Sherbino, et al., 2008;Bandiera, et al., 2006).This assessment format is also advantageous since it provides control over the patient and examiner variables, and allows the exam to be repeated with high fidelity (Harden, et al., 1975).Achieving an adequate level of standardization for an OSCE assessment can be time consuming, however the creation of an OSCE station bank with standardized instruction and assessment sheets can facilitate the easy set-up of future OSCE administrations.
Another advantage of the OSCE format is that case complexity and rarity can be controlled (Harden, et al., 1975).This can be of particularly utility in residency training as Residents may not encounter certain clinical situations often, and as a result, there can be a wide range of CanMEDS competency levels when dealing with these complex or rare clinical scenarios.This is consistent with our finding that the CanMEDS OSCE scores from the Health Advocate station varied with postgraduate year in the 2011 administration (ANOVA,p=0029).This station evaluated the Residents ability to advise a patient regarding radiation safety upon discovering that they were pregnant soon after x-ray imaging had been taken.Residents may have limited experience dealing with these less common scenarios, and the CanMEDS OSCE provides a direct assessment method for these specific competencies.Interestingly, in a survey of residency program directors from across Canada, it was revealed that they were least satisfied with the assessment of the Health Advocate Role (Chou, et al., 2008).This further highlights the potential utility of the CanMEDS OSCE as examiners can control case content to specifically and directly evaluate Roles that may not necessarily occur on a regular basis during day-to-day practice.
The main limitation of the CanMEDS OSCE is that it is time and resource intensive, and as such may not seem like a feasible assessment method.Our design has managed this issue by incorporating PGY-5 participation into the examination, acting as standardized patients, health professionals and examiners.This minimizes the cost of acquiring standardized patients and health professionals, and is advantageous as the assessment and teaching competencies generally are not included in CanMEDS OSCEs (Jefferies, et al., 2007).Through the PGY-5 residents' involvement as examiners during the CanMEDS OSCE and as teachers during the debriefing session, this exam design additionally functions to assess the Scholar Role for the PGY5s.
Finally, the CanMEDS OSCE is an effective summative assessment tool that also generates objective, formative feedback of the Intrinsic Roles that is generally difficulty to gather.Formative feedback identifies areas for improvement as well as grading trainee performance (Gunderman, 2012).Radiology education has been largely focused on summative assessments that occur at the end of a period of training, such as ITERs, and that function by attempting to determine what score a Resident should receive (Gunderman, 2012).However, the CanMEDS OSCE, which we designed with an end-of-day take-up session, allows time for immediate verbal feedback regarding each Resident performance.This also allows the program director, and the PGY-5 Residents, to highlight key general learning points for the resident group and respond to trends, issues or questions that may have arisen throughout the day.
Our CanMEDS OSCE has proven particularly beneficial as it provided a formal assessment method for nontraditional content, and engaged Residents in competency Role awareness and education.The use of multiple assessment methods to gauge the Intrinsic CanMEDS competencies is favourable over the use of a single tool, such as a sole ITER (Kassam, et al., 2014).We must continue to explicitly teach and assess the Intrinsic CanMEDS Roles, including the particularly challenging Roles such as Health Advocate and Scholar, and create appropriate assessment methods to help Residents improve their performance.We have formally implemented the CanMEDS OSCE into our residency program design as a biennial exam.Thus far, we have received highly positive feedback across the postgraduate years, and were able to modify CanMEDS station content to facilitate an OSCE for nuclear medicine, neuroradiology and pediatric radiology Residents.