Research article
Open Access

An Evaluation of Implicit Bias Training in Graduate Medical Education

Utsha Khatri[1][a], Amy Zeidan[2][b], Michael LaRiviere[1], Sarimer Sanchez[3], Lauren Weaver[1], Jenny Lynn[1], Frances Shofer[1], Barbara Todd[1], Jaya Aysola[1]

Institution: 1. Hospital of the University of Pennsylvania, 2. University of Kentucky, 3. MGH/Harvard
Corresponding Author: Dr Amy Zeidan ([email protected])
Categories: Educational Strategies, Teaching and Learning, Research in Health Professions Education
Published Date: 23/05/2019


Objective: Mounting evidence reveals that health care disparities stem from a combined effect of structural bias within our health system and the unconscious bias of well-intentioned health care professionals. The authors designed and evaluated a novel educational intervention to introduce the concept of unconscious bias to front-line providers.


Methods: The authors designed and implemented an educational curricula for providers from three different programs at a single large, urban, tertiary-care academic institution. The intervention consisted of participants taking the implicit attitudes test (IAT), which was followed by a facilitated discussion.  The discussion was audio recorded, transcribed, and coded for emerging themes. An online survey assessed participant awareness of these topics before and after the intervention and was analyzed using paired t-tests.


Results: The authors analyzed the results by focus group. There were 19 participants in Focus Group 1 (FG1), 6 in Focus Group 2 (FG2), and 42 in Focus Group 3 (FG3). The majority of participants were white, between the ages of 26 to 35 and female. When analyzed in aggregate, authors found a statistically significant improvement in self-reported domains on whether the intervention changed participant understanding of healthcare disparities and implicit bias. While the authors’ qualitative results indicated varying acceptance of the implicit attitude test, most participants acknowledged that implicit bias exists.


Conclusion: Our educational intervention was successful at engaging front-line clinicians on the role of implicit bias on the development of health care disparities. However, prior to scaling our intervention to other institutions, many improvements should be considered including session structure, choice of facilitator and introduction of strategies to mitigate biases in clinical practice.


Keywords: Implicit Bias; Healthcare Disparities; Interprofessional Education; Graduate Medical Education


Mounting evidence reveals that health care disparities stem from a combined effect of structural bias within our health system and the unconscious bias of well-intentioned health care professionals (Smedley et al., 2003)(Hall et al., 2015). A study by Green et al. found that among internal medicine and emergency medicine residents, significant pro-White bias exists despite no explicitly reported preference for Whites over Blacks (Green et al., 2007). Another study by Sabin et al. examined data from Harvard’s Project Implicit website and found that the 2,535 website participants reporting an MD degree demonstrated significant pro-White bias (Sabin, Rivara, and Greenwald, 2008).


Social scientists assert that such unconscious attitudes and stereotypes acquired through socialization can be unlearned, or inhibited, by countervailing influences (Dovidio et al., 2008). Yet, there are limited studies to date that evaluate the use of countervailing influences in medical education.  One study demonstrates that dedicated implicit bias curricula for undergraduate, medical, and physician assistant (PA) students raised learners’ awareness of their implicit biases (Archambault et al., 2008). However, to date there lacks formal training in implicit bias in graduate medical education (GME) and training for residents, fellows, and advanced practice providers (APPs).  In addition, we are unaware of prior assessments of residents’ or APPs’ perceptions on whether implicit bias affects their own patient care. 


To address this critical gap, we designed and evaluated an educational intervention with the three objectives.  First, to empower residents and APPs, to develop and implement implicit bias curricula for their peers.  Second, to utilize the implicit attitudes test (IAT) to introduce the concept of unconscious bias and encourage self-reflection.  Lastly, to engage residents and APPs in a facilitated discussion of implicit bias, how these biases affect medical care, and what can be done to combat bias.


Intervention Development and Design

The study team consisted of two Emergency Medicine residents, one Internal Medicine resident, one Radiation Oncology resident, two Advanced Practice Providers (APPs), and three senior advisors, all of whom were members of the University of Pennsylvania Graduate Medical Education Housestaff and APP (HAP) Quality Council, an organizing body of residents and APPs focused on patient safety and quality. The team engaged key stakeholders in an iterative process to develop an educational intervention for residents and APPs that focused on understanding health care disparities and the role of implicit bias. The study team engaged residents and APPs across all specialties to determine areas of interest and gaps in knowledge.  Ideas were generated, presented to residents/APPs and revised based on continuous feedback from all stakeholders at the HAP Quality Council monthly meeting.


The study population selected included three health professional training programs based at a large academic and urban tertiary care hospital serving a diverse patient population. The participating programs were in-part self-selection as interested members of the HAP quality council sought approval from their respective program leadership. The three programs included: Internal Medicine Residency Program, Radiation Oncology and APPs. There were 19 participants in the Internal Medicine Residency session, 6 in the Radiation Oncology session, and 42 in the APP session. Facilitators were selected faculty mentors who had expertise in implicit bias training and healthcare disparities. The facilitator for each session was distinct from the study team and considered a participant. 


The session structure consisted of a one-hour facilitated discussion. The session started with participants completing the IAT on Race on their cellular phone or laptop computer.

Instructions were provided to participants at the start of the session, detailed the link to access the IAT.  After completion of the IAT, groups participated in a 45-minute facilitated discussion on implicit bias and its potential effects on clinical care. The study team adapted a previously operationalized facilitator guide (Baylor College of Medicine Facilitator Guide) and met with facilitators ahead of time to review the guide and study protocol and revise as needed.  The facilitator guide (See Supplementary File 1) included ground rules for participation, questions for discussion, and outlined the specific learning objectives below:   


  1. Discuss what implicit bias is and identify how it may affect clinical care, specifically in their specialty
  2. Understand the difference between health care disparities and health disparities and how implicit bias relates to one or both of these terms
  3. Discuss potential ways of mitigating provider implicit bias in the clinical setting

At the start of the session, participants were required to review a statement of research and thereby sign-in to opt into taking the IAT test, participating in the discussion, and completing the post-session survey.  The statement of research described that discussions would be audio-recorded, observed, and transcribed by The University of Pennsylvania Mixed Methods Research Laboratory (MMRL).  Two trained experts from the MMRL observed each session to evaluate for findings that would be difficult to interpret from recorded comments. Participants were made aware of this prior to the sessions. Given the potential for sensitive data collection and the need to ensure confidentiality during each session, written documentation was not obtained. In addition to trainee participants, all facilitators provided consent to participate in this study, where facilitated discussions were audiotaped, transcribed, and analyzed for emerging themes


Data Sources

Quantitative Data Collection

After the session, we provided participants a link to a voluntary anonymous post-session survey via the University of Pennsylvania secure REDCap™ database. The study team developed the survey to assess the participants’ interest on the topic, how the session affected their knowledge/perceptions of implicit bias, and feedback on session logistics and implementation. The survey consisted of five Likert scale questions (1=never, 2=rarely, 3=sometimes, 4=often, 5=always or strongly disagree=1 to 5=strongly agree) that assessed respondent perceptions before and after the session. It also included eight evaluation questions, including two open ended questions, to assess participant interest and feedback of the session. In addition, the survey included demographic questions. When possible the study team adapted questions previously operationalized surveys or prior work in this area.


Qualitative Data Collection

The discussion sessions were observed and audio recorded by The Penn Mixed Methods Research Lab (MMRL) trained experts observed and took notes on the facilitated discussion sessions.  In addition, they facilitated audio recorded the sessions and subsequent transcription of those recorded sessions.


Statistical Analysis

Quantitative Data Analysis

We first analyzed frequencies and percentages for categorical variables and means with standard deviations for continuous variables.  Second, to determine differences in respondent perceptions of before and after the session, we conducted a 2-factor analysis of variance by focus group and time (pre/post). We performed post-hoc Tukey Kramer tests on all pairwise comparisons.  All analyses were performed using SAS statistical software (Version 9.4, SAS Institute, Cary NC).


Qualitative Data Analysis

The Penn Mixed Methods Research Lab (MMRL) observed and audio recorded the three facilitated discussions. Recordings were transcribed, coded, and analyzed using a content analysis framework; deriving codes from a thorough reading and understanding of the data and employing a summative content analysis approach to key themes identified. All transcripts were coded by two coders with a final inter-rater reliability of 0.9962. Inter-rater reliability was derived from the two coders double coding one transcript, which constituted 25% of the data source. Due to inter-group differences noted in this analysis, each group’s data was analyzed independently. The top themes from each group were summarized, along with limitations of the analysis and potential directions for future discussions on implicit bias.


Demographics of Survey Participants

There were 19 participants in Focus Group 1 (FG1), 6 in Focus Group 2 (FG2), and 42 in Focus Group 3 (FG3). The majority of participants regardless of group were between the ages of 26 to 35 and female.  Of the FG2 and FG3 participants, 50.0% and 81.0% self-identified as non-Hispanic White respectively in contrast to FG1 participants, where only 21.0% self-identified as non-Hispanic White. (Table 1)


Table 1. Demographic of Participants


Focus Group 1

Focus Group 2

Focus Group 3


N (%)

N (%)

N (%)









0 (0)

0 (0)

3 (7.1)




11 (57.9)

1 (16.7)

15 (35.7)




2 (10.5)

4 (66.7)

12 (28.6)




1 (5.3)

0 (0)

5 (11.9)




0 (0)

0 (0)

3 (7.1)




0 (0)

0 (0)

3 (7.1)




5 (26.3)

1 (16.7)

1 (2.4)









5 (26.3)

1 (16.7)

4 (9.5)



Black /African American

1 (5.3)

0 (0)

3 (7.1)




1 (5.3)

0 (0)

0 (0)




3 (15.8)

0 (0)

0 (0)




4 (21.1)

3 (50)

34 (81)




5 (26.3)

2 (33)

1 (2.4)









10 (52.6)

3 (50)

36 (85.7)




4 (21.1)

2 (33)

4 (9.5)




5 (26.3)

1 (16.7)

2 (4.8)


Years of Experience

3.1 (3.03)

3.8 (0.84)

5.7 (8.4)



Prior knowledge of health care disparities and the IAT

Participants who completed the survey were asked a series of questions to assess their prior knowledge of health care disparities and the amount of healthcare disparities training they had received. In this section we describe the results of the surveys that were administered to the three groups. 


In all three groups, the majority of participants characterized their prior knowledge of healthcare disparities as “somewhat knowledgeable” (FG1 57.9%, FG2 83%, FG3 78.6%. (Table 2).


Prior to this session, 68.4% of FG1 and 71.4% of FG3 respondents reported receiving formal education on healthcare disparities while 66.6% of FG2 respondents reported no prior formal education on healthcare disparities. Of the respondents that reported receiving formal training, 82% responded with examples in a free text question. Formal education ranged from training during college to medical/NP/PA/Graduate school to during graduate medical training. Prior training often occurred as part of a lecture, an elective selected based on personal interest or as a special session during a seminar or conference focused on diversity.


Prior to the intervention, most respondents had not taken an IAT (FG1 68.4%, FG2 66.6%, FG3 85.7%). Of those who had taken an IAT, most had taken the “race” IAT. After taking the IAT test, nearly all participants expressed being slightly or very interested in their results. (Table 2)


Table 2. Prior Knowledge of Healthcare Disparities


Focus Group 1

Focus Group 2

Focus Group 3


N (%)

N (%)

N (%)


Prior knowledge of health disparities






Very knowledgeable

1 (5.3)

0 (0)

4 (9.5)



Somewhat knowledgeable

11 (57.9)

5 (83.3)

33 (78.6)



Very little knowledge

7 (36.8)

1 (16.7)

 4 (9.5)



No knowledge

0 (0)

0 (0)

1 (2.4)


Prior formal education on healthcare disparities

13 (68.4)

2 (33.3)

30 (71.4)


Have you taken the IAT prior to this session

6 (31.6)

2 (33.3)

6 (14.3)



Educational Intervention: Knowledge and Attitudes Before and After

Participants who completed the survey were asked a series of questions to determine how the session changed their understanding/perception of healthcare disparities and how implicit bias influences care. (Figures 1a-e) FG1 showed statistically significant improvements in four of the five domains we assessed. The training thus improved how often participants think about health care disparities (p = 0.006), how often disparities occur in their care (p = 0.02), how frequently they think about their implicit bias (p <0.0001), and that knowledge of the IAT results will influence their care (p <0.0001). FG2 (n=6) demonstrated modest improvements that were not statistically significant but notably had a smaller sample size.


Figure 1: Pre and Post Session Perceptions


Knowledge and Attitudes Before and After the Educational Intervention

There was heterogeneity among the groups with respect to themes that arose during discussions.  FG1 and 3 shared skepticism of the validity of the testing format and results. Specifically, FG3 participants expressed confusion as well as angst about the outcome of the test. They suggested the test could be manipulated to obtain the desired result but also were not surprised by their test results. One participant suggested that as a result of their skepticism, the test result wouldn’t change their behavior.


All focus groups suggested that implicit bias exists and may affect patient care. FG1 participants saw a complicated relationship between bias, pattern recognition and treatment offered to patients. Pattern recognition can be important and is often reinforced. However, some patterns may strengthen biases with negative outcomes if not challenged. FG2 challenged the idea that implicit biases must always have an impact on treatment or care, but did agree that, left unchecked could result in worse care. FG2 suggested that implicit bias could apply to other factors in addition to race, such as income, class and age. FG3 participants commented on the negative potential impacts of implicit bias on care to patients, especially if they felt dissimilar to patients or couldn’t identify with the patient and/or situation.


Unique themes arose during FG1 and FG2 group discussions. FG1 participants struggled with their own internal biases, feeling frustrated or embarrassed when realizing there is dissidence between an individual’s bias and reality. FG2 suggested that implicit bias exists on a personal and systemic level and there are possibly ways to mitigate bias at both levels.


Implementation Suggestions

From open-ended survey questions, observational and qualitative data we solicited valuable feedback from participants on implementation strategies. Participants were asked to provide suggestions on the logistics and content of the sessions as well as to provide recommendations for subsequent sessions. We have detailed a summary of those findings below and in Table 3 with representative quotes. 


Table 3. Focus Group Themes: Qualitative Findings on Discussions of Bias


FG1 Quote


FG2 Quote

FG3 Quote

Response to Test

“I went into it with the preconception that I could predict what my outcome would be and that I would be able to go faster when I saw  black associated with good and be able to compensate and make sure that I would have a preference for black over white.”






“…it made me nervous because I felt like I was going to- I have a conceived notion of myself  I guess. And I was concerned about what the outcomes would show.”


Implicit Bias Effects

“Well, I think biases are just like our brain trying to put together random facts and trying to make a pattern, and that’s what we do as physicians – pattern recognition.  And I think the problem with biases is sometimes these pattern recognition processes are wrong, but a grand majority of our pattern recognition processes are correct.  That’s why we’re able to function as humans and as physicians


“I am a little more interested in the sort of income disparities and class disparities than I am racial disparities...I think we tend to downplay class very often…I think that class has a much bigger impact.”

“Oh, I think it's about…how you identify.  So you may be more sympathetic or empathetic to one – race, gender or what have you.  And you may not be able, despite your best efforts, to do that to something that you potentially don't identify with.  And that could jeopardize care if you're not able to appropriately communicate with that person.”

Unique Themes

“…feelings of frustration and embarrassment are recurrent feelings that you feel over and over again as a provider when you realize the dissidence between your bias and reality.”


“I think that's where like the policies and procedures aren't going to address those micro situations.  There's classic things that I think you talked about before about the amount of pain medications that people get in the ER.  And that's like a – that's going to be hard to regulate on any type of macro level.  But some people might just be more – an individual provider might be just changing their prescribing habits differently based on race and without realizing it.”






Participants from FG1 and FG3 recommended smaller group sizes to facilitate an environment for more candid discussion on a sensitive topic. Other recommendations centered on choosing an appropriate size room and seating configuration to facilitate interactive discussions.  Participants felt that AV equipment during the sessions made the discussion unnatural and advised we remove it in the future. Additionally, participants recommended more detailed, step-by-step instructions for taking the IAT. 


Facilitators recommended that residents have protected time away from patient care duties during the session to allow them to fully engage. Most participants thought the session length was just right (64%). A small portion of participants thought the length was too long (14%), although notably, 19% of participants did not answer this question



Overall, the majority of participants rated the session as good to excellent (79%) and participants largely felt that the content covered was appropriate. One suggestion provided was to expand the content material to include strategies to mitigate bias.


Implementation Feedback and Future Recommendations

Table 4 outlines key feedback we received on implementation.  Participants in the focus groups recommended including clinical or case based scenarios to provide realistic examples of implicit bias. Additional recommendations included reviewing journal articles on disparities, role playing during sessions, and implementing hospital wide group discussions around implicit bias. Facilitators recommended a priming activity prior to the session to improve participant engagement and contributions.


Table 4. Implementation Feedback




Observed Data/Qualitative Data


Smaller group sizes

Improve seating arrangements

Eliminate observers and microphones

Improve instructions on IAT

Allow computer use for IAT

Smaller group sizes

Eliminate microphone

No patient care during sessions (pagers) 

Avoid theater or auditorium setting Utilize focus group model

Avoid use of AV equipment


Training on how to overcome/mitigate biases

Information on institution specific disparities

Include bias towards providers

Include practical ways to mitigate bias

Provide strategies in facilitator guide


Include strategies on how to address and combat bias

Future Recommendations

Clinical scenarios with standardized patients

Review of journal articles on disparities

Hospital wide group discussions

Case based scenarios

Role playing

Priming activity

Include scenarios pertinent to practice type and/or location

Ensure facilitator does not dominate conversation

Ensure facilitator is comfortable with audience and setting 


Our mixed methods evaluation revealed that the novel educational intervention was successful in exposing front-line clinicians in training to the existence of their personal implicit bias and encouraging them to explore how it affects the delivery of patient care. The novelty of this intervention stems from the fact that it was developed and implemented for clinicians in training by clinicians in training, with feedback from their peers on how to best incorporate this sensitive topic into clinician education.  


A key finding of this intervention was a participant-reported increase in self-awareness of biases and therefore, served to meet a key recommendation that the intervention raise awareness of healthcare disparities among trainees.  While our qualitative results indicated varying acceptance of the implicit attitude test, most participants acknowledged that implicit bias exists. Although studies have demonstrated pro-White biases among healthcare providers, recent work question whether implicit bias affects clinical care (Dehon et al., 2017). Interestingly, in our study, participants did describe clinical situations in which they felt their implicit bias affected the care they provided, both positively and negatively. Further, the majority of participants expressed interest in receiving additional training on implicit bias suggesting the educational intervention successfully engaged frontline clinicians on the topic of implicit bias and health care disparities.


Over the last several decades, there has been growing evidence to support that implicit bias in clinician decision-making may perpetuate health care disparities. Developing interventions to reduce the impact of provider biases is therefore paramount (Chapman, Katz, and Carnes, 2013). While other studies have measured rates of implicit bias among residents and have used the IAT as an educational intervention among medical students (van Ryn et al., 2015), to our knowledge, no study has measured the effectiveness of the IAT as an educational intervention to raise awareness of implicit bias among medical residents and APPs.


Prior studies have demonstrated the value of utilizing the IAT as a method of introducing medical students to the concept of unconscious bias. While beginning such work early in medical training has great potential value, one could argue that encouraging “graduate-level” clinicians, such as residents and APPs - those who may be relying on their biases while they diagnose and treat patients - is of even greater value. In fact, in a study by Haider et al. that looked at unconscious race and social bias with vignette-based clinical assessments by medical students, authors found that while the majority of first-year medical students consistently demonstrated an implicit preference for White patients, when asked to make decisions in vignette-based clinical assessments, there was no association with the students’ implicit preferences and their clinical assessments (Haider et al., 2011). This is contrary to the many studies that have revealed the strong association between provider bias and poorer patient outcomes. Many factors, including the relative lack of fatigue, time pressure and cognitive load burden, may be protective in shielding medical students from being influenced their biases when compared with resident and attending physicians. The relaxing of work-hour restrictions and a return to overnight in-house call, and their accompanying fatigue, may further uncover underlying biases. As Haider et al. concluded, while initiating implicit bias training during medical school has great value, continuing such training at the physician-level/APP trainee-level could be an important intervention point to reduce disparities in health care. Our study sought to target the clinicians who are likely to be most susceptible to acting on their biases. 


As Teal et al. describe, the process of becoming aware of one’s implicit or unconscious bias is fluid, and not every learner has the same baseline insight (Teal et al., 2010). Thus, it is reasonable to conclude that interventions such as ours should be continued throughout the training of a resident or APP, rather than delivered as a one-time intervention. Additionally, as a one-time intervention, we cannot comment on the long-term effects these sessions have on patient care and provider interest regarding issues related to implicit bias. This study also does not describe nor track the department-specific initiatives or further sessions this intervention may have sparked.



Our study should be interpreted with the recognition of a number of limitations. First, each session had a different facilitator who guided the conversations to varying degrees. To reduce this variation, the evaluation team session provided facilitators with verbal instruction on the goals of the session and a written facilitator guide.  Additionally, there was inherent heterogeneity in the structure of the sessions due to differences in group size, room size, and participant seating, all of which may have contributed to varying degrees of participant comfort and participation.


Participants were explicitly instructed not to reveal their individual IAT results. This raises the possibility that participants had information on their biases that may have led them to either over- or under-participate in the discussion based on their acceptance of their results. Similarly, because participation in the discussion was optional, the themes that emerged may not be representative of the sentiments held by the participants. An additional option to express reactions in written form may have better captured the sentiment of the entire group.


Our findings drew exclusively from an educational intervention at a single academic institution and may not be generalizable.  We acknowledge that there may be an institutional culture that uniquely contributed to our results. However, we posit that our implementation lessons and thematic observations provide broadly applicable knowledge.  Additionally, our participants were from non-surgical specialties and predominantly self-identified as non-Hispanic white and female, which may limit our ability to extrapolate results to a more diverse group. Future studies should explore similar interventions with more demographically diverse groups and across other specialties, notably surgical specialties.


Educational Implications: Incorporation of implicit bias training in graduate education

The ACGME, through its Clinical Learning Environment Review (CLER) assessments, aims to measure how institutions engage residents on the reduction of health care disparities. It establishes the expectation that trainees and faculty receive education on identifying and reducing health care disparities and recommends that trainees be engaged in quality improvement activities that address health care disparities for vulnerable populations (ACGME, 2017). However, despite establishing this expectation, specific educational models through which to achieve these goals are limited.  While APP programs are not required to include such training, as frontline clinicians, this group would also benefit. This study provides a model through which other residency and APP programs can pilot and develop an intervention to create awareness of implicit bias on the individual level as a starting point to address health care disparities within the institution. Both the qualitative and quantitative results demonstrate that this intervention successfully met the goals of the study to empower the development of an end-user driven curriculum, to utilize the IAT tool to introduce individuals to their personal biases, and to engage residents and APPs on a discussion regarding the impact of implicit bias.


Research Implications: Strategies to Address and Mitigate Implicit Bias

The recognition that unconscious bias affects opportunities and outcomes for individuals is not limited to the field of medicine. The last two decades have resulted in implicit bias research in the areas of criminal justice, education, employment, and even professional sports (Sen, 2014)(Clark and Zygmunt, 2014)(Reeves, 2014)(Ingraham, 2014). In fact, in line with many police departments across the country, the U.S. Justice Department has gone as far as mandating implicit bias training for its 33,000 federal agents and prosecutors.  Despite these advances, strategies to address and mitigate implicit bias are limited.


As described by Stone & Moskowitz, a workshop on unconscious bias should also teach participants how to reduce the activation of their implicit bias (Stone and Moskowitz, 2011). As emerging research demonstrates, individuals can use their explicit processes to change and control their implicit responses. While the specifics of those described strategies are beyond the scope of this paper, it is important to note that existing research suggests that future interventions on raising awareness of implicit bias should include instruction on strategies to reduce the effect of bias on patient care. In one study, authors developed a 12-week longitudinal habit-breaking intervention that showed dramatic reductions in implicit race bias among participants as well as increases in concern about discrimination and personal awareness of bias over the study duration (Devine et al., 2012). Chapman et al. describe in their perspective piece, “As with any behavioral change, individuals need to become aware of their habitual engagement in an undesirable behavior and be provided with strategies to increase self-efficacy to engage in a new desirable behavior.” While our intervention has proven successful at increasing awareness, it was not designed to provide clinicians with instruction on how to combat the effects of their bias. Further research assessing implementation of and long-term effects of current strategies to mitigate bias is necessary. Defining metrics of assessment and implementation is challenging but will assist in successful incorporation of implicit bias training into residency education.


Our educational intervention was successful at engaging front-line clinicians on the role of implicit bias on the development of health care disparities. However, prior to scaling our intervention to other institutions, many improvements should be considered.  First, logistical improvements to the session structure were described by participants and facilitators. These include appropriate room size, smaller group sizes, and limited technical equipment. Second, choice of facilitator is important and challenging. Participants recommended facilitators who provided sufficient instruction without dominating the discussion. Finally, modifications to the curriculum should include instruction on how to combat the effect biases play in clinical practice. 

Take Home Messages

  • Front-line providers can be effectively engaged in training on the effect of implicit bias on health care disparities
  • Utilization of the validated IAT (Implicit Association Test) is a useful tool in introducing the concept of implicit bias to resident physicians and advanced practice providers
  • Future interventions should involve teaching providers how to mitigate the effect of biases in order to provide just and equitable care

Notes On Contributors

Utsha Khatri: Conceptualized and designed this study; developed and designed the tools for the acquisition of the quantitative data and analysis and interpretation; drafted the manuscript and performed critical revisions of the manuscript; provided final approval of the version to be published.

Dr. Khatri is an Emergency Medicine Resident at the Hospital of the University of Pennsylvania.


Amy Zeidan: Conceptualized and designed this study; developed and designed the tools for the acquisition of the quantitative data and analysis and interpretation; drafted the manuscript and performed critical revisions of the manuscript; provided final approval of the version to be published.

Dr. Zeidan is Clinical Instructor and Ultrasound fellow and the University of Kentucky. ORCID:


Michael LaRiviere: Conceptualized and designed this study, both the intervention described and the rigorous measurement and analysis; spearheaded the acquisition of data from the Radiation Oncology group and helped in the analysis and interpretation of the collected data; revised the drafted article critically for important intellectual content; provided final approval of the version to be published.

Dr. LaRiviere is Radiation Oncology Resident at the Hospital of the University of Pennsylvania.


Sarimer Sanchez: Assisted with implementation of the intervention in the Internal Medicine focus group; helped to identify and coordinate with the session’s facilitator; assisted in the acquisition of data from the APP group; reviewed the manuscript and provided feedback during the revision phase; provided final approval of the version to be published.

Dr. Sanchez is an Infectious Disease Fellow at MGH/Harvard


Lauren Weaver: Assisted with implementation of the session to the APP group; coordinated with the session facilitator; participated in the review of the manuscript and provided valuable feedback on both the qualitative and quantitative analysis of the data; provided final approval of the version to be published.

Lauren is critical care advanced practice provider at the Hospital of the University of Pennsylvania.


Jenny Lynn: Assisted with study structure, recruited APP team members, evaluated, and ultimately helped to review data that was collected as well as editing drafts of various dissemination information; reviewed and edited the manuscript; provided final approval of the version to be published.

Jenny is a surgical advanced practice provider at the Hospital of the University of Pennsylvania.


Frances Shofer: assisted with survey development as well as the collection and analysis of the data; provided all of the statistical analysis and interpretation of the quantitative results; created the figures that depict our results and reviewed the tables; reviewed the manuscript; provided final approval of the version to be published.

Dr. Shofer is the Director of Epidemiology and Biostatistics and an Adjunct Professor of Emergency Medicine at the Hospital of the University of Pennsylvania


Barbara Todd: assisted with implementation of the program in the APP group; provided financial and programmatic support to the project at the inception phase; reviewed the facilitator guide as well as the survey; assisted with data interpretation and analysis; reviewed and edited the manuscript; provided final approval of the version to be published.

Barbara Todd is the Director of the CMS Graduate Nurse Education (GNE) Demonstration Project, Senior Fellow at the Center for Health Outcomes & Policy and Adjunct Assistant Professor of Nursing, University of Pennsylvania School of Nursing.


Jaya Aysola: Assisted with defining the problem and conceptualizing the study design; provided guidance and feedback on every step of the pre-implementation process, including development of facilitator guide, training for facilitators and critical review of the survey content and design;  garnered the institution’s executive leadership support; provided critical feedback on the data analysis as well as the creation of figures and tables; contributed substantially to revisions of the article; provided final approval of the version to be published.

Dr. Aysola is an Assistant Professor of Medicine at the Hospital of the University of Pennsylvania Faculty Affiliate, Center for Health Incentives and Behavioral Economics (CHIBE), University of Pennsylvania, a Senior Fellow at the Leonard Davis Institute of Health Economics, the Associate Designated Institutional Official for Health Equity and Inclusion, Hospital of University of Pennsylvania, the Assistant Dean, Graduate Medical Education, Perelman School of Medicine and the Research Director, Office of Inclusion and Diversity, Perelman School of Medicine


The authors wish to thank our executive sponsors, Dr. Jennifer Myers, the Housestaff and APP Quality Council, the University of Pennsylvania Health System (UPHS) Graduate Medical Education office, Dr. Eve Higginbotham and Dr. Jeffrey Berns for their support on implementing this project. The authors would also like to thank Dr. Eve Higginbotham and the Office of Inclusion and Diversity and Dr. Barbara Todd for their financial support of the project. The authors would like to thank the Penn Mixed Methods Research Lab for their contributions to the qualitative analysis of the results. Lastly, the authors would like to thank Dr. Bernard Lopez, Dr. Janet A. Deatrick and Dr. Carmen Guerra for their roles in facilitating the sessions.


Archambault, M.A., Van Rhee, J.S., Marion, G., Crandall, S. (2008) ‘Utilizing Implicit Association Testing to Promote Awareness of Biases Regarding Age and Disability’ The Journal of Physician Assistant Education, 19(4), pp. 20-26.

Chapman, E.N., Kaatz, A., Carnes, M. (2013) ‘Physicians and Implicit Bias: How Doctors May Unwittingly Perpetuate Health Care Disparities’, J Gen Intern Med, 28(11), pp. 1504-1510.

Clark, P., and Zygmunt, E. (2014). ‘A Close Encounter with Personal Bias: Pedagogical Implications for Teacher Education’, J Negro Educ, 83(2), pp. 147-161.

CLER Evaluation Committee. (2017). ‘CLER Pathways to Excellence: Expectations for an Optimal Clinical Learning Environment to Achieve Safe and High Quality Patient Care, Version 1.1’, Accreditation Council for Graduate Medical Education.

Dehon, E., Weiss, N., Jones, J., Faulconer, W., Hinton, E., Sterling, S. (2017). ‘A Systematic Review of the Impact of Physician Implicit Racial Bias on Clinical Decision Making’, Acad Emerg Med, 24(8), pp. 895-904.

Devine, P.G., Forscher, P.S., Austin, A.J., Cox, W.T.L. (2012). ‘Long-term reduction in implicit race bias: A prejudice habit-breaking intervention’, J Exp Soc Psychol, 48(6), pp. 1267-1278.

Dovidio, J.F., Penner, L.A., Albrecht, T.L., Norton, W.E., et al. (2008). ‘Disparities and distrust: the implications of psychological processes for understanding racial disparities in health and health care’, Soc Sci Med, 67(3), pp. 478-486.

Green, A.R., Carney, D.R., Pallin, D.J., Ngo, L.H., et al. ‘Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients’ Journal of Gen Internal Medicine, 22(9), pp. 1231-1238.

Haider, A.H., Sexton, J., Sriram, N., Cooper, L.A. (2011). ‘Association of unconscious race and social class bias with vignette-based clinical assessments by medical students’, JAMA, 306(9), pp. 942-951.

Hall, W.J., Chapman, M. V., Lee, K.M., Merino, Y.M., et al. (2015). ‘Implicit Racial/Ethnic Bias Among Health Care Professionals and Its Influence on  Health Care Outcomes: A Systematic Review’, American Journal of Public Health, 105(12), pp. 60-76.

Ingraham, C. (2014). What the NBA can teach us about eliminating racial bias, The Washington Post. Available at:  (Accessed: January 31, 2019).

Reeves, A.N. (2014). Written in Black & White: Exploring Confirmation Bias in Racialized Perceptions of Writing Skills’, Yellow Paper Series.

Sabin, J.A., Rivara, F.P., Greenwald, A.G. (2008) ‘Physician implicit attitudes and stereotypes about race and quality of medical care’ Med Care, 46(7), pp 678-685.

Sen, M. (2014). ‘How Judicial Qualification Ratings May Disadvantage Minority and Female Candidates’, J Law Court, 2(1), pp. 33-65.

Smedley, B.D., Stith, A.Y., and Nelson A.R. (2003). ‘Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care’, Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities in Health Care.

Stone, J., and Moskowitz, G.B. (2011). ‘Non-conscious bias in medical decision making: what can be done to reduce it?’, Med Educ, 45(8), pp. 768-776.

Teal, C.R., Shada, R.E., Gill, A.C., Thompson, B.M., et al. (2010). ‘When best intentions aren’t enough: helping medical students develop strategies for managing bias about patients’, J Gen Intern Med, 25(2), pp. 115-118.

van Ryn, M., Hardeman, R., Phelan, S.M., Burgess, D.J., et al. (2015). ‘Medical School Experiences Associated with Change in Implicit Racial Bias Among 3547 Students: A Medical Student CHANGES Study Report’, J Gen Intern Med, 30(12), pp. 1748-1756.




There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (

Ethics Statement

IRB Approval was granted by the University of Pennsylvania Institutional Review Board, IRB# 826743, Titled: Implicit Association Testing for Residents & Advance Practice Providers.

External Funding

This article has not had any External Funding


Please Login or Register an Account before submitting a Review

Ken Masters - (15/08/2019) Panel Member Icon
An interesting paper on evaluating an implicit bias training intervention in graduate medical education.

While the study has been mostly well-conducted, there are some issues that need to be addressed:
• “within our health system” and references to “pro-White bias”. The geographical context of these statements is unclear. With the “our”, and the bias, are the authors referring to the whole world or (with a cautious inference based on the authors’ institutions), North America, or the USA, or part/s of the USA? The authors really do need to clarify this.
• The authors should also give some more information about the IAT. I (again, a cautious inference) am assuming that the authors are referring to the one at the Implicit project, but this is not stated explicitly in the paper. This should be clearly identified and the URL given.
• On these lines, the authors should also give some idea of the usage, credibility, generalisability and limitations of the IAT that they have used.
• Where figures are given in the text (especially where they are not given in the tables), they should be written as raw numbers, followed by the percentages in parentheses, not only as percentages.
• The focus of the study needs to be clarified more. On the one hand, the title and the intervention (Supplementary File) appear to be on bias in general. But the introductory comments on “pro-White bias” and the test that the participants took was not on bias in general, rather on racial bias. So, I partially wish to recommend that the authors alter their title to reflect “racial bias”, as that was the only bias tested. But the intervention appears to cover bias more broadly, although the focus was probably still on race, given that one of the participants specifically refers to the fact that the focus should not be on race, and that economic and class bias also plays a role, and many other biases (e.g. gender, religious, etc.) may also exist. Unless the authors are using racial bias as a proxy for all bias, but that would be dangerous territory and would need some discussion. This is really confusing. So, I would strongly recommend that this be teased out a little more.
• The authors mention a limitation that this study may not be generalisable outside of the institution or to other specialties. Given that the racial and gender groups were so heavily imbalanced, it might also not be even generalisable even within the specialities from which the participants were drawn. One way to reduce the uncertainty is to find the overall gender and race composition of the specialties from which the participants were drawn, and then statistically compare them to the sample. This may not be possible at this late stage, but, if possible, would give clarity on this issue.

Minor issues:
• In Table two, the last question should be phrased as a statement, otherwise the reader has to refer back to the text to determine whether that number and percentage reflected is “Yes” or “No” to the question.

So, an interesting study, but the paper does require clarity on several issues.

Possible Conflict of Interest:

For Transparency: I am an Associate Editor of MedEdPublish

Felix Silwimba - (23/05/2019)
I have enjoyed reading this research report. although its contents and settings are the first world healthcare practice I still find it relevant to third world healthcare. really do we discuss implicit biases and healthcare biases even though they are prevalent. I recommend this report to all medical educators .
Trevor Gibbs - (23/05/2019) Panel Member Icon
I very much enjoyed reading this well written and well structured paper about a very difficult and sensitive issue. I must admit that I had to read through the paper twice for a full understanding, but I think that reflects my lack of knowledge in the area discussed rather than any problem with the paper.
I like the way that the authors felt that their methodology required modification and were not prepared to puss ahead regardless and were honest in their limitations section.

I think that this is a piece of research that needs to be followed to make sure that any change as a result of the intervention is long lasting

I would recommend this paper to all those engaged in the area, although saying that I feel the content interjects with many teaching and learning activities and health development.
Possible Conflict of Interest:

For transparency purposes, I inform others that I am the Associate Editor of MedEdPublish