Research article
Open Access

Analysis of OBGYN Resident In-Training Exam Scores After Implementation of a Flipped Classroom Curriculum

Ana Valente[1], Rajiv Gala[1], Beverly Rueb[1]

Institution: 1. Ochsner Clinic Foundation
Corresponding Author: Dr Ana Valente ([email protected])
Categories: Curriculum Planning, Research in Medical Education, Postgraduate (including Speciality Training)
Published Date: 01/05/2018

Abstract

Background: There has been a push for flipped classroom use in graduate medical education in recent years although a paucity of objective data exists to support its' superiority in resident and fellow education. During a "flipped classroom" learners are responsible for mastering material prior to scheduled didactic time and are challenged to apply acquired knowledge during the session.

Objective: We hypothesized that there would be a statistically significant increase in OBGYN Residents' in-training exam scores with implementation of a flipped classroom compared with previous traditional classroom.

Methods: We implemented a pilot flipped classroom curriculum and performed an ANOVA analysis to determine if a statistically significant difference existed in OBGYN in-training exam scores for years using flipped (2012-2014) vs. traditional classroom (2015-2016).

Results: When using p < 0.05 as significance level, there was a significant improvement in in-training exam scores when comparing first year (P=.0468) and second year (p=.0018) residents in flipped vs. traditional classroom. Statistical significance was lost after controlling for USMLE score, which might be due to the small sample size.

Conclusions: There was a trend towards improvements in OBGYN in-training exam scores when using the flipped classroom model, but statistical significance was lost when controlling for USMLE scores.

Keywords: Flipped Classroom; OBGYN Resident Education; CREOG Exam; In-training Exam

Introduction

The traditional lecture-based educational model has persisted in American education since the early 1900s (Freeman et. al, 2014). Recent data in educational literature suggests this is not the most effective learning strategy (Freeman et. al, 2014) (McLaughlin et al., 2014). There is a push for active learning where educators present learners with questions that challenge them to use knowledge they should attain prior to entering the classroom (McLaughlin et al., 2014).   This model is known as the “flipped classroom” and its’ use and acceptability is rapidly expanding in medical education. (Prober and Heath, 2012) (Tan et. al, 2015). On literature review, there is a paucity of objective data to support the use of the flipped classroom in graduate medical education. We found no studies that evaluated the effects of the flipped classroom on OBGYN (Obstetrics and Gynecology) residency in-training exam scores.

In our OBGYN residency program we implemented a flipped classroom as our primary educational tool used during weekly scheduled two-hour didactic time starting in July 2014. We hypothesized that in-training CREOG (Council of Resident Education in Obstetrics and Gynecology) exam scores of residents in the flipped classroom would increase significantly compared with scores of residents in the previous traditional classroom.

Methods

We created a flipped classroom curriculum that covered a variety of OBGYN topics. As part of the flipped classroom, residents were e-mailed recorded lectures, textbook chapters and other reading assignments (i.e. ACOG practice bulletins) one week before the scheduled didactic time. The expectation was that they would review and master the content prior to entering the classroom. The scheduled didactic session was two hours. It was structured in the same way each week and was lead by the same attending physician. Upon entering the classroom, residents were immediately quizzed using an interactive online app.  The quiz was 10 questions and took approximately 15 minutes. Results were revealed immediately. Following the quiz, residents broke up in to groups of 5. In these groups of 5, they were assigned a series of cases related to the topic to work through as a team. The case-series part of the session lasted approximately 1 hour and 15 minutes.  During this time, the instructor circled the room and helped to facilitate discussions in the groups. The last 15 minutes of the session were used for all groups to come together and engage in an interactive discussion to highlight their discoveries as a small group and share ideas about what they believed to be the most testable topics covered during the session.

The weekly flipped classroom was used for eighteen months (July 2014-January 2016) and spanned the time of two annual in-training examinations.  Of the residents included in the analysis (n=25), 10 never experienced the traditional method of teaching.  Therefore, we performed an ANOVA to analyze the difference between CREOG scores among the PGY 1 and 2s taught using traditional methods (2012-2014) and those with flipped classrooms (2015-2016). Significance level was set at p <.05.  Residents were excluded if they attended less than 75% of the flipped classroom sessions.  We also used USMLE Scores to control for residents who may be good test takers at baseline. Institutional IRB approval was granted by the Ochsner Clinic Foundation IRB #2015.293.  All subjects consented to participate.

Results

Comparison of PGY-1 Scores:

Twenty-five subjects had taken the CREOG test as first year residents therefore the comparison of first-year CREOG scores included 25 subjects in 2 groups: traditional classroom (15 subjects, enter year: 2012, 2013, 2014) and flipped classroom (10 subjects, enter year: 2015, 2016). ANOVA (analysis of variance) was used to analyze the difference of CREOG scores between the two groups. Annual CREOG Exam scores are reported in two values, a score comparing residents to all PGY years and a score comparing them to their respective PGY year. Without any controlling, the results showed that first-year CREOG score compared to all years was significantly higher in flipped program than that in regular program (p=0.0468). The CREOG score compared to PGY year 1 trended towards being higher in flipped program but was not significantly different (p=0.0716).  After controlling for USMLE Step 1 and Step 2, the results showed that there was no difference between first-year CREOG score compared to all year (p=0.2316) and CREOG score compared to PGY year (p=0.3259) between the two groups (Table 1).

Comparison of PGY-2 Scores:

The comparison of second-year CREOG scores included 20 subjects in 3 groups: traditional classroom only (10 subjects, enter year: 2012, 2013), 1 year flipped classroom (5 subject, enter year: 2014) and two years flipped classroom (5 subjects, enter year: 2015). Without any controlling, ANOVA analysis showed that second-year CREOG scores compared to all year and CREOG scores compared to PGY-2 year were significantly different among the three groups (p=0.0018 and p=0.0015 respectively). Total percent correct were not significantly different (p=0.0511). After controlling for USMLE Step 1, Step 2 and PGY-1 CREOG score, the results showed that there was a trend towards increase in second-year CREOG score compared to all year and compared to PGY year, but the differences were not significant (p=0.0987 and p=0.0882 respectively) (Table 1).

Discussion

This is the first report of the objective impact by changing OBGYN residency curriculum to adopt evidence based adult learning techniques. Overall, we found the change in curriculum yielded significantly higher CREOG scores among the PGY 1 and 2 residents compared to all years and in PGY-2 residents compared to their peers. These findings are consistent with those of other specialties who have measured objective data. Girgis and Miller describe significant increase in Neurosurgery board examination scores after implementing the flipped classroom (2018). Martinelli et al describe increase in long term material retention after flipped classroom use in Anesthesia resident education (2017).

Our study has several limitations. First, our sample size was small with only ten residents in each PGY year enrolled in the new curriculum. In addition, the average USMLE scores for applicants entering Ob/Gyn training increased by 9 points in 5 years (2011=220, 2016=229). (NRMP, 2012). After controlling for USMLE Step 1 and Step 2 scores the findings are not statistically significant. While the small sample size may have contributed to this limitation, we also acknowledge that residents who are good test takers will perform well on all types of standardized tests.  A further limitation in comparison of the two classrooms is that in prior years the traditional classroom lectures were given by varying attendings, where our flipped classroom curriculum had a set schedule and was lead by the same attending weekly.

Strengths of launching this new curriculum included that we observed increased engagement of the residents with the didactic presentations. We also felt residents incorporated more resources into studying and working through cases than they would have otherwise utilized in the traditional classroom. Residents reported they were satisfied with the technique. We feel it could be easily replicated in other specialties.

Conclusion

In conclusion, we found there was a trend towards improvement in OBGYN in-training exam scores when using a novel flipped classroom curriculum but statistical significance was lost when controlling for USMLE scores. This study was the first to evaluate in-training exam scores when using the flipped classroom in OBGYN resident education. We believe that the flipped-classroom model may be superior to traditional lecture. Future studies comparing objective data are needed to identify the differences in the educational efficacy between these two models in graduate medical education.

Table 1

          Table 1. Comparison of PGY-1 and PGY-2 CREOG scores controlling for USMLE

Program

Traditional Classroom

1 year flipped Classroom

2 years flipped Classroom

P-value

PGY-1 n

15

10

n/a

 

PGY-1 CREOG Score (standardized compared to all years)

176.8 ± 3.0

183.7 ± 3.9

n/a

0.2316

PGY-1 CREOG Score compared to PGY-1 year

191.3 ± 3.6

197.9 ± 4.6

n/a

0.3259

PGY-2 n

10

5

5

 

PGY-2 CREOG Score (standardized compared to all years)

194.2 ± 3.8

211.1 ± 6.9

205.0 ± 6.3

0.0987

PGY-2 CREOG Score compared to PGY-2 year

194.7 ± 4.2

213.4 ± 7.6

207.7 ± 6.8

0.0882

Take Home Messages

  • The Flipped Classroom is an educational tool that may be of value in graduate medical education and warrants further study. 

 

Notes On Contributors

Ana Valente is a current Chief OBGYN resident at the Ochsner Clinic Foundation in New Orleans, LA.

Rajiv Gala is the current OBGYN Residency Program Director at the Ochsner Clinic Foundation in New Orleans, LA.

Beverly Rueb is the current OBGYN Residency Program Manager at the Ochsner Clinic Foundation in New Orleans, LA.

Acknowledgements

The authors would like to acknowledge statistician Ruijuan Gao.

Bibliography/References

Freeman S, Eddy SL, McDonough M et al. (2014). Active learning increases student performance in science, engineering and mathematics. PNAS; 111 (23) 8410-8415.

https://doi.org/10.1073/pnas.1319030111   

Girgis F, Miller JP. (2018). Implementation of a "Flipped Classroom" in Neurosurgery Resident Education. Can J Neurol Sci; 45(1):76-82

https://doi.org/10.1017/cjn.2017.234   

Martinelli SM, Chen F, DiLorenzo AN et al. (2017). Results of a Flipped Classroom Teaching Approach in Anesthesiology Residents. Journal of Graduate Medical Education; Vol. 9, No. 4, pp. 485-490.

https://doi.org/10.4300/JGME-D-17-00128.1   

McLaughlin JE, Roth MT, Glatt DM et al. (2014). The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Academic Medicine; 89 (2); 236-43

https://doi.org/10.1097/ACM.0000000000000086   

National Resident Matching Program. (2018). Report Archives - The Match, National Resident Matching Program. [online] Available at: http://www.nrmp.org/report-archives/ [Accessed 13 Mar. 2018].

Prober CG, Heath C. (2012). Lecture halls without lectures—a proposal for medical education. N Engl J Med. 2012;366(18):1657–1659

https://doi.org/10.1056/NEJMp1202451   

Tan E, Brainard A, Larkin GL. (2015). Acceptability of the flipped classroom approach for in-house teaching in emergency medicine. Emerg Med Australas; 27 (5); 453-9

https://doi.org/10.1111/1742-6723.12454

Appendices

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Reviews

Please Login or Register an Account before submitting a Review

Subha Ramani - (04/05/2018) Panel Member Icon
/
The author used a scientific approach to assess the impact of a flipped classroom technique in postgraduate education. They used an appropriate methodology to answer the study questions and an overall scholarly approach. writing was clear and they referenced the topic well. Using the USMLE scores as a surrogate for previous exam taking traits was a good idea.
A couple of comments. The small sample size, as the authors mentioned, was probably skewing results on one side or the other. Though part of the class was used for testing, a large portion of the session seemed to have case-based discussions. these discussions are inherently active learning, so it cannot be categorically said that any trend towards improvement is due to the flipped classroom approach. comparisons between historic groups who may have had a completely different overall curriculum could have yielded results that maybe are not directly comparable.
Those interested in adopting a flipped classroom approach in their curricula would find this paper interesting, the references useful and the methodology helpful.
Finally, I would state that flipped classroom is not the only active learning method there is, so medical education should not lose sight of other interactive methodologies while being fascinated by a currently popular one
Jamie Patel - (03/05/2018)
/
Student Perspective:

This paper studied the hypothesis that a “flipped classroom” would improve test scores of PGY1 and 2 residents. The researchers adopted a scientific approach to their method by controlling the structure and director of the weekly teaching sessions. Moreover, the use of USMLE scores to control for residents that were already good test takers strengthened the study. The exclusion of residents who attended less than 75% of the flipped classroom sessions also added to this.

In saying this, there were flaws. As cited by the authors, their small sample size of 25 participants may have lead to the loss of statistical significance after controlling for USMLE score. Additionally, they were unable to compare residents from the same years: the control group of residents that used the traditional teaching method were from 2012-2014 whilst those using flipped classrooms were from 2015-2016. Therefore, they did not account for differences in overall difficulty of the exam taken by each of the groups. This may have contributed to the discrepancy in scores that they recorded, making conclusions regarding the supremacy of a certain teaching method void.

The hypothesis studied by these researchers is indeed interesting to anyone passionate about medical education. Though there were flaws in its design, it opens discussion to the objective quality of the “flipped classroom” approach to teaching. Moreover, it provides a framework from which future research about this topic can be based. It is therefore a valuable paper and one that inches us forward in optimising medical education.