A Comparison of Simulation-Based Education and Problem-Based Learning in Pre-Clinical Medical Undergraduates

This article was migrated. The article was marked as recommended. Introduction: Despite the increasing popularity of high-fidelity simulation (SIM), few studies compare the effectiveness of SIM with established small-group modalities such as problem-based learning (PBL) for factual learning among pre-clinical medical undergraduates. Methods: 162 second-year medical students were randomized to two groups. The first group encountered teaching points related to the management of hypertensive emergency during a traditional PBL session, while the second group covered the same teaching points through SIM. Both groups then completed a post-test. The students were subsequently crossed-over to the opposite modality and assessed on teaching points related to the management of chronic obstructive pulmonary disease exacerbations. Results: A total of 162 students participated in the study. For the hypertension case, the average proportion of correct questions for the SIM intervention was 83% compared to 70% for the PBL intervention (OR = 3.1, 95% CI [1.3, 7.2], p=0.009). For the COPD case, the average proportion of correct questions for the SIM intervention was 73% compared to 51% for the PBL intervention (OR = 2.6, 95% CI [1.8, 3.8], p < 0.001). Conclusion:Students scored significantly higher with SIM in both cases, indicating that SIM may be superior to PBL for factual learning among students with limited clinical exposure.


Introduction
Since its inception nearly a half-century ago, problem-based learning (PBL) remains a staple of small-group medical education (Kinkade, 2005;Hartling et al., 2010;Poulton et al., 2014).Originally developed to foster the ability "to evaluate and manage patients with medical problems effectively, efficiently, and humanely", PBL also allows students to identify areas for continued exploration and lifelong learning (Barrows and Tamblyn, 1980;Neville and Norman, 2007;Neville, 2009;Norman, 2012).With the advent of new and more readily accessible technologies, however, the use of simulation-based medical education (SBME) has become an increasingly popular small-group educational modality across all spheres of health education, including graduate and undergraduate medical education, nursing education, and pharmacy education (Seybert et al., 2012;Cohen et al., 2013;Lopreiato and Sawyer, 2015;Shin, Park and Kim, 2015;Alluri et al., 2016).
As the utilization of SBME modalities has increased, so too has the evidence for the efficacy of SBME.Within the realm of graduate medical education alone, SBME has been shown to improve trainee performance in a variety of procedural and interpersonal tasks across a broad range of disciplines, including colonoscopy, central venous catheter insertion, laparoscopic surgical procedures, and ethical dilemmas (McGaghie et al., 2011;Barsuk et al., 2012;Bearman et al., 2012;Smith et al., 2012;Cohen et al., 2013;Schmidt et al., 2013;Singer et al., 2013).SBME has also become increasingly popular in the realm of undergraduate medical education, but studies on the efficacy of simulation to date have largely been limited to students at the clinical level (traditionally third and fourth year students who are completing clinical clerkships) (Ten-Eyck, Tews and Ballester, 2009;Ko et al., 2011;Littlewood et al., 2013).Results from these studies have mostly validated the use of SBME as an effective teaching modality in this population, but some results have been equivocal (Wenk et al., 2009;Kerr et al., 2013).High-fidelity patient simulation (SIM) in particular is increasingly utilized in the context of pre-clinical undergraduate medical education, but its efficacy as it compares to traditional teaching modalities of problem-based and didactic-based education is just now being elucidated (Harris, Ryan and Rabuck, 2012;Littlewood et al., 2013;Alluri et al., 2016).
Despite the growing popularity of SBME, there is limited data to quantify how SBME compares in efficacy to the other small-group educational modalities with which it directly competes, namely PBL, for knowledge acquisition (Alluri et al., 2016).Several studies have demonstrated a significant improvement in factual learning with SIM as compared to paper-based PBL, but these studies have been limited to advanced medical students at the clinical level (Steadman et al., 2006;Schwartz et al., 2007).Therefore, the purpose of this study is to compare small group high-fidelity simulation with PBL for factual learning in pre-clinical medical students.

Population Setting
All second-year (preclinical) medical students enrolled in the 2014-2015 academic year of the Cardiovascular, Renal, and Respiratory Medicine II (HBD409) course at the David Geffen School of Medicine at UCLA were included in this randomized, controlled cross-over study conducted within the parameters of an established problem-based learning curriculum.Student participation was voluntary through their enrollment in the course.All study activities including completion of multiple choice self-assessment questions to assess learning were conducted within the context of the course and no extracurricular activities were required.UCLA Human Subjects Protection Committee approved exemption from full IRB review for this study as all data were de-identified and analyzed after all students had completed and received a final grade for the course.It should be noted that this study was separate from a previous study at our institution by one author (JN) and involved a different cohort of students and separate IRB approval (Alluri et al., 2016).

Study Design
At the beginning of the course, 162 second-year medical students were divided into PBL groups of 8-9 students, as is the routine with every course at our medical school.This made 20 groups, 10 of which were randomly assigned to group A and 10 of which were assigned to group B. The traditional PBL curriculum requires a two-hour meeting at the start of each week and a two-hour meeting for the resolution of the patient case on the last day of each week.The teaching sessions analyzed in our study took place during this second meeting of both the third and sixth weeks of the course.During the third week of the course, group A encountered structured teaching points related to the diagnosis and management of hypertensive emergency in the traditional PBL format.A specific addendum to the existing PBL patient case was utilized for this purpose.Group B encountered identical structured teaching points related to an identical case in the format of a 10-minute SIM with subsequent debriefing session.After the completion of their respective modalities, all students completed three written multiple-choice questions to assess their learning of the teaching points.
During the sixth week of the course, the students were crossed over to the opposite modality.During this week, Group B encountered structured teaching points related to the management of a chronic obstructive pulmonary disease (COPD) exacerbation in a traditional PBL format while Group A completed an identical SIM case with identical teaching points.All students again individually completed three written multiple-choice questions as a post-test to assess the learning of the teaching points.Each post-test was scored from 0 to 3 via a web-based course-management system.All PBL sessions were completed as part of a stable group (the students and tutors remained in the same group for all eight weeks of the course).Two faculty members with experience conducting and debriefing simulation sessions proctored all the simulation sessions.

Simulator Sessions
The SIM sessions were conducted in the same manner as other sessions at our institution (Alluri et al., 2016).As in previous studies, a METI (Medical Education Technologies, Sarasota, FL) high-fidelity mannequin was used for all SIM components of this study (Alluri et al., 2016).All students were familiar with the simulator system through their prior participation in simulator sessions as a part of their preclinical curriculum.
Each SIM scenario in this study lasted ten minutes.At the start of each session the students were introduced to the patient's chief complaint, brief history, and initial vital signs by the faculty proctor.Students were then able to obtain further history by interviewing the mannequin and information was provided by a simulation technician or the faculty proctor according to a written simulation script.The students were also able to perform examination maneuvers, request clinical data, and administer therapeutic interventions such as medications.A telemetry monitor provided real-time physiologic data according to the case script and student interventions.Faculty proctors participated in the simulation in the role of a bedside nurse and were present throughout the entire session.
Upon completion of the simulation all students received a ten-minute debriefing session led by the faculty proctor.During the debriefing session proctors emphasized pre-determined teaching points which were identical to the teaching points found in the tutor notes for instructors leading the traditional PBL sessions.A single proctor ran each SIM session with a simulation specialist in the room controlling the mannequin.Simulations were run in two rooms simultaneously with a new group of 8-9 students coming into each simulation room every 20 minutes.
Problem-Based Learning Sessions PBL sessions were conducted according to the existing PBL curricular framework at our institution.All PBL sessions for this study were conducted on the second meeting of the problem-based learning component for each respective week.A specific case addendum was written to existing paper-based PBL cases.Each addendum consisted of one page added to the existing handouts in which students were given a patient encounter identical to the simulation case.Students were encouraged to explore how they would manage the patient through small-group discussion.All PBL faculty facilitators were given specific instructions to reinforce the teaching points related to the management of the given presentation prior to moving forward with the next section of the PBL case.

Post-Test Assessment
All students were individually assessed on their learning of the predetermined teaching points through a total of six written multiple-choice post-test assessment questions-three for each case.Upon completion of each case the students encountered three multiple-choice questions incorporated into the existing weekly self-assessment framework for the course.These weekly self-assessments were administered through a web-based course-management system and completed at a time of the student's choosing between 5:00PM on the last day of the week and 8:00AM on the first day of the following week.These assessments were open-book format, although students were encouraged to complete these assessments on their own and without assistance from outside resources.All assessments were scored by the course management software according to an answer key written prior to administration and each student received an individual score 0 through 3 for each post-test.All student scores were de-identified and all students were randomly assigned a number through 1 through 162 for the purposes of examining the effect of the crossover intervention.Student performance on the assessment did not impact their final grade for the course.

Data Analysis
The average number of questions correct were calculated for each post-test and compared between the PBL and SIM groups for both the hypertension and COPD cases.Additionally, a mixed effects logistic regression model was used to examine between subject differences in educational intervention (SIM vs PBL) at each week and to examine between subject differences in test ordering (Week 6 vs Week 3).Within-subject differences in the educational intervention and in test ordering were also examined using data combined from both weeks and both interventions respectively.A p-value of <0.05 was considered statistically significant.All analyses were conducted in Stata Version 13, StataCorp LP (College Station, Texas).

Results/Analysis
A total of 162 students were studied.Eighty-one students were assigned to group A, which completed the PBL intervention for the hypertensive emergency case and the SIM intervention for the COPD exacerbation case.The other eighty-one students (Group B) completed SIM for the hypertensive emergency case and PBL for the COPD case.All students completed both post-tests except for one student in Group B who did not complete the COPD post-test.In both cases, the students who completed SIM scored significantly higher on the post-test questions than those who completed PBL (Table 1).For the hypertension case (week 3), the average proportion of post-test questions correct across students completing the SIM intervention was 83% compared to an average post-test score of 70% for students completing the PBL intervention (odds ratio = 3.1, 95% CI [1.3, 7.2], p=0.009).For the COPD exacerbation case (week 6), the average proportion of post-test questions correct across students completing the SIM intervention was 73% compared to an average post-test score of 51% for students completing the PBL intervention (odds ratio = 2.6, 95% CI [1.8, 3.8], p < 0.001).For both the PBL and SIM groups, the combined post-test scores for the hypertension case (86% in week 3) were significantly higher (p < 0.001) than for the COPD case (62% in week 6).A description of the students' correct responses to the post-test questions is provided in Table 1 and a graphical depiction is provided in Figure 1.
Given that the study permitted a cross-over analysis with repeated within-subject measurements, a mixed effects logistic regression model was used to determine if the effects of the educational intervention (SIM vs PBL) differed depending on the type of clinical scenario (hypertension in week 3 vs COPD in week 6).Modeling the interaction of week by intervention yielded a p-value of 0.76, indicating no significant difference in the performance of the two simulation groups in their respective weeks.Thus, the differences between the two interventions (SIM vs. PBL) did not differ between the COPD and hypertension cases.

Discussion
This study demonstrated a significant improvement in factual learning and knowledge acquisition in students who completed high-fidelity simulation training compared with problem-based learning.In fact, when looking at both weeks of the study, the odds of answering the post-test questions correctly after having completed SIM were over three times more likely than they were after completing PBL.These results also demonstrate that post-test performance was improved with simulation regardless of the week in which the students received the simulation intervention.After completing the analysis, it was apparent that the questions in week 6 were more difficult than in week 3, but ultimately there was no other significant difference between the two weeks.This also demonstrates that ordering of the interventions played no significant role and that it did not matter whether the students performed SIM or PBL first.This study, to our knowledge, is one of the largest randomized studies comparing high-fidelity simulation with problem-based learning for the purposes of knowledge acquisition at the preclinical level.Previous studies in clinical-level medical students have shown similar improvements in post-test scores with simulation as compared to PBL (Schwartz et al., 2007).
We chose to compare SIM to PBL as both modalities are interactive small group activities that revolve around working through a patient case.PBL is pervasive in modern medical education, with greater than 70% of medical schools in 2005 reporting the use of PBL in some capacity in the preclinical years (Kinkade, 2005;Wood, 2008).Yet, despite the ubiquity of PBL, the data surrounding its efficacy and outcomes as compared with traditional pedagogical methods remains mixed (Koh et al., 2008;Okuda et al., 2009).While PBL has been shown to improve abilities such as clinical reasoning (Neville and Norman, 2007;Neville, 2009;Norman, 2012), PBL has not proved to be a universally superior curricular modality (Steadman et al., 2006;Koh et al., 2008;Wenk et al., 2009).Intuitively this is logical, as no one educational intervention or pedagogical method is likely to perfectly encompass all the nuanced aspects of medical education.Thus, it is fair and prudent, given the pervasive use of PBL and its widespread endorsement within the medical education community, to compare it as a small group teaching modality with simulation.
While both PBL and SIM are small-group modalities, a key difference is that the high-fidelity simulator uses physical equipment and technology that provides students with psychomotor stimuli and real-time feedback.The SIM milieu may evoke greater emotional engagement and concentration than a classroom-based PBL session.This study corroborates support for the hypothesis posited by others that such visual, auditory, and tactile stimuli engage the learner beyond the purely cognitive aspects of PBL (Issenberg et al., 2005).Importantly, SIM provides the learner with a safe environment in which to make decisions and analyze the consequences of those decisions (Issenberg et al., 1999;Wood, 2008).Such engagement may reinforce learning concepts beyond oral discussion of a written case.
Our study has some limitations.First, some may contend that PBL was not designed to optimize factual learning.We, however, would assert that most PBL cases and tutor guides have specific learning points written into them.Therefore, it is often expected that students will assimilate new knowledge during PBL sessions alongside the opportunities to ask and answer clinical questions and to strengthen communication and teamwork skills.That stated, we are not arguing that SIM should replace all PBL.However, the activation of the SIM lab may make certain facts "stickier" and therefore more easily learned by students.More work remains to be done to determine which concepts would be better served through PBL and which would be better served through simulation.
Second, despite our sessions taking place in the "preclinical" years, our learning points for this study focused on a combination of patient management and basic science principles, rather than purely focusing on basic science.Additionally, all of our PBL groups and SIM sessions had one or more clinicians as a facilitator.Our approach might not yield the same results for medical schools not using faculty with clinical experience to facilitate their sessions.
Similarly, the high-fidelity simulators used in our study are expensive and the SIM sessions require intensive investment of time by faculty.Faculty at other medical schools might wonder whether it is worth the investment in equipment and faculty to incorporate high-fidelity simulation into the first one or two years of medical school to obtain a moderate improvement in immediate recall of teaching points.Future research might look at whether similar results to our study could be obtained using cheaper, screen-based simulation linked with debriefing and whether improved immediate recall translates into improved long-term knowledge retention.

Conclusion
In conclusion our study demonstrates that SIM was superior to PBL for knowledge acquisition in pre-clinical medical students studying hypertension and COPD.Given the limited amount of educational time in the pre-clinical years, more work remains to be done to determine which of these modalities (or whether a combination of both) is best for teaching numerous physiologic, pathophysiologic and pharmacologic concepts, and whether these benefits of knowledge acquisition lead to improvement in performance in the clinical years and beyond.

Take Home Messages
Factual learning can be achieved through both high-fidelity simulation and problem-based learning Simulation-based education may be superior to problem-based learning for factual acquisition Figure 1.Source: Dr. Nicholas J. Jackson (author).

Bibliography/References
Alluri, R. K., Tsing, P., Lee, E. and Napolitano, J. ( 2016 This review has been migrated.The reviewer awarded 5 stars out of 5 This is a very interesting and important study in that it compares in a rigorous fashion the learning of key concepts of basic clinical entities between two modes of learning, problem based discussion, which is time tested and universally available, and newer SIM based learning which is less widely distributed, requires investment in time, training and material, but nonetheless is shown in this study to improve the ability of learners to master concepts required, at a minimum, to pass written examinations on the subject matter.The authors conduct a crossover study which demonstrates the superiority of SIM based learning in either order of presentation when compared to PBL sessions.Although one might have intuitively thought that SIM based learning, while superior for hands on, so called "practical skills", would not show superiority for a more cognitive -question answering assessment.The authors provide a plausible explanation for why that may be so, in that being placed into a (simulated) clinical situation evokes the emotional connection with the material, the more or less visceral understanding of its clinical importance, which makes the material stick.The authors point out that SIM technology is not universally available.I wonder whether the good bedside teacher could formulate questions to learners in order to take advantage of this now known beneficial effect of a SIM model.Perhaps the creation in the learners mind and in real time of possible realistic imaginary clinical scenarios, having the student imagine, for example, during the case presentation of a chest pain patient, that the patient is dropping their blood pressure, skipping beats, having more chest pain, and the like.In that manner, pending creation of a SIM program, teachers can utilize the knowledge, confirmed in this study, that SIM improves learning, even in pre-clinical years and for basic pathophysiologic concepts.A very intriguing and interesting study!
Competing Interests: No conflicts of interest were disclosed.

Table 1 .
Comparison of Proportion correct between PBL and SIM with 95% Confidence Intervals Figure 1.Post-Test Proportion Correct by week.
Cohen, E. R., Barsuk, J. H., et al. (2011).R., Barsuk, J. H., et al. (2011)Does simulation-based medical education with deliberate practice yield better results than traditional clinical education?A metaanalytic comparative review of the evidence.Academic Medicine.86(6),pp.706-711.This review has been migrated.The reviewer awarded 5 stars out of 5Thanks, an interesting study.Many people have tried moving away from a paper-based PBL 'case' to video or SPs, so perhaps this paper aligns with that.As the authors state, PBL is not just about learning facts, it's as much about finding out what you don't know.It would be interesting to know what the students were thinking as they went through the scenario.A de-brief which included a 'think-aloud' session might be revealing.Notwithstanding that the Simulation was in a 'safe-space' it probably gets the adrenalin flowing more than the average PBL session.Is it that that makes the facts 'sticker'?What if the adrenalin goes too high-one would anticipate less learning perhaps?The resources used are interesting, particularly staff time.If I interpret it correctly the PBL time lasted 4 hours but the Simulation 20 mins.In these comparisons it seems important to look for efficiency of learning; how much learning is acquired per unit time.On the face of it Simulation was not only better but highly efficient.However I'm not sure if that conclusion can be drawn here on the data.This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.