Determining the Instructional Effectiveness of Online Vaccine Education Modules: A Focus-Group Analysis

This article was migrated. The article was marked as recommended. Vaccine education for pediatric and family medicine residents is inadequate. Implementation of evidence-based instructional design methods for medical education is understudied. We conducted four focus groups with residents who had completed a novel immunization curriculum to explore their satisfaction with design, content and impact on confidence. Data were analyzed using thematic content analysis. Overall satisfaction with the curriculum was high. Residents valued the interactive design and content, reported improvement in confidence in discussing vaccines with parents, and shared recommendations for future iterations of the modules. Technical challenges were reported with the learning management system. Medical education modules developed using best practices in instructional design were well-liked by trainees and future modules should be developed using these principles.


Introduction
Childhood vaccination is an effective method to reduce morbidity and mortality.However, vaccination rates in the U.S remain below goals.(Office of Disease Prevention and Health Promotion, 2018) Strong recommendations from healthcare providers have been shown to positively impact vaccination decisions (Opel et al., 2013), yet, immunization and communication education during pediatric residency training is inadequate.(Williams and Swan, 2014) To address this deficiency, we created the Collaboration for Vaccine Education and Research (CoVER) to augment residents' immunization knowledge and vaccine communication competency.A 4-module curriculum was developed using empirically supported instructional design principles based on Cognitive Load Theory (CLT) (Sweller, 2010), Multimedia Learning (MML) (Mayer, 2009), and the Attention, Relevance, Confidence, Satisfaction Model (ARCS) (Keller, 2010).We delivered the CoVER curriculum ("CoVER") using a flipped-classroom approach and Internet Based Learning (IBL), which has been associated with greater positive effects in learning compared to traditional or non-IBL educational techniques.(Wittich et al., 2017) As there is little data evaluating the instructional effectiveness of flipped-classroom technology in training programs (Cook et al., 2010), this study seeks to address this deficit through evaluation of the instructional effectiveness of the CoVER curriculum.

Methods
This qualitative study used focus group interviews to obtain pediatric and family medicine residents' ("residents") educational experience with the four CoVER modules.The study was exempt by the Institutional Review Board at Children's Mercy, Kansas City.Implied consent was obtained through voluntary focus group participation.

Study Setting and Participants
In Fall 2018 we recruited residents for focus groups from a convenience sample of four residency programs that received CoVER during a 1-year pilot program.Participants included only those who had completed >1-of-4 CoVER modules.Recruitment was conducted via e-mail, through residency program announcements, or during resident conferences.No monetary incentives were provided.

CoVER Curriculum
Five experts in pediatric infectious disease, immunization, immunization education, residency education, and adult learning developed four 1-hour modules incorporating best practices in instruction design principles.We applied the ARCS Model to promote learning and sustain motivation in the learning process, and incorporated principles of MML and CLT to manage intrinsic load (working memory load) and maximize storage of new knowledge in long-term memory.
CoVER focused on key aspects of vaccines and vaccination (vaccine fundamentals, vaccine preventable diseases, vaccine safety, communication) and was piloted July 2017 through June 2018.

Instrument
The focus group moderator's guide included thirteen open and closed format questions focusing on instructional content and design, the learning platform's usability, and suggestions for improvement.Clarifying questions were utilized when needed.Because our primary goal was to evaluate instructional effectiveness, questions were grounded in these concepts.

Data Collection and Analysis
Group interviews were conducted at each institution and facilitated by a moderator.Focus groups were audio recorded and transcribed for review and analysis.Two coders read the transcripts multiple times to gain a sense of the entire context of material and to discover quotes and examples to illuminate key concepts and themes.A list of codes was developed prior to review using the focus group guide and then applied to quotes within transcripts.Coded responses were grouped into thematic domains.Additional codes were developed during review to capture additional domains as needed.Data were analyzed using thematic content analysis and open, axial, and selective coding procedures from grounded theory principles.Differences in coder interpretations were resolved by discussions leading to consensus, or the inclusion of a third coder.Charts and tabulations were used to reduce researcher bias.

Results/Analysis
The study cohort comprised 28 residents representing all years of training.One focus group was conducted at each of four academic institutions: Vanderbilt University Medical Center, Children's Mercy, Kansas City, University of Kansas Medical Center, and Truman Medical Center.Identified reflections were clustered into six main themes.

Content Design
Respondents reported that the material was well planned and clear, sharing that CoVER content "wasn't the same thing" as every other required training.Residents valued the broad range of content with many specifying the importance of learning about public and/or patient vaccine misinformation, parental concerns, "odd ball" or out-ofthe-ordinary vaccine questions, negative media stories, delayed vaccination schedules, and vaccine components.Additionally, residents responded positively to learning about vaccine "basics" and resources for additional information.One resident shared: "What we would want in a vaccine module is pretty much what you guys gave us."

Module Structure/Learning Engagement
Residents liked the content's logical sequencing with relevant "small chunks" of information and the variety of interactive activities such as flashcards, simple games, knowledge checks, and multimedia integration of content (videos and links).Respondents believed that the content was "very visually appealing", "fairly simple", and presented in a "userfriendly" manner.

Perceived Learning and Confidence
Most participants reported the modules greatly improved their own confidence in their vaccine knowledge and comfort in discussing and recommending vaccines.One resident recalled utilizing new knowledge to convince a previously vaccine hesitant parent to vaccinate her child.Several residents shared that modules were helpful to "prepare for concerns from parents."One resident described the impact on his confidence as now: "Being able to ask them why they're hesitant and be more confident that I'll have an answer to whatever reason."

Challenges
Several participants reported technology challenges with the Learning Management System (LMS) that housed CoVER, such as difficulty with logging out, losing prior work, or being uncertainty if they had completed a module: "That was the only thing I didn't like about it.I didn't know if I was done." 5. Recommendations for the Future Several ideas were presented for future iterations of CoVER (Table1).Suggestions included broadening content, changing the LMS, and adding new access options.

Overall Satisfaction
Residents had high overall satisfaction with CoVER.Several participants reported the modules were the best they had completed in training thus far.Almost all participants agreed the modules incorporated a clear, clean layout and visual design.They found the modules user-friendly and valued the interactive nature and embedded resources.

Table 1 .
Residents' Recommendations for Future Iterations of CoVER More frequent vaccine instruction (e.g., module of the month) nonbranded presentations.DM -Advisory boards for Pfizer, GlaxoSmithKline, and Sanofi Pasteur, lectures for Pfizer and Seqirus, co-principal investigator for a CDC grant.SEW -Additional grant funding from the American Academy of Pediatrics and Pfizer for an Independent Grant for Learning & Change SGH -Consultant to Immunization Action Coalition (nonprofit organization) and Sanofi Pasteur.