Mock interviews with video-stimulated recall to prepare medical students for residency interviews

This article was migrated. The article was marked as recommended. Purpose: The residency interview is the most important factor for residency program directors when deciding on how to rank medical student applicants. With the residency match becoming increasingly competitive, it is more important than ever for students to perform well in this high-stakes interview. Video-stimulated recall (VSR) has been shown to be an effective tool for facilitating reflection on performance and behaviors. As such, we conducted mock interviews with and without video-stimulated recall to gauge its effect on student perceptions of preparedness and confidence for residency interviews. Methods: Students completed a pre-mock interview survey followed by a video recorded interview with faculty. All students received verbal feedback on their performance immediately after the interview. Students were randomized to receive their feedback from their faculty interviewer either while reviewing their video or without the video review. Post-mock interview and post-residency interview surveys were completed. Wilcoxon signed-rank was used to compare median aggregate scores between pre/post surveys. Wilcoxon rank-sum was used to compare pre/post aggregate scores between the video review vs. no-video review groups. Results: 33 of 70 students participated (47%). 14 students (42%) reviewed their video and 19 (58%) received feedback without video. Likert scores for pre- and post-mock interview and post-residency interview surveys revealed median aggregate scores of 10 (interquartile range, or IQR=8,11), 12 (IQR=12,13), and 13 (IQR=12,13) (p <0.001, p<0.001). The change in median aggregate score between pre/post-mock interview surveys in the video review group vs. no-video review group was 3 (IQR=3,5) and 1 (IQR=0,3) (p<0.01) and from pre-mock interview to post-residency interview in the video review vs. no-video review groups was 3 (IQR=3,5) and 2 (IQR=1,4) (p=0.04). Conclusions: The mock interview for residency application improved students’ perceptions of preparedness and confidence. Reviewing the video of the interview while receiving verbal feedback increased students’ confidence in their interview skills.


Introduction
The residency application process for medical students has become increasingly competitive in recent years.The 2016 Main Residency Match resulted in over 600 US graduating senior medical students lacking residency positions after the Supplemental Offer and Acceptance Program.(Signer, 2016) There are several reasons for this.Predictions of future physician shortages has led American medical schools to increase class sizes with a goal of an overall increase in enrollment by 30% by 2017.(AAMC, 2013) In addition, the number of US allopathic medical school seniors participating in the match has grown by 18% between 2005 and 2014.Other groups in the applicant pool including osteopathic graduates and foreign medical graduates have also grown substantially in recent years.(NRMP, 2014) Despite the considerable increase in the number of medical school graduates over the past 10 years, the number of residency positions has remained relatively constant.As a result, in the 2016 residency match there were 42,370 registered applicants competing for only 30,750 available positions making the 2016 match the most competitive to date.(NRMP, 2016) It is therefore incumbent upon medical schools to prepare senior students as much as possible to maximize their chances of success in the match.
The interview has been shown to be possibly the most important factor in residency program directors' decision making and a significant influence on the ranking order of residency candidates.(Gong et al., 1984) Interactions with faculty and house staff during the interview were cited as two of the most important factors in ranking a candidate according to the 2014 National Residency Matching Program Survey of Program Directors.(NRMP, 2014) These data suggest that improving interviewing skills may favorably affect candidates' chances of securing a residency position.
Despite this, there is little in the literature to determine the most effective way to prepare students for residency interviews.Ryan et al. demonstrated that conducting mock interviews for senior medical students in a group setting could improve students' perceptions of preparation and confidence for residency interviews.(Ryan, Ewing and O'Brien, 2014)Several studies in the pharmacy literature have shown that implementation of a mock interview program including individual practice interviews with faculty and verbal feedback can not only help students feel more confident and prepared for the real interview process(Caballero et al., 2012), but can potentially improve match success rates.(Koenigsfeld et al., 2012) But despite the high-stakes nature of the residency interview, to the authors' knowledge, there is no standardized approach to preparing students and practices vary widely across medical schools.
As noted above, limited evidence does exist that practice interviews are of value in preparing medical students for residency interviews.However, it is unclear what is the optimal way to provide feedback and how best to encourage reflection on interview performance.Previous studies in medical education have shown that giving immediate verbal feedback is a valuable learning tool and can improve preparedness, confidence and performance in various clinical and educational settings.(Ende, 1983) (Hodder et al., 1989) (Garner, Gusberg and Kim, 2014) Video stimulated recall (VSR), or the review of video-recorded events of an individual's behavior has been used extensively in social science research and has been shown to be helpful in facilitating reflection on social behaviors.(Lyle, 2003) In educational research, VSR has been shown to be an effective tool for enabling instructors and students to articulate their thinking and feelings and even uncover cognitive processes.(Powell, 2005) Furthermore, evidence from the psychology literature suggests that individuals can learn more effectively by watching their own performances.(Knoblich and Sebanz, 2006) However, to the authors' knowledge, no evidence exists demonstrating the effectiveness of VSR in the preparation of medical students for residency interviews.
In our study, we established a voluntary mock interview session for senior medical students at our institution.We then utilized VSR in providing feedback to a subset of the students.Our aims were to: (1) determine whether conducting individual mock residency interviews for senior medical students increases perceived preparedness, confidence, and performance in residency interviews; and (2) to determine if receiving verbal feedback while reviewing the video recording of the mock interview with faculty immediately after the mock interview enhances the effectiveness of the mock interview experience.

Study design:
The study was approved by and conducted in accordance with the University of Arizona Institutional Review Board -Human Subjects Protection Program.All senior medical students who were on target to graduate and planned to enter the 2016 residency match were invited by email to participate in a mock interview activity prior to the start of the residency interview season.Students received a disclosure attached to the email stating that participation in the investigation was voluntary and they had the option of participating in the mock interview activity without participating in the research study.Students who agreed to the disclosure statement were directed to the pre-mock interview survey (Survey #1) assessing demographic information, their intended residency specialty choice, and their perceptions about their preparedness and confidence level for residency interviews using a 5-point Likert scale (Appendix A).After completing the survey, they were directed to a web-based sign-up for the mock interview activity.Students who did not agree to the disclosure bypassed the survey and were directed to a web-based sign up for the mock interview.Students completing the survey were assigned a unique identifier which was used on all surveys going forward so that all student information was de-identified.
The mock interview activity was conducted on two separate evenings to maximize students' availability and students could choose which night to attend.Each mock interview activity consisted of two components: a mock interview portion and a roundtable discussion with faculty and local residency program directors so that students could ask specialtyspecific questions about the residency interview process.For the mock interview, each student had a 20-minute, individual mock interview with a faculty member who may or may not have been from the students' chosen specialty.Students were encouraged to bring their CV and personal statement to the mock interview.Each interview was followed immediately by verbal feedback with the faculty interviewer.All interviews were video-recorded.Students were alternately assigned in a randomized fashion to either view their video with their faculty interviewer immediately after the mock interview while receiving feedback (video review group), or to receive verbal feedback without reviewing their video (no-video review group).All faculty members were given a standardized feedback form (Appendix E) and were instructed to give feedback to the students addressing all of the elements indicated on the form.Since watching the video would require additional time, the video review group was given 20 minutes for feedback in order to allow time to cover all of the feedback elements and the no-video review group was given 10 minutes for feedback.
Immediately following feedback, students were asked to complete a post-mock interview survey (Survey #2) again assessing perceptions about confidence and preparedness for residency interviews and to gauge the effectiveness of the feedback they received (Appendix B).
A third survey was administered after the completion of the residency interview season (Survey #3) to assess students' perceptions of preparedness, confidence, and performance in their actual residency interviews (Appendix C).All three surveys contained the same three questions to assess students' perceptions of preparedness, knowing what types of questions to expect, and confidence for residency interviews using a 5-point Likert scale (1=Strongly disagree, 5=Strongly agree).
All participating students and faculty received the following information prior to the mock interview session: 1) details about the format of the session 2) a list of commonly asked interview questions (Appendix D), and 3) a PowerPoint presentation providing tips for successful interviewing and links to interviewing skills resources.
The faculty who were invited to participate in the event were from the departments in which students planned to pursue residencies, as well as members of the Student Affairs and Admissions teams.The individuals chosen as faculty interviewers all had experience both with interviewing student or residency candidates and with giving feedback to students and house staff.Faculty who were assigned to interview students in the video review group all had experience in providing feedback to students using video review as instructors in the Doctoring course for first and second year medical students where standardized patients and video review are used to teach medical students history taking and physical exam skills.Since the number of faculty who were deemed qualified to do this were limited, the number of students who were randomized to the video review group had to be limited accordingly.
Statistical methods: For students who participated in the mock interview program, student characteristics including age, gender, and specialty were summarized using median and interquartile range (IQR) for continuous variables and frequency and proportion for categorical variables, and were compared between the subjects who received immediate video-based faculty review and those who did not, using either the Wilcoxon rank-sum test (for continuous variables like age) or the Fisher's exact test (for categorical variables like gender and specialty).
All three surveys shared three identical questions assessing preparedness, knowing what type of questions to expect, and confidence in residency interviews (Questions 4, 5 and 6 of Survey #1, Questions 3, 4 and 5 of Survey #2, and Questions 6, 7 and 8 of Survey #3).For each survey an aggregate score was calculated as the sum of the responses to these three questions, with the five levels of the response for each question coded as integers 1 through 5. To evaluate the effect of the mock interview, for subjects who completed all three surveys, the Wilcoxon signed-rank test was used to compare the baseline pre-mock interview aggregate score with the post-mock interview score and also separately with the postresidency interview score.The tests were also performed separately for the score of each individual question.To evaluate the effect of the immediate video review, the Wilcoxon rank-sum test was used to compare the change in aggregate score from baseline to the post-mock interview survey between the two groups of subjects receiving or not receiving the video review.All statistical tests were two-sided with significance level 0.05.

Results/Analysis
Twenty-eight faculty members from the departments of Obstetrics and Gynecology, Internal Medicine, Family and Community Medicine, Child Health, Anesthesiology, Radiology, Psychiatry, Emergency Medicine, Radiation Oncology, and Surgery as well as members of the Student Affairs and Admissions teams participated in one or both of the mock interview sessions by conducting interviews, being a part of the roundtable discussion or both.A total of 39 of the 70 eligible students (56%) participated in the mock interview activity and 33 students (85%) completed all 3 surveys.Among these 33, 14 students (42%) were randomized to the video review group and 19 students (58%) were randomized to the no-video review group (Figure 1).
Complete demographic information regarding the participants can be found in Table 1.The median age of students was 27 (IQR=26, 28) and 16 students (49%) were men and 17 (51%) were women.Thirty out of the 33 participants (91%) had done no preparation prior to the mock interview.Students who participated planned to apply for residencies in the following specialties: anesthesiology, emergency medicine, general surgery, internal medicine, obstetrics and gynecology, ophthalmology, orthopedic surgery, otolaryngology, pediatrics, psychiatry, radiology, dermatology, and family medicine (Figure 2).There were no significant differences in baseline characteristics between the students in the video review and the no-video review groups.
Our primary analysis focused on the three questions common to all three surveys: assessing preparedness, knowing what type of questions to expect, and confidence in residency interviews.The Likert scores for each question were compared from pre-mock interview, post-mock interview, and post-residency interview surveys as were the total aggregate Likert scores for those three questions from pre-mock interview, post-mock interview, and post-residency interview surveys (Table 2).
Additional exploration into the responses to the questions on the post-mock interview survey showed a trend for students in the video review group to have a more positive assessment of the effectiveness of the mock interview activity and the feedback they received.Students in the video review group were more likely to feel that their interviewing skills improved as a result of the mock interview session (57% strongly agreeing in the video review group vs 42% in the no video review group) and were more likely to feel that the feedback they received was helpful (93% strongly agreeing in the video review group vs 79% in the no video review group).These values did not reach statistical significance (p =0.71, p=0.28) (Table 4).

Discussion
The residency interview is a high-stakes evaluation as it represents a key determinant for residency program directors to select candidates in an increasingly competitive match environment.Several recent studies have investigated the use of a mock interview program to prepare students but with some notable differences from our study in both design and outcomes.In 2014, Ryan et al. conducted practice interviews for senior medical students in groups of four with one student undergoing the interview while the other three students in the group observed.They concluded that conducting mock interviews in a group setting was an effective and time-efficient way to improve students' perceptions of confidence and preparedness for residency interviews but they did not investigate if there was any longer term benefit of the activity with follow up surveys.In 2016, a similar mock interview program was implemented but with individual mock interviews.They found that such a program was feasible and may have improved students' comfort and confidence level in the residency interview process.(Multerer et al., 2016) However, the program was offered only to students interested in pediatrics and therefore was not representative of the medical school class as a whole.Another similar mock interview program was conducted at the Medical College of Wisconsin in 2016 and demonstrated that students felt that individual mock interviews followed by verbal feedback helped them identify their strengths and weaknesses and that the students' perceptions of the benefit of the program were still present five months later.They further noted that the group of students who participated in the mock interview program demonstrated a higher match rate than those who did not, suggesting a potential benefit not only in perceptions but in actual interview performance.(Hueston and Holloway, 2016) The validity of this conclusion is unclear however since the mock interview program was voluntary and students who participated were self-selected therefore creating a potential volunteer bias that could have affected these results.To the authors' knowledge, ours is the first study to evaluate the use of immediate video review with verbal feedback from faculty after an individual mock interview to prepare students for residency interviews.Our data demonstrate that conducting individual mock interviews with immediate verbal feedback is an effective approach for improving students' perceptions of confidence and preparedness for residency interviews, consistent with what previous studies have shown.Our data further suggest that the addition of video review to the feedback process, although challenging because of the need for more time and resources, may be very worthwhile since the students who received feedback on their mock interview performance with video review demonstrated greater perceptions of confidence and preparedness for residency interviews both immediately after the mock interview activity and after they completed their actual residency interviews.Furthermore, students who received feedback with video review trended toward being more likely to feel that their interviewing skills had improved and that the feedback they received was helpful.These data suggest that the process of watching oneself while receiving feedback may have a greater potential to effect change in performance and behavior.
These findings are consistent with what has long been suspected.The importance of feedback towards the acquisition of clinical skills has been clearly demonstrated in the literature and these same principles likely apply to the acquisition of interviewing skills as well.In JAMA in 1983, Ende commented that "feedback occurs when a student is offered insight into what he or she actually did as well as the consequences of his or her actions.This insight is valuable insofar as it highlights the dissonance between the intended result and the actual result, thereby providing impetus for change."He further stated that "as a compendium of cognitive, psychomotor, and affectual behaviors, clinical skill is easier demonstrated than described" and that "like ballet, it is best learned in front of a mirror." The neurobiology behind the use of video review to promote self-reflection and change in behavior has also been investigated in the psychology literature.For example, Knoblich and Sebanz showed that dancers demonstrate increased brain activity when watching videos of themselves dancing rather than watching others perform the same dance.(Knoblich and Sebanz, 2006) This suggests that there is selective activation of the internal repertoire that allows us to distinguish our own actions from those of others and that seeing ourselves in action has a greater resonance in our neurons.
This concept has been similarly demonstrated in the literature across various disciplines where VSR has been shown to be an effective educational tool.For example, in 2015 Chen et al. used VSR to help attending surgeons reflect on the guidance they were giving to surgical residents in the operating room (OR), which is a key element in developing resident autonomy and directly influences resident confidence and the ability to operate independently when transitioning from resident to attending surgeons.Attending surgeons tended to underestimate the amount of guidance they were providing to residents in the OR and the residents had a significantly different perception about the amount of OR guidance they were receiving.VSR facilitated attending surgeons' reflection on their behavior and decision making in the OR that influenced the amount of guidance they were providing to their trainees.(Chen, Williams and Smink, 2015) VSR has also been used in the primary care realm, where it has been implemented to help primary care physicians reflect on their In keeping with these studies, our data suggest that receiving verbal feedback while simultaneously reviewing the video recording of their mock interview aided the students' reflection on their verbal and non-verbal communication leading to a greater understanding of their behaviors and social interaction in the mock interview.This likely resulted in what was  perceived by the students to be more meaningful feedback and identification of specific areas for improvement in interviewing skills leading to a greater sense of confidence and preparedness for their residency interviews.
Several limitations of our study must be acknowledged.Since the mock interview activity consisted of both the roundtable discussion with residency program directors and the individual mock interviews with feedback, it is difficult to know if the students' increased perceptions of confidence and preparedness were from the mock interview itself or from the activity as a whole.Also, our sample size was small and did not allow for detailed regression to adjust for potential confounders that could explain the differences observed between the video review and no-video review groups.However, there was no evidence to suggest imbalance in the randomization as shown in Table 1.One potential confounder is the fact that the students randomized to the video review group all interviewed with and received feedback from faculty who were experienced in providing feedback using video review.Therefore, the enhanced effectiveness of the experience for the video review group could be due to the increased skill level and experience in giving feedback that these faculty had, rather than the video review itself.Most notably, our study was designed only to assess student perceptions of their confidence and preparedness -not their actual performance in residency interviews or match success.Since the mock interview program was voluntary, there is also a potential volunteer bias.Despite these limitations, we feel that our study is the first to demonstrate the feasibility and utility of VSR in the process of preparing medical students for residency interviews.

Conclusion
In light of the highly competitive residency match climate, attempts to best prepare senior medical students for success in the match will remain of the utmost importance in the coming years.Our study reveals that a mock interview program including individual mock interviews and verbal feedback with faculty is a feasible and effective approach to improve students' confidence and perceived preparedness for residency interviews.The increased resource utilization required to provide feedback with video review seems to be justified given the enhanced effectiveness of the mock interview activity that it provided.Future studies designed to assess interview performance and outcomes by match success rate would be most helpful.
<0.01 0.04 1 Wilcoxon rank-sum comparing median change in aggregate score between video review and no video review groups.

Take Home Messages
Students must perform well in residency interviews for success in the match.
There is no standardized approach to prepare students.
Mock interviews followed by verbal feedback increased students' perceptions of confidence and preparedness.
Video review increased students' confidence in their interviewing skills.
Mock interviews with video review may be an effective way to prepare medical students for residency interviews.The authors describe a well-designed research study which has the potential to positively impact medical students' residency interview preparation in the future.The number of students who chose to participate was relatively small and therefore the comparative data from the two groups (video review versus non video review) was limited by statistical power, but there was a demonstrable positive effect of doing the preparatory interview in general.I wonder whether this would have been better designed with a "placebo" arm (i.e.students who choose to participate randomized to either mock interview with video review versus no mock interview at all) to better gauge the overall impact of the mock interview.The fact that all students who chose to participate did receive some interview practice was probably helpful to the students themselves but may dilute the true impact of interview practice versus none when the comparison was limited to one type of feedback versus the other.Nevertheless, this was certainly an interesting study which could be pursued further at the authors' institution as well as at others.I wish the authors had spent a bit more time on a "next steps" section discussing their plans (or lack thereof) for expansion of the mock interview process to all medical students planning to apply for residency.

Tina Schade Willis Indiana University
This review has been migrated.The reviewer awarded 4 stars out of 5 The authors implemented a feasible process to provide video-stimulated recall interview preparation for residency match.Although small and at a single center, this was nicely done with a straightforward process that could be adapted at other programs.My main criticism is the outcome measure being perception of preparedness rather than some measure of actual performance in residency interviews.This limitation was identified by the authors in the discussion.I am also interested if the program continued to use video-stimulated recall after the study was complete.The time and resources needed for implementing this program would be helpful follow up in for readers that are interested in sustainability of such a mock interview program in medical schools.Overall this was nicely done and reinforces the importance of a standardized approach to residency interview preparation.
Competing Interests: No conflicts of interest were disclosed.

Trevor Gibbs AMEE
This review has been migrated.The reviewer awarded 4 stars out of 5 Pre-residency interviews for selection and training have become more common now in many parts of the world, so this paper covers an important area for development.The structure and clarity of the paper is very good and the methodology chosen is appropriate.The conclusion section provided by the authors "says it all" and although I don't feel that the paper introduces us to anything new, it does bring out some very important issues related to the power of effective feedback-a subject or topic that still has more questions than answers.The addition of the video as a method of the faculty showing and clarifying their observations certainly seemed to add to the overall value of the feedback, but I think that the process of feedback, and using faculty, used to structuring effective feedback, made the exercise very worthwhile.I am not sure that the pre and post test questionnaires gave much new information, and as the authors suggest, they gave perceived improvements in confidence; a much more extended evaluation is required in this situation and I hope that the authors can follow this group of students through.I also wondered how much feed-forward was provided to the students; how to improve and move forward.On another note regarding the methodology, I wondered as to the value of collecting so much demographic information-was it really necessary and what extra did it bring to the paper?A good paper for all involved in interviewing to read.

Figure 1 .
Figure 1.a).Student participation and randomization.b).A student receiving verbal feedback with video review (both individuals gave consent for this image to be published)

Figure 2 .
Figure 2. a).Specialty choices of students participating in the mock interviews; Numbers indicate how many students interviewed for each specialty.b).Specialty choices of all students in the class; Numbers indicate how many students planned to apply to each specialty.

Figure 3 .
Figure 3. a) Median (IQR) aggregate Likert scores for Pre-Mock Interview, Post-Mock Interview, and Post-Residency Interview surveys respectively.b) Change in aggregate score from Pre-Mock Interview to Post-Mock Interview and Post-Residency Interview surveys respectively.

Table 1 .
Demographic information and baseline characteristics 1Wilcoxon rank-sum comparing continuous variables and Fisher's Exact comparing categorical variables.

Table 2 .
Comparison of preparedness, question expectation, confidence, and the aggregate score at different time points 1 Wilcoxon signed-rank test comparing Post-Mock Interview scores vs Pre-Mock Interview scores. 2 Wilcoxon signed-rank test comparing Post-Residency Interview scores vs Pre-Mock Interview scores

Table 3 .
Comparison of change in preparedness, question expectation, confidence, and the aggregate score by video review status

Table 4 .
Comparison of response to additional post-mock interview survey questions by video review status Feedback in clinical medical education.Journal of the American Medical Association.250, pp.777-781.Reference Source Garner, M.S., Gusberg, R.J., and Kim, A.W. (2014) The positive effect of immediate feedback on medical student education during the surgical clerkship.Journal of Surgical Education.71, pp.391-397.Reference Source Gong, H. Jr., Parker, N.H., Apgar, F.A., and Shank, C. (1984) Influence of the interview on ranking in the residency selection process.Medical Education.8, pp.366-369.
National Resident Matching Program (NRMP).(2016) Results of the 2016 NRMP Main Residency Match Largest on Record as Match Continues to Grow.Available at: Reference Source (Accessed: 12 December 2016).Paskins, Z., McHugh, G., and Hassell, A.B. (2014) Getting under the skin of the primary care consultation using video stimulated recall: a systematic review.BMC Medical Research Methodology.14, p. 101.
This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Reviewer Report 19 August 2018 https://doi.org/10.21956/mep.19627.r29211© 2018 Gibbs T. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.