Research article
Open Access

The Relationship between Students’ Probationary History, Failure to Follow Career Counseling and Entry into the Supplemental Offer Acceptance Program (SOAP): A Comparison between Two Medical Schools

Scott Cottrell [1], Julie Phillips [2], Jason Hedrick[3], Linda S. Nield[4], Jeffrey Vos[5], Brian Mavis [6], Norman D. Ferrari III[7]

Institution: 1. West Virginia University School of Medicine, 2. Michigan State University College of Human Medicine, 3. West Virginia University School of Medicine, 4. West Virginia University School of Medicine, 5. West Virginia Univesity School of Medicine, 6. Michigan State University College of Human Medicine, 7. West Virginia University School of Medicine,
Corresponding Author: Dr Scott Cottrell [email protected]
Categories: Medical Education (General), Students/Trainees, Postgraduate

Abstract

Introduction: A probationary history reported on students’ Medical Student Performance Evaluations (MSPE) may have implications on whether medical students match with a residency program in the National Residency Matching Program (NRMP) in the United States (US).  Students who fail to follow academic advisors’ advice and apply a considered residency application plan may be in jeopardy of needing to enter the Supplemental Offer Acceptance Program (SOAP).  The SOAP is a much less desirable means to securing a position in a US residency program.  The purpose of this study was to examine how a probationary history reported on the MSPE and failing to follow academic advice may put students at risk for entering the SOAP. 

Methods: The NRMP results for 3 graduating classes (N=380) at West Virginia University School of Medicine (2013-2016) and 2 graduating classes (N= 378) at Michigan State University College of Human Medicine (2015-2016) in the US were examined.  A Pearson’s chi-square was calculated to determine whether students with a probationary history were more likely to enter the SOAP.   The numbers and percentages of students in the SOAP who failed to engage a back-up specialty plan or interview at an appropriate number of programs were also identified.  

Results: There was a statistically significant association between a probationary history and whether a student was required to enter the SOAP at both West Virginia University (X2(1) = 11.15,p < .001) and Michigan State University (X2(1) = 32.91, p < .001).  The majority of students who entered the SOAP at both schools also failed to follow advice. 

Conclusion: While a probationary history may put students at risk for entering the SOAP, failing to listen to career advisors’ counsel during the application process may be more important in achieving success in the NRMP.  Future investigations may explore the characteristics of those students who do not adhere to career advice.

Keywords: career advising, residency match, mentoring

Introduction

Each year, graduating medical students in the United States (US) secure a residency position by participating in the National Residency Matching Program (NRMP).   The NRMP match, also referred to as just “the match,” is becoming more and more competitive.  Particularly competitive programs (e.g., plastic surgery) receive substantially more student applications than there are available residency positions.  If a student fails to match to a residency position, then the student enters the Supplemental Offer Acceptance Program (SOAP).  The SOAP is a less desirable means to secure a residency position for the following reasons: it occurs after the match and includes only those positions left unfilled in the US;  not all specialities will have unfilled positions after the match; and most SOAP positions are only for one year preliminary programs, such as surgery and internal medicine.  While completing a one year position will allow residents to apply for a medical license in the US, the limited one-year programs do not allow residents to become board certified in a medical specialty which would then force a student to re-enter the match again the following year.

Despite increasing numbers of available residency positions, hundreds of senior medical students have been unable to secure a position in the match and therefore have participated in the SOAP for the past several academic years (Greenberg, 2015).  In general, students can no longer limit their number of interviews and expect to match in a US residency, particularly when applying to competitive specialties without a “back-up” or “parallel” plan, which requires students to apply to both preferred, competitive programs (e.g., neurosurgery) and to less competitive programs (e.g., family medicine) (Kirch, 2015).  Academic advisors are encouraged to apply several resources to help students evaluate their competitiveness, such as the NRMP Charting Outcomes in the Match, the Results of Program Director Survey, and the Association of American Medical Colleges (AAMC) Careers in Medicine website (NRMP, 2014).  These efforts and resources aim to maximize students’ ability to secure a residency program in the match. 

Most US medical schools also have a committee that determines whether medical students are promoted to the next academic year.  These committees also decide whether students should receive a probationary status or be dismissed from the program because of academic or professionalism deficiencies.  Members of academic standards and promotion committees may be concerned whether promotion, probationary status, and remediation decisions made today will impact students’ career options for tomorrow.  For example, earning a probationary status may prevent a student from being considered for a residency position.  However, many educators also believe it is crucial to communicate honest information in academic records and in the medical student performance evaluations (MSPE), which is a summary of students’ academic progress that is necessarily sent to every residency program to which a student applies (Edmond et al, 1999).  Some educators argue that it does no good to suppress negative information, which only serves to obfuscate the residency selection process.  Indeed, guidelines give clear instructions on the content and organization of the MSPE, including information about whether a student was required to repeat any coursework or received an adverse action, such as probation (AAMC, 2016). 

The purpose of this exploratory study was to examine factors that may be associated with students failing to match at two allopathic US medical schools: West Virginia University School of Medicine (WVU) and Michigan State University College of Human Medicine (MSU). Similarities and differences between the two schools’ match statistics may suggest how probationary histories reported on the MSPE and students’ ability to follow academic counsel may put students at risk for entering the SOAP.

The two schools use different approaches to evaluate students’ academic histories, which are reflected in the MSPE. While both WVU and MSU utilize an honors/pass/fail grading system, MSU also utilizes a “conditional pass” (CP) designation.  A CP grade is the result of a student who has failed a specific component of a course.  The student can remediate the failed component without repeating the entire course.  Students who fail a course at WVU must repeat the entire course, regardless of which specific component or components fall below the passing standard.  

Both WVU and MSU distinguish between academic and professionalism probation status, and report a student’s probation as an adverse action on the MSPE.  The schools’ promotion and academic standards committees, however, have different interpretations of probation and consider different conditions that must occur before placing a student on probation.  The Committee on Academic and Professionalism Standards (CAPS) at WVU places students on academic probation for failing any one of the required courses and clerkships in the curriculum, which is a strict interpretation of a probationary status.  The student performance committee (SPC) at MSU has a different interpretation of academic probation.  MSU implements an academic probation status when a student has academic issues beyond failing only one course.  For example, if a student fails two courses or fails one course and earns a CP, then an academic probation status may be designated.  Professionalism probation may also be designated on a student’s MSPE as an adverse action.  The types of student behaviors and incidents that trigger a professionalism probation vary across circumstances.  CAPS and SPC will place a student on professionalism probation if the behavior was egregious enough to warrant discipline and alert the student that any additional irregularities in professional behavior may lead to dismissal. 

At both WVU and MSU, a probationary status requires students to meet with a faculty advisor who helps guide assigned students toward successful remediation.  Most students who fail a course or clerkship successfully remediate and are subsequently removed from a probationary status.  Students’ removal from probation is also reported in the MSPE.  In addition, professionalism probation is often remediated as advisors collaborate with students to correct misconduct or unprofessional behavior. 

WVU and MSU implement a robust career advising system.  Both schools have multiple career advising events.  Students are paired with an academic advisor to help prepare a residency application plan.  There are two fundamental recommendations that are emphasized to both WVU and MSU students.  Firstly, students who are applying to a competitive residency specialty should consider a parallel plan. That is, students should also apply to programs in a less-competitive specialty as a back-up plan.  Considering an alternate specialty may be particularly necessary for students who did not perform well on national standardized examinations (e.g., United States Medical Licensing Step 1 examination).  Secondly, students are advised to interview and rank at least eight or more residency programs.   These recommendations are aimed at minimizing the number of students who initially fail to match.  Students who enter the SOAP may have to abandon original specialty preferences and must select from only the available SOAP positions, the majority of which are one year preliminary programs.  If students follow academic counsel and follow-through with a measured application plan, then they may be more likely to avoid the consequences of entering the SOAP.      

Methods

Institutional review board approval for this study was granted by West Virginia University.  NRMP match results for both WVU and MSU students were examined. De-identified data were collected from the student deans who directly confer with students who must enter the SOAP.  It is a common educational practice at WVU and MSU for student deans to monitor when students are placed on a period of probation.  Specifically, deans were asked to indicate whether students had an “adverse action” reported on their MSPE (i.e., reflecting a history of probation status).  A Pearson’s chi-square analyses was also conducted to determine whether students with a probationary history are more likely to enter SOAP.   The student deans help counsel students who must enter the SOAP and often examine students’ ability to follow-through with a residency application plan.  The student deans at WVU and MSU compared their data to determine if there was an association between students’ non-compliance with academic advice and their need to participate in the SOAP. Specifically, two points of academic advice were examined: whether students had applied to a back-up plan specialty (if recommended), and whether students interviewed with at least eight residency programs. This information was summarized descriptively, using counts and proportions.

Results

The match statistics for two graduating classes at MSU (2015 to 2016) and four graduating classes at WVU (2013 to 2016) were included in the analyses (see Table 1).  Since MSU has a much larger class size, the number of graduates included in the analyses at each school were comparable.  As expected, WVU’s policy to place students on probation for failing only one course contributed to a much higher percentage of adverse actions. Both MSU and WVU had approximately 6% of their students enter the SOAP.  Both schools also had relatively low number of students who had a probationary history and entered the SOAP.  

There was a statistically significant association between a probationary history and whether a student was required to enter the SOAP at both WVU (X2(1) = 11.15, p < .001) and MSU (X2(1) = 32.91, p < .001).  As a measure of effect size, a calculated odds ratio suggested that WVU students were 4.15 times more likely to require participation in the SOAP if designated with a probationary history, while MSU students with a probationary history were 18 times more likely to enter the SOAP. In regards to students who participated in the SOAP, 9 (43%) of WVU students and 19 (79%) of MSU students did not have a probationary history reported on the MSPE.

The majority of students who entered the SOAP at both schools failed to follow the recommendations of career advisors.  Sixty-four percent of the WVU students who required SOAP participation applied to competitive programs. All of these students (100%) failed to implement a parallel plan, despite counsel to the contrary.  Further, 59% of WVU students and 75% of MSU students who entered the SOAP ranked fewer than eight programs.  These findings are consistent with extant research that concluded that unmatched students are statistically significantly less likely to rank a mix of competitive and less-competitive programs, and rank one or more less-competitive programs as a back-up plan.

Conclusions

The major limitation of this exploratory study is the reliance on a sample of students from two US medical schools.  Future investigations should examine whether these findings are consistent at other schools.  As a retrospective study, it was also not possible to identify whether WVU and MSU students who did not require the SOAP followed academic counsel.  Prospective studies may carefully track whether students’ ability to follow academic counsel influences match results.  Additional variables and their influence on match results should also be examined.  For example, we suspect that it may be more imperative for students with relatively low United States Medical Licensing Examination (USMLE) Step 1 scores to follow academic counsel.  Extant research has observed that students who fail to match have a lower mean USMLE Step 1 score (mean = 225.2) than students who have matched (mean = 233.6) (Liang et al, 2016).   

Our findings have laid the groundwork for future work.  First, we plan to investigate whether our efforts to alert students about the value of academic advice ultimately prevent student from entering the SOAP.  We plan to explore why some students refuse to listen to academic advice and jeopardize their career options.  Students need to understand the purpose of academic advising.  Advisors do not discount students’ goals of pursuing a particular specialty.  Advisors help students judge their competitiveness.  Students may be encouraged to reach for competitive specialties but also follow a measured residency application plan to find a “good fit” (Aagaard, 2015)  

We also plan to reinforce the message that while a probationary history may put students at risk for requiring SOAP, failing to follow-through with a back-up plan and ranking enough programs are common characteristics of the unmatched student.  Future investigations may also examine why students fail to follow academic advice.  Aagaard’s work suggests that failing to have a back-up plan is the result of an “overinflated” sense of applicant confidence (Aagaard, 2015).  It is also important to create an environment where students feel comfortable following-through with match advice, especially if there is concern such advice runs counter to what program directors prefer.  It’s important, then, to share these results program directors and engage in a dialogue with undergraduate medical education advisors (Chretien et al, 2015).   For example, some program directors may object to the idea of being a “back-up” for a student’s residency application plan. It is important that we explain why students are counseled to apply to back-up programs and how program directors may in fact benefit from matching an exceptional student who was not able to secure a preferred program.  

Overall, our findings suggest that while a probationary history may put students at risk for entering the SOAP, the failure to heed academic advice appears to be an even more important risk factor.  While WVU places more students on probation and, in turn, more students have a reported adverse action on the MSPE, a strikingly similar percentage of WVU and MSU students entered the SOAP.  This suggests that program directors are able to correctly judge the information reported on the MSPE.  For example, the MSPE should include information about whether a student must remediate a course or clerkship.  While students at WVU are necessarily put on probation for failing any course, students are also removed from probation for successfully remediating.  Program directors, then, are able to discern that a probation history is merely a consequence of remediation and not a sign that the student is inevitably a risk for being a successful resident.  This level of transparency is advisable and helps program directors to examine applicants more holistically in the pre-interview stage.        

The suggestions for effective medical student counseling generated from this study can be employed by institutions across the globe, no matter the process implemented for securing residency training positions.  We would argue that a fundamental educator role is to monitor students’ academic progress and professionalism without any competing biases.  Indeed, many of our advisors view a probationary status as an opportunity to leverage additional resources and counsel students who are struggling, which could ultimately prevent additional issues and dismissal from the program.  Moreover, the majority of students who listen to advisors can often overcome a probationary history and successfully match.  We plan to relay these findings at our career counseling events, which occur at multiple times throughout the curriculum.  Students should be cautioned that carefully considering how they can compete for specific specialties and following-through with a realistic residency application plan are the keys to avoiding the need to participate in the SOAP.

Take Home Messages

Risk factors for students failing to secure a residency position in the National Residency Matching Program in the United States include a history of academic probation, failing to comply with career advice to implement a "back-up" or "parallel" plan, and not ranking enough programs.

Students with a history of academic difficulties, including a history of probation, improve their chances to successfully secure a residency position if they comply with the career advice to implement a "back-up" or "parallel plan."

Notes On Contributors

Dr. Cottrell is Associate Dean for Student Services and Curriculum and Associate Professor in the Department of Medical Education, West Virginia University School of Medicine.

Dr. Phillips is Assistant Dean for Student Career and Professional Development and Associate Professor in the Department of Family Medicine, Michigan State University College of Human Medicine.

Mr. Hedrick is Director of Academic Operations and Accreditation in the Department of Medical Education, West Virginia University School of Medicine.

Dr. Nield is Assistant Dean of Admissions and Professor in the Departments of Medical Education and Pediatrics.

Dr. Vos is Associate Professor in the Department of Pathology and the Chair of the Committee on Academic and Professionalism Standards, West Virginia University School of Medicine

Dr. Mavis is Professor and Director of the Office of Medical Education Research and Development, Michigan State University College of Human Medicine

Dr. Ferrari is the Vice Dean for Education and Academic Affairs, West Virginia University School of Medicine 

Acknowledgements

Bibliography/References

1. Aagaard E. Medical student advising: Informed individualized advice is the key. Journal of Graduate Medical Education 2015:7(3):486-488.

https://doi.org/10.4300/JGME-D-15-00230.1

2. Association of American Medical Colleges. Recommendations for revising the medical student performance evaluation (MSPE); 2016. https://www.aamc.org/download/470400/data/mspe-recommendations.pdf. Accessed December 1, 2016.         

3. Chretien KC, Elnicki DM, Levine D, Aiyer M, Steinmann A, and Willett LR. What are we telling our students? A national survey of clerkship directors' advice for students applying to internal medicine residency. Journal of Graduate Medical Education: September 2015:7(3):382-387.\

https://doi.org/10.4300/JGME-D-14-00552.1      

4. Edmond M, Roberson M, Hasan N. The dishonest dean's letter: An analysis of 532 dean's letters from 99 U.S. medical schools. Acad Med 1999:74(9):1033-1035.

https://doi.org/10.1097/00001888-199909000-00019               

5. Greenberg, R; Association of American Medical Colleges. Despite Successful Match Day, concerns remain over GME slots. https://www.aamc.org/newsroom/reporter/april2014/378168/match-day.html Accessed August 26, 2015.    

6. Kirch D.G; Association of American Medical Colleges. A word from the president: Taking a lesson from the 2013 match. https://www.aamc.org/newsroom/reporter/june2013/346238/word.html Accessed August 26, 2015.

7. Liang ML, Curtin LS, Signer MM, Savoia MC. Unmatched U.S. Allopathic seniors in the 2015 main residency match: A study of applicant behavior, interview selection and match outcome. [published online ahead of print November 29, 2016]. Acad Med.

https://doi.org/10.1097/ACM.0000000000001501

8. National Resident Matching Program. Results and Data: 2014 Main Residency Match®. National Resident Matching Program, Washington, DC. 2014

Appendices

Table 1.  Number and Percentage of West Virginia University Graduates and Michigan State University Graduates across Match Variables

Match Variables

West Virginia University

N = 380

N (%)

Michigan State University

N = 378

N (%)

Probationary History

99 (26%)

10 (3%)

Entered SOAP

21 (6%)

24 (6%)

Probation History and

Entered SOAP

 

12 (3%)

 

5 (1%)

Probation History and DID NOT

Enter SOAP

 

87 (23%)

 

5 (1%)

There are no conflicts of interest.

Please Login or Register an Account before submitting a Review

Reviews

Trevor Gibbs - (20/03/2017) Panel Member Icon
/
Although this might be a US centric paper, I think that it does add value to our understanding of the "struggling " student, and like my co-reviewer I thought that the outstanding finding was the apparent reluctance of the struggling / problem student to follow advice. I wondered in fact if both schools mentioned had already missed the boat in that they need an earlier exploration of the student, why they were struggling and then perhaps earlier remediation.
Richard Hays - (20/03/2017) Panel Member Icon
/
For me the most interesting finding in this study is that a majority of students with identified problems did not follow the advice of their support team. That raises the issue of insight, a familiar topic for those dealing with students who have performance problems. It is not surprising that students with problems continue to have problems - that fits with the literature on students crossing levels , where prior academic performance best predicts performance at the next level - but this is a relatively unusual example of what happens after medical school on entering specialty training. Generalisation beyond the two medical programs is difficult, so more examples are needed. One point to note is that the proportion of students with a probationary history varies considerably between the two programs (26% vs 3%). Is this related to policies or how they are interpreted? It would seem that there is a much lower thresh hold in one program, so what impact does this have?