Does 360° Multisource Feedback Influence the Obstetrics and Gynaecology Residency Selection Process?

Objective The 360° multisource feedback (MSF) is an assessment tool used to assess a physician’s competency, skills and communication by gathering feedback from multiple groups of assessors. Studies on its use in the residency selection process is limited. This study aims to determine if 360° MSF affects faculty perception of a candidate and its influence on residency selection. Methods For the selection of the 2015 SingHealth Obstetrics and Gynaecology Residency Programme, 360° MSF was gathered and collated prior to the interview. During the ranking exercise, selection panel members voted on a candidate based on their performance during the interview and review of their curriculum vitae. The selection panel was then given each candidate’s 360° MSF and was asked to vote for a second time. Results of the voting before and after 360° MSF were analysed. Results Comparing scores before and after 360° MSF, the mean difference was 0.193 with a standard deviation of 0.380. Paired t-test of the candidate’s scores before and after 360° MSF showed a significant difference after 360° MSF was revealed (p=0.018). Conclusion Faculty perception is influenced by 360° MSF and can lead to changes in voting patterns of a residency selection process. Ang J, Lee J, Mathur M MedEdPublish https://doi.org/10.15694/mep.2019.000220.1 Page | 2


Introduction
Selection of residents for a residency programme is an important process as it impacts the quality of patient care, academic progress and research output in the sponsoring institution. In addition, there are limited positions and resources to ensure high quality education for the chosen few. Workplace based assessments such as the 360°m ultisource feedback (MSF) may be useful in the selection process as it gives a well-rounded impression of a candidate.
The 360° MSF refers to an assessment process in which individuals receive feedback from multiple groups of assessors such as supervisors, peers, subordinates and from other staff such as allied health professionals (Antonioni, 1996;Garavan and McCarthy, 2001). It has been used widely in residency training in the assessment of a doctor's competencies, communication and professionalism (Archer, Norcini and Davies, 2005;Wood et al., 2006;Tham, 2007;Miller and Archer, 2010;Sasaki et al., 2015;Yama et al., 2018). The majority of studies on 360° MSF involve doctors already in training. Studies on its use in the residency selection process is limited.
The SingHealth Obstetrics and Gynaecology (OBGYN) Residency Programme introduced 360° MSF as part of its selection process. This study aims to determine if 360° MSF affects faculty perception of a candidate and if it influences residency selection. To our knowledge, this is the first study on 360° MSF and its impact on the selection process.

Methods
This study was undertaken as part of a larger study to understand residency selection in OBGYN. Results on the use of clickers in the residency selection process and the figures used have been published in The Asia Pacific Scholar by our colleagues Lee et al. (Lee et al., 2019) The SingHealth OBGYN Residency Programme is the largest OBGYN training programme in Singapore. On average, 29 post-graduate year ones (PGY-1s) and beyond and 13 final year medical students apply for OBGYN via the national residency application portal. Of which, on average of 24 PGY-1s and beyond and 6 final year medical students are interviewed for a position in our residency programme. The number of candidates that matched successfully with the programme has declined from 17 to six over nine years in view of nation-wide manpower planning and limited resources. In addition, the SingHealth OBGYN Residency Programme has limited the intake to candidates from PGY-1 and beyond from cohort 2018.
The selection process in the SingHealth OBGYN residency programme has also changed through the years. From cohorts 2011 to 2013, the selection process involved a single-panel interview. From cohort 2014, the residency programme adopted a multi-panel round robin style of interview which included a large-panel, a small-panel and a "bullpen" style interview. In addition, from the selection of the 2014 cohort, the use of 360° MSF was introduced.
An email was sent out to all levels of staff including faculty members, senior and junior residents and allied health professionals a few weeks prior to the interview day. This document included the list of all candidates applying for SingHealth OBGYN residency that year. The document had each candidate's name and photograph and a checkbox to indicate if a candidate was "Highly recommended", "Neutral" or "Not recommended". Qualitative feedback and comments were also sought. A sample of the document can be found in Figure 1.This feedback was collected, anonymised and collated by a programme executive. The SingHealth selection panel comprised ten interviewers; the Program Director (PD), two Associate Program Directors (APDs), Academic Chair, four core faculty and two chief residents. After completion of the three-station round robin interview which included a large panel interview, a small panel interview and a less formal "bull-pen" interview, the selection panel convened to discuss each candidate's performance.
Candidates' names and photographs were shown on a screen ( Figure 2) and all members of the selection panel were asked to vote using a 5-point Likert scale with 1 being "Highly Not Recommended", 2 being "Not Recommended", 3 being "Neutral", 4 being "Recommended" and 5 being "Highly Recommended" with an electronic Audience Response System (ARS). Their decision was made based on information from the candidate's curriculum vitae and interview performance whilst blinded to 360° MSF. After completion of the first round of voting, the selection panel was then given the 360° MSF results of each candidate. The 360° MSF collated prior to the interview was shown on a screen with the candidate's name and photograph ( Figure 3). The number of "Highly Recommended", "Neutral" and "Not Recommended" votes from the 360° MSF were tabulated and qualitative feedback was included. Positive feedback was in black font while negative feedback was highlighted in red font. The selection panel was then asked to vote again using the same 5-point Likert scale described above with the electronic ARS. Their decision now was based on the candidate's curriculum vitae, interview performance and 360° MSF. Figure 3 showing a sample slide for the selection panel to vote for each candidate in the second round of voting with 360° feedback Photo: Colourbox.com / Ohla Klochko Data analysis including paired t-test was conducted using SPSS Statistics Version 19 to analyse differences in scoring patterns prior to and after 360° MSF. This study was undertaken as part of a larger study to understand residency selection in OBGYN. The SingHealth Centralised Institutional Review Board (CIRB) exempted this study from further review.
After revealing the 360° MSF, 10 candidates had improved scores in which 90% (n=9/10) had at least 1 "Highly Recommended" feedback from the 360° multisource. Eight candidates had their scores reduced and 75% (n=6/8) had at least 1 "Not Recommended" feedback from the 360° multisource. Seven candidates had no change to their scores after 360° MSF.
Comparing scores before and after 360° MSF, the mean difference was 0.193 with a standard deviation of 0.380. Paired t-test of the candidate's scores before and after 360° MSF showed a significant difference after 360° MSF was revealed (p=0.018) ( Table 1).

Discussion
360° MSF has been established as a recognised way of assessing a doctor's performance in practice. It is comprehensive and is able to assess professional competence, behaviours, communication, collaboration with other allied healthcare professionals and attitudes in the workplace (Lockyer, 2003;Donnon et al., 2014). Having multiple assessments of a doctor allows for different evaluators to focus on different characteristics which provides a more comprehensive evaluation than what would be derived from a single source alone (Sala and Dwight, 2002).
Whilst there are a wealth of literature on the reliability, validity and feasibility of 360° MSF in assessing a physician's clinical performance and competence (Wood et al., 2006;Allerup et al., 2007;Massagli and Carline, 2007;Pollock et al., 2008;Donnon et al., 2014), less has been studied in its use in the residency selection process. A study published by Hsu et al (Hsu et al., 2018) concluded that 360° MSF especially from nursing staff can influence a postgraduate year 1's choice to pursue a speciality. In Hsu et al's paper, using Cooley's theory of the self-looking glass (Cooley, 1998), the authors theorize that we perceive ourselves through others' attitudes towards us and it is possible that positive assessment might influence our future hypothetical identity of who we might be (Hsu et al., 2018). This study gives insights on how 360° MSF can influence a candidate's perception and their career choice. Our study discusses the opposite, a candidate's 360° MSF on faculty perception and voting during a selection process.
The selection process is varied between institutions, residency programmes and specialities. In OBGYN, the interview has been considered the most important element in ranking the candidates for selection (Taylor, Weinstein and Mayhew, 1995;Olawaiye, Yeh and Withiam-Leitch, 2006;Stephenson-Famy et al., 2015). While a single-panel interview may give the selection panel some insight of a candidate's communication and interpersonal skills, the SingHealth OBGYN residency programme adopted a multi-panel approach to ensure a balanced assessment of candidates. In addition, the authors believe that workplace-based assessment is essential in evaluating a candidate prior to selection into a residency programme. This allows the selection panel to assess non-cognitive attributes of a candidate such as professionalism, integrity, teamwork and work output on a day-to-day basis. This 360° MSF also gives the selection panel privy to the "fit" of the candidate in the programme. This may explain changes in voting patterns after 360° MSF was given to the selection panel.
The process of selecting a resident is an important task and choosing the best candidate is an extremely complex task. Moreover, members from the selection panel may have different criteria when selecting a candidate for residency programme. These criteria may differ between programme directors, the Academic Chair, core faculty members and resident representatives. Lee et al discussed that the interview panel may prioritise group performance traits such as being a team player, hardworking or having good emotional quotient as they are concerned about the needs of colleagues. On the other hand, the programme director and ACP chairman may prioritise individual abilities such as intelligence quotient, academic results and research output as they are concerned about the needs of the department and the academic unit. (Lee et al., 2014) Having the 360° MSF may be useful as it gives a global assessment of a candidate which may not be assessed during one interview.
To our knowledge, this is the first study to assess the influence of 360° MSF on faculty's perception on candidates applying for residency and its impact on the selection process. This finding is novel and to our knowledge is the first time this association has been demonstrated. In addition, our study utilised the electronic ARS system for voting. This system allowed for anonymity of the voting process which has been shown to reduce the bias of swayed votes via the traditional manual voting process (Lee et al., 2019).
This study has its limitations. Firstly, its small sample size of a single cohort may lead to type I errors. The authors propose that larger studies including other residency programmes would allow us to better understand the role of 360 feedback in faculty perception and the residency selection process. In addition, we did not quantify the number of positive versus negative feedback or qualify its effect to cause changes in the voting scores of a candidate.

Conclusion
Selection of a candidate for a residency programme is a complex task. 360° MSF is important as it is able to give a comprehensive assessment of the candidates applying for a programme. Faculty perception is significantly influenced by 360° MSF and can lead to massive changes in voting patterns of a residency selection panel. Having the right 360° MSF may sway the selection panel into accepting or rejecting a candidate's application. Collection and collation of 360° MSF should be done for all residency selection.

Take Home Messages
Selection of a candidate for a residency programme is a complex task 360° MSF can give a comprehensive assessment of a candidate Faculty perception can be influenced by 360° MSF Collection of 360° MSF should be done for residency selection

Notes On Contributors
Dr Joella Xiaohong Ang is a Senior Resident of the SingHealth Obstetrics and Gynaecology Residency Programme. She is currently in her final year of residency.