Research article
Open Access

Does 360° Multisource Feedback Influence the Obstetrics and Gynaecology Residency Selection Process?

Joella Xiaohong Ang[1], Jill Cheng Sim Lee[1], Felicia Yiqian Ang[2], Manisha Mathur[1]

Institution: 1. Division of Obstetrics and Gynaecology, KK Women's and Children's Hospital, Singapore, 2. Obstetrics and Gynaecology Academic Clinical Programme, SingHealth Duke-NUS, Singapore
Corresponding Author: Dr Joella Xiaohong Ang ([email protected])
Categories: Postgraduate (including Speciality Training), Selection
Published Date: 29/11/2019

Abstract

Objective

The 360° multisource feedback (MSF) is an assessment tool used to assess a physician’s competency, skills and communication by gathering feedback from multiple groups of assessors. Studies on its use in the residency selection process is limited.  This study aims to determine if 360° MSF affects faculty perception of a candidate and its influence on residency selection.

Methods

For the selection of the 2015 SingHealth Obstetrics and Gynaecology Residency Programme, 360° MSF was gathered and collated prior to the interview. During the ranking exercise, selection panel members voted on a candidate based on their performance during the interview and review of their curriculum vitae. The selection panel was then given each candidate’s 360° MSF and was asked to vote for a second time. Results of the voting before and after 360° MSF were analysed.

Results

Comparing scores before and after 360° MSF, the mean difference was 0.193 with a standard deviation of 0.380. Paired t-test of the candidate’s scores before and after 360° MSF showed a significant difference after 360° MSF was revealed (p=0.018).

Conclusion

Faculty perception is influenced by 360° MSF and can lead to changes in voting patterns of a residency selection process.

Keywords: 360°; Feedback; Multisource; Residency; Selection

Introduction

Selection of residents for a residency programme is an important process as it impacts the quality of patient care, academic progress and research output in the sponsoring institution. In addition, there are limited positions and resources to ensure high quality education for the chosen few. Workplace based assessments such as the 360° multisource feedback (MSF) may be useful in the selection process as it gives a well-rounded impression of a candidate.

The 360° MSF refers to an assessment process in which individuals receive feedback from multiple groups of assessors such as supervisors, peers, subordinates and from other staff such as allied health professionals (Antonioni, 1996; Garavan and McCarthy, 2001).  It has been used widely in residency training in the assessment of a doctor’s competencies, communication and professionalism (Archer, Norcini and Davies, 2005; Wood et al., 2006; Tham, 2007; Miller and Archer, 2010; Sasaki et al., 2015; Yama et al., 2018). The majority of studies on 360° MSF involve doctors already in training. Studies on its use in the residency selection process is limited.

The SingHealth Obstetrics and Gynaecology (OBGYN) Residency Programme introduced 360° MSF as part of its selection process. This study aims to determine if 360° MSF affects faculty perception of a candidate and if it influences residency selection. To our knowledge, this is the first study on 360° MSF and its impact on the selection process.

Methods

This study was undertaken as part of a larger study to understand residency selection in OBGYN. Results on the use of clickers in the residency selection process and the figures used have been published in The Asia Pacific Scholar by our colleagues Lee et al. (Lee et al., 2019)

The SingHealth OBGYN Residency Programme is the largest OBGYN training programme in Singapore. On average, 29 post-graduate year ones (PGY-1s) and beyond and 13 final year medical students apply for OBGYN via the national residency application portal. Of which, on average of 24 PGY-1s and beyond and 6 final year medical students are interviewed for a position in our residency programme. The number of candidates that matched successfully with the programme has declined from 17 to six over nine years in view of nation-wide manpower planning and limited resources. In addition, the SingHealth OBGYN Residency Programme has limited the intake to candidates from PGY-1 and beyond from cohort 2018.

The selection process in the SingHealth OBGYN residency programme has also changed through the years. From cohorts 2011 to 2013, the selection process involved a single-panel interview. From cohort 2014, the residency programme adopted a multi-panel round robin style of interview which included a large-panel, a small-panel and a “bullpen” style interview. In addition, from the selection of the 2014 cohort, the use of 360° MSF was introduced.

An email was sent out to all levels of staff including faculty members, senior and junior residents and allied health professionals a few weeks prior to the interview day. This document included the list of all candidates applying for SingHealth OBGYN residency that year. The document had each candidate’s name and photograph and a checkbox to indicate if a candidate was “Highly recommended”, “Neutral” or “Not recommended”.  Qualitative feedback and comments were also sought. A sample of the document can be found in Figure 1.This feedback was collected, anonymised and collated by a programme executive.

Figure 1 showing the document sent to supervisors, senior and junior residents and allied health professionals to obtain 360° feedback

Figure 1 showing the document sent to supervisors, senior and junior residents and allied health professionals to obtain 360° feedback

Photo: Colourbox.com / Ohla Klochko

 

The SingHealth selection panel comprised ten interviewers; the Program Director (PD), two Associate Program Directors (APDs), Academic Chair, four core faculty and two chief residents. After completion of the three-station round robin interview which included a large panel interview, a small panel interview and a less formal “bull-pen” interview, the selection panel convened to discuss each candidate’s performance.

Candidates’ names and photographs were shown on a screen (Figure 2) and all members of the selection panel were asked to vote using a 5-point Likert scale with 1 being “Highly Not Recommended”, 2 being “Not Recommended”, 3 being “Neutral”, 4 being “Recommended” and 5 being “Highly Recommended” with an electronic Audience Response System (ARS). Their decision was made based on information from the candidate’s curriculum vitae and interview performance whilst blinded to 360° MSF.

Figure 2 showing a sample slide for the selection panel to vote for each candidate in the first round of voting

Figure 2 showing a sample slide for the selection panel to vote for each candidate in the first round of voting

Photo: Colourbox.com / Ohla Klochko

 

After completion of the first round of voting, the selection panel was then given the 360° MSF results of each candidate. The 360° MSF collated prior to the interview was shown on a screen with the candidate’s name and photograph (Figure 3). The number of “Highly Recommended”, “Neutral” and “Not Recommended” votes from the 360° MSF were tabulated and qualitative feedback was included. Positive feedback was in black font while negative feedback was highlighted in red font. The selection panel was then asked to vote again using the same 5-point Likert scale described above with the electronic ARS. Their decision now was based on the candidate’s curriculum vitae, interview performance and 360° MSF.

Figure 3 showing a sample slide for the selection panel to vote for each candidate in the second round of voting with 360° feedback

Figure 3 showing a sample slide for the selection panel to vote for each candidate in the second round of voting with 360° feedback

Photo: Colourbox.com / Ohla Klochko

 

Data analysis including paired t-test was conducted using SPSS Statistics Version 19 to analyse differences in scoring patterns prior to and after 360° MSF. This study was undertaken as part of a larger study to understand residency selection in OBGYN. The SingHealth Centralised Institutional Review Board (CIRB) exempted this study from further review.

Results/Analysis

In 2014, 25 candidates applied for the SingHealth OBGYN residency programme and participated in the selection process for cohort 2015. The mean 360° MSF score for all 25 candidates was 3.51 (SD 0.86, 1.50-4.73). The mean 360° MSF score for the 10 successful candidates was 4.28 (SD 0.27, range 3.86-4.73), compared to 2.99 (SD 0.71, 1.50–3.79) for the 15 unsuccessful candidates (p<0.001).

After revealing the 360° MSF, 10 candidates had improved scores in which 90% (n=9/10) had at least 1 “Highly Recommended” feedback from the 360° multisource. Eight candidates had their scores reduced and 75% (n=6/8) had at least 1 “Not Recommended” feedback from the 360° multisource. Seven candidates had no change to their scores after 360° MSF.

Comparing scores before and after 360° MSF, the mean difference was 0.193 with a standard deviation of 0.380. Paired t-test of the candidate’s scores before and after 360° MSF showed a significant difference after 360° MSF was revealed (p=0.018) (Table 1).

Table 1 showing the results of the paired t-test comparing candidate scores before and after 360° MSF

Paired T-test for the Before and After 360° feedback on candidates score

 

Paired Differences

     
       

95% Confidence Interval of the Difference

     
 

Mean

Std. Deviation

Std. Error Mean

Lower

Upper

t

df

p

Before & After score

0.193

0.380

0.076

0.036

0.350

2.541

24

0.018

Discussion

360° MSF has been established as a recognised way of assessing a doctor’s performance in practice. It is comprehensive and is able to assess professional competence, behaviours, communication, collaboration with other allied healthcare professionals and attitudes in the workplace (Lockyer, 2003; Donnon et al., 2014). Having multiple assessments of a doctor allows for different evaluators to focus on different characteristics which provides a more comprehensive evaluation than what would be derived from a single source alone (Sala and Dwight, 2002).  

Whilst there are a wealth of literature on the reliability, validity and feasibility of 360° MSF in assessing a physician’s clinical performance and competence (Wood et al., 2006; Allerup et al., 2007; Massagli and Carline, 2007; Pollock et al., 2008; Donnon et al., 2014), less has been studied in its use in the residency selection process. A study published by Hsu et al (Hsu et al., 2018) concluded that 360° MSF especially from nursing staff can influence a postgraduate year 1’s choice to pursue a speciality. In Hsu et al’s paper, using Cooley’s theory of the self-looking glass (Cooley, 1998), the authors theorize that we perceive ourselves through others’ attitudes towards us and it is possible that positive assessment might influence our future hypothetical identity of who we might be (Hsu et al., 2018). This study gives insights on how 360° MSF can influence a candidate’s perception and their career choice. Our study discusses the opposite, a candidate’s 360° MSF on faculty perception and voting during a selection process.

The selection process is varied between institutions, residency programmes and specialities. In OBGYN, the interview has been considered the most important element in ranking the candidates for selection (Taylor, Weinstein and Mayhew, 1995; Olawaiye, Yeh and Withiam-Leitch, 2006; Stephenson-Famy et al., 2015). While a single-panel interview may give the selection panel some insight of a candidate’s communication and interpersonal skills, the SingHealth OBGYN residency programme adopted a multi-panel approach to ensure a balanced assessment of candidates. In addition, the authors believe that workplace-based assessment is essential in evaluating a candidate prior to selection into a residency programme. This allows the selection panel to assess non-cognitive attributes of a candidate such as professionalism, integrity, teamwork and work output on a day-to-day basis. This 360° MSF also gives the selection panel privy to the “fit” of the candidate in the programme. This may explain changes in voting patterns after 360° MSF was given to the selection panel.

The process of selecting a resident is an important task and choosing the best candidate is an extremely complex task. Moreover, members from the selection panel may have different criteria when selecting a candidate for residency programme. These criteria may differ between programme directors, the Academic Chair, core faculty members and resident representatives. Lee et al discussed that the interview panel may prioritise group performance traits such as being a team player, hardworking or having good emotional quotient as they are concerned about the needs of colleagues. On the other hand, the programme director and ACP chairman may prioritise individual abilities such as intelligence quotient, academic results and research output as they are concerned about the needs of the department and the academic unit. (Lee et al., 2014) Having the 360° MSF may be useful as it gives a global assessment of a candidate which may not be assessed during one interview.

To our knowledge, this is the first study to assess the influence of 360° MSF on faculty’s perception on candidates applying for residency and its impact on the selection process. This finding is novel and to our knowledge is the first time this association has been demonstrated. In addition, our study utilised the electronic ARS system for voting. This system allowed for anonymity of the voting process which has been shown to reduce the bias of swayed votes via the traditional manual voting process (Lee et al., 2019).

This study has its limitations. Firstly, its small sample size of a single cohort may lead to type I errors. The authors propose that larger studies including other residency programmes would allow us to better understand the role of 360 feedback in faculty perception and the residency selection process. In addition, we did not quantify the number of positive versus negative feedback or qualify its effect to cause changes in the voting scores of a candidate.

Conclusion

Selection of a candidate for a residency programme is a complex task. 360° MSF is important as it is able to give a comprehensive assessment of the candidates applying for a programme. Faculty perception is significantly influenced by 360° MSF and can lead to massive changes in voting patterns of a residency selection panel. Having the right 360° MSF may sway the selection panel into accepting or rejecting a candidate’s application. Collection and collation of 360° MSF should be done for all residency selection.

Take Home Messages

  • Selection of a candidate for a residency programme is a complex task
  • 360° MSF can give a comprehensive assessment of a candidate
  • Faculty perception can be influenced by 360° MSF
  • Collection of 360° MSF should be done for residency selection

Notes On Contributors

Dr Joella Xiaohong Ang is a Senior Resident of the SingHealth Obstetrics and Gynaecology Residency Programme. She is currently in her final year of residency.

Dr Jill Cheng Sim Lee is a core faculty member of the SingHealth Obstetrics and Gynaecology Residency Programme. She was the Chief Resident for Education and has a special interest in Medical Education.

Ms Felicia Yiqian Ang is a programme executive for the SingHealth Obstetrics and Gynaecology Residency Programme and the SingHealth Duke-NUS Academic Clinical Programme.

Dr Manisha Mathur is the Programme Director of the SingHealth Obstetrics and Gynaecology Residency Programme.

Acknowledgements

We would like to acknowledge the continued support and administrative contributions of Ms Wileen Lim, Miss Charlene Ong, Ms Yvonne Tang, programme executives and administrators at SingHealth OBGYN Residency Programme and Division of Obstetrics and Gynaecology, KK Women’s and Children’s Hospital, Singapore. In addition, we would also like to acknowledge the authors of Lee et al (Lee et al., 2019), Dr Muhammad Fairuz Abdul Rahman, Dr Ho Weng Yan, Mr Ng Mor Jack, Prof Tan Kok Hian and Prof Bernard Chern Su Min for their contributions to the study on selection processes in OBGYN and the figures from their paper. Photo: Colourbox.com/Ohla Klochko

Bibliography/References

Allerup, P., Aspegren, K., Ejlersen, E., Jørgensen, G., et al. (2007) ‘Use of 360-degree assessment of residents in internal medicine in a Danish setting: a feasibility study’, Medical Teacher, 29(2–3), pp. 166–170. https://doi.org/10.1080/01421590701299256.

Antonioni, D. (1996) ‘Designing an effective 360-degree appraisal feedback process’, Organizational Dynamics, 25(2), pp. 24–38. https://doi.org/10.1016/S0090-2616(96)90023-6.

Archer, J. C., Norcini, J. and Davies, H. A. (2005) ‘Use of SPRAT for peer review of paediatricians in training’, BMJ (Clinical research ed.), 330(7502), pp. 1251–1253. https://doi.org/10.1136/bmj.38447.610451.8F.

Cooley, C. H. (1998) On self and social organization. Chicago: University of Chicago Press.

Donnon, T., Al Ansari, A., Al Alawi, S. and Violato, C. (2014) ‘The reliability, validity, and feasibility of multisource feedback physician assessment: a systematic review’, Academic Medicine: Journal of the Association of American Medical Colleges, 89(3), pp. 511–516. https://doi.org/10.1097/ACM.0000000000000147.

Garavan, T. N. and McCarthy, A. M. (2001) ‘360° feedback process: performance, improvement and employee career development’, Journal of European Industrial Training, 25(1), pp. 5–32. https://doi.org/10.1108/03090590110380614.

Hsu, C.-M., Hsiao, C.-T., Chang, L.-C. and Chang, H.-Y. (2018) ‘Is there an association between nurse, clinical teacher and peer feedback for trainee doctors’ medical specialty choice? An observational study in Taiwan’, BMJ open, 8(4), p. e020769. https://doi.org/10.1136/bmjopen-2017-020769.

Lee, J. C. S., Ho, W. Y., Ly, P. H., Wong, M. C. Y., et al. (2014) ‘Criteria for Selection of OBGYN Residency Applicants: Do we all Agree?’ An International Association for Medical Education.

Lee, J. C. S., Muhammad, F., Ho, W. Y., Ng, M. J., et al. (2019) ‘Clickers: Enhancing the residency selection process’, The Asia Pacific Scholar, 4(1), pp. 51–54. https://doi.org/10.29060/TAPS.2019-4-1/SC1052.

Lockyer, J. (2003) ‘Multisource feedback in the assessment of physician competencies’, The Journal of Continuing Education in the Health Professions, 23(1), pp. 4–12. https://doi.org/10.1002/chp.1340230103.

Massagli, T. L. and Carline, J. D. (2007) ‘Reliability of a 360-degree evaluation to assess resident competence’, American Journal of Physical Medicine & Rehabilitation, 86(10), pp. 845–852. https://doi.org/10.1097/PHM.0b013e318151ff5a.

Miller, A. and Archer, J. (2010) ‘Impact of workplace based assessment on doctors’ education and performance: a systematic review’, BMJ (Clinical research ed.), 341, p. c5064. https://doi.org/10.1136/bmj.c5064.

Olawaiye, A., Yeh, J. and Withiam-Leitch, M. (2006) ‘Resident selection process and prediction of clinical performance in an obstetrics and gynecology program’, Teaching and Learning in Medicine, 18(4), pp. 310–315. https://doi.org/10.1207/s15328015tlm1804_6.

Pollock, R. A., Donnelly, M. B., Plymale, M. A., Stewart, D. H., et al. (2008) ‘360-degree evaluations of plastic surgery resident accreditation council for graduate medical education competencies: experience using a short form’, Plastic and Reconstructive Surgery, 122(2), pp. 639–649. https://doi.org/10.1097/PRS.0b013e31817d5fbd.

Sala, F. and Dwight, S. A. (2002) ‘Predicting executive performance with multirater surveys: Whom you ask makes a difference’, Consulting Psychology Journal: Practice and Research, 54(3), pp. 166–172. https://doi.org/10.1037/1061-4087.54.3.166.

Sasaki, H., Archer, J., Yonemoto, N., Mori, R., et al. (2015) ‘Assessing doctors’ competencies using multisource feedback: validating a Japanese version of the Sheffield Peer Review Assessment Tool (SPRAT)’, BMJ Open, 5(6), p. e007135. https://doi.org/10.1136/bmjopen-2014-007135.

Stephenson-Famy, A., Houmard, B. S., Oberoi, S., Manyak, A., et al. (2015) ‘Use of the Interview in Resident Candidate Selection: A Review of the Literature’, Journal of Graduate Medical Education, 7(4), pp. 539–548. https://doi.org/10.4300/JGME-D-14-00236.1.

Taylor, C. A., Weinstein, L. and Mayhew, H. E. (1995) ‘The process of resident selection: a view from the residency director’s desk’, Obstetrics and Gynecology, 85(2), pp. 299–303. https://doi.org/10.1016/0029-7844(94)00388-T.

Tham, K.-Y. (2007) ‘360 Degrees feedback for emergency physicians in Singapore’, Emergency medicine journal: EMJ, 24(8), pp. 574–575. https://doi.org/10.1136/emj.2007.047530.

Wood, L., Wall, P. D., Bullock, A., Hassell, A., et al. (2006) ‘“Team observation”: a six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gynaecology training in the UK’, Medical Teacher, 28(7), pp. e177–e184. https://doi.org/10.1080/01421590600834260.

Yama, B. A., Hodgins, M., Boydell, K. and Schwartz, S. B. (2018) ‘A qualitative exploration: questioning multisource feedback in residency education’, BMC medical education, 18(1), p. 170. https://doi.org/10.1186/s12909-018-1270-7.

Appendices

None.

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

This study was undertaken as part of a larger study to understand residency selection in OBGYN. As this study was an anonymous educational survey, the SingHealth Centralised Institutional Review Board (CIRB) has noted less than minimal risks to participants and has exempted this study from further review.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Ian Wilson - (29/11/2019)
/
This is an interesting paper that hints at some significant ebenfit for multisource feedback. I do have a number of concerns.

It is not clear from the paper what criteria are used in selection. It does seem the interview scores from various interviews are the primary (only) source on information used in selection prior to the sue of the MSF. I presue somewhere there is some checking of the resident or students competence. this needs to beclarified.

The style of MSF is very constrained. Most MSF have detailed criteria for reporting.

It is not clear from the paper how many MSF assessors, in total and of each type of reporter there were per candidate and if this varied significantly betweencandidates.

It would also be interesting to know if the MSF made any difference to who was selected. If it did not change who was selected then its value may be limited.

Thsi si an interesting start and continuing to research the use of MSF in selesction would be very valuable and I urge the authors to continue doing so.