Research article
Open Access

MMI - An unbiased approach to health education selection?

Gerens Curnow[1]

Institution: 1. University of Exeter Medical School
Corresponding Author: Mr Gerens Curnow ([email protected])
Categories: Selection
Published Date: 28/05/2018

Abstract

Background

Widening participation to health science degrees is a vital part of our efforts to ensure a high-quality pool of graduates in clinical fields. With the increasing usage of multiple mini interviews for selection to these courses, it is crucial that we recognise the biases within these selection processes.

Aims

This paper aims to examine recently published literature to determine the extent to which demographic factors (age, race, sex and socio-economic background) impact on MMI scores for applicants to health science degrees.

Methods

A literature search was conducted using the Medline and SCOPUS databases for literature published from 2015 to present. Relevant papers were identified through a Boolean search, and individually analysed to determine their relevance to this review.

Results

This review identified nine relevant papers. Biases were identified in all four domains, but the evidence was mixed and of varying quality. The strongest evidence for a bias was found in papers looking at socio-economic background.

Conclusion

Further research is required to determine the extent to which the MMI approach is biased against certain groups of applicants, and to identify ways to address these imbalances, as evidence of the impact of demographics on MMI score has been identified.

Keywords: Multiple Mini Interview; Bias

Introduction

The topic of Widening Participation (WP) is experiencing a lot of attention at present. A recent consultation by the Department of Health on expanding undergraduate medical school places includes a focus on how universities will ensure they are recruiting underrepresented demographics, which highlights the current atmosphere with regards to WP (Department of Health 2017). Given the relatively low attrition rates in many health science degrees (Rees et al. 2016), ensuring that the selection process for entry onto these courses is reliable and valid is of paramount importance.

A broad range of attributes and skills, both cognitive and non-cognitive, is vital for everyone working in the clinical environment (Ferguson et al. 2002). Deciding which attributes to test poses a challenge for health science institutions. Of all the tools available to selection committees, from standardised admission tests, such as the UK Clinical Aptitude Test or the Pharmacy College Admissions Test, to personal statements and academic achievement, the interview has perhaps undergone the most change in recent years.  

Interviews allow institutions to assess candidates against an extremely wide range of criteria, depending on the assessment aims of each institution; from biomedical knowledge and statistical ability, to communication skills and empathy. It is perhaps surprising therefore that there are only two main interview modalities that are currently employed by health science institutions: Multiple Mini Interviews (MMI’s) and panel interviews (Rees et al. 2016).

MMI’s were first developed in 2004 to improve upon the relatively poor predictive validity of the contemporary non-cognitive assessments, such as panel interviews and personal statements (Rees et al. 2016; Reiter et al. 2007). Their uptake by health science degree providers has been rapid, and much research has been carried out into their efficacy as a selection tool.

Given the current emphasis on WP, and the importance of the selection process to health science degrees in determining the make-up of the workforce in health professions, it is vital that any hidden biases within the MMI system are well understood. Bias in health science selection procedures, which refers to the phenomenon wherein attributes other than those designed to be assessed (e.g. race, age, socio-economic background) affect performance in an assessment (Rees et al. 2016), has the potential reduce the quality of the workforce by eliminating potentially high-performing candidates.

This review will examine the recently published literature to investigate the extent to which MMI’s can be discriminatory in terms of age, sex, socio-economic background (SEB), and race. This will update the findings of the BEME Guide No. 37, which provided an insight into the literature on this topic from 2004-2014 (Rees et al. 2016). While differences in the success rates of these candidates may not necessarily indicate a bias within the system (e.g. if desirable traits tend to exist more commonly in one group than another), it will provide a useful metric for assessing the validity of the assessment and provide a grounding for further research on this topic.

Search Methods

The literature search for this review was conducted on Medline and SCOPUS on 15th October 2017. The search used to generate the literature for this review was:

(mmi OR "multiple mini interview*" OR "multiple-mini-interview*" OR "multiple-mini interview*") AND (sex OR socioeconomic OR socio-economic OR bias OR race OR aboriginal OR age)  

The search was limited to papers published since 2015 to ensure it provided an update to the BEME Guide No. 37, the literature search for which was updated in 2014 (Rees et al. 2016). This is a narrower search than the one used for the BEME review, as the present study is looking only at bias and not at face validity, acceptability etc. This search revealed 154 papers, which were assessed against inclusion and exclusion criteria (table one), leaving nine papers for use in the review (see Figure 1).

Table 1 – Inclusion and Exclusion Criteria

Inclusion Criteria

Exclusion Criteria

  • Primary research
  • Examining bias within MMI interviews in a health sciences setting
  • At least one of the following examined in terms of bias:
    • Race
    • Age
    • Sex
    • Socioeconomic Background
  • Published since 2015
  • Not primary research
  • Not health sciences specific
  • Published before 2015
  • Paper not in English

Results

The nine papers identified examined entry onto three different health science courses: six for medicine, two for nursing, and one for pharmacy. A summary of these papers can be found in table 2.

Discussion

Age

Six papers investigated the impact of applicants’ age on MMI score; three in applicants to medicine, two to nursing, and one to pharmacy  (Traynor et al. 2017; Leduc et al. 2017; Kim et al. 2017; Pau et al. 2016; Gale et al. 2016; Cox et al. 2015). Two studies found age did impact on MMI score, while four found it didn’t.

Traynor et al. (Traynor et al. 2017) found that being older provided a small but significant increase in MMI score in nursing students at Queen’s University Belfast (n=110). They reported that, for every 10 extra years of age, candidates’ MMI scores improved by 5 points (an increase of 0.5 SD, p=0.008). This study was conducted on 1st year students, who had already passed a panel-based interview and gained entry to the course, which may have impacted the results. It was also heavily dominated by female students (92%), which limits the generalisability of the findings.

Leduc et al. (Leduc et al. 2017) used the self-reported socio-economic and demographic data from 1,089 applicants to three French-speaking Canadian medical schools, to examine the extent to which these factors impacted MMI scores. Their analysis revealed that candidates aged 25-29 had a small but significant advantage over those aged <20 years (p=0.001). While this study has the advantage of a large sample size, all MMI’s were conducted in a single centre. Further, it is unclear what impact the fact that the interviews were conducted in French might have on the study’s generalisability.

Four other studies (total n=754) found no impact of age on MMI result (Kim et al. 2017; Pau et al. 2016; Gale et al. 2016; Cox et al. 2015). These findings mirror the BEME Guide No. 37, which also reported conflicting results regarding the impact of age on MMI score (Rees et al. 2016). These conflicting results, coupled with the lack of adjustment for confounding factors in these studies, suggests that this is an area that requires further research.

Sex

Seven papers investigated the effect of sex on MMI score. Four papers examined medical school admissions; two, nursing school admissions; and one, pharmacy admissions. Two papers identified a significant impact of sex on MMI score, while the other four did not.

Leduc et al (2017) found that female students performed significantly better in MMI’s for three French-Canadian medical schools than their male counterparts, but only in the subgroup that spoke French at home (p=0.003). This implies a previously unexplored interaction between sex and language, which warrants further study.

Ross et al (2017) examined the MMI scores of 526 medical applicants at the University of Calgary. It found that female students achieved a higher score than their male counterparts (p<0.001), even when they controlled for confounding variables (e.g. MCAT score, applicant age etc). Two interpretations for this difference are proposed. Firstly, that female applicants are more likely to possess the attributes of interest in the MMI assessment (communication skills, critical thinking, ethical decision making etc) which, if true, this would support the validity of the MMI process. The other is that the interviewers are victims of confirmation bias, in that they expect female applicants to possess these skills, and therefore mark them more highly. The authors of this study recommend a longitudinal, health outcome based approach to investigate this further. None of the other studies included in this analysis showed any significant effect of sex on MMI scores (Traynor et al. 2017; Kim et al. 2017; Pau et al. 2013; Gale et al. 2016; Cox et al. 2015).

Race

Four studies investigated the impact of race on MMI scores; three on medical school applications, and one on pharmacy (Leduc et al. 2017; Pau et al. 2016; Cox et al. 2015; Jerant et al. 2015). One of these studies reported a significant impact. Leduc et al (Leduc et al. 2017) found that applicants who self-declared as “Chinese” or “Southeast Asian” performed significantly worse than all other ethnic groups (p=0.013, 0.003 respectively). It also found that applicants who identified as “White/Caucasian” performed significantly better than other ethnic groups (p=0.012). In a qualitative investigation of the potential reasons behind this, it was reported that many of the interviewers had come across a stereotype that Asian students are “shy”, and that this may have impacted on the scores given through a confirmation bias. The other three papers reported no significant impact.

Socio-economic Status

Three papers analysed the impact of socio-economic background (SEB) on MMI scores (Leduc et al. 2017; Taylor et al. 2015; Jerant et al. 2015).

Leduc et al. used a self-assessment form to categorise applicants into those with a family income of either <$100,000, $100,000 to $250,000, or >$250,000, and compared the MMI scores for applicants within these categories. It found that students who declared a parental income of >C$250,000 had significantly higher MMI scores (M=252.8) than those in the C$100,000 to C$250,000 (M=243.3, p=0.035) and <C$100,000 (M=237.1, p<0.001) groups. This study is limited by the fact that it relies on the applicants’ self-assessment of family income, but nonetheless draws into question the validity of the process.

Jerant et al. (Jerant et al. 2015) examined eight different factors, including parents’ education level, whether the applicant had contributed to family income, and whether the applicant was in receipt of income support, to create a composite score for SEB. This was applied to 1,420 MMI interviewees at the University of California. They found that candidates from a more deprived background had a significantly lower MMI score (p=0.03), but that this was only a small a small impact of less than 0.2 S.D. However, being from a more disadvantaged background was associated with a greater probability of being accepted onto the course (OR 3.28, p>0.001), since the final decision was made on a wide range of factors beyond MMI score, including personal statements and MCAT scores. This shows that biases within the MMI system can be compensated for by the admissions process as a whole.

Taylor et al. (Taylor et al. 2015) found no significant relationship between the MMI score achieved by an applicant and the status of the applicant’s school (selective/fee paying vs non-selective and non-fee paying). Nor was there a relationship between MMI score and Higher Education participation rates in the area in which the applicant lived.

Study Limitations

This study has several limitations. As discussed previously, it does not examine the reasons behind differences in success rates among applicants with different demographics. While confirmation bias in the examiners may explain these differences, in many instances the true reasons could be more complex, with an interplay of valid and invalid factors influencing the score candidates receive. Further, this review has not examined the impact of the demographics of the assessor on the levels of bias within the MMI system, which presents an interesting topic for further study. Lastly, this review has presented no evidence regarding the ways in which these impacts should be mitigated.

Conclusion

This review has found mixed evidence for the impact of demographics on MMI score for the selection of health science students. For all aspects examined in this review (age, sex, socio-economic background, and race), there is some evidence of bias within the MMI process, most convincingly in terms of candidates’ socio-economic backgrounds. This demands further research, with the aim of determining the long-term impact of bias on health outcomes, and the methods through which these biases can be mitigated.

Take Home Messages

  • MMI’s have been increasingly employed in recent decades
  • The last major investigation into the validity of this assessment type was completed in 2014
  • This review has found mixed evidence, with some studies indicating the presence of bias
  • Educators should take steps to mitigate the impact of these biases
  • More research is needed to investigate the potential reasons behind this data

Notes On Contributors

Gerens Curnow is an undergraduate medical student at the University of Exeter, currently undertaking an MSc in Clinical Education.

Acknowledgements

Prof Karen Mattick and the Clinical Educaiton team, for your support and helping me develop educational research skills.

Bibliography/References

Cox, W.C. et al., 2015. Development and assessment of the multiple mini-interview in a school of pharmacy admissions model. American Journal of Pharmaceutical Education, 79(4).

Department of Health, 2017. Expansion of Undergraduate Medical Education,

Gale, J. et al., 2016. Student nurse selection and predictability of academic success: The Multiple Mini Interview project. Nurse Education Today, 40, pp.123–127.

https://doi.org/10.1016/j.nedt.2016.01.031

Jerant, A. et al., 2015. How Medical School Applicant Race, Ethnicity, and Socioeconomic Status Relate to Multiple Mini-Interview-Based Admissions Outcomes: Findings from One Medical School. Academic Medicine, 90(12), pp.1667–1674.

https://doi.org/10.1097/ACM.0000000000000766

Kim, K.-J., Nam, K.-S. & Kwon, B.S., 2017. The utility of multiple mini-interviews: experience of a medical school. Korean journal of medical education, 29(1), pp.7–14.

https://doi.org/10.3946/kjme.2017.48

Leduc, J.M. et al., 2017. Impact of sociodemographic characteristics of applicants in multiple mini-interviews. Medical Teacher, 39(3), pp.285–294.

https://doi.org/10.1080/0142159X.2017.1270431

Pau, A. et al., 2013. The Multiple Mini-Interview (MMI) for student selection in health professions training – A systematic review. Medical Teacher, 35(12), pp.1027–1041.

https://doi.org/10.3109/0142159X.2013.829912

Pau, A. et al., 2016. What does the multiple mini interview have to offer over the panel interview? Medical Education Online, 21(1).

https://doi.org/10.3402/meo.v21.29874

Rees, E.L. et al., 2016. Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37. Medical Teacher, 38(5), pp.443–455. Available at:

https://doi.org/10.3109/0142159X.2016.1158799

Reiter, H.I. et al., 2007. Multiple mini-interviews predict clerkship and licensing examination performance. Medical Education, 41(4), pp.378–384. Available at:

https://doi.org/10.1111/j.1365-2929.2007.02709.x

Ross, M. et al., 2017. Are Female Applicants Rated Higher Than Males on the Multiple Mini-Interview? Findings from the University of Calgary. Academic Medicine, 92(6), pp.841–846.

https://doi.org/10.1097/ACM.0000000000001466

Taylor, C.A., Green, K.E. & Spruce, A., 2015. Evaluation of the effect of socio-economic status on performance in a Multiple Mini Interview for admission to medical school. Medical teacher, 37(1), pp.59–63.

https://doi.org/10.3109/0142159X.2014.923562

Traynor, M. et al., 2017. Identifying applicants suitable to a career in nursing: a value-based approach to undergraduate selection. Journal of Advanced Nursing, 73(6), pp.1443–1454.

https://doi.org/10.1111/jan.13227

Appendices

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Reviews

Please Login or Register an Account before submitting a Review

Trevor Gibbs - (23/02/2019) Panel Member Icon
/
A very well written and balanced paper with some new references on the subject. Although there are quite a few papers now in circulation regarding the MMIs, there seems to be work still to be done, especially in the area of how MMIs probably need to be adapted to the widening access admission programmes.
A paper all involved in student selection should read
David Bruce - (31/05/2018)
/
This is a well written and interesting paper which covers an important topic. The MMI is increasingly used in student selection, and this paper which covers research from 2014 onwards adds to the evidence produced in the BEME guide. Although the results show a mixed picture in how the age, gender, race and socio-economic status of applicants can effect success at MMI, some evidence of decreased performance in those from lower socio-economic groups is an important finding in the drive to widen access to the health professions. This calls for more research and a balance of admissions criteria. An enjoyable read that will be of great interest to all involved in student admissions.
Sateesh Babu Arja - (29/05/2018)
/
Very thoughtful paper and I enjoyed reading it. The four demographics selected that were selected for this study are Age, sex, race, and socioeconomic background are very interesting. Mni-multiple interviews itself is a very curious topic in the current medical students' selection and admission process. There are many articles in favor of MMI vs Traditional interviews. But the same way we have some papers in the literature offering no special added advantage with MMI. One such paper published recently in Academic Medicine Journal is "Do Multiple Mini-Interview and Traditional Interview Scores Differ in Their Associations With Acceptance Offers Within and Across Five California Medical Schools?"

Thank you for providing the detailed analysis based on the literature regarding the bias associated with age, sex, race, and socio-economic background for scoring in MMI. There was no conclusive evidence as author quoted. It requires further research. This paper is interesting for interviewers and admissions committees of the medical programs.
P Ravi Shankar - (28/05/2018) Panel Member Icon
/
This is an interesting study examining the impact of four demographic characteristics (age, gender, race and socioeconomic background) on performance in multiple mini-interviews (MMIs). The methodology of the study/review is well described. Limiting the search to the Medline and SCOPUS databases may have excluded some studies dealing with this topic. MMI is being increasingly used for admission to health profession schools. The paper is well written. Educators involved in admissions to health professions schools will find this manuscript of special interest.