Research article
Open Access

Traditional Panel Interview versus Multiple Mini-Interview (MMI) in Medical School Admissions: Does Performance differ by Age, Gender, Urban or Rural, or Socioeconomic Status (Findings from one medical school)

Wanda Parsons[1], Janet McHugh[1], Yanqing Yi[1]

Institution: 1. Memorial University
Corresponding Author: Dr Wanda Parsons ([email protected])
Categories: Assessment, Students/Trainees, Curriculum Evaluation/Quality Assurance/Accreditation, Selection, Undergraduate/Graduate
Published Date: 04/12/2018

Abstract

Introduction:

Globally, medical schools are trying to widen access and to increase the diversity of their student body to be more representative of the population and to meet the future heath care needs of society. Selection methods must not disadvantage the applicants from priority groups. In Memorial University’s Faculty of Medicine, rural applicants and applicants from low socioeconomic status are priority groups.

 

Methods:

Since 2013, Memorial University has used a combination of traditional panel interviews and MMIs to interview candidates for medical school. We wondered whether applicants who participate in this medical school interview process perform differently on the MMIs compared to the traditional panel interview process and whether performance differs on either of the two interview processes based on age, sex, origin(urban or rural), or socioeconomic status.

 

Results:

The mean score on the traditional panel interview was higher than that on the MMI. Females scored higher than males on both the traditional panel interview and the MMI.  Applicants aged 22 and younger performed worse on both the traditional panel interview and the MMI than the other age groups. Neighborhood socioeconomic status, and urban/rural living status were not significantly related with applicants’ performance on the traditional panel interview or MMI.

 

Discussion:

The type of interview is not disadvantaging applicants from Memorial University’s priority areas.

 

Keywords: Selection; medical school admissions; MMI; panel interviews; SES; urban/rural; gender

Introduction

The Faculty of Medicine at Memorial University is committed to social accountability. The medical school is situated on an island in the North Atlantic with distributed learning across the province of Newfoundland and Labrador, and other areas of Canada. A large proportion of the population in the province lives in rural and remote communities. To meet the health care needs of the province, the medical school is committed to the equitable selection of a diverse student body in three priority areas:

 

  • Aboriginal peoples
  • Students from rural and remote areas
  • Economically disadvantaged students

The interview process is part of an inclusive, holistic approach to admissions at Memorial University. It takes personal characteristics as well as academic history into consideration in the selection process. Since 2013, this medical school has used a hybrid of the traditional (semi-structured) and MMI (structured) (TaMMI).  All applicants are selected for interview based on review of their written application, including academics (GPA and MCAT), personal statement, work experience, extracurricular activities and references. Interviewed applicants participate in eight (ten minute) MMI stations with one assessor at each station, and a 30-minute two-person traditional panel interview.

Structured multiple mini-interviews (MMIs) as an interview process for assessing applicants to medical schools has been used since 2002. McMaster University, which has championed the process in Canada, has shown that candidates who are admitted based on scoring well on the MMI perform better on subsequent licensing exams than those who are admitted to medical school based on other assessment processes (Eva et al, 2012). Studies of the effect of socioeconomic status (SES) on interview scores have yielded mixed results. One study found SES had no influence on panel interview scores (Lumb, Homer and Miller, 2010). Another study found lower MMI scores for applicants with lower SES (Jerant et al, 2015).  A group from Manitoba, Canada reported on a study that suggested medical school applicants who graduated from rural high schools fare worse than candidates from urban high schools on the MMI, despite comparable GPA scores (Raghavan et al, 2013). At the University of Iowa, a team compared interrater reliability and test-retest reliability for both structured and unstructured interviews and found that the unstructured format proved more reliable in both instances (Axelson et al, 2010). Also of note for this current study, a combination of both structured and unstructured interviews proved more reliable than either type alone.

There is also evidence that more (6) stations with a single assessor (interviewer) at each station provided better reliability than fewer (3) stations with two assessors at each station (Donnon and Paolucci, 2008). As well, interviewer training improves the validity and reliability of the scoring of MMIs (Roberts et al, 2008).

Demographic variables such as ethnicity, gender and SES have been shown to affect performance on cognitive tests (Jensen, 1980; Suzuki, Short, and Lee, 2011; Davis et al, 2013).  Some studies show women do significantly better on interview (Griffin and Hu, 2015; Ross et al, 2017)).   Other studies have shown older candidates have higher scores (Jerant et al, 2015; Reiter et al, 2012).  Given Memorial’s need to avoid disadvantaging applicants from priority areas, and findings from previous research, the current study focuses on whether demographic variables, especially SES, and urban/rural origin affect performance on the TaMMI process.  Analysis was also done of two other standard variables, age and gender (as self-identified by the applicant).  The number of Aboriginal applicants was too small for meaningful statistical analysis.

Methods

As noted above, the Faculty of Medicine at Memorial University uses a hybrid of the traditional (semi-structured) and MMI (structured) (TaMMI) interview process.  Interviewed applicants participate in eight (ten-minute) MMI stations with one assessor at each station, and a 30-minute two person panel traditional interview.

 

Based on four years of data on the TaMMI process, we present analyses to answer the following research questions:

 

  • Do applicants who participate in the medical school interview process at Memorial University perform differently on the MMIs compared to the traditional interview process?
  • Does performance on either of the two interview processes differ based on age, gender, origin (urban or rural), or socioeconomic status?

Automated Geographic Coding Based on the Statistics Canada Postal Code Conversion Files was used to determine SES. (Wilkins and Peters, 2012). Statistics Canada definitions were used to determine rural and urban origin. An urban area was defined as Census Metropolitan Area (CMA) or Census Agglomeration (CA) with more than 50,000 people (including the 50% metropolitan influenced zone surrounding communities). Other areas were defined as rural.

Results/Analysis

Pearson correlation

The mean score on traditional interview is significantly higher than that of MMI scores (p-value <0.001). The 95% confidence interval for the difference is (8.6, 10.2).  The mean difference between traditional interview score and MMI score is 9.4 (SD: 11.4).

 

The Pearson correlation between the score on traditional interview and the MMI score is 0.311. This correlation is significantly not zero (p-value <0.0001), but the correlation is not strong. This suggests that the traditional and MMI interviews are not redundant.

 

Multiple regression results

MMI score: Gender and age group are significant when adjusted for urban/rural origin and neighbourhood SES.

  • Gender:  0.0041 (p-value)
  • Age group: <0.0001 (p-value)

Notes: 

  1. Females performed better than males.
  2. Compared with the group of ages 27 and above, the young group of ages 22 and below performed worse and the group of ages 23-26 performed not significantly differently.
  3. Urban/rural origins and neighborhood SES are not significant. The p-values are as follows:

           Urban/rural: 0.1963

           Neighborhood SES: 0.1841

 

Traditional interview score: Gender and age group are significant when adjusted for urban/rural origin and neighbourhood SES.

  • Gender: <0.0001 (p-value)
  • Age group: 0.0352 (p-value)

Notes: 

  1. Females performed better than males.
  2. Compared with the group of ages 27 and above, the young group of ages 22 and below performed worse and the group of ages 23-26 performed not significantly differently.
  3. Urban/rural root and neighborhood SES are insignificant. The p-values are as follows:

           Urban/rural: 0.0787

           Neighborhood SES: 0.4501

 

Table 1: Demographics and scores for all applicants

 

Characteristic

 

Traditional Score

p-value

MMI Score

p-value

Age group

      <=22

     23 ~ 26

     >=27

 

 

  467 (61.8%)

  223 (29.5%)

  66 (8.7%)

 

 

61.2±10.4

62.8±9.7

63.7±11.4

 0.054

 

 

 

51.3±8.8

54.2±8.8

54.8±8.8

 

 <0.0001

 

 

Gender

   Female

   Male

 

 

  428 (56.6%)

  328 (43.4%)

 

 

63.6±10.0

59.6±10.3

 <0.0001

 

53.2±9.0

51.6±8.7

 0.013

 

Urban/rural

   Urban

   Rural  

 

  584 (77.4%)

  171 (22.7%)

 

61.6±10.0

63.1± 9.9 

0.087

 

52.7±8.9

51.7±9.0

 

 0.198

Neighborhood SES

   Highest

   Second highest

   Middle

   Second lowest

   Lowest

 

 

  304 (40.3%)

  159 (21.1%)

  101 (13.4%)

  84 (11.1%)

  107 (14.2%)

 

 

62.1±10.5

62.1±10.2

61.4±10.6

60.0±10.9

63.1±9.3

 0.328

     

 

53.2±8.7

52.7±9.1

50.7±9.0

51.6±10.1

52.5±8.1

 0.139

Application year

   2013

   2014

   2015

   2016

 

 

  229 (30.3%)

  190 (25.1%)

  170 (22.5%)

  167 (22.1%)

 

 

62.2±10.3

62.1±9.3

61.8±11.0

61.3±10.7

0.849

 

52.6±8.6

53.3±9.0

52.7±8.6

51.2±9.6

0.182

The scores are reported as mean ± standard deviation.

Discussion

The mean score on the traditional panel interview is higher than that on the MMI.

Females scored higher than males on both the traditional panel interview and the MMI. This has been shown in other interview domains where females were found to have more effective coping orientations for interviews than did males (Feeney, McCarthy and Goffin, 2015). Applicants aged 22 and younger performed worse on the MMI and traditional panel interview than the other age groups.

Neighborhood socioeconomic status, and urban/rural origin were not significantly related with applicants’ performance on the traditional panel interview or MMI.

Conclusion

These results are important as we attempt to widen access to medical school since they show SES and origin (urban /rural) do not affect performance on the MMI and traditional panel interview, suggesting that the interview process is not disadvantaging these applicants.

Take Home Messages

  • SES and origin (urban /rural) do not affect performance on the MMI and traditional panel interview.
  • The interview process is not disadvantaging these applicants.
  • Applicants score higher on traditional panel interviews than MMI.
  • Females perform better on both traditional panel interviews and MMI interviews.
  • Younger applicants score lower on both traditional panel interviews and MMI interviews.

Notes On Contributors

Dr. Wanda Parsons is Assistant Dean for Admissions and Associate Professor of Family Medicine, Faculty of Medicine, Memorial University. 

 

Ms. Janet McHugh is Past Admissions Officer, Faculty of Medicine, Memorial University.

 

Dr. Yanqing Yi is Associate Professor of Biostatistics, Community Health and Humanities, Faculty of Medicine, Memorial University.

Acknowledgements

The authors wish to thank Ms. Donnamarie Khalili for her valuable contribution to the study.

Bibliography/References

Axelson, R., Kreiter, C., Ferguson, K., Solow, C., et al. (2010) ‘Medical school preadmission interviews: are structured interviews more reliable than unstructured interviews?’ Teach Learn Med. Oct;22(4):241-5. https://doi.org/10.1080/10401334.2010.511978

 

Davis, D., Dorsey, J.K., Franks, R.D., Sackett, P.R., et al. (2013) ‘Do racial and ethnic group differences in performance on the MCAT exam reflect test bias?’ Academic Medicine. 88: 593–602. https://doi.org/10.1097/ACM.0b013e318286803a

 

Donnon, T. and  Paolucci, E.O. (2008) ‘A generalizability study of the medical judgment vignettes interview to assess students' noncognitive attributes for medical school.’ BMC Med Educ. Dec 10;8:58.   https://doi.org/10.1186/1472-6920-8-58

 

Eva, KW, Reiter, HI, Rosenfeld, J, Trinh, K, et al. GR (2012). ‘Association between a medical school admission process using the multiple mini-interview and national licensing examination scores.’ JAMA. Dec 5; 308(21):2233-40. https://doi.org/10.1001/jama.2012.36914

 

Feeney, J. R., McCarthy, J. M., and Goffin, R. (2015). ‘Applicant Anxiety: Examining the sexlinked anxiety coping theory in job interview contexts.’ International Journal of Selection and Assessment, 23: 295-305. https://doi.org/10.1111/ijsa.12115

 

Griffin, B. and Hu, W. (2015). ‘The interaction of socio-economic status and gender in widening participation in medicine.’ Medical Education, 49(1), 103–113. https://doi.org/10.1111/medu.12480

 

Jensen, A. R. (1980). Bias in mental testing. New York: The Free Press & Lee.

 

Jerant, A., Fancher, T., Fenton, J. J., Fiscella, K., et al. (2015). ‘How medical school applicant race, ethnicity, and socioeconomic status relate to multiple mini-interview-based admissions outcomes: Findings from one medical school.’ Academic Medicine, 90(12), 1667–1674. https://doi.org/10.1097/ACM.0000000000000766

 

Lumb, A. B., Homer, M.,and Miller, A. (2010). ‘Equity in interviews: Do personal characteristics impact on admission interview scores?’ Medical Education, 44(11), 1077–1083. https://doi.org/10.1111/j.1365-2923.2010.03771.x

 

Raghavan, M., Martin, B.D., Burnett, M., Aoki, F., et al. (2013) ‘Multiple mini-interview scores of medical school applicants with and without rural attributes.’ Rural Remote Health. Apr-Jun;13(2):2362. https://www.ncbi.nlm.nih.gov/pubmed/23574402

 

Reiter, H. I., Lockyer, J., Ziola, B., Courneya, C. A., et al. (2012). ‘Should efforts in favor of medical student diversity be focused during admissions or farther upstream?’ Academic Medicine, 87(4), 443–448. https://doi.org/10.1097/ACM.0b013e318248f7f3

 

Roberts, C., Walton,. M, Rothnie, I., Crossley, C., et al (2008) ‘Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school.’ Medical Education. 2008 Apr;42(4):396-404. https://doi.org/10.1111/j.1365-2923.2008.03018.x

 

Ross, M., Walker, I., Cooke, L., Raman, M., et al. (2017). ‘Are female applicants rated higher than males on the multiple mini-interview? Findings from the University of Calgary.’ Academic Medicine, 92(6), 841–846. https://doi.org/10.1097/ACM.0000000000001466

 

Statistics Canada, Census Metropolitan Area and Census Agglomeration definitions. https://www12.statcan.gc.ca/census-recensement/2016/ref/dict/geo009-eng.cfm (Accessed March 3, 2017)

 

Suzuki, L., Short, E., and Lee, C. (2011). ‘Multicultural perspectives of racial and ethnic group differences on intelligence in the United States.’ Cambridge Handbook of Intelligence. Cambridge: Cambridge University Press.

 

Wilkins, R. and Peters, P. (2012) PCCF+ Version 5K* User’s Guide. Automated Geographic Coding Based on the Statistics Canada Postal Code Conversion Files, Including Postal Codes through May 2011. Catalogue 82F0086-XDB. Health Analysis Division, Statistics Canada, Ottawa.

Appendices

None.

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

This project was reviewed by the Ethics Board (Interdisciplinary Committee on Ethics in Human Research (ICEHR)) who granted an exemption for this project as it was program evaluation and quality assurance.

External Funding

This paper has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Richard Hays - (14/03/2019) Panel Member Icon
/
This is a nice study that is important to the institution involved, and probably there as well. It is important that changes in selection processes should not detract from the mission of the school, and that was found. Indeed, selection processes should be designed to reflect the mission. I wonder how much the thinking about attracting students with certain characteristics influenced the design of the MMI? It should have been foremost in the minds of the designers, as selection should be aligned with the mission. Other medical programs should consider conducting similar studies when selection processes are altered.
Possible Conflict of Interest:

I am Editor of MedEdPublish

Balakrishnan(Kichu) Nair - (13/12/2018) Panel Member Icon
/
Medical school selection is major area of research now. Medical schools have societal responsibilities and therefore select students who can meet the societal expectation. However the big question is whether our current selection process is rigorous enough to select the students with those non-cognitive skills to become good doctors to serve the local communities?

In this study , from one medical school , they have shown a combination of traditional and MMIs to select students, did not disadvantage students from poor socioeconomic or rural areas. It is good news for all of us who are involved in medical school selection .

It will be good to know why they have used both formats of the interviews for the admission process and whether they will abandon the traditional interview which may introduce some bias because of the small sample size of the interviewers
Barbara Griffin - (12/12/2018) Panel Member Icon
/
This paper adds to the growing evidence that interviews do not discriminate against disadvantaged groups. Unfortunately, information on the number of participants, their ages, gender etc was missing as was information on the scoring system for each type of interview making it difficult to assess what the significant difference in scores between the MMI and traditional interview actually meant. I would be really interested to see if participants changed their rank on one interview compared to the other, i.e., was there a difference in z-scores?
Linda Nield - (10/12/2018)
/
Dr. Parsons and her Admissions Team should be commended for their work in evaluating the fairness of their admissions process. Their brief report answers the question posed in the Title of the work, and a helpful introduction with pertinent references to MMI and traditional interviewing are included for the reader. Although the findings cannot be generalized to all schools' admissions' processes, their work should serve as an impetus for all committees to continually engage in quality improvement.
Tripti Srivastava - (05/12/2018) Panel Member Icon
/
Congratulations to the authors for a novel study. It would have been worthwhile to detail the process of MMT and traditional panel interview and what are their attributes and limitations. The rationale of using both methods should be justified. The data is analysed independently for two methods and records no difference in performance through both methods on the basis of origin and SES. The authors may consider comparison of data to see the co-relation between scores by both methods, pattern of scores and maybe some qualitative data regarding stakeholders perception and preference about the two methods.
Lot of scope in this aspect for sure!