Review of the literature
Open Access

Assessment of Higher Ordered Thinking in Medical Education: Multiple Choice Questions and Modified Essay Questions

Arslaan Javaeed[1]

Institution: 1. University of Ottawa
Corresponding Author: Dr Arslaan Javaeed ([email protected])
Categories: Assessment
Published Date: 12/06/2018

Abstract

Background: Multiple choice questions and Modified Essay Questions are two widely used methods of assessment in medical education. There is a lack of substantial evidence whether both forms of questions can assess higher ordered thinking or not.

Objective: The objective of this paper is to assess the ability of a well-constructed Multiple-Choice Question (MCQ) to assess higher ordered thinking skills as compared to a Modified Essay Questions (MEQ) in medical education.

Methods: The medical education literature was searched for articles related to comparison between multiple choice questions and modified essay questions, looking for credible evidence for using multiple choice questions for assessment of higher ordered thinking.

Results and Conclusion:  A well-structured MCQ has the capacity to assess higher ordered thinking and because of many other advantages that this format offers. Multiple choice questions should be considered as a preferable choice in undergraduate medical education as literature shows that different levels of Bloom’s taxonomy can be assessed by this assessment format and its use for assessing only lower ordered thinking i.e. recall of knowledge, is not very convincing.

Keywords: multiple choice questions; modified essay questions; assessment higher ordered thinking

Introduction

What is Higher ordered thinking?

Higher ordered thinking is usually defined in reference to the cognitive domain of Bloom’s Taxonomy (Fig I). First two levels, which are considered as lower ordered thinking, include remembering and understanding whereas rest of the four levels, constituting higher ordered thinking, include application, analysis, evaluation, and creation of knowledge in an ascending order (Anderson, Lorin, Krathwohll, & Bloom., 2001).

 

1. Bloom’s Taxonomy to Revised Bloom’s Taxonomy

Bloom’s Taxonomy described and published in 1956 had permeated teaching for almost 45 years before it was modified in 2001. In Table 1, these existing taxonomies of cognition are discussed.

 

2. MCQs and MEQ in Medical Education

There has been a considerable revision in undergraduate medical curriculum particularly in the assessment and teaching methodology. Written tests are an essential component of medical education. Objectivity is gradually replacing subjective assessment. Long essay type questions have been substituted by MEQs and MCQs.  There is an ongoing debate on which assessment format should be administered to test higher ordered thinking (Mehta, Bhandari, Sharma, & Kaur, 2016).

Assessment formats are mere tools and their usefulness can be hampered by their poor design, proficiency of its user, deliberate abuse and unintentional misuse (Tom Kubiszyn, 2013). To establish usefulness of a particular assessment format, the following five criteria should be considered: (1) reliability (2) validity (3) influence on future thinking and practice (4) suitability to learners and teachers (5) expenses (to the individual student and institution) (Vleuten, 1996). Reliability is the degree to which a measurement produces consistent results (Salkind, 2006) and validity means that how well the test measures which it intends to measure (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014).

Discussion

MCQs are extensively used for assessment in medical education owing to their ability to offer a broad range of examination items that incorporate several subject areas. They can be managed in a relatively short period of time. Moreover, they can be marked by a machine which makes the examination standardized (Epstein, 2007). There is a general perception that MCQs emphasize on knowledge recall i.e. Level I of revised Bloom’s Taxonomy and MEQs are capable of testing higher ordered thinking. The criticism against MCQs is basically due to its poor construction rather than the format itself. A study reveals that in assessing cognitive skills, MCQs significantly correlate with MEQs when their assessment’s content is matched (Palmer & Devitt, 2007).

The modified essay question is a compromise between an essay and a multiple-choice question. Although it is well documented that a well-constructed MEQ tests higher ordered thinking, it is appropriate to ask if MEQs in undergraduate medical education are well-constructed and test higher ordered thinking. Mostly MEQs only test factual knowledge i.e. lower ordered thinking and at the same time risk significant variation in standards of marking as they are usually hand-marked (Palmer & Devitt, 2007), rendering it unreasonable as an assessment format for testing a large number of students (Sam, Hameed, Harris, & Meeran, 2016).  Besides, it is a difficult task to construct an MEQ capable of testing higher ordered thinking in students and is more frequently associated with item writing flaws (Khan & Aljarallah, 2011).

In contrast to MEQ, MCQs are suitable for testing a large number of students as they are machine scored (Morrison & Walsh Free, 2001). Research shows that multiple choice questions assessing comprehension, application and analysis have been identified. This suggests that the ability of MCQs to assess higher ordered thinking is persistently undervalued and indicates that MCQs have the potential to assess higher ordered thinking (Scully, 2017). Examples of multiple choice question assessing higher ordered thinking i.e. application (Table 2), analysis (Table 3) and evaluation levels (Table 4,5 and 6) are as follows:

For a number of purposes, the significance of measuring higher ordered thinking is well renowned in medical education. It has been debated that multiple choice format is useful because it is reliable, objective, unbiased and efficient, cost-effective in nature but incapable of measuring higher ordered thinking. This is not true. A more correct declaration would be that MCQs measuring higher ordered thinking are rarely constructed and MCQs assessing lower ordered thinking are over-presented. One of the reasons of this over presentation is that the most item writers are not formally trained. This emphasizes that format itself is not limited to the assessment of lower ordered thinking. In undergraduate medical education, a well-constructed MCQ can easily assess a student’s ability to apply, evaluate and judge medical education knowledge (Vanderbilt, Feldman, & Wood, 2013). Nevertheless, writing MCQs capable of assessing higher ordered thinking are challenging (Bridge, Musial, Frank, Thomas, & Sawilowsky, 2003) but can be developed by following certain guidelines, especially ensuring that item writers are competent in their fields (Haladyna, & Downing, 2006).

Scully (2017) invalidated the perception that MCQs can only assess lower ordered thinking and Palmer EJ and Devitt (2007) illustrated that the percentage of question testing lower ordered thinking is same in both MCQs and MEQs. It also shows that a well-constructed MCQ is a better tool to assess higher ordered thinking in medical students than an MEQ (Palmer & Devitt, 2007). There is nothing innate in the MCQ assessment format which prevents testing of higher-ordered thinking (Norcini, Swanson, Grosso, Shea, & Webster, 1984). Besides, medical schools are training their faculty members to develop multiple-choice questions which ensure assessment of higher ordered thinking of their students. (Vanderbilt et al., 2013).

Conclusion

The higher ordered thinking in undergraduate medical students can be better assessed through well-constructed multiple-choice questions as compared to modified essay questions. Therefore, well-constructed MCQS should be considered a reasonable substitute for MEQs because of a variety of other advantages it provides over MEQs.

Take Home Messages

A well-constructed MCQS should be considered a reasonable substitute for MEQs because of a variety of other advantages it provides over MEQs.

Notes On Contributors

Dr. Arslaan Javaeed is an assistant professor of Pathology in Poonch Medical College, Rawalakot, Pakistan and is doing his masters in Health Profession Education from Faculty of Education, University of Ottawa, Ottawa, Canada. 

Acknowledgements

Thanks are due to Dr. Katherine Moreau for her guidance at every step of this review article. 

Bibliography/References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing. (2014). Standards for educational and psychological testing. Washington, DC: AERA

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.   

Bloom, B. S., Engelhart, M. D., & Committee of College and University Examiners. (1956). Taxonomy of educational objectives: The classification of educational goals. London: Longmans.

Bridge, P. D., Musial, J., Frank, R., Thomas, R., & Sawilowsky, S. (2003). Measurement practices: Methods for developing content-valid student examinations. Medical Teacher.

https://doi.org/10.1080/0142159031000100337   

Case, S. M., & Swanson, D. B. (2002). Constructing written test questions for the basic and clinical sciences. Director, 27(21), 112.   

Collins, J. (2006). Writing multiple-choice questions for continuing medical education activities and self-assessment modules. RadioGraphics, 26(2), 543–551.

https://doi.org/10.1148/rg.262055145   

Epstein, R. M. (2007). Medical education - Assessment in medical education. New England Journal of Medicine, 356(4), 387–396.

https://doi.org/10.1056/NEJMra054784   

Khan, Z., & Aljarallah, B. M. (2011). Evaluation of modified essay questions and multiple choice questions as a tool for assessing the cognitive skills of undergraduate medical students. International Journal of Health Sciences, 5(1), 39–43.   

Mehta, B., Bhandari, B., Sharma, P., & Kaur, R. (2016). Short answer open-ended versus multiple-choice questions : A comparison of objectivity, 52(3), 173–182.   

Morrison, S., & Walsh Free, K. W. (2001). Writing multiple-choice test items that promote and measure critical thinking. The Journal of Nursing Education, 40(1), 17–24.   

Norcini, J. J., Swanson, D. B., Grosso, L. J., Shea, J. a, & Webster, G. D. (1984). A comparison of knowledge, synthesis, and clinical judgment. Multiple-choice questions in the assessment of physician competence. Evaluation & the Health Professions, 7, 485–499.

https://doi.org/10.1177/016327878400700409   

Oermann, M. H., & Gaberson, K. B. (2009). Evaluation and testing in nursing education. New York, Springer.   

Palmer, E. J., & Devitt, P. G. (2007). Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Medical Education, 7(1), 49.

https://doi.org/10.1186/1472-6920-7-49   

Radan Čapek, Peter McLeod, C. C. & J. R. B. (2005). Do accompanying clinical vignettes Improve student scores on multiple choice questions testing factual knowledge? Medical Science Educator, 20(2), 110–119.   

Salkind, N. J. (2006). In Tests & measurement for people who (think they) hate tests & measurement. Thousand Oaks, Calif: SAGE Publications.   

Sam, A. H., Hameed, S., Harris, J., & Meeran, K. (2016). Validity of very short answer versus single best answer questions for undergraduate assessment. BMC Medical Education, 16(1), 266.

https://doi.org/10.1186/s12909-016-0793-z

Scully, D. (2017). Constructing multiple-choice items to measure higher-order thinking. Practical Assessment, Research & Evaluation, 22(4), 1–13.   

Haladyna, T. M., & Downing, S. M. (2006). Handbook of test development. Mahwah, N.J: L. Erlbaum.   

Tiemeier, A. M., Stacy, Z. A., & Burke, J. M. (2011). Innovations in pharmacy using multiple choice questions written at various Bloom's Taxonomy levels to evaluate student performance across a therapeutics sequence. INNOVATIONS, 2(2).   

Kubiszyn, T., & Borich, G. D. (2013). Educational testing and measurement: Classroom application and practice. New York, NY: Wiley.   

Touchie, C. (2010). Medical Council of Canada Guidelines for the Development of Multiple-Choice Questions.   

Vanderbilt, A. A., Feldman, M., & Wood, I. K. (2013). Assessment in undergraduate medical education: a review of course exams. Medical Education Online, 18(1), 20438.

https://doi.org/10.3402/meo.v18i0.20438   

Vleuten, C. Van Der. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education, 1, 41–67.

https://doi.org/10.1007/BF00596229

Appendices

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Reviews

Please Login or Register an Account before submitting a Review

sathyanarayanan varadarajan - (13/03/2019) Panel Member Icon
/
This interesting article is about assessing the ability of a well-constructed Multiple-Choice Question (MCQ) to assess higher order thinking skills as compared to a Modified Essay Questions (MEQ) in medical education. This has been done by the author by searching and analyzing the medical education literature available. The author has compared MEQ and MCQ with suitable Examples of multiple choice questions assessing higher order thinking i.e. application, analysis and evaluation levels. The author concludes that well-constructed MCQs should be considered as a reasonable substitute for MEQs because of a variety of advantages it provides over MEQs.

But the real challenge lies in constructing MCQs appropriately that can assess the higher order thinking skills truly and effectively.

This paper will be useful for all medical educators, who are involved in Evaluation and Assessment.

Sateesh Babu Arja - (24/06/2018)
/
I enjoyed reading this paper. It is an interesting paper which is in favor of multiple choice questions over modified essay questions. I agree with authors it is required to see how well constructed is the assessment rather than the format of the assessment. As long as the assessments fulfill the important traits like Validity, Reliability, feasibility, educational effect, and catalytic effect, they are good practices. Validity can be determined by the Miller's pyramid, the content of the exam, and mapping with learning objectives. It is known that MCQs are clearly helpful in assessing application and analysis levels. But it is interesting to see that MCQs can be used to assess evaluation level too. This paper is good for faculty members, academic administrators, and assessment committees.
P Ravi Shankar - (17/06/2018) Panel Member Icon
/
The author discusses an important point in this article. He describes how a well-constructed multiple choice question (MCQ) can test higher-order skills. The author has provided some examples of MCQs testing higher order skills and has described both the original Bloom’s taxonomy and its recent revision. The article could be further strengthened by the author providing tips to construct MCQs testing higher order skills.
In offshore Caribbean medical schools MCQs are widely used for assessment and are modeled after the United States Medical Licensing exam (USMLE) pattern. A problem is MCQs are the predominant method of assessment and other methods are not widely used. Many of these schools concentrate on preparing students for success in the USMLE which predominantly uses single response MCQs. MCQs are objective, unbiased and can be evaluated by machine and are widely used in various licensing and other high stakes exams. All medical educators will find this article to be of interest.
Tayyaba Khan - (17/06/2018)
/
I thoroughly enjoyed reading this research article. Mostly multiple choice questions are used to assess factual knowledge only and this article indicates need of wellconstructed mcqs for their proper usage.