Metacognitive awareness and the link with undergraduate examination performance and clinical reasoning

Theory: Metacognitive awareness is a component of self-regulated learning and helps us to understand and control our thinking and learning. Thinking about thinking is also an important aspect of the clinical reasoning process for medical practitioners. Hypotheses: This pilot study researched the link between metacognitive awareness and undergraduate examination performance. The Metacognitive Awareness Inventory (MAI) is a validated 52 item survey instrument for measuring metacognitive awareness. It has eight sub-scales grouped into two domains: Knowledge of Cognition and Regulation of Cognition. It was hypothesised that MAI scores would increase between first and fifth-year undergraduate medicine students and secondly that MAI scores would correlate with undergraduate examinations results. Method: Medical students at James Cook University, Australia were invited to complete the MAI and consented to give access to their examination scores. Results: The results of this pilot study found that metacognitive awareness was not significantly different between first and fifth-year undergraduates in this sample. For first-year medical undergraduates there were correlations between the Knowledge of Cognition domain and their end of year examination results, but not with the Regulation of Cognition domain. For fifth-year students there were correlations between both the Knowledge and Regulation of Cognition domains and their end of year examination results. Conclusion: This study identified that metacognitive awareness is not significantly different between first and fifthyear medical students. This may cause concern given that the study identified the importance of both MAI domains in undergraduate medical examinations. This study should be repeated on a larger scale and may confirm that raising metacognitive awareness levels among students is desirable. Increasing metacognitive awareness may raise Welch P, Young L, Johnson P, Lindsay D MedEdPublish https://doi.org/10.15694/mep.2018.0000100.1 Page | 2 examination performance and better prepare students for developing clinical reasoning skills.


Introduction
Clinical reasoning failures are a noteworthy cause of patient mortality and morbidity and result in approximately 40,000 deaths per year, as well as being the cause of 28.6% of malpractice claims in the USA (Newman-Toker & Pronovost, 2009;Saber Tehrani et al., 2013).In Australia 27.6% of adverse events were due to a failure to synthesise, decide or act upon available information, or failing to arrange an investigation, procedure or consultation (Wilson, Harrison, Gibberd, & Hamilton, 1999).Clinical teachers consider clinical reasoning problematic to conceptualise and teach, as well as difficult for students to grasp, but it is considered central to the medical education process (Charlin, 2012;Cutrer, Sullivan, & Fleming, 2013).
In medical practice, doctors gather clinical data about a patient and then make decisions based on the meaning they attribute to this information.These clinical reasoning skills can be defined as the "ability to sort through a cluster of features presented by a patient and then to accurately assign a diagnostic label, with the development of an appropriate treatment strategy being the end goal" (Eva, 2005, p. 98) .Helping medical students and junior doctors develop clinical reasoning skills is a central aim of medical education and requires knowledge and experience (Cutrer et al., 2013).
Seeking to better understand self-regulated learning among medical students has attracted attention from medical educators in recent years for two reasons (Bruin, Dunlosky, & Cavalcanti, 2017;Song, Kalet, & Plass, 2011).Firstly, self-regulated factors have been identified as a source of achievement differences between students (Zimmerman & Pons, 1986).Secondly, self-regulated learning (SRL) has been demonstrated as an effective means of raising student achievement (Schunk, 1981).In the 1980s, Schunk showed that students who are proactive selfregulators set goals, devise and implement effective learning strategies, create an effective learning environment, seek feedback and help when necessary, show tenacity as well as self-monitoring and being able to assess their progress towards specific goals (Zimmerman & Schunk, 2011).Studies have shown that metacognitive skills, a component of SRL, increase during adolescence, plateau during early adulthood and then decline in older age (Palmer, David, & Fleming, 2014;Weil et al., 2013).
Metacognition, or thinking about your thinking, is vital to the development of clinical reasoning skills due to its role in monitoring and regulating cognitive processes (Colbert et al., 2015;Croskerry, 2003aCroskerry, , 2003b;;Dunphy et al., 2010;Marcum, 2012).Medical accreditation authorities recognise metacognitive skills as important and seek to mandate their development by practising clinicians (CPMEC, 2016;ACGME, 2015;GMC, 2016).Although metacognitive skills are widely regarded as important for the development of clinical reasoning, they are seldom directly taught or assessed at medical school (Colbert et al., 2015).Metacognitive skills are a component of SRL which manage the cognitive aspects of learning and thinking (Bruin et al., 2017;Gönüllü & Artar, 2014).Effective SRL skills are positively associated with superior levels of student achievement, as well as expertise in clinical reasoning (Zimmerman & Schunk, 2011).
Cognitive skills are needed to perform a task, whereas metacognition is required to manage these cognitive skills (Garner, 1987).Metacognitive awareness skills include the Knowledge of cognition, which enable a person to have knowledge about their thinking and learning processes (Gönüllü & Artar, 2014;Schraw, 1998).The second component of metacognitive awareness is the Regulation of cognition.Being able to regulate their cognition enables an individual to self-monitor the effectiveness of their learning and decision making, regarded as vital to all clinicians, regardless of whether they are students or experts (Bruin et al., 2017).
Knowledge of cognition comprises declarative, procedural and conditional knowledge.Declarative knowledge encompasses knowing about things and covers facts, information, events, rules and processes.It involves networks of facts, is public knowledge and originates from what teachers state or declare (Anderson, 1982).Knowledge is encoded declaratively and then interpreted for it to be understood.For knowledge to be converted into behaviour, it must first go through this interpretive stage.Declarative knowledge may be thought of as conceptual, propositional, descriptive or explicit knowledge.
When declarative knowledge is put to work and involves knowing how to do something, it becomes procedural knowledge which undergoes a process of continual refinement with increases in processing speed.Procedural knowledge refers to the performing of knowledge or implicit knowledge and is a behaviour or skill.If knowing when and why to use a particular skill is important it becomes conditional knowledge which involves the regulation of memory, thought and learning (Ackerman & Zalmanov, 2012).
Regulation of cognition helps the student control or manage their learning or decision making e.g. a medical student may ask themselves if they understand the significance of a specific piece of clinical data, or if they comprehend the rationale behind a clinical decision or management plan.Regulation of cognition comprises of planning, information management, monitoring, debugging strategies and evaluation (Schraw & Dennison, 1994).
Planning involves choosing the best strategy alongside managing the required information to achieve the desired outcome.Monitoring refers to the ability to self-test the progress being made on a task.Debugging strategies concerns intentionally looking for disconformity evidence to reduce the risk of confirmation biases.Finally, evaluation refers to the ability to globally assess the progress towards solving a clinical problem or gaining mastery in learning.Figure 1 shows how Regulation of cognition sub-domains interacts with all stages of the clinical reasoning cycle as the individual plans, monitors and evaluates new information in the light of existing knowledge.As the doctor in training makes use of debugging strategies they test their reasoning for its robustness (Croskerry, 2003a).The Knowledge of cognition sub-domains help the clinician to encode and interpret new information by linking it to their existing knowledge structures.This new knowledge may then be used in the clinical reasoning process.Knowledge already possessed by the doctor in training guides them to seek new patient data, as well as helping to make sense of the clinical findings.Clinical reasoning is an iterative process (Charlin, 2012).It may take place rapidly and with little metacognitive guidance input, but this may increase the clinical reasoning error rate (Croskerry, 2003b;Graber, 2003;Kiesewetter et al., 2016).
Clinical reasoning errors appear not to be predominantly due to incompetence or a lack of knowledge, but arise due to the complexity of the case, or under conditions of uncertainty or time pressure (Scott, 2009).The various errors and biases that impact on clinician judgement have been well documented, and commonly include premature closure whereby a diagnosis or management plan is decided upon before all the possibilities have been adequately considered (Nendaz & Perrier, 2012;Scott, 2009).
Medical school examinations test student knowledge and clinical skills with the goal of producing intern doctors prepared for clinical practice.Part of this preparation includes developing the clinical reasoning capabilities of their graduates and by inference, their metacognitive capacities, in readiness for clinical reasoning in the workplace.In this paper, the authors test the hypothesis that there is a correlation between the metacognitive awareness of medical students and their examination performance and secondly, that there will be an increase in metacognitive awareness levels between first and fifth-year students of a medical course.
This research took place with volunteer undergraduate medical students at James Cook University (JCU), Australia.
The university and hospitals used for clinical placements are in regional and rural northern Queensland, Australia.
The first year of the medical program at JCU is regarded as the first of three pre-clinical years.Although there are several clinical placements, most of the time is spent in class or laboratory sessions.For the last three years of the program, students are primarily taught while on clinical placements.During their medical course, students are periodically required to write reflectively about their clinical experiences, but they are not explicitly taught metacognitive techniques.

Research questions
The research questions hypothesised firstly, that metacognitive awareness levels would positively correlate with undergraduate examination scores.Secondly, it was hypothesised that there would be an increase in metacognitive awareness scores from the first to the fifth year of the medical course.

Method
Ethical approval was obtained for this pilot study from James Cook University Human Research Ethics Committee reference H6008.This study used a between-subject design with data collected at one point in time.
Students were invited to participate in this study by email.For participating first-year medical undergraduates (43 from a total of 197)), their mean age was 19.19 (SD = 2.7) years with 19 male and 24 female students.For the fifthyear students (13/177) their mean age was 25.69 (SD= 6.16) years, with seven male and six female students.This study was undertaken in September 2015.Two types of measurement instrument were used, medical school examination results and the metacognitive awareness inventory (MAI) (Schraw & Dennison, 1994).

Measurement instruments Medical School Examinations
A variety of examinations are used by the medical school to test knowledge and clinical skills, and these include the Objective Structured Clinical Examination (OSCE), Key Features Papers (KFP) and Multi-Station Assessment Task (MSAT).An OSCE is a focused examination consisting of approximately 20 stations where students may be asked, for example, to take a patient history, examine a part of the body or interpret a laboratory report (Harden, 1988).These tasks form part of the information gathering and analysis components of the clinical reasoning process (Welch et al., 2017).The OSCE is a commonly used and highly regarded method of assessment around the world, both in undergraduate and postgraduate examinations (Khan, Ramachandran, Gaunt, & Pushkar, 2013).
Key Features Papers (KFP) were designed as a means of evaluating clinical problem solving and decision-making skills (Page & Bordage, 1995;Page, Bordage, & Allen, 1995).The examination is a written test consisting of between 15-20 brief cases where the student is required to make qualified decisions based on the information presented.The Multi-Station Assessment Tasks (MSAT) have strong similarities to the OSCE examination but are adapted for medical students in years 1-3 at JCU.There is a debate in the literature about how best to measure clinical reasoning (Gruppen, 2017).The absence of a 'gold standard' test means that reliance on testing depends on several different types of examinations including the OSCE and KFP examinations (Ilgen et al., 2012;Rencic, Durning, Holmboe, & Gruppen, 2016).
Studies have shown that metacognitive awareness allows individuals to plan, sequence and monitor their learning in a way that improves their overall examination performance (Swanson, 1990).Metacognitive skills are essential for any complex learning process, but there is a lack of evidence connecting metacognitive awareness with performance in medical school examinations.This study investigates how metacognitive skills are associated with performance in undergraduate medical examinations.

The Metacognitive Awareness Inventory (MAI)
The Metacognitive Awareness Inventory (MAI) was developed to quantify the different components of metacognitive awareness see Table 1 (Schraw & Dennison, 1994).The MAI has sub-scales within the Knowledge of Cognition domain including declarative knowledge, procedural knowledge, and conditional knowledge.From the perspective of clinical reasoning, it is important for the clinician to understand how knowledge is added to what they already know about the signs and symptoms of a patient.This knowledge about a patient medical data is declarative knowledge (Schraw & Moshman, 1995).Procedural knowledge is concerned with how practical procedures are sequenced and executed, for example how a patient's abdomen is palpated to feel for an enlarged liver.Conditional knowledge is knowing why and when to apply various cognitive actions, for example knowing which heart sounds to listen for, based on the result of a patient's ECG and history (Schraw & Moshman, 1995).
The second domain of the MAI is Regulation of Cognition and is associated with controlling one's thinking and learning.There are five sub-scales in this domain including Planning, Information Management, Monitoring, Debugging and Evaluation (Schraw & Dennison, 1994).Planning involves controlling one's thinking and actively developing a strategy for solving a clinical problem.Information Management means actively reflecting on whether there is enough information gathered to enable a decision to be made.If there is insufficient information, the clinician can develop a strategy to obtain the extra information he/she needs to make a decision.Monitoring means the real-time awareness of how the clinician is utilising the information they are receiving and checking how they are performing clinically.Debugging techniques may be used to correct understanding and performance errors.For example, a student may think why they placed undue emphasis on a particular clinical detail which then took their thinking to an incorrect conclusion.Finally, the evaluation sub-scale is linked with analysing the whole decisionmaking process and reflecting on its effectiveness.This evaluation step is particularly important in enabling the learner to deliberately and consciously make better decisions next time (Ericsson, 2004).

Table 1 -Domains and sub-scales of the Metacognitive Awareness Inventory (Schraw & Dennison, 1994)
The Metacognitive Awareness Inventory (MAI) was deployed via the online tool Smart Sparrow and consisted of a series of 52 short statements, for example, "I consider several alternatives to a problem before I answer, and I consciously focus my attention on important information" (Schraw & Dennison, 1994;Smartsparrow, Accessed Sept 2014).
the MAI Knowledge of cognition domain is calculated as the mean score of the declarative, procedural and conditional knowledge scores, while the Regulation of cognition domain score is the mean score for the planning, information strategies, monitoring, debugging strategies and evaluation sub-scales.
The inventory required participants to indicate their level of agreement with each statement by positioning a sliding point on a 10-point scale bar.The far right of the scale bar (10) indicated strong agreement while the far left (1) denoted strong disagreement.Individual statement item scores ranged from 1-10, and scores for each sub-scale were totalled and recorded as a percentage of the maximum available for each sub-scale.Following online completion and submission of the MAI, participants were given a breakdown of their percentage scores in each of the eight subscales and feedback about how they could improve skills in each domain.
The following student examination scores were obtained from consenting participants, depending on their year of the undergraduate medical program.All academic performance data was de-identified by the College of Medicine and Dentistry Assessment Unit before analysis by the researchers.The data was stored and retrieved on password protected electronic files.

Results
Results are summarised in Table 2 below

Discussion
For first year students, the Knowledge of Cognition domain and some of its sub-scales were correlated with examination performance.However, the relative unimportance of the Regulation of Cognition domain would infer that performing well in first-year examinations is not significantly correlated with the skills of self-monitoring, selfevaluation and information management strategies (some of the sub-scales of the Regulation of Cognition domain).
When undergraduate students reach the fifth-year, both the Knowledge and Regulation of Cognition domains show important correlations with examination performance.The most notable difference between the first and fifth year students is the increased importance of the Regulation of Cognition domain.For the fifth-year students, the results of this research shows an increasing necessity to regulate their cognition in clinical examinations like the KFP and OSCE.
The purpose of the OSCE examination is to assess clinical competence in a planned and structured manner, while Key Features Papers are designed to assess clinical reasoning.(Harden, 1988;Hrynchak, Glover Takahashi, & Nayer, 2014).This study provides evidence that performance in both the KFP and OSCE examinations is correlated with performance in the Regulation of Cognition domain of the MAI.
The second important finding from this research is that there was no significant increase in metacognitive awareness from first to fifth-year students.From the findings of our earlier results, we can see that metacognitive awareness becomes increasingly correlated with performance in the fifth-year examinations.If there is no significant change in a student's metacognitive awareness from first to the fifth year, a lower scoring student may struggle in the later years of medical school where metacognitive awareness is more important.Secondly, and of relevance to graduates, the literature indicates there is a strong link between high levels of metacognition and clinical reasoning expertise -a crucial factor in clinical practice (Croskerry, 2003a).
The limitations of this study include the small number of students participating.Due to the small number of participants in this pilot study and the data not being normally distributed, Spearman's correlation was used.Another limitation may be that the MAI has not been used extensively with medical undergraduates before.The overall examination performances of the first and fifth-year students were not significantly different from the overall mean results for their cohorts, meaning the participants in this study were representative of their year cohort.The single site location of this study means the result cannot be generalised.
Future research should aim to confirm that MAI scores do not vary significantly between first and fifth-year medical students and that there are correlations between components of the MAI and the OSCE and KFP examinations for fifth-year students.Future research may extend this study to include students from other medical schools in both Australia and internationally.Practising clinicians may also be studied to investigate the relationship between their performance in Script Concordance Tests or postgraduate clinical fellowship examinations and their MAI scores (Boulouffe, Doucet, Muschart, Charlin, & Vanpee, 2013).

Conclusion
This pilot study found there was no significant difference in metacognitive awareness between first and fifth year medical students.This may be a cause for concern given the importance of metacognitive awareness in selfregulated learning and expertise in clinical reasoning.The positive correlations between the sub-scales of the Regulation of Cognition domain and the fifth year KFP and OSCE examinations highlight the importance of metacognitive awareness for undergraduate clinical examination performance.Our evidence along with the literature, support reviewing the need to support raising the metacognitive awareness of medical students which may benefit both their examination performance and their clinical reasoning skill acquisition (Berkhout et al., 2015;Bruin et al., 2017).

Take Home Messages
Clinical reasoning is core to medical practice.Metacognitive awareness is a key trait of clinical reasoning experts.
Metacognitive awareness levels appear to be correlated with some undergraduate examinations.Supporting the learning of metacognitive awareness skills may assist undergraduate examinees and help foster the development of clinical reasoning expertise.

Paul Welch
Paul has over twenty years of experience in education in both the secondary and tertiary sectors.

Declaration of Interest
The author has declared that there are no conflicts of interest.

Figure 1
Figure 1 Shows how the components of metacognitive awareness may interact with four stages of the clinical reasoning cycle.

First
year: Overall year mark, end of year Multi-Station Assessment Task (MSAT) examinations, Key Features Papers (KFP) examination scores.Fifth year: Overall year mark, end of year Objective Structured Clinical Examination (OSCE) and their Key Features Paper (KFP).

.
Table 2 -Resultsshowing the significant correlations between the domains and sub-scales of the Metacognitive Awareness Inventory and Year 1 and 5 student examination results.coefficient.Mann-Whitney tests were performed to examine any differences in MAI scores between first and fifth year students.The first research question hypothesised that MAI scores would correlate with examination scores.For the first-year students, their overall examination result for the year had a low but significant correlation with the Knowledge of Cognition domain scores of the MAI (r=0.32,p<0.04) and their Key Features Paper result also had a low correlation with the Knowledge of Cognition domain score (r=0.32,p<0.03).There were no significant correlations within the Regulation of Cognition domain score for the first-year student examination results.For fifth-year students, their overall end of year examination result was moderately correlated with the Knowledge of Cognition domain score (r= 0.69, p<0.04) and their Key Features Paper (KFP) was highly correlated with the Knowledge of Cognition domain (r= 0.82, p<0.01).The Regulation of Cognition domain was moderately correlated with performance in the KFP examination (r= 0.62, p<0.02) and there was a low correlation with the Regulation of Cognition domain score for their OSCE examination (r= 0.48, p<0.04).The fifth-year overall MAI score was moderately correlated with performance in the KFP examination (r= 0.63, P<0.01).
His qualifications include BSc.(Hons), PGCE, MA and he is currently a PhD candidate at James Cook University, Queensland in the College of Medicine and Dentistry.His particular interest is in the development of clinical reasoning skills amongst doctors-in-training.Paul has taught and presented both within Australia and internationally.