The Effect of Curricular Phase on the Clinical Reasoning Skills of Medical Students at the University of Nottingham

Introduction There are many factors affecting the development of clinical reasoning (CR) skills. It has been suggested that the stage of undergraduate study of the student can affect their level of CR. The marks of CR questions in the summative exams of students studying medicine at the University of Nottingham (UoN) at different curricular phase were used to ascertain further insight into the theory.


Introduction
There are many factors affecting the development of CR skills. It has been suggested that the stage of Undergraduate (UG) study can affect the medical students' level of CR skills (Nafea, 2015). Neufeld concluded in his study that the CR process remains relatively constant from UG to postgraduate level, with only the measure of content in the students' hypothesis correlating with both outcome and educational level. This implies that medical students' knowledge is developing but the development of their CR skills remains relatively constant during this period (Neufeld et al., 1981). On the other hand, Groves and Da Silva stated that CR skills develop throughout the successive progress of the students (Groves, O'rourke and Alexander, 2003) (Silva, 2013). The studies of (Hmelo, 1998) (Norman and Schmidt, 2000) (Koh et al., 2008) also reported that the clinical phase of the curriculum has a strong impact on students' CR.
By comparing the results of 'CR questions in the summative written paper exams' after the first clinical phase (CP1) and after the final (third) clinical phase of students (CP3) at the University of Nottingham studying for a Bachelor of Medicine Bachelor of Surgery degree (M.B.,B.S), it is hoped to ascertain further insight as to whether the level of progression through curricular phase influences the CR skills of the students.

Structure of the Medical Curriculum 2016/2017
There are three routes at Nottingham by which medical students graduate as doctors: the five and six year UG courses and the graduate entry four-year course. On all three courses the educational objectives are the same: to acquire the knowledge, skills and behaviour to allow the graduate to practice as a new doctor on the foundation training programme with the M.B.,B.S. However, all routes of entry congregate and converge at the start of the clinical phase of the course, from that point on they are treated as one single cohort.

Clinical Phases
Clinical Phase 1 Students spend 17 weeks in one of five partner NHS sites. During this time students are taught the basic principles of taking a history and undertaking a clinical examination in both medicine and surgery placements. At the end of the attachment students undertake knowledge based multiple choice question examination and clinical skills examination.

Clinical Phase 2
Currently there are 4 x 10 week placements in this phase covering a range of specialties i.e. Obstetrics and Gynaecology, Child Health, Psychiatry Health, Care of Later Life, Community Based Medicine, Specials (Ophthalmology, ENT and Dermatology, each 2 weeks) and a special study module (4 weeks).
Clinical Phase 3 CP3 comprises two main components -the Advanced Clinical Experience (ACE) module and the Transition to Practice Module. ACE clinical placements rotate around partner NHS sites and comprise 8 weeks critical illness and general practice, 8 weeks musculoskeletal disorders and disease, 8 weeks medicine, 8 weeks surgery. At the end of the module students take the ACE knowledge and clinical examinations.
In this study, only CP1 and CP3 are compared because the subjects that cover in these two phases are similar whereas CP2 studies more specialities. The effects of different curricular phases on CR is measured by comparing the students' marks of those questions in the summative written exam that are considered to be predominantly CR questions in CP1 and CP3. This study looked at three cohort groups in 2 stages: CP1 (2012,2013,2014) and CP3 (2014CP3 ( ,2015CP3 ( , 2016. CP1 2012 cohort becomes CP3 in 2014, CP1 2013 cohort becomes CP3 in 2015 and CP1 2014 cohort becomes CP3 in 2016. There are between 318-351 students per cohort. Statistical analysis will then be carried out in order to determine if there is an improvement in student's ability to answer CR questions. The ethics committee confirmed that ethics approval was not required for this kind of project since it is classed as service evaluation.

Research Designs and questions
This leads to a research question and associated hypothesis.
RQ: Does the curricular phase of the curricular have an impact on the development of CR? H0 1 : There is no significant effect on CR from the curricular phase as measured by CR score in the summative written exams. Ha 1 : There is a significant effect on CR from the curricular phase as measured by CR score in the summative written exams.

Data Collection
The data for this study is from increasingly challenging clinically orientated summative assessment of CP1 and CP3 written papers in different cohort years.

Before the exam
The written papers are reviewed and categorised at standard setting meetings as CR questions or non-CR questions before the exams. The total number of experts who participated in standard setting meetings varies between 15-25. The experts come from a wide range specialities and roles such as, consultants, GPs, clinical teaching fellows, medical educators, Director of clinical skills, module leads and some junior doctors.
When grading, if the experts have different opinions about CR components on the questions, the team maps these questions against the 3 statements from the 'Outcomes for graduates from Tomorrow's Doctors from GMC; (8c,8g and 14f), considered in Bloom's taxonomy of learning domains, and the matter is discussed until a mutually acceptable decision was made (GMC, 2015). The Bloom cognitive domain involves knowledge and the development of intellectual skills. This includes the recall or recognition of specific facts, procedural patterns, and concepts that serve in the development of intellectual abilities and skills. There are six major categories of cognitive processes, starting from the simplest to the most complex (Anderson and Krathwohl, 2001 In these summative written exam papers, only the questions that assess the third category (apply) to the sixth category of Bloom's cognitive processes are accepted as CR questions. In the final data, CR questions can cover: being given a history and being asked to formulate the diagnosis for each case; being given physical findings and being asked to choose the most likely diagnosis; being given investigation results and being asked to find a diagnosis and provide the treatment plan; being given a diagnosis and being asked to choose the matching case vignette or history; being given a history and being asked to match the investigation findings to the interpretation of the findings.

After the exam: Psychometric evaluation
Routine psychometric analysis of the medical exam is carried out for each exam paper in order to provide an assessment tool with high quality. 'The post examination psychometric analysis of exam data' is conducted using Item Response Theory (IRT) and Classical Test Theory (CTT). Knowledge papers are analysed using test-score reliability (Cronbach's alpha), item discrimination index (ID), standard error of the measurement (SEM).
Frequency and discrimination (U-L) analysis and learning objective analysis are done for each paper. Item difficulty (p) and discrimination value (d) is calculated for each item. The reliability of the test was measured using generalizability (G) theory (Student× item). The hope is that students' marks on the test reflect true or consistent differences between students with respect to their knowledge and skills. G study was used to address error variance among questions.
These papers are also reviewed by the external examiners and internal peers. They gave comment for some questions and these comments are carefully considered and decided whether action is needed to be taken or not. The final scores are provided after these steps are taken. The tables below provide the overall results for the CP1 and CP3. The CP3 has 2 written papers and comparison was made with both papers.   2014  151  41  192  79  87  83  170  51  2015  85  85  170  50  91  79  170  54  2016  113  57  170  66  98  77 175 56

Data Analysis Methodology
This study involves the use of parametric statistical tests of independent sample t-test to address the research objective of the study. The parametric test requires the data of the dependent variable to be normally distributed.
The dependent variables include the CR scores, the non-clinical reasoning (NCR) scores, and total scores of summative written exams for each of the dataset of CP1 and CP3 datasets. Normality testing was conducted by investigation of the skewness and kurtosis statistics and histogram to check the distribution of data of the different dependent variable.
An independent sample t-test was conducted to determine whether the outcomes as measured by the CR scores in the summative written exams were significantly different in the each of the three periods between the CP1 and CP3 dataset. For instance, year 2012 CP1 students are the same as the year 2014 CP3 students. This longitudinal analysis was conducted to determine how CP1 developed CR to CP3. A level of significance of 0.05 was used in the independent sample t-test.
Mean comparisons are conducted if significant differences are observed when comparing cohorts CR results dependent on their clinical phase.

Normality
To determine whether the data follows normal distribution, skewness statistics greater than three indicate strong non-normality and kurtosis statistics between 10 and 20 also indicate non-normality (Kline, 2005). As can be seen in Table 4 and 5, the data of all dependent variables in both the CP1 and CP3 datasets exhibited normality. Thus, the parametric statistical analyses can be conducted.

Results of Analysis of CR Scores in the Summative Exams by Different Years between CP1 and CP3
This section presents the summaries of the data analysis using descriptive statistics, independent sample t-test. IBM © SPSS ® Statistics Version 22 was utilized to conduct the data analysis.         These results show that clinical practice has an important effect on the development of CR. This supports what has been reported previously in the literature (Hmelo, 1998) (Norman and Schmidt, 2000) (Boshuizen, 2003) (Koh et al., 2008). However, this finding contradicts partially the finding that CR remains relatively constant from medical school entry to practice by Neufeld et al (Neufeld et al., 1981). In order to confirm these findings, a longer longitudinal study would be required, with regular data collection moments, using more in-depth methodologies such as protocol analysis to access students' CR process.
The previous research and literature shows that learning from the practice is not simple cause and effect phenomenon (Kolb, Boyatzis and Mainemelis, 2000) (Ericsson, 2004); educators should carefully plan students' experience in practice to ensure that these opportunities are created.

Conclusion
As a summary, the CR scores of summative written exam of students were significantly higher in the CP3 dataset than in the CP1 dataset for 3 cohorts. With these results, the null hypothesis that there is no significant effect in CR score from the curricular phase on outcomes as measured by CR score in the summative written exams is rejected. CP3 students score higher than CP1 students and it supports the statement that CR is improved and developed from CP1 to CP3.

Limitation of the study
In the quantitative study, there are many data sets with different numbers of students, different summative knowledge exam papers and different components of CR marks in each cohort. Due to the nature of the data we have, we are conducting an independent sample t-test wherein we are comparing the group means instead of conducting a pairwise analysis of the difference of scores. The total raw scores reflect the weight of the correct answers.