Research article
Open Access

What do trainee doctors and their trainers say about workplace based assessments? A survey of UK based junior doctors and their supervisors in a rural hospital

Mohammad Sharif Razai [1], Byron Theron[2]

Institution: 1. St George's Hospital , 2. North Devon District Hospital ,
Corresponding Author: Dr Mohammad Razai [email protected]
Categories: Assessment

Abstract

Purpose:  To determine the opinion of recently-qualified trainee doctors and their assessors on workplace based assessments in postgraduate training in one UK rural hospital.

Design: Two 10-item questionnaires (one for trainers and one for trainees) were completed anonymously by 22 trainers conducting the review and 17 trainees (postgraduate year 1 & 2) in the Northern Devon Hospital.

Results: Most supervisors had formal medical education training (68%) and in giving structured feedback to trainees (82%). Most trainers (81%) spent 10 – 20 minutes on the assessment and almost all (95%) conducted it through face-to-face meeting. Most trainers (67%) believed that the assessment was useful for facilitating feedback, determining engagement with the training programme and assisting a poorly performing trainee (52%). Eighty-two percent of trainers believed the assessment was a good use of their time. Poor IT service and lack of time (50%) were believed to be the biggest challenges for trainers. Trainees rated the assessment useful for engagement with the training programme (94%), assessment of overall performance (88%), recording training and achievements (88%) and allowing structured feedback (82%). Most trainees believed feedback helped boost their confidence (58%) and made them reflect on their performance (52%). A majority (52%) found it easy to arrange the assessment and many were satisfied with the process (76%).

Conclusion: This study provides insight into an important workplace based assessment for the first time. A majority of trainees and trainers viewed the assessment useful. This survey suggests that good IT infrastructure and protected time for trainers could improve the quality of the process and its outcome.

Keywords: Workplace based assessments, Feedback, Postgraduate training

Introduction

As part of Modernising Medical Careers (MMC) programme for postgraduate training in the UK, since 2005 newly-qualified doctors enter a two-year structured Foundation Programme (FP), during which they must complete six 4-month placements.1-2 In order to progress through the FP, foundation doctors must undergo mandatory portfolio-based reviews to assess performance in the workplace at the end of each placement. These competency-based summative assessments take account of a wide of range of skills, behaviours and performance based on the FP curriculum. These include workplace-based assessments (WBAs) known as Supervised Learning Events (SLEs) [mini-clinical evaluation exercises (mini-CEXs), case-based discussions (CbDs) and direct observation of procedural skills (DOPS)], 360-degree multi-source feedback from colleagues, trainer's own observations and records of trainees' other continuing professional development work.3 During the assessment meeting an online report must be completed on an online system called e-portfolio (figures 1-2).

There are two assessments at the end of each placement, one with clinical supervisor and one with educational supervisor. Although there is general guidance on how these assessment meetings should be conducted and used for professional development, 3-4 in practice there is likely to be considerable variation. At the start of their posts, each foundation trainee is allocated an educational supervisor for the whole 12-month period who sets objectives and overviews the overall progress of the trainee. In addition, each trainee has a clinical supervisor in each placement who is responsible for training, support and assessment for the 4-month period.

Previous studies on this area have focused on the role and impact of WBAs on trainee doctors' education and performance5-6 and some have reported trainees' views on the use and usefulness of portfolio-based learning7-8 and assessments such as the Annual Review of Competence Progression (ARCP) in specialty training9. There is no study, to our knowledge, that has consulted either trainees or trainers for their views on how end of placement assessments are conducted and used in doctors' professional development in the first two years post graduation. The aim of this study is to explore foundation trainers' and trainees' perspectives on the mandatory workplace assessments of overall performance. 

Method

A 10-item questionnaire was sent to all foundation trainers (clinical and educational supervisors) undertaking the end of placement assessments and another 10-item questionnaire was sent to all doctors in their first two years post graduation (Foundation year 1 & 2) in the Northern Devon Hospital, following the final end of placement assessments in the summer of 2015. Questionnaires were completed anonymously online.

Trainers were asked about whether they had training for their roles as supervisors, providing feedback to trainees and for their views about the process of the assessment. Trainers were also asked how useful they found the assessment and the challenges they faced in completing it using a four-point Likert scale with textual anchors. Three questions collected demographic data.

Trainees were asked to rate the usefulness of the assessment and feedback on a three-point Likert scale with textual anchors. In addition, the questionnaire comprised of items related to trainees' experience of the assessment process, understanding and perception of its significance to them and whether they were satisfied with the outcome in each placement. One question collected demographic data.  Data from both questionnaires were analysed using simple descriptive statistics, frequencies and percentages on Microsoft Excel SpreadSheet. 

Ethical approval: This study was a review of a current process; therefore, it did not require formal ethical approval. Electronic surveys were completed anonymously by the participants, which was taken to indicate consent to participate. There was no need to access confidential data. All information about the study was included in the introduction page of the survey.

Results

There was 61% (22/36) response rate amongst trainers to the questionnaire and 48% (17/36) response rate amongst the foundation trainees.  Most trainers were male consultant (55%) and most were physicians (63%). Thirty two percent were surgeons and one was both physician and surgeon. Most trainee respondents were female UK graduate (53%), there were two female international graduates and one male international graduate.

Trainers' Questionnnaire: 

All 22 respondents had undertaken end of placement assessments as clinical supervisors and 14 out of 22 also as educational supervisors. Sixty-eight percent of trainers had training for their roles as clinical and educational supervisors and 82% in giving structured feedback to foundation trainees. Most trainers (67%) believed assessments were useful or very useful for facilitating feedback and determining engagement with the Foundation Programme (FP) and a majority reported (52%) that it would assist a poorly performing trainee.  Lack of access to online e-portfolio due to poor internet and IT service and lack of time occasionally or often (58%) were the biggest challenges reported by the trainers. Most trainers (81%) spent 10 – 20 minutes on assessment with the trainee and all conducted the assessment through face-to-face meeting. Ninety percent of trainers believed the assessment was a good use of their time.

Trainees' Questionnnaire: 

Foundation trainees rated the assessment useful for engagement with the FP (94%), assessment of overall performance (88%),  recording training and achievements (88%) and allowing structured feedback (82%). Most believed feedback helped boost their confidence (58%) and made them reflect on their performance (52%). Most trainees regarded the assessment moderately important (53%) and a majority (52%) found it easy to arrange it with their supervisors and many were satisfied with the process (76%). For most trainees the assessment process was before end of placement (67-100%), conducted through face-to-face meeting (93-100%), with more than 5 minutes spent on feedback and discussion (75-93%) across the three placements for the whole year. 

Discussion

Views on the process

It is surprising that in spite of reported lack of time by trainers, most spent more than 10 minutes for the assessment through a face-to-face meeting with the trainee. In addition, the trainees did not find it difficult to arrange an assessment meeting before the end of placement (table 2). Furthermore, the time spent on feedback and discussion of performance with the trainee also reported to be more than 5 minutes. This is despite the reported poor IT and eportfolio access.

Perspectives on the outcome and usefulness

Most trainees regarded the assessment moderately important (53%), this is in spite of the fact that a large number rated it useful for a wide range of professional development domains. The perceived level of importance of this assessment mechanism is significant as it may determine trainees' engagement with the process and affect the outcome. One reason may be lack of awareness of the role the assessment plays in progression through the foundation programme (FP) and satisfactory outcome at the end of the year.

Almost a quarter of trainees were never given any plans for future development during the assessment feedback and discussion by the trainers; however, the proportion who reported that it was given occasionally or often increased in the second and third placement (table 2). There is some overlap between trainers' and trainees' views as both reported the assessment useful in determining engagement with the FP, facilitating structured feedback and discussion on the performance in the workplace. It is significant that almost all trainers regarded the assessment  as good use of their time in spite of the challenges discussed below.

Challenges and issues

Although a majority of supervisors reported to have the required training for their roles, a significant number (32%) did not have any formal training to meet the GMC and Academy of Medical Educators standards10 (table 1). It is therefore important to address this aspect to ensure a consistent training experience by all foundation trainees. Furthermore, support for trainers could also address problems with 'lack of time' as reported by a significant number of trainers (more than 50% as occasionally or often). Protected teaching time for trainers may be a useful step in improving the quality of supervision, training and feedback. Poor IT infrastructure such as slow internet connection, incompatible software, missing login details for access to  online system (eportfolio) and other training material are significant problems reported by the trainers (table 1) that could affect the validity of the assessment mechanism. Some of these challenges including consultants computer literacy are issues that have previously been reported by other studies 9.

Study limitations and further work

The authors acknowledge that the study provides a snapshot of recently-qualified  (foundation year 1 & 2) doctors in one hospital; therefore, the results may not be generalisable to a wider population.  The second important limitation is that although the response rate is reasonably good for this kind of study,  the sample size is small, which could potentially skew the results. Therefore, we do not draw firm conclusions from this opinion survey. Nonetheless, the survey provides valuable insight into previously unexplored summative assessment of overall performance from both trainers' and trainees' perspectives. We believe this survey is not only of interest locally but will be useful to wider including international readership. This is because the assessment mechanisms of overall performance in the workplace is a widely used method in postgraduate training in the UK and elsewhere. However,  further regional and national surveys exploring the reliability, consistency and validity of assessments such as this is required for developing and evaluating postgraduate trainees' assessment mechanisms.

Conclusion

Workplace assessments are important in postgraduate training as they determine trainees suitability for progression. This study provides evidence for the first time that both trainers and trainees have positive views of the process leading to more satisfied outcome for supervisors and doctors in training. To better equip trainers to harness the opportunities that the assessment offers better IT infrastructure and protected teaching time for trainers could be considered in improving the quality of this assessment mechanism. 

Take Home Messages

The assessment is a good opportunity to provide useful, structured and specific feedback that would improve trainees' performance in the workplace. However, better infrastrutcture such as IT service and more protected time as well as guidance for trainers will significantly improve the quality of the process. 

Notes On Contributors

Mohammad Razai contributed to the conception and design of the study as well as with the data collection, interpretation and writing up of this article.

Byron Theron contributed to the conception and design of the study, data interpretation and writing up of this article. 

Acknowledgements

We are grateful to trainers and trainees who completed the questionnaires and to David Preston, Foundation Programme Administrator for his assistance. 

Bibliography/References

1. Modernising Medical Careers, the Next Steps. The future shape of Foundation, Specialist and General Practice Training Programmes. April 2004. Available at http://www.dh.gov.uk/assetRoot/04/07/95/32/04079532.pdf. [Accessed on 21 February 2016]  

2. FP Curriculum 2012 with minor amends August 2015 & FP Reference Guide (2012) updated for 2014. Available at http://www.foundationprogramme.nhs.uk/pages/home/about-the-foundation-programme. [Accessed on 21 February 2016] 2012 with minor amends August 2015   

3. Overview of meetings in each placement. Foundation Programme e-portfolio. Available at http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/curriculum2012 [Accessed on -28 February 2016]   

4. Assessment. Frequently Asked Questions. UK Foundation Programme and Academy of Medical Royal Colleges. Available at http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/curriculum2012 [Accessed on -28 February 2016]   

5. Bindal N., Goodyear H., Bindal T., Wall, D. DOPS assessment: A study to evaluate the experience and opinions of trainees and assessors. Medical Teacher 2013; 35, 6, E1230-E1234

https://doi.org/10.3109/0142159X.2012.746447   

6. Bindal, T., Wall, D., Goodyear, HM. Trainee doctors' views on workplace-based assessments: Are they just a tick box exercise? Medical Teacher 2011; 33, 11, 919-927.

https://doi.org/10.3109/0142159X.2011.558140   

7. Hrisos S. Illing JC, Burford BC. Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Medical Education 2008; 42, 214-223.

https://doi.org/10.1111/j.1365-2923.2007.02960.x   

8. Vance, G., Williamson, A., Frearson, R., O'Connor, N., Davison, J., Steele, C. and Burford, B. Evaluation of an established learning portfolio. The Clinical Teacher 2013; 10:21–26.

https://doi.org/10.1111/j.1743-498X.2012.00599.x   

9. Goodyear, H., Wall, D. and Bindal. Annual review of competence: trainees' perspective. The Clinical Teacher 2013; 10:394–398.

https://doi.org/10.1111/tct.12040   

10. General Medical Council. http://www.gmc-uk.org/education/10264.asp

Appendices

Figure 1. Clinical Supervisor assessment form (first part) online (e-portfolio).

 

Figure 2. Educational Supervisor assessment form (first part) online (e-portfolio).

 

Table 1 summarises results of trainers' questionnaire

 

Table 2 summarises results of trainees' questionnaire

There are no conflicts of interest.

Please Login or Register an Account before submitting a Review

Reviews

Megan Anakin - (28/10/2017) Panel Member Icon
/
This is a brief and well written article that outlines a deeper look into workplace-based assessment for newly-qualified doctors. By taking a ‘business-as-usual’ approach to the data collection and analysis, the authors present portrait of current assessment practice from the perspective of the participants in that process.
Because this article is so brief, a wider and deeper discussion of the findings has not been presented. Ideas to consider may include how the findings can be related to the wider literature on workplace-based assessment to a greater extent than has been done in this article; especially the training of assessors and how assessor training can impact their ability to make meaningful observations and provide useful feedback. One other suggestion that could enhance the contribution of this work might be to include a theoretical perspective that frames the collection and interpretation of the data. For example, if a sociocultural view of learning and assessment is adopted, then providing feedback might be more than a trainee learning about the rules and expectations about the work required of a doctor. The conversation between assessor and trainee may be viewed as a process of developing a shared language and negotiating understanding of what it means to be a doctor.
I recommend this article be read by practitioners interested in understanding more about workplace-based assessment. It also provides a contribution to our understanding of assessment from the perspective of those involved in the process.
Trevor Gibbs - (12/10/2017) Panel Member Icon
/
An interesting paper that appears to have been well researched. It does reflect only the views from one rural hospital in the UK, and on a small number of participants, which in itself may provide some reasons for this apparent lack of quality in workplace-based assessment.
However, results are results and although this paper does not add a great deal to what we already know / understand, it is worth reading . I would have liked to have seen the so what element to this paper- as a result of this paper what will happen?
Alan Jaap - (10/10/2017)
/
This short paper reports the opinions of Foundation trainees and their supervisors (from a single centre) on the conduct and usefulness to the training process of end of placement review meetings, rather than workplace based assessment as suggested by the title and abstract. The choice of the term ‘assessment’ for the end of placement review meeting does lead to some confusion (at least for this reader!).

The authors are careful to discuss the potential limitations of their study and the need to replicate the results in a larger and more diverse sample.

The discussion could have been enhanced by making reference to the following related publications:

Rees CE, Cleland JA, Dennis A, Kelly N, Mattick K, Monrouxe LV. Supervised learning events in the Foundation Programme: a UK-wide narrative interview study. BMJ Open 2014;4:e005908.
Weston PS, Smith CA. The use of mini-CEX in UK Foundation training six years following its introduction: lessons still to be learned and the benefit of formal teaching regarding its utility. Med Teach 2014;36:155-63.