Podcasting of Practical Summative Assessment Feedback : An Evaluation of Practice

Introduction: feedback is a current higher education and National Student Survey hot topic. Student ratings of experience are powerful tools in university rankings. Use of podcasts has increased in education with the majority of work considering written assignment feedback. Few investigations have addressed feedback podcast for summative practical assessments. Methods: in an educational development project three modes of feedback (face-to-face, individual and group podcasts) were compared. Evaluation considered accessibility, student utilisation and perceptions of different feedback modes via a bespoke questionnaire. Staff time in production/delivery of feedback modes was evaluated. Results: response rate was high (83% N51). Students rated individual feedback higher than group modes in all aspects. Individual podcast was rated higher for informing future performance. However, face-to-face was the most preferred. More students (96%) accessed the individual podcast. Staff found a benefit of time for podcasts feedback. Discussion: findings for podcast feedback utilisation for future performance are in agreement with other literature on written assessments. Although podcasts proved an efficient method of ensuring everyone received feedback staff were unaware of the context, setting or emotional state in which the individual student received the content. Staff in our study found podcasting more time efficient.


Background
Feedback is a current topic in education and one of the items of interest on the National Student Survey (NSS).This feeds into the key performance indicators on which a University is measured (QAA 2012, King et al 2008).Research shows that feedback on assessment performance is best given close to the assessment (QAA 2012) and should provide the student with constructive feed-forward feedback to help improve their performance in future assessment (Eva et al 2012).This requirement to provide feedback which is feed-forward (FbFf) in nature presents a challenge of time to both the student and lecturer, particularly in the case of verbal one to one feedback on performance and where appointments between busy lecture schedules need to be made (King et al 2008).In Higher Education the challenge of doing more with less time, but not impacting on the quality of the learning experience is ever present (Brookes 2010).
In the Division of Physiotherapy and Rehabilitation Sciences at the University of Nottingham first year undergraduate students undertake a 10 minute group verbal and practical summative assessment (known as topic tests).The content of the assessment was human anatomy and handling skills based.These occur every three weeks throughout the academic year with approximately 45 -50 students in each cohort and one or two lecturers required to offer feedback.This is a time consuming activity with approximately 8 hours of contact time after each test, longer that the assessment itself.Fitting timely feedback around the lecture load is always a challenge for both student and lecturer.
Reflecting on several years of experience the assessment team noted that only the most motivated students would present for feedback, often these were the students who had done well in the assessment and not necessarily the students that would potentially benefit most from the feedback on offer.Poorer performing students tended not to book a feedback session.Possible reasons might be avoiding being told what they already knew, apprehension or undervaluing of the role of feedback (Eva et al 2012).Eva et al (2012) in their qualitative work fond that students do not wish to 'look stupid' (p23) or lose face with their educator.However, in avoiding feedback sstudents may miss a valuable opportunity to improve their performance for the next assessment and for their course of study.
With the advances in e-learning and technology the lecturers involved with the delivery of this module (CMM and SWW) wondered whether this could be exploited to improve the mode with which the students received feedback and address take up.One such mode which might facilitate improved access was podcasts.Podcasts can be described as audio recordings that can be distributed through an electronic medium (Brookes 2010).The student can listen to the podcast anywhere and at any time and this can potentially support students with diverse lifestyles (Brookes 2010, Meade et al 2009).Thus it was decided to pilot a method of delivering feedback using podcasts.There was a notion at the outset that students would prefer face to face contact from a lecturer rather than an audio feedback.
Therefore the aim of this project was to compare three modes of providing feedback for undergraduate students.The three modes were face to face contact, individual podcast and due to the group nature of the topic test a group podcasts.Evaluation considered: utilisation and convenience, for example take up student perceptions of the feedback content including implementation of the feedback students' report of feedback mode preference staff resource This was an educational development project which aimed to inform future feedback practice.

Methods
As an educational development project, discussion with the ethics committee at the University of Nottingham medical School found no requirement for formal ethical approval.All first year undergraduate Physiotherapy (n=51) students were invited to take part in the project.This summative assessment comprises three topic tests (TT) throughout each semester of study.The test is devised so that it is takes place within class time and students are tested in small groups of four or five.The group is awarded a mark based on an average of each individual's performance.Individual achievement marks are not disclosed however previously the feedback is individual aimed to provide constructive direction for future assessment performance.
Following each of the three TT in semester 1 a different mode of feedback was implemented as follows: Following TT1 -Optional face to face individual meetings with the lecturers who undertook the assessments (existing practice) Following TT2 -Individual podcast (distribution via email) Following TT3 -Group podcast (distribution via email) Language from the assessment criteria to indicate the individual's performance was used in both the face to face and individual podcast feedback modes.For example if the individual had been awarded an A grade; terms such as 'an excellent demonstration' were used.Whereas if it were a C grade; 'satisfactory' would be used.For the group podcast, group grade language was used.
Evaluation of the project utilised a bespoke questionnaire administered after each mode of feedback.The questionnaire comprised three domains (the utilisation and convenience, individual's perceptions of feedback content and preferred feedback mode).There was space for free text.The final questionnaire explored comparisons of the three modes of feedback including ranking them for preference.

Results
Out of a cohort of 51 students 43 (84%) responded (F 63%; M 37%) to the first two questionnaire data points with 26 (51%) responding to the final questionnaire data point.Seventy-six percent of the original sample from questionnaire 1 were aged between 18 -20; with 24% over 21 years of age.Seventy-nine percent had prior experience of feedback, 94% of those having experienced this within the previous 3 years.
The majority of students had experience of written feedback (65% individual; 12% group) with some experience of verbal feedback (54% individual; 14% group) none had had experience of podcast mode of feedback.Ninety-seven per cent of those students who had experience of feedback reported having used their feedback to inform future performance.Interestingly 21% reported never having experienced any feedback.

Student perceptions of modes of feedback utilised in this project
Podcasts were the most frequently used mode of feedback both individual and group with 96% of participants accessing their individual podcast compared to 71% accessing face-to-face feedback (table 1).All modes of feedback utilised in this project were reported as convenient and of appropriate duration (Table 1).Most students agreed that the individual modes of feedback (face-to-face 80% or podcast 98%) were useful (table 2).However, again this was not the case for the group feedback mode, where the majority (54%) of students disagreed with its usefulness.
Question subject; concerns addressed/not addressed For reliability purposes we asked a negative question for which mode of feedback did NOT address students' assessment concerns (table 2).Once again individual feedback modes were rated more favourably than group modes (individual podcast 68%, face-to-face 74%, Group 41%)(table 2).
Question subject; questions about the assessment addressed Students were asked whether the content within the various feedback modes addressed questions they had about their assessment.Their responses indicate that they favoured individual modes with face-to-face mode rated most favourably (69%) with individual podcasts at 59% (table 2).Here there was a notable difference in favour of faceto-face feedback; the difference from face-to-face to group podcasts being 52% and to individual podcasts 8%.

Question subject; concerns about individual performance
As this was a verbal and practical skills assessment students were asked which feedback mode addressed questions about their performance.Individual feedback modes were rated more highly than group (face-to-face 69%, individual podcast 60%).Fifty-six per cent of students disagreed that the group podcast addressed questions about their performance (table 2).
Question subject; preparation for next assessment One of the aims of feedback is to help students prepare for their next assessment therefore students were asked to rate each mode of feedback delivery for their perceptions of its value for this.Individual podcasts are clearly more valued for potentiating future performance (Individual podcast 96%, face-to-face 65%, group podcast 56%) (table 2).
Preferred feedback mode: Finally students were asked for their preferred mode of feedback.The most preferred feedback mode was face-toface (57%) least was group mode (3%) with individual podcast receiving support from 43%.
Gender and Age bias: All domains data were scrutinised further for gender and age groups with no trends evident.

Staff resources
For the purposes of informing the discussion around implementation of the differing feedback modes the process for making individual podcasts was timed.In terms of the time taken to complete the podcasts, upload and email to students this entire process was achieved in between 1-2 hours.Students received podcast feedback within 5 working days of their assessment.
To undertake the individual face-to-face feedback two educators were required each allocated 50% of the cohort for 10 minute tutorials.As time had to be available for each of the 51 student's whether they took up the tutorial or not this took a total of 8.5 hours with 20% for over-run, total 10.2 hours.

Summary of results:
The response rate to the three questionnaires was high (first two 83%, third 51%).Most students had prior experience of feedback (79%) with 97% of those reporting use of it to inform future performance.It is clear that, in general students rated individual modes of feedback higher than group modes in all aspects.There is some variability in ranking of face to face and individual podcast modes in comparison to each other; with individual podcast rated higher for informing future performance than face to face.Face to face mode being rated as most preferred (with no gender or age group bias).Over the duration of the study (one academic semester) 96% of students accessed the individual podcast with less (77%) accessing the individual face to face mode.

Discussion
Students in this evaluation reported a preference for face to face feedback.Others have found that where there is a discourse between the students' perceived ability and their self-appraisal of it and the external data being presented regarding their performance, doubt in the credibility of the feedback given arises (Eva et al 2012).In a face to face setting this issue can be sensed and addressed but via a podcast (a one-way only means of communication) this is impossible to resolve.This may be why preference for face to face feedback was greater than the other modes.In this sense the quality of the feedback is in the dialogue however; interestingly students' behaviour is not in-line with the stated preference as they are offered follow up individual appointments for further feedback but rarely take up the opportunity.
In a face to face encounter the giver of feedback can read the facial responses of the individual and adjust the delivery of the feedback accordingly.In a podcast, the author may be more direct as they are dictating into a machine in isolation of the human interaction.This can be constructive but also potentially destructive if the feedback is not congruent with the students' expectations (Eva et al 2012).Being unable to perceive the students' reaction to the information therefore may have a destructive effect on the impact of the feedback.
The podcast does have the advantage of reducing deliverer and recipient 'awkwardness' which may occur where work is a fail or of poor quality (King et al 2008).In their qualitative study King et al (2008) reported students viewed the honesty of this medium as positive, citing amount and depth of feedback as quality factors.
Brookes (2010) identified that the use of podcasts can help generate inclusivity.This was demonstrated in this evaluation where accessing feedback was achieved with 100% of students for the podcast mode.In contrast Brookes (2010) found that not all students accessed podcasts, this may be because the podcasts from this current evaluation where individual to each student rather than a summary of each lecture as in Brookes' work.Indeed according to King et al (2008) feedback which is well constructed should be more personal.
Although group feedback was by far the quickest to produce this was the least preferred mode of feedback.When feedback is aimed at a group much more general statements have to be used to describe the performance and to offer aspects for improvement losing the personal nature of the feedback (King et al 2008).The listener may find it difficult to select from this the information which is relevant to them as an individual.
Podcasts have the potential to ensure everyone receives feedback in a timely manner.We cannot know how it is received or in what setting, context, mood etc. the person receives the feedback therefore we cannot make judgments on the impact of the feedback to inform future practice.Eve et al ( 2012) discuss the importance of the emotional state of the individual when they receive the feedback in order for the recipient to be receptive to feedforward changes.Unlike in the face to face encounter, with podcasts there is less control over this emotional state.To alleviate the potential anxiety and distraction grade disclosure may illicit (Macfarlane and Wakeman 2011) the grade is stated at the start and end of the podcast in wording based on the criteria (e.g.excellent, very good, good etc).
Abt and Barry (2007) argue that podcasts offer little benefit over written feedback for improving student performance.However, the time and access benefits found in this evaluation demonstrated that podcasts were the preferred mode for informing future performance from the students' perspective.In our results there was no evidence of any detriment to the student by the use of podcasts.Although Abt and Barry (2007) and King et al (2008) were comparing written with podcast feedback which is in contrast to this evaluation which compared podcast to face to face feedback no other literature in this area was found.
In terms of the time savings for staff and students podcasts were an extremely efficient mode of delivering feedback.Students were able to access podcasts anywhere and at any time so this negates the need to make appointments around lecture schedules.This is at odds with King et al (2008) who reported greater depth and richness of feedback without an overall time saving.In contrast they found an average increase in time of 14.5%.This may be explained by both the nature of their feedback being on written assignments and the improvement in technology since this time.
All students in this evaluation were UK students where English was their first language this means this is not generalisable to other groups of students.Schlairet (2010) did find differences in utilisation of podcasts where English was the students' second language.With the increase in international students accessing healthcare courses in the UK this needs to be taken into account when providing feedback via a podcast.The evaluation was based on one cohort only and may not be as appropriate for other cohorts of students.Meade et al (2011) found a mature student population appreciated podcasts for their learning support whereas we found no population sub group trend.

Conclusion
Podcasting improves the time taken to produce and accessibility of feedback to students following practical assessment.Students also report that it provides them with feedback that informs their future assessment preparation.However, this study could not offer any conclusions regarding the value of the feedback delivered via this mode specifically.Individual feedback is deemed more acceptable than group feedback.This may be accounted for by the difficulty that students have in identifying their own feedback when it is aimed at a group.Students prefer face to face feedback which may be due to the two way nature of the encounter allowing for questions about the assessment to be answered, something not possible in a podcast.Further research of a qualitative nature exploring the value and acceptability of podcasts would add to the understanding on podcast impact as a means of feedback.
Other areas of exploration may include the use of audio-visual modes.Carefully crafted podcast feedback allows for more frank and useful feedback with the flexibility of access and repetition.

Notes On Contributors Sarah Westwater-Wood
Director of Assessments MPhil, SFHEA Sarah is currently working as an educator, module convener and School of Health Sciences Director of Assessments.She has experience in educational development, implementation and eLearning innovation.

Course Leader BSc Hons Physiotherapy
Catherine is presently undergraduate course leader in Physiotherapy and as such has been involved in curriculum development and implementation, including innovation in assessment/feedback.

Mode of feedback and % of respondents Face to Face (n=43) Individual podcast (n=43) Group pod cast (n=26) Did you take the opportunity for feedback? Yes 71% 96% 93% Were the times convenient for you?
Most students agreed that the individual feedback modes (either face to face 80% or individual podcast 100%) were appropriate.This was not the case for group feedback, where 50% of students disagreed that the group podcast contained appropriate feedback for them (table 2).In considering appropriateness of feedback content by mode only individual feedback via podcast had 100% tallies in agreement (table2).For face-to-face and group podcast it was 80% and 46% respectively, showing a stronger preference towards individual podcast mode and content of feedback.

Table 2 :
Responses to Questionnaire