Observational study evaluating online procedural skills

ABSTRACT Introduction ELearning has gained popularity amongst doctors in training. We undertook an observational study assessing the quality of online resources for intra osseous (IO) needle insertion. Methods IO needle insertion videos were identified using "You-Tube". 21 pre-defined essential criteria were agreed by the study group based on the Advanced Paediatric Life Support recommendations, considered to be gold standard. Each video was independently scored 0-21 for quality. Results 34 IO videos, 14 manual and 20 electrical were identified. The quality score for the 34 videos, based on the mean score from the 4 raters, ranged from 5.5 to 17.5, mean 11.7, median 12.25. The inter observer variability was 0.76 , 95% confidence intervals 0.65 to 0.86. Conclusion Online resources can have a substandard educational content. To avoid developing poor practice a standard resource for procedural skills in the curriculum should be developed and endorsed by the Royal College of Paediatrics and Child Health (RCPCH).Introduction ELearning has gained popularity amongst doctors in training. We undertook an observational study assessing the quality of online resources for intra osseous (IO) needle insertion. Methods IO needle insertion videos were identified using "You-Tube". 21 pre-defined essential criteria were agreed by the study group based on the Advanced Paediatric Life Support recommendations, considered to be gold standard. Each video was independently scored 0-21 for quality. Results 34 IO videos, 14 manual and 20 electrical were identified. The quality score for the 34 videos, based on the mean score from the 4 raters, ranged from 5.5 to 17.5, mean 11.7, median 12.25. The inter observer variability was 0.76 , 95% confidence intervals 0.65 to 0.86. Conclusion Online resources can have a substandard educational content. To avoid developing poor practice a standard resource for procedural skills in the curriculum should be developed and endorsed by the Royal College of Paediatrics and Child Health (RCPCH). Vallabhaneni P, Jenkins C, James-Ellison M, Mason B, Edwards D MedEdPublish https://doi.org/10.15694/mep.2017.000096 Page | 2


INTRODUCTION
The pursuit of excellence in delivering high quality patient care has meant that post graduate medical education has changed over the last twenty years. The introduction of the European Working Time Directive (EWTD) has contributed to the changes with less time spent on the traditional methods of clinical teaching at the bedside. New innovative ways of teaching and learning are being developed with e-learning and the use of the World Wide Web playing an increasingly important role.
Quality of care and patient safety has become a significant focus of medical practice, [1]. It is suggested that doctors in training may expose patients to harm because of lack of experience, knowledge and technical skills while working in complex and dynamic clinical settings, [2]. Procedural skills are important competencies to be attained in run through paediatric training in United Kingdom. Life support courses teach some procedural skills but not all mandatory procedural skills as stipulated by Royal College of Paediatrics and Child Health [RCPCH] in the training curriculum, [3] Trainees may not be exposed and shown how to perform all the procedural skills required during a working day and may resort to learning these skills from web based sources. The quality of the are variable and not subject to any quality control measures and therefore, may not be of the high quality standard required to be used on patients.
This observational study focussed on one of the required procedural skills for paediatric training, the insertion of an intra osseous [IO] needle, and is a level 2 and 3 training requirement in the RCPCH curriculum, [3]. The purpose of the study was to review web based videos relating to IO needle insertion and quality assess the videos against the gold standard practice based on the technique taught on the courses run by the Advanced Life Support Group (ALSG).

METHODOLOGY
A "Youtube" video search over a two year period, August 2013 to August 2015 was carried out using the search term "intraosseous infusion". The search was further restricted to "video" content only. For the purpose of this study videos were included if they had an educational content and were IO needle insertions carried out on the tibial bone. Videos were excluded for the following reasons: videos were incomplete, duplicates, real time demonstrations that were not instructional, fluoroscopy was used, the videos were on animals, dentistry, news reports, irrelevant discussion, not in English, intramuscular or subcutaneous cannulation infusions, not medically related at all, medical but not IO, near needle, BIG gun (military), injection into humerus, IO into an egg, distal tibia or sternum.
The "gold standard" method for the insertion of the IO needle was based on the practical procedures taught on the Advanced Paediatric Life Support Course, [4]. The study group agreed a set of criteria for reviewing the quality of the videos. Essential criteria were identified for: the set up and equipment; performing the procedure; teaching beyond the procedure; ease of following instructional dialogue; visual and audio quality. Twenty one essential criteria were identified, each of which were scored 0 if absent, or 1 if present. (Table 1).
Four members of the study group reviewed all the videos independently and recorded a score for each of the 21 essential criteria on an excel spreadsheet. The length of the video and a number of non-essential criteria were also recoded. The four raters included two consultant paediatricians, one paediatric ST8 registrar and a medical student. The intra-observer and inter-observer variability of the quality ratings were measured using Intraclass correlation (ICC) calculated using Stata (Stata Statistical Software release 11.2. Texas: Stata Corporation, 2011).

RESULTS
The search yielded a total of 1072 videos. 1038 videos were excluded for the following reasons: 477 were non video resources; 139 not medically related; 123 were medical videos but did not show technique for IO insertion; 61 insertion into a bone other than the tibia; 47 were non English; 45 showed intramuscular or subcutaneous infusions; 42 were not instructional; 34 replicas; 19 were news reports; 18 used Big gun military; 10 videos were incomplete; 9 used fluoroscopy; 8 were animal videos; 3 dental videos; 2 had irrelevant discussion and 1 used near needle. Thirty four videos, 14 manual IO needle insertion and 20 electric IO needle insertions, remained for review.
The length of the video ranged from 1 to 14 minutes for manual IO insertion (mean 7 minutes, median 6 minutes) and for the electric IO insertion ranged from 1 to 12 minutes (mean 5 minutes, median 4 minutes).
The quality score for the 34 videos, based on the mean score from the 4 raters, ranged from 5.5 to 17.5, mean 11.7, median 12.25. The quality scores for the manual IO insertion ranged from 5.5 to 17.25, mean 12.8, median 13.25, and for the electric IO insertion ranged from 5.5 to 17.5, mean 1.7, median 9, (Image 1).
Intraclass correlation (ICC) analysis demonstrated that the inter-rater reliability for the 34 videos scored by the 4 different raters was 0.76, 95% confidence intervals 0.65 to 0.86. The correlation among mean team ratings with those that might be produced by another team of 4 raters was 0.93, 95% confidence intervals of 0.88 to 0.96.

DISCUSSION
This observational study highlights the variability in available video teaching resources for this essential procedural skill. Many of the available videos were of low quality. A doctor undertaking self-directed learning may easily retrieve low quality videos from an internet search, with potential implication if any learning was used in clinical practice.
The strengths of our study included a sensitive video search strategy which should have resulted in us identifying most of the available videos on this topic area. The independent review of videos by four observers was reliable with high intraclass correlation (ICC). The correlation among mean team ratings with those that might be produced by another team was 0.93 (95% confidence intervals 0.88 to 0.96) approaches a perfect correlation of 1.0. One potential limitation of our study is the absence of a gold standard against which the videos could be judged; however we develop essential criteria from the ALSG course manual which is recognised as definitive teaching.
Internet-based learning has positive effects compared with no educational intervention, and effectiveness is similar to that of traditional learning methods, [5]. However, the variable quality of video material for teaching anatomy, [6] and clinical examination, [7] in medical education has been described.
Our study raises concerns that trainees in the clinical setting may access poor quality videos, with sub standard educational content, that demonstrate these essential procedural skills. This may lead to poor practice which could impact on patient care and safety.

CONCLUSION
Procedural skills training for doctors are an important area of training which has potential implications for patient safety. We recommend that nationally RCPCH accredited video resources that are validated for teaching procedural skills to doctors in training.

WHAT IS KNOWN ABOUT THIS TOPIC
Online resources are increasingly used to assist junior doctors with their educational needs. 1.
The quality of these resources is variable. 2.

WHAT THIS STUDY ADDS
This study looks specifically at the quality of the videos for the insertion of the intraosseous needle a Royal 1.
College of Paediatrics training requirement. The educational quality of the videos is variable and may not be of the standard required to practice on 2. patients.

Notes On Contributors
PV-worked on the first draft of the article and orginially concieved the idea.
CJ-was one of the reviewers as a part of the study.
MJE-was one of the reviewers as a part of the study.
BM-analysed the results and provided key statastical support to the study group.
DE-overall lead for the project and produced the final summary of the article.