Research article
Open Access

Effects on cognitive load of tutoring in virtual reality simulation training

Steven Arild Wuyts Andersen[1], Martin Frendø[1], Mads Sølvsten Sørensen[1]

Institution: 1. Rigshospitalet
Corresponding Author: Dr Steven Arild Wuyts Andersen ([email protected])
Categories: Educational Strategies, Educational Theory, Clinical Skills, Simulation and Virtual Reality
Published Date: 26/03/2020

Abstract

Aims: According to the guidance hypothesis, tutoring during technical skills training can result in tutoring over-reliance, reflected in a negative effect on performance when tutoring is discontinued. In this study, we wanted to explore if similar effects would be found for cognitive load.

Methods: Two cohorts of novice medical students were recruited for distributed virtual simulation training (five practice blocks of three procedures): 16 participants received intermittent simulator-integrated tutoring and 14 participants served as a reference cohort and did not receive simulator-integrated tutoring. Cognitive load during simulation was estimated using secondary task reaction time. Linear mixed models were used to account for repeated measurements.

Results: Overall, the tutored cohort had a significantly higher cognitive load than the reference cohort (mean difference = 7 %, p=0.006). Simulator-integrated tutoring did seem to lower cognitive load when active but also caused the tutored cohort to have a substantially higher cognitive load in subsequent performances where it was turned off (mean difference = 7 %, respectively, p<<0.001).

Conclusions: Concurrent feedback by simulator-integrated tutoring causes tutoring over-reliance and modifies cognitive load. This suggests that tutoring, in addition to degrading motor skills learning also affects the cognitive processes involved.

Keywords: cognitive load; tutoring; directed, self-regulated learning; instructional design; distributed practice; simulation-based training; surgical skills training; virtual reality simulation; temporal bone surgery; otorhinolaryngology

Introduction

Virtual reality (VR) simulation training of surgical procedures is often self-directed—especially when training requires prolonged repeated practice. Self-directed practice allows the learner to practice according to individual needs and schedule but also requires clear training goals along with supports to scaffold the learning experience (Brydges et al., 2009). Consequently, feedback is an important learning support in simulation-based education (Cook et al., 2013). VR simulation training offers unique possibilities for automated feedback that can be used both for summative assessment but also for real-time feedback and tutoring.

A review of feedback in simulation-based procedural skills training found that concurrent feedback can result in tutoring over-reliance, and that terminal feedback results in better long term learning (Hatala et al., 2013). Tutoring over-reliance can be understood from a motor skills learning perspective by the guidance hypothesis: continuous feedback during skills acquisition degrades learning (Schmidt and Wulf, 1997). Distributing practice seems to partially counteract this negative effect on performance in simulation-based training (Andersen et al., 2015). This moderate protective effect of distributed practice could be a result of spacing the practice: distributed practice allows memory consolidation to occur in the interim periods between training sessions, which is not possible under massed practice conditions (Shea et al., 2000).

Tutoring over-reliance can also be understood in the context of cognitive load (CL) theory: feedback during skills training can potentially modify CL, resulting in either an increased CL due to information overload, or a decreased CL due to improved structuring of for example learning instructions for easier processing (Hatala et al., 2013). The effects of tutoring on CL can therefore be mixed and also hypothesized to vary when it comes to immediate effects (performance) and longer term effects (learning). However, almost no research has directly investigated the role of CL during different feedback conditions (Hatala et al., 2013).

In a study on the effects of intermittent tutoring in a distributed VR simulation training program of temporal bone surgery, we aimed to explore the effects of tutoring on CL.

Methods

Study design and participants
This was a prospective study of a distributed, temporal bone virtual reality simulation training program in the period Oct. 2015–Jan. 2017 as previously reported (Andersen, Mikkelsen and Sørensen, 2020). One cohort of 16 participants (tutored) and another cohort of 14 participants were recruited among medical students at the University of Copenhagen. The participants were recruited similarly for the two cohorts and represent complete novices without previous exposure to any temporal bone procedure in real-life or simulated. During the training program, two participants dropped out of the tutored cohort and one participant out of the reference cohort because they could not find time to complete the training program.

Training program
Participants trained in the Visible Ear Simulator, an academic freeware virtual reality temporal bone surgical simulator (Sørensen, Mosegaard and Trier, 2009). The training program consisted of five blocks of three identical procedures in the VR simulator (mastoidectomy—a common but complex procedure with sequential drilling of the temporal bone near vital anatomical structures such as the facial nerve and dura)(Andersen, Konge and Sørensen, 2018). Each practice block lasted between 2 and 3.5 hours and was separated by at least a week. Both cohorts were self-directed and had access to a written, on-screen, step-by-step guide of the procedure for instructions.

In addition to this, the tutored cohort received simulator-integrated tutoring by green lighting of the target volume (Andersen et al., 2015) for the first of the three consecutive procedures in each block. The reference cohort did not receive this tutoring.

Outcomes and data analysis
To estimate cognitive load, secondary task reaction time (hundredths/s) was measured manually using a reaction timing device (American Educational Products LLC, USA) as previously reported (Andersen, Konge and Sørensen, 2018). This task consisted of the participant pressing a foot switch in response to an auditory stimulus (a beep) at baseline (immediately before and after the training block) and at two times during simulation (t=5 min and t=15 min). At each measurement time, reaction time was measured in a series of four consecutive stimulus/response times. The individual relative change in reaction time (unitless) during simulation compared to baseline mean was calculated. Data were analyzed in SPSS (SPSS Inc., IL, USA) version 23 for MacOS X using linear mixed model due to repeated measurements as outlined by Leppink (Leppink, 2015). Procedure number (repetition 1–15), measurement time (t=5/15 min), and measurement number (1–4) constituted the repeated effects. The final model further included cohort (tutored/reference) and tutoring during the procedure (yes/no) as fixed effects. Estimated marginal means and 95 % CI intervals are reported. P-values <0.05 are considered statistically significant.

Ethics
The regional ethics committee for the Capital Region of Denmark deemed this study to be exempt (H-15011780). All participants volunteered for the study, signed informed consent, and participation was not part of their study programme.

Results/Analysis

Relative reaction time decreased with each repetition (parameter estimate -0.01, 95 % CI [-0.009 – -0.012], p<<0.0001). A significantly lower relative reaction time was found during procedures guided by the simulator-integrated tutor-function (1.21, 95 % CI [1.18–1.25]) than during non-tutored procedures (1.28, 95 % CI [1.25–1.30])(p<<0.001). Importantly, the tutored cohort had a significantly higher overall relative reaction time (1.28, 95 % CI (1.25–1.32)]) than the reference cohort that had never received tutoring (1.21, 95 % CI [1.16–1.25])(p=0.006). In other words, even though less CL was induced during tutored procedures this also resulted in a considerably higher CL for the tutored cohort when tutoring was not available as reflected in the substantial difference between the two cohorts.

Discussion

In this study, we investigate the effects on cognitive load of using a simulator-integrated tutor-function for intermittent guidance during VR temporal bone surgical simulation. In agreement with the positive immediate effects and negative learning effects on different performance outcomes (i.e. tutoring over-reliance) previously reported (Andersen, Mikkelsen and Sørensen, 2020), we find a similar pattern for CL: concurrent feedback using a simulator-integrated tutor-function results in less CL being induced in the learner when active but when the tutor-function is discontinued in subsequent procedures, CL is increased above the level of never tutored novices.

The immediate effect of tutoring during the procedure—a reduced CL—could potentially be explained by a reduction of the extraneous component of CL by the spatial split attention principle (sources of information being physically separated, requiring the learner’s attention to shift between these by a change of gaze), and redundancy principle (two self-contained sources of information being replaced with a single source of information)(van Merriënboer and Sweller, 2010). This could potentially have been further substantiated if it had been possible to measure the learners’ use of the written on-screen instructions in our setup.

The negative effects on CL in the post-tutored performances and the overall effect on the tutored cohort suggest that tutoring might also interfere on other mechanisms involved in learning. For example it can be hypothesized that tutoring reduces the learners cognitive engagement in the learning task and that the reduction in CL is due to less germane load (used for schema formation and actual learning) being induced. This could potentially explain tutoring over-reliance from a CL theory point of view. However, a major limitation in using secondary task reaction time to estimate CL is that the different components of CL cannot be separated. Measuring germane load (Klepsch, Schmitz and Seufert, 2017) and separating it from intrinsic load is difficult and still debated (Schnotz and Kürschner, 2007), so whether tutoring causes a reduction in the extraneous load component and/or the germane load component remains to be established.

Conclusion

Concurrent feedback by simulator-integrated tutoring during distributed practice of a surgical procedure causes tutoring over-reliance even when tutoring is only used intermittently. This affects not only performance but also modifies cognitive load, adding to the current literature that tutoring not only degrades motor skills but also affects the cognitive processes involved. The problem of tutoring over-reliance in simulation-based training can therefore be viewed from both a motor skills learning perspective (guidance hypothesis) and a cognitive load perspective: tutoring might degrade motor skills learning and at the same time modify CL for example by causing less cognitive engagement, thereby reducing the germane load. Future investigations on the relative contribution of different components of CL might shed further light on the exact mechanisms.

Take Home Messages

Concurrent tutoring as a learning support during directed, self-regulated learning affects cognitive load similarly to the effect seen on performance: there are immediate positive effects while tutoring is active during practice at the cost of increase in cognitive load (and decrease in performance) when tutoring is subsequently discontinued.

Notes On Contributors

Steven A. W. Andersen, MD, PhD, is resident at the Dept. of Otorhinolaryngology, Rigshospitalet, Copenhagen, Denmark, and postdoc at Copenhagen Academy for Medical Education and Simulation (CAMES). ORCiD: https://orcid.org/0000-0002-3491-9790

Martin Frendø, MD, is PhD research fellow at the Dept. of Otorhinolaryngology, Rigshospitalet, Copenhagen, Denmark, and Copenhagen Academy for Medical Education and Simulation (CAMES). ORCiD: https://orcid.org/0000-0003-4301-347X

Mads Sølvsten Sørensen, MD, DMSc, is professor at the Dept. of Otorhinolaryngology, Rigshospitalet, Copenhagen, Denmark, and has developed the academic freeware virtual reality temporal bone surgical simulator, the Visible Ear Simulator.

Acknowledgements

We would like to acknowledge Peter Trier Mikkelsen, the Alexandra Institute, Aarhus, Denmark, for his continued contribution to the development of the Visible Ear Simulator freeware software.

Bibliography/References

Andersen, S.A.W., Konge, L., Cayé-Thomasen, P., Sørensen, M.S. (2015) ‘Learning Curves of Virtual Mastoidectomy in Distributed and Massed Practice’, JAMA Otolaryngol Head Neck Surg. 141(10), pp. 913–918. https://doi.org/10.1001/jamaoto.2015.1563

Andersen, S.A.W., Konge, L., Sørensen, M.S. (2018) ‘The effect of distributed virtual reality simulation training on cognitive load during subsequent dissection training’, Med Teach, 40(7), pp. 684–689. https://doi.org/10.1080/0142159X.2018.1465182

Andersen, S.A.W., Mikkelsen, P.T., Sørensen, M.S. (2020) ‘The effect of simulator-integrated tutoring for guidance in virtual reality simulation training’, Simul Healthc, 10 Jan 2020 [Epub ahead of print], pp. 1–7. https://doi.org/10.1097/SIH.0000000000000414

Brydges, R., Carnahan, H., Safir, O., Dubrowski, A. (2009) ‘How effective is self-guided learning of clinical technical skills? It’s all about process’, Med Educ, 43(6), pp. 507–515. https://doi.org/10.1111/j.1365-2923.2009.03329.x

Cook, D.A., Hamstra, S.J., Brydges, R., Zendejas, B., Szostek, J.H., et al. (2013) ‘Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis’, Med Teach, 35(1), pp. e867–e898. https://doi.org/10.3109/0142159X.2012.714886

Hatala, R., Cook, D.A., Zendejas, B., Hamstra, S.J., Brydges, R. (2013) ‘Feedback for simulation-based procedural skills training: a meta-analysis and critical narrative synthesis.’ Adv Health Sci Educ Theory Pract. 19(2), pp. 251–272. https://doi.org/10.1007/s10459-013-9462-8

Klepsch, M., Schmitz, F., Seufert, T. (2017) ‘Development and Validation of Two Instruments Measuring Intrinsic, Extraneous, and Germane Cognitive Load’, Front Psychol, 8:1997. https://doi.org/10.3389/fpsyg.2017.01997

Leppink, J. (2015) ‘Data analysis in medical education research: a multilevel perspective’, Perspect Med Educ, 4(1), pp. 14–24. https://doi.org/10.1007/s40037-015-0160-5

Schmidt, R.A., Wulf, G. (1997) ‘Continuous concurrent feedback degrades skill learning: Implications for training and simulation’, Hum Factors, 39(4), pp. 509–525. https://doi.org/10.1518/001872097778667979

Schnotz, W., Kürschner, C. (2007) ‘A Reconsideration of Cognitive Load Theory’, Educ Psychol Rev, 19(4), pp. 469–508. https://doi.org/10.1007/s10648-007-9053-4

Shea, C.H., Lai, Q., Black, C., Park, J-H. (2000) ‘Spacing practice sessions across days benefits the learning of motor skills’, Hum Mov Sci. 19(5), pp. 737–760. https://doi.org/10.1016/S0167-9457(00)00021-X

Sørensen, M.S., Mosegaard, J., Trier, P. (2009) ‘The visible ear simulator: a public PC application for GPU-accelerated haptic 3D simulation of ear surgery based on the visible ear data’, Otol Neurotol, 30(4), pp. 484–487. https://doi.org/10.1097/MAO.0b013e3181a5299b

van Merriënboer, J.J.G., Sweller, J. (2010) ‘Cognitive load theory in health professional education: design principles and strategies’, Med Educ, 44(1),  pp. 85–93. https://doi.org/10.1111/j.1365-2923.2009.03498.x

Appendices

None.

Declarations

There are some conflicts of interest:
The Visible Ear Simulator is academic freeware software which was developed by Dr. Mads Sølvsten Sørensen in collaboration with the Alexandra Institute, Aarhus Denmark. The authors report no declarations of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

The regional ethics committee for the Capital Region of Denmark deemed this study to be exempt (H-15011780).

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Sateesh Babu Arja - (28/03/2020) Panel Member Icon
/
Thank you for asking me to review this manuscript. It is an interesting manuscript. Research methodology is appropriate. But the conclusions drawn based on results is inconclusive. Thank you.
David Taylor - (27/03/2020) Panel Member Icon
/
The authors have taken great care with this study and seem to have used the appropriate statistical approaches to account for relatively small changes in a small to moderate sample size. have read this paper carefully, and I am still unsure about the author’s interpretation of the results. I can see that the students who had been tutored have a longer reaction time to a secondary stimulus than those who were not tutored. The authors attribute this to an increase in cognitive load for the tutored group. I could be wrong, of course, but could this also be attributed to the tutored group being more focussed on the task than those who were not tutored. The proxy measure for cognitive load, could therefore be due to careful and deliberate practice in the tutored group because they were more aware of the problems that they might generate. It would then be a positive outcome, rather than the negative outcome which seems to me to be implied by the authors.