Clinical training traditionally relies on an apprenticeship model that occurs through immersion in the clinical environment without a formal structure or prior planning (Bleakley, 2006), while newer apprenticeship education models describe a process of constructing new knowledge on the foundations of what is already known to the learner (Bleakley, 2006; Taylor and Hamdy, 2013; Morris and Blaney, 2014). Deliberate teaching tools are “frameworks that enable clinicians to have a purposeful and considered approach to teaching encounters by incorporating elements identified with good teaching practice” (Sidhu and Edwards, 2018). No deliberate teaching tool has been designed for or evaluated in clinical anaesthesiology (Sidhu and Edwards, 2018). The objectives of this research project were to: 

1. Identify desirable teaching practices in clinical anaesthesiology for incorporation into a deliberate teaching tool
2. Perform a pilot study to evaluate the effect of a deliberate teaching tool on the educational environment, when applied as an ‘intention-to-treat’ intervention

Design of deliberate teaching tool

We performed a literature review (Ovid MEDLINE®) on 2/12/15 to identify ideal teaching practices specific to clinical anaesthesiology, with no date and or language limits, combining 1 AND (2 OR 3 OR 4) below:

1. Teaching (MeSH) OR feedback (keyword) OR debrief* (keyword)
2. Anesthesiology (MeSH) OR anaes* (keyword) OR anes* (keyword)
3. Education, Medical, Graduate (MeSH)
4. Operating Rooms (MeSH) OR operating room (keyword) OR operating theatre (keyword) OR operating theater (keyword)

Two authors (NS and EC) identified peer-reviewed articles that outlined desirable teaching practices in clinical anaesthesiology, with disagreements resolved by discussion. Items were categorised according to feasibility for incorporation into a deliberate teaching tool designed for clinical anaesthesiology.

Study protocol

We piloted a pre-post interventional study in two institutions, recruited after approval from the clinical directors and supervisors of training. A site investigator was appointed at each institution. Specialist anaesthesiologists and learners (defined as any trainee or resident/medical/house officer employed in the department) were allowed to opt out of the study, with remaining individuals designated as either participant specialists or participant learners. Participant specialists were provided with brief written material describing the deliberate teaching tool and advice on how to facilitate its implementation, including feedback delivery. Participant learners were provided with basic information on what to expect with the educational intervention and advice on how to facilitate specialist involvement. Reminder cards with a summary of relevant information were handed out to all participants, and A4-sized posters were placed in every operating theatre as cognitive aids (Figure 1). The deliberate teaching tool was applied during every in-theatre teaching encounter occurring in-hours (Monday to Friday, 0700-1800) that involved participant learners paired with participant specialists, over a four-week period. We defined a teaching encounter as either a half-day or full-day theatre list with the same specialist and learner. If a workplace-based assessment was required for training purposes, the study was suspended for that teaching encounter. At the end of each week, the site investigator contacted participant learners to ascertain compliance and reasons for non-compliance. Results were analysed with an ‘intention-to-treat’ approach, regardless of compliance levels. No new educational interventions were applied nor existing ones removed during the study period.

Change in the educational environment was determined by administering the Measure for the Anaesthesia Theatre Educational Environment (MATE) (Sidhu and Clissold, 2018), pre- and post-intervention. The MATE is a validated survey tool used to measure the anaesthesiology educational environment in the operating theatre, comprising 33 items derived sequentially from a systematic literature review, a modified Delphi approach, and exploratory factor analysis (Sidhu and Clissold, 2018). The items are categorised into four domains: Teaching Preparation and Practice, Assessment and Feedback, Procedures and Responsibility, and Overall Atmosphere. Respondents are required to rate each item on a 0-6 scale (0 = strongly disagree, 6 = strongly agree), with overall and domain scores presented as percentage scores. Pre-intervention demographic information was collected and a post-intervention survey was administered to provide information on compliance and participant experience. 

Data collection and statistical analysis

An online survey tool (Survey Monkey) was used to collect data anonymously. Data was analysed with IBM SPSS 24. A p value of 0.05 or less was considered statistically significant. We applied the Kolmogorov-Smirnov test to determine if data were normally distributed. Paired comparisons were carried out using either the paired sample t-test (for normally distributed data) or the Wilcoxon Signed-Rank test (non-parametric data). Unpaired comparisons were carried out using the a two-tailed unpaired t-test. Group comparisons were carried out using one-way analysis of variance (parametric data) or the Kruskal-Wallis test (non-parametric data). If the Kruskal-Wallis test generated a p value of < 0.05, Dunn’s non-parametric pairwise comparisons with Bonferroni-adjusted significance were carried out.

Design of the deliberate teaching tool

The search protocol yielded 6830 results. 63 papers were identified for furtherscrutiny after review of abstracts, and a further 35 identified after bibliographicreview. Of these 98 papers, six were identified as containing items related to desirable teaching practices in clinical anaesthesiology (Table 1). These items were combined with a model for delivering feedback (Rudolphet al., 2006; Rudolph, Raemer and Shapiro, 2013)to develop the deliberate teaching tool, dubbed the Concise Regular Assessment and Feedback for Teaching and learning (CRAFT) framework (Figure 1). The three phases of the CRAFT framework are based on a previously described instructional structure (Irby, 1992), utilised by almost 75% of subsequently described deliberate teaching tools (Sidhu and Edwards, 2018).The design did not prohibit opportunistic teaching of other clinical teaching points as they emerged during a case. While the framework advocated using a specific feedback delivery model, supporting material stated that participant specialists were free to use any feedback model familiar to them in order not to discourage feedback provision.

Figure 1: The Concise Regular Assessment and Feedback for Teaching and learning (CRAFT) framework, version 1

Participant demography and participation rates

Of the two study sites, Site A delivered anaesthesia in 20 locations (operating theatres, interventional suites, etc) and Site B in 22 locations. Sixty-six of 84 specialists (79%) responded to a brief survey on baseline teaching practice (Table 2). Fifty of 84 specialists consented to participate in the study, the participation rate of 60% being equal to both sites. Thirty-eight of 50 (76%) participant specialists completed the post-study survey. There was a participant specialist gender imbalance between the sites (female specialist participants 37% for Site A and 65% for Site B). Levels of prior training on teaching and feedback delivery were similar for specialists across both sites, with 54% having had no formal instruction on teaching, 42% attending a course/workshop without any formal qualification, and 4% possessing a formal qualification in medical education. Fifty-three of 75 eligible learners (71%) consented participate, with 45/53 (85%) completing the post-intervention measure. A participant learner gender imbalance between sites was noted (female participant learners 41% for Site A and 72% for Site B; 53% overall). Study sites were allowed to determine themselves, if fellows were to be classified as ‘learners’ or ’specialists’ for the purposes of the study, depending on their role and usual level of supervision in the department.

Change in educational environment

We observed a statistically significant improvement in the overall educational environment and in three of four domains (Table 3). Differences in the magnitude of improvement were noted between study sites (p = 0.271) and participant learner gender (p = 0.069), though these were not statistically significant in this pilot study (detailed in Table 4). There was no apparent difference when comparing age (p = 0.518), clinical experience (p = 0.806), or time in the current department (p = 0.888). Results for individual MATE items are presented in the appendices.

Compliance with study protocol

There was a 15% learner drop-out rate due to non-completion of the post-intervention survey. Compliance figures for Site A were not available and described as ‘variable’. Site B reported 100 eligible teaching encounters with 75% compliance for use of CRAFT framework. The reasons cited for non-compliance were as follows: ‘specialist forgot and was not reminded by trainee,’ ‘trainee seeing patients preoperatively therefore did not have time,’ ‘senior trainee under level 2 supervision,’ ‘busy list,’ and ‘list cancellation.’ Almost 60% of participant specialists stated that they provided feedback using the model described in the CRAFT framework, similar across both sites. Eight percent used an alternative model while just under 30% stated they did not utilise any particular feedback model. Five percent of specialists stated they did not provide feedback (11% in Site A; none in Site B). 

Feasibility and Acceptability of CRAFT framework

Participants were surveyed on their opinion of the CRAFT framework, to inform feasibility of implementation (mean ± SD on 0-6 perception scale; 0 = strongly disagree, 6 = strongly agree). Learners were significantly more positive towards the use of a deliberate teaching tool compared to specialists (‘CRAFT framework is a feasible method of enhancing teaching and learning,’ Learners 4.6 ± 1.3 vs Specialists 4.1 ± 0.8, p = 0.043; ‘CRAFT framework should be standard practice for every teaching list,’ Learners 4.3 ±1.5 vs Specialists 3.0 ± 1.3, p = <0.001). These are detailed by study site and participant gender in Table 5. Specialists did not consider the CRAFT framework to significantly impact levels of vigilance or productivity (‘CRAFT framework affected level of vigilance’ = 1.6± 1.4; ‘CRAFT framework did not impact theatre productivity’ = 4.1 ± 1.3). Learners held positive views towards feedback provision (‘When given, positive feedback reinforced my current performance’ = 4.9± 1.0; ‘When given, corrective feedback helped me identify deficient areas of practice’ = 4.9 ± 0.9).

This was a pilot study to evaluate the effect of a deliberate teaching tool on the educational environment in clinical anaesthesiology, with results revealing a statistically significant 5% positive effect. We consider this effect to be educationally significant, as 64.2% (one standard deviation) of all educational environment scores with this measure have a relatively narrow range of ± 15.6% (Sidhu and Clissold, 2018). Significant positive effects were observed in three of the four educational environment domains, with the ‘Procedures and Responsibilities’ domain remaining unchanged (p = 0.866). The CRAFT framework design did not specifically address this domain as related items were not identified as a major theme in the literature review. This domain is consistently rated highest by learners (Sidhu and Clissold, 2018), and therefore least in need of an intervention. That this domain did not exhibit an improvement while the others did, is an indication of the validity of the MATE tool in measuring the effect of the intervention. 

There was a clear signal of female learners favouring the intervention in both study sites, compared to their male peers, in terms of efficacy, feasibility, and acceptability. This difference was not anticipated and the pilot study was not powered to detect this difference nor designed to investigate the possible underlying causes. There is recent evidence from the surgical specialty of unconscious/implicit bias against female learners (Meyersonet al., 2017; Nebekeret al., 2017), and it is possible that the implementation of a deliberate teaching tool resulted in more equal treatment of learners by removing some elements of bias. Another possibility may be that certain personality traits, perhaps those associated with men, are less responsive to a structured teaching encounter or any change from current teaching practice. An adequately powered larger study with multiple sites could determine if this trend is observed universally, and a qualitative study may be able to investigate the possible underlying causes.

There were a number of differences between the study sites. Site A had a larger number of participants – both learners and specialists – who took part in the study. Pre-intervention, specialists in Site A were twice as likely to not provide feedback (‘infrequently’ or ‘never’) to learners, compared to Site B. The magnitude of positive change was higher in Site B, compared to Site A, despite the former having a higher baseline educational environment score. There was a gender difference between sites, with Site B having more female specialists and learners. The CRAFT framework was significantly more acceptable to specialists in Site B, who were also more likely to adhere to the study protocol. Both sites were located in geographically distinct locations. A multi-site study design with a larger number of sites would be required to investigate the effects of these confounders.

We selected a pre-post interventional study design because randomisation and separation of participants in the same institution would be unfeasible, as there would be no guarantee that individuals in the control group would not implement aspect of the deliberate teaching tool when it was being used by their colleagues. We did not randomise whole institutions as controls as we were uncertain of our ability to recruit a sufficient number of sites. Recruiting other institutions as matched comparisons would be challenging due to complexities in matching specialist and learner backgrounds. We employed an ‘intention-to-treat’ analysis to factor in the variable application of new teaching initiatives in a clinical environment. We measured the effect of the intervention on the educational environment instead of directly measuring teaching or learning. The educational environment is defined as “a social system that includes the learner (including the external relationships and other factors affecting the learner), the individuals with whom the learner interacts, the setting(s) and purpose(s) of the interaction, and the formal and informal rules/policies/norms governing the interaction” (American Medical Association, 2009). Measuring teaching requires direct observation (or video recording) by trained observers of every teaching encounter, and scoring using a validated tool. This would only be feasible with a limited number of participants and teaching encounters, and observed teaching practice would be subject to the Hawthorne effect. The current accepted gold standard of measuring learning is performance in a high-stakes examination, for which there would be too many confounders. 

In our study, approximately 60% of specialists ‘infrequently’ or ‘never’ discuss learning goals with learners prior to the start of each list and almost half ‘infrequently’ or ‘never’ provide feedback at the end of each teaching encounter. The use of learning goals and routine feedback delivery were key components of this deliberate teaching tool, supporting its implementation at these sites. It is possible that this level of baseline practice varies across different countries and training regions but there is no evidence that these desirable teaching practices are prevalent in clinical anaesthesiology.

Limitations

This was a pilot study conducted in two training sites, and results may not be applicable in other sites due to the limited nature of the study. This is compounded by the difference in the participant demographics and effect size between the two sites. The pilot was not designed to evaluate the underlying factors for the differences between sites and participant gender. Although baseline teaching practice (Table 2) and specialist teacher training were similar between the sites, inherent differences in the teaching culture of the two departments may exist. Use of other positive teaching practices already in place were not measured or taken into account by this study. Learner expectations may have been different between the sites and possibly influenced by gender effects towards teaching and learning.

The study relied on participant self-reporting for data on compliance and feedback delivery. We cannot be certain that specialists delivered feedback as described by the feedback model. Data collected on feedback delivery were confined to incidence and type of feedback model, but not effectiveness or quality of feedback. Direct observation (or video recording) of all teaching encounters by a trained external observer would supply more accurate data, but requires significant additional resources and results would be exposed to the Hawthorne effect. However, any positive or negative effect of feedback delivery should have been (and was) detected by a change in the ‘Assessment and Feedback’ domain of the educational environment measure, with an ‘intention-to-treat’ analysis.

With pre-post interventional studies, it is possible that any measured change may have occurred even without the intervention but a number of factors make this unlikely for our study. Prior to the pilot study, we applied the educational environment measure on a small sample of 10 trainees, revealing no difference in scores after a four-week period without any intervention. In addition, study protocol required that no other educational interventions be applied or removed during the study period. Finally, the one educational environment domain that was not specifically addressed by the intervention did not show any change over the intervention period, supporting the notion that no change is likely to have occurred over the four-week period without the intervention.

Future research

Results from this pilot indicate that a number of underlying factors may influence the effect of the intervention, including geographic location (as an approximate surrogate for teaching culture) and gender effects. Our research group aims to evaluate the effect of this deliberate teaching tool in different training regions, in order to account for differences in teaching culture and practice. A qualitative arm will investigate gender effects in more detail. The CRAFT framework may require minor modifications to improve uptake and compliance. Additional resources for teaching feedback delivery will be required, possibly in the form of an online video.

The use of a deliberate teaching tool significantly improved the educational environment in clinical anaesthesiology for learners in two study sites. Differences in the effect were observed between study sites and participant genders. Possible reasons for this include gender effects in teaching and learning, inherent differences in teaching culture (possibly linked to geographical location), pre-existing use of unquantified teaching elements, ability to embrace change, learner expectations, and level of compliance. We plan to launch a definitive study with larger number of sites in multiple training regions, to further evaluate the effect of the CRAFT framework on the educational environment in clinical anaesthesia.