The Day 1 evaluation had two purposes:
to establish a baseline attitude towards human factors issues and related training;
to identify the extent to which the course met needs
The first of these generated largely expected outcomes, as follows:
Table 1: Baseline evaluation of need
Familiarity with HF and NTS issues
|
3.4/4
|
HF training is a positive development
|
3.8
|
My team can benefit from HF training input
|
3.7
|
HF training should be mandatory
|
3.5
|
There is a claimed high level of familiarity with key issues and strong support for training for colleagues and this may indicate a combination of over-confidence in self and under-confidence in others, which may be an example of the Dunning-Kruger effect (Kruger and Dunning, 1999), described as follows:
People tend to hold overly favourable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it.
The final question in this opening section invited delegates to consider the extent to which their team would benefit from training, by inviting a response to:
“I have concerns about how colleagues work effectively in teams”
and the score for this item indicated a general level of disagreement, with an average score of 2.6/4. This may represent a degree of defensiveness and a failure to recognise a possible deficit in their colleagues’ performance. Whatever the interpretation, there seems to be a degree of inconsistency in expressed attitudes and some of this will be explored when looking at open comments that emerged from both Day 1 and Day 2 evaluations and comments made in focus group interviews and the very few online responses.
The remainder of the questionnaire focussed on the extent to which participants considered the programme to be effective and scores for these were generally very favourable:
Table 2: Sessions evaluation Day 1
Effectiveness: Strongly Agree = 4
Introduction to human factors in Obstetrics & Gaenocology
|
3.7
|
Communications
|
3.7
|
Situation Awareness
|
3.9
|
Cognition and decision making
|
3.7
|
Teamwork
|
3.8
|
Leadership
|
3.6
|
Workshop – small group
|
3.6
|
Workshop – plenary
|
3.7
|
Similar levels of enthusiasm were evident in the evaluations of Day 2 programme elements, as follows:
Table 3 Sessions evaluation Day 2
Effectiveness: Strongly Agree = 4
Stress effects on human performance
|
3.7
|
Resilience
|
3.7
|
Human factors and ergonomics
|
3.7
|
Human Factor analysis
|
3.7
|
Issues in evaluation
|
3.7
|
Closing discussion
|
3.7
|
Delegates were also asked a number of open questions, and these produced some expected outcomes, but also some more interesting insights into the challenges of implementing changes in practice, as follows:
Responses to open questions
Q1. Name up to three issues from the course that may have the most impact on your thinking about Human Factors so far.
While open responses are sometimes difficult to interpret, there was ample evidence in these data to indicate the issues that delegates (n=125) considered important, specifically:
Human factors analysis (45 mentions)
Resilience (28)
Stress (23)
Understanding of the issues/challenges (15)
Teamwork, reflection on own performance and leadership all had significantly fewer mentions (6, 4 and 3).
Q2. What would you consider to be your most pressing next step in terms of addressing the concerns of the BTBB programme?
The dominant issue to emerge from this stimulus was the view that the course needed to be rolled out to junior members of staff (28 responses)
Other recurring themes are:
Analysis of incidents (and sharing with colleagues) (18)
Teamwork and motivation (11)
Handover (5)
Reflection and personal learning (4)
Resilience management (3)
Q3. What do you consider to be the nature of the challenges you would face in implementing change in your practice?
By far the most significant response to this question related to cultural obstacles and there were 49 mentions of this phenomenon, one respondent writing:
“Culture change is very difficult in the NHS especially within maternity settings.”
Another wrote:
“Implementing change and changing people’s thought processes … encouraging personal reflection and self-awareness will be difficult."
Other issues identified were:
Time (23)
Staffing (21)
Other resource issues (7)
Follow up responses
The third stage of evaluation was planned to be conducted in greater depth, through three possible options:
Participation in focus group discussion
Online questionnaires
Entries on CPDme
and delegates were asked for their preference. Among those who responded to this request, the preferred mechanism for further engagement was online questionnaire, followed by focus group, and finally CPDme. As it turned out, there was significant lack of engagement in any of these, with the exception of some focus group interviews and some individual discussions, all of which were recorded and are summarised below. Participation in these processes was as follows:
Table 4: Data collection indication of preference
Data collection
|
Preferences
|
Participation
|
Focus group
|
37
|
12 (+ 2 paper based)
|
Online questionnaires
|
63
|
2
|
CPDme
|
25
|
3
|
Despite the preferences indicated in Table 4 above, there was little enthusiasm for engaging in the evaluation process beyond the group sessions (which had timetabled evaluation slots). There may be many explanations for this and they will be explored in the Discussion section below, as will alternative models for identifying the extent to which the training experience has impacted on practice.
Online questionnaires
There were two responses to follow-up emails. The first was a depiction of a clinical emergency that while having good patient outcomes, was unsatisfactory from the point of view of team management. It would seem that the course experience provided a structure and vocabulary to describe the team failings, but failed to establish leadership (“Could you leave if not required?” for example) or situation awareness. These issues were identified in subsequent debrief.
The e-mail responses were limited to expressions of enthusiasm and commitment to the idea of human factors and NTS training and there were no reflections on the impact of either individual or shared experience on practice.
Focus groups
The focus group interviews, however, elicited richer material, that while endorsing the general high levels of satisfaction, commented on the impact on behaviour in managing complex situations. Particular mention was made of:
Improved patters of communication (e.g. changes from “Can someone get me …” to “[Name] could you get me …”; “Can I have some quiet?”) and more effective communication across disciplines. Specific mention was made in respect of who spoke to who (initiation and response) and paralinguistic features – indicating attitude as well as meaning.
Enhancing teamwork, moving from “noisy and messy” to a more systematic and psychologically safe environment (particularly for the patient aware of staff behaviour). This feature, however, was considered to be more evident among those more senior staff who had experienced the course, while junior members are “… a bit clueless”.
There was suggestion of a more flexible and fluid perception of leadership – being able to hand over responsibility for both specific actions and overview.
The view that changing culture (evident in comments on evaluation forms) was a challenge within the NHS.
There was widespread agreement on the value of rolling the programme to junior staff and there was a distinct preference for simulation-based work, with the opportunity to explore complex issues, but in role. This latter point was considered an important component of effective simulation experience, as was the importance of “expert” feedback and debrief.