Research article
Open Access

Group Trust in the Setting of Competency Committee Decisions: A Qualitative Observational Study

Jason Sapp[1], Kelsey Larsen[1], Dario Torre[1], Eric Holmboe[2][a], Steven Durning[1]

Institution: 1. Uniformed Services University of the Health Sciences, 2. Accreditation Council for Graduate Medical Education
Corresponding Author: Dr Jason Sapp ([email protected])
Categories: Learning Outcomes/Competency, Curriculum Evaluation/Quality Assurance/Accreditation, Behavioural and Social Sciences, Research in Health Professions Education, Undergraduate/Graduate
Published Date: 25/03/2019

Abstract

Introduction: Trust is an important foundational component of a competency committee.  Little work has been done in health professions education to look at trust and group decisions.  A recent scoping review defined “group trust,” examined factors that may influence group trust, and proposed a model to illustrate the relationship between trust at the individual and group levels.  The purpose of this study was to test this model in the context of competency committee decisions and assignment of the ACGME Milestones.

Methods: The authors conducted structured observations of competency committee meetings in internal medicine and pediatrics residency and subspecialty programs within the National Capital Consortium.  Data were analyzed using a constant comparison method.

Results: The authors observed six meetings from May to June 2018 (15 hours) reaching thematic saturation by the final meeting.  The proposed model served as an organizing framework for reporting findings into the following categories: individual and group level factors, interpersonal interactions, and environmental factors.  Results suggest that diversity of opinion promotes a more complete understanding of trainee competence.  Varied faculty experiences, interactions, and actions by committee chairs helped to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones.  Meetings generally had similar structure based upon the size of the training program, and groupthink was more of a risk for larger training programs, especially when assigning individual Milestones.  An important environmental factor identified was decision making fatigue, which was most prevalent towards the end of committee meetings.

Conclusions: As training programs develop policies, procedures, and determine the membership of competency committees, utilizing these findings may help improve the design and execution of these groups, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions.

Keywords: Trust; Group; Decision; Competency Committee; Clinical Competency Committee; CCC; Milestone; Accreditation Council for Graduate Medical Education; ACGME

Introduction

Developing a process to assess clinical competence is an essential component of all graduate medical education (GME) programs.  Built upon the fundamentals of competency-based medical education, the Accreditation Council for Graduate Medical Education (ACGME) Milestones help GME programs determine which of their trainees are on appropriate paths towards unsupervised clinical practice and which are not.  Within a GME program, the clinical competency committee (CCC) forms a fundamental part of determining trainees’ progress in the context of the Milestones, and some recent studies describe data sources that CCCs use and how they weigh this information in their collective decisions (Choe et al., 2016; Shumway et al., 2015).  Several other studies have also looked at group decision making processes and how these concepts may be utilized by CCCs (Chahine et al., 2017; Hauer et al., 2016; Hemmer and Kelly, 2017). 

 

Given the important decisions shouldered by the CCC (e.g. overall progress, trainee promotion, and remediation), trust is an important foundational component of these committees, enabling open and collaborative discussions of strengths and areas of improvement, to include recommendations of remediation and dismissal.  In the context of individual trainee-supervisor relationships, health professions education research has explored trust in decisions that faculty make about the level of supervision granted for different clinical responsibilities and the factors that influence those decisions (Damodaran, Shulruf, and Jones, 2017; Hauer, 2015; Cate et al., 2016).  Entrustable Professional Activities (EPAs) are an example of these decisions and address tasks that trainees are permitted to execute once they attain a sufficient level of competence; the level of supervision for each EPA depends upon how much a supervisor trusts a trainee to complete that specific task (Cate, 2013).  While individual decisions of trust are important, little work has been done in health professions education to look at trust as it relates to group decisions, such as trust within a CCC, trust in the context of how characteristics and processes of these committees might impact collective decisions, and group member trust in these decisions. 

 

In a recent scoping review, we defined group trust (which encompasses both trust within a group as an aggregate or combination of individual trust and trust in the context of group decisions) as a group-directed willingness to accept vulnerability to actions of the members based on the expectation that members will perform a particular action important to the group, encompassing social exchange, collective perceptions, and interpersonal trust.  We also examined literature outside of health professions education to determine what factors may influence trust from the perspective of those who rely on decision making groups and proposed a model (Figure 1) to illustrate the relationship between trust at the individual and group levels (Sapp et al., 2019).  In this model, we delineate individual level factors, group level factors (i.e. group structure and processes), interpersonal interactions, and environmental factors.  As empirical data in a health professions education context has not been collected for this literature-informed model, we conducted the current study with two primary purposes: a) test our proposed model in the context of CCC collective decisions and assignment of the Milestones in primary care (i.e. internal medicine and pediatrics) and subspecialty settings and b) assess what factors are important to individual faculty members regarding trust in the setting of CCC decisions. 

 

Theoretical Framework

 

For decision making groups to reach a consensus, communication and interactions between and among people, objects (those items that extend human capabilities in a learning or task-oriented setting, such as learning management systems), and the environment are believed to be essential.  We therefore used situated cognition as the underlying theoretical framework for our model (Sapp et al., 2019).  Situated cognition contends that thinking is viewed as situated (or located) within the larger social and physical context of the environment (Durning and Artino, 2011).  This theory suggests that a group setting is highly complex with multiple components (i.e. social, cultural, and physical) and opportunities for interactions between these components.  It recognizes the complex interplay between participants, objects that augment individuals’ cognitive capabilities (i.e. artifacts), and the environment.  Situated cognition places emphasis on how these various interactions lead to thinking and decisions, and this framework informed our model.

Methods

We conducted a qualitative observational study involving GME program CCCs at our institution to determine how collective decisions about trainee progress within these programs are made in the context of the Milestones.  We focused on internal medicine and pediatrics residency and subspecialty fellowship programs within the National Capital Consortium because they represent the largest population of GME trainees within our institution.

 

Given the social and group dynamics underpinning CCCs and their decisions, along with the multiple inputs to our model, we believe that a qualitative approach was optimal for this study.  Our qualitative approach also allows for iterative construction of the results based upon the emergence of conceptual categories and descriptive themes.

 

Data Collection

 

Before collecting data we developed a structured observation guide (included as a supplemental file) based on our literature review and proposed model (Figure 1) to measure various components of trust and to focus data collection on a CCC’s structure and processes.  We also held conversations with subject matter experts who informed revisions to the guide.  The structured guide included the following headings along with a sample of items within those sensitizing concepts: individual-level factors (i.e. observed behaviors that may affect interactions amongst group members, group decisions, etc.), group structure (e.g. group size and diversity), group processes (e.g. group conflict, procedures utilized by the CCC, overall climate of the group, and leadership), information sharing/interpersonal interactions, and environmental factors (e.g. task complexity, uncertainty, and threats).  Next we pilot tested the guide by observing four CCC meetings in internal medicine and adult endocrinology (amounting to 10.5 hours of observation) from September 2017 to May 2018 and made revisions based upon observations in these meetings.  Observers were part of the research team, had the opportunity to practice observing during these four CCC meetings, and helped to revise the observation guide.

 

We sent out requests to a purposive sample of programs to observe their CCC meetings during the June 2018 end-of-year Milestone reporting period.  For programs that elected to participate, we conducted structured observations of the CCC meetings using the revised observation guide.  We had two observers in the room, and they observed as spectators (as opposed to a participant observer).  To minimize interference in the CCC meeting, observers sat outside the group circle along the perimeter of the room.  Observations were from an outside perspective, and the inquiry of the observation was directed entirely by the observers (i.e. there was no collaboration with those being studied).  Observers provided full disclosure of their role to those being observed and independently recorded field notes utilizing the structured observation guide.  At the end of each CCC, the lead observer asked the committee, “what do you think contributes to trust in CCC decisions?”  Observers collected information provided by individual committee members on their field notes.   

 

Data Analysis

 

Field notes were independently read and coded by the two CCC meeting observers (JS, KL).  To code and analyze the data, we used a constant comparison method (Fram, 2013) to identify and confirm themes and/or topics that emerged from the observations and discussions.  We looked at published approaches to group decision making (i.e. individual and group level factors, environmental factors, the interactions between all of these elements, and measurable components of trust), how CCCs operationalize this information, components of trust in the setting of decisions made by these groups, and the emergence of new concepts or themes.  We sought counter-examples to determine how new concepts or categories may vary from the published literature and how this impacts the process of group decision making.  Data analysis occurred simultaneously with the collection of data in order to monitor for thematic saturation and the need for any new directions in data collection.  The research team (JS, KL, DT, SD) met regularly to discuss coding and resolve all differences.  During analysis, memoranda and diagrams were used to keep written records of the process.  Techniques to enhance trustworthiness of the qualitative analysis process per the Standards of Reporting Qualitative Research included member checking, maintaining an audit trail, and collection techniques to allow triangulation of the data (O’Brien et al., 2014).

 

Ethics

 

Our research was approved by the Walter Reed National Military Medical Center Institutional Review Board (IRB; WRNMMC-2018-0137).  All participating training program directors and individual CCC faculty members provided written consent prior to their participation in the study.

Results/Analysis

We observed six CCC meetings in the following specialties from May to June 2018: general pediatrics, pediatric endocrinology, neonatal-perinatal medicine, internal medicine, and adult cardiology and hematology-oncology.  During this time, we observed a total of 15 hours’ worth of meetings amounting to 36 pages of data, and we reached thematic saturation before the sixth CCC meeting.  Our proposed model served as an organizing framework for reporting our findings.  See Appendix 1 for themes that were observed during these meetings, with representative quotes, which are organized under various components of the model (Figure 1) and discussed below.  We aggregated data from the observations and the debrief discussions into the sections below and contend that the different components of the model may affect group trust.

 

Figure 1.  Proposed Group Trust Model        

 

 

Group Level Factors

 

Group Structure

       

The size of the committees varied from 5 to 11 individuals (average 8, standard deviation 2.8).  All assigned faculty members were present for the meeting with two exceptions.  An absent faculty member on one CCC provided her scoring sheet to the CCC chair in advance of the meeting.  On another CCC, the committee had to table the discussion of an absent faculty member’s trainees because he had not provided recommendations about the updated Milestone levels for these individuals.  Besides physician faculty members, some of the CCCs also included a diabetic nurse educator, nurse practitioner, chief residents who had completed a core residency program, and training program administrative personnel.  CCC members commonly reported that a diversity of opinions from different faculty members contributes to a more complete collective understanding/picture of competence for each trainee.

 

Group Processes

 

Group Conflict

 

In all of our observations, we did not witness any instances of affective (relationship) conflict.  Rather, during each meeting we observed at least one episode of task (cognitive) conflict, which tends to arise from individual differences in viewpoints, ideas, or opinions when group members participate in a shared undertaking.  Observers did not specify how frequently these episodes occurred during a single meeting, and CCC faculty members in general seemed to offer personal experiences that were largely confirmatory and agreeable relative to points about learners raised by the CCC chair or other members.  However, “individuals at times volunteered differing comments, but [these experiences] were usually treated as apples-and-oranges differences rather than conflicting experiences (KL).”  When assigning individual Milestones, some differences of opinion occurred.  “Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; the majority of inquiries were collegial, open-ended, and were interpreted as clarifications rather than challenges (KL).”

 

Some faculty members felt that dominant voices/personalities may negatively impact a CCC.  One member remarked, “It is difficult to achieve a confident consensus when there is a forceful personality in the group, especially if other members of the committee may be more conflict averse.”  In one CCC, however, a faculty member consistently emerged as a dominant personality in both volume and quantity of commentary.  Unless other CCC members were specifically addressed, this individual volunteered, and at times interrupted, conversation an average of four times per each trainee discussion (a contrast to other members generally volunteering thoughts zero to two times per trainee).  These comments consisted of generally confirming/restating the point of others, injected with personal experiences from working with various trainees.  The other CCC members never attempted to prevent this individual from speaking, nor did they give any indication of annoyance or frustration (rather, there was an indication that they simply were comfortable and did not mind having the forceful personality present).

 

Group Procedural Fairness

 

Regarding the assignment of individual Milestones, procedures varied with each training program observed.  For smaller programs, such as pediatric endocrinology (3 trainees), the group generally went through each individual Milestone in greater detail with each trainee, and individual trainees were not assigned to faculty members.  In cases where trainees had previously achieved high scores on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered (“let’s talk globally… I don’t want to go line by line…”) and at times would explicitly state that those Milestones did not need to be considered (“if it’s mostly 4’s then we don’t really have to do it [as in depth as for others]”).  However, this was not universally true; on another CCC, several trainees had their Milestone levels decreased given some faculty concerns about professionalism and completing administrative tasks on time.   

 

For training programs with a larger number of trainees (i.e. internal medicine and pediatrics with 74 and 29 trainees respectively), faculty members were assigned individual trainees and utilized a standardized CCC data collection form that they filled out in advance of the meeting, which included proposing updated Milestone levels for each trainee.  During the meeting, faculty members took turns presenting the information off of these forms, which seemed to help standardize the process of discussing trainees and assigning Milestones.  Faculty presentations were followed by varying degrees of discussion.  In some circumstances, “the group seemed reluctant to make changes to the assigned faculty recommendations (JS),” and the discussion was more likely to involve groupthink (i.e. group members refrained from expressing doubts and judgments or disagreeing with the consensus).

 

In other CCCs, there also seemed to be less discussion for certain Milestones, especially in areas where the faculty members may not have any observational experience or evaluation data for specific trainees.  In some committees, “very rarely were specific Milestone numbers discussed; rather the discussions focused almost entirely on [trainee] skill-sets and general scoring (KL).”  Multiple CCC members also indicated that they felt they should not progress trainees in the context of the Milestones just for the sake of progression.  Members stated that they should be comfortable keeping trainees at their same level if no clear evidence exists to suggest that the learner has progressed in that specific Milestone.

 

Group Climate

 

The atmosphere of the meetings was “collegial and positive (KL)” and “comfortable (JS).”  Each meeting seemed to be a safe environment where all opinions were welcome and openly discussed. One member stated that “respect for each other is important to build trust, which includes listening to other members when they are talking and sharing.”  In one CCC, the discussion repeatedly made use of the term “we” and referenced what the team could do as a collective to improve the trainees’ performance, which is the hallmark of a common group identity.  Further, the cardiology CCC chair stated that consensus in this setting helped to support “[cardiology] community standards.”  The group agreed that determining when fellows are ready for independent practice provides reassurance to society that these trainees graduate with a basic level of skills to practice cardiology in any setting.  Finally, it was clear during some of the group discussions (especially in smaller programs with fewer than 15 trainees) that all of the faculty members knew the trainees well and had worked with them frequently.

 

Group Leadership

 

The two most important leader behaviors that we observed were the abilities of the CCC chair to conduct the meeting and to facilitate a shared mental model.  In most of the meetings we observed, the chair provided an overview of the Milestones to the group and the structure of the session.  “The pace was immediately established by the chair as quick; he explicitly stated the order in which [trainees] would be evaluated (third years first, ‘anticipat[ing] it will go quickly’); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones’ meaning (deficiencies, ongoing progress, and aspirations) (KL).” 

 

Two different meeting formats emerged during our observations.  Some CCCs went through all of the trainees in order (e.g. by post-graduate year, alphabetically, or based upon other external factors) discussing the trainee in general terms and assigning the Milestones at the same time.  “It seemed like the faculty who had other competing schedule requirements were given priority to present their [trainees] early (JS).”  For other CCCs, meetings generally had two distinct phases.  In phase 1 the chair led a discussion to generate formative comments for each trainee, and in phase 2 the group assigned the Milestones for each trainee.

 

Generating a shared mental model seemed to be one of the most important parts of making group decisions and assigning Milestones, and the CCC chair appeared critical in facilitating this.  In CCCs of smaller programs (i.e. fewer than 15 trainees), the chair usually introduced and provided some information about each trainee.  “[The CCC chair] began by summarizing the past activities and future plans of each of the [trainees], in addition to comments about exam performances and most recent one-on-one experiences with the [trainee] (KL).”  In larger programs, assigned faculty members generally provided this information as discussed above.  In all of the meetings, group members tended to freely share their opinions; for instance, in one meeting “the CCC chair did not need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion (JS).”  When one of the members commented about a trainee’s performance, other members tended to jump into the conversation to provide their individual experience with the trainee.

 

Individual Level Factors

 

Several faculty members felt that individuals on a CCC should have some baseline knowledge and experience in medical education.  One CCC member noted that “individuals who serve on the committee should have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance.”  Additional areas identified by the observers include buy in (“faculty should recognize that they bought into training these residents/fellows, and the overall agenda of the group should be to make the learners successful, to the maximum extent possible”); having an open mind (“members should come to the meeting with no preconceived notions or ‘labels’ for specific trainees”); and advocacy for the trainees (“the group felt that they were advocates for the trainees, and the group setting allowed them to identify mentors for individual trainees who may be struggling (JS)”).  We did not observe any specific comments related to patient advocacy and if any tension may result from advocacy for patients versus trainees.  

 

Interpersonal Interactions

 

Imported Information

 

Information brought to the meeting by individual faculty members is important to helping build a shared mental model.  Two main themes that emerged from the observations include information obtained from faculty pre-work assignments and individual experiences with trainees.  For CCC decisions, the latter seemed to be the most frequently relied upon source of information.  Especially for smaller programs, CCCs frequently used specific individual stories and/or experiences with trainees as a means of evaluation.  During the general discussion of each trainee, CCC members would most frequently offer specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding trainee performance. 

 

Shared Mental Model

 

During our observations we noted three general themes whereby CCCs build a shared mental model: assignment of the Milestones, perceptions regarding the developmental level and overall clinical competence of various trainees, and the work and processes of the CCC.  When building a shared mental model, CCCs discussed a number of different topics to include trainee global strengths/weaknesses (e.g. praises of bedside manner, maturity and concerns of professionalism, and personality issues), instances in which feedback was given to trainees (and whether they did/did not respond to it), and general discussions of what the CCC members could do and actions they could take in the future in order to aid the trainees and foster additional growth.  Indeed, when discussing solutions for questions and/or concerns about certain trainees, CCC members either described specific instances in which they attempted to correct behavior one-on-one or discussed team actions in abstract terms (e.g., “we have to somehow make it clear to [the trainee]…” or “we might need to consider program-level remediation…”).  A sample of other topics discussed are listed in Appendix 1.

 

When there was quick group consensus regarding the Milestones, CCC chairs would generally move the discussion forward to the next trainee.  In instances when a member disagreed with any of the proposed numbers, he or she would usually either implicitly ask the CCC member who offered the number why (“really?”) or offer a contradictory opinion (“I’d say [the trainee] is still probably at a x…”), at which point CCC members would offer specific examples, explanations, or evaluations that explain their points.  Additionally, there was an occasional voiced reluctance to offer “perfect” scores, as the committee generally agreed that trainees may never achieve some aspirational competencies.

 

Conversations seemed to be key to the process of discussing trainee competence and building a shared mental model.  Members used the analogy of a puzzle in which each faculty member had pieces of data about experience with trainees, and together through discussion and information sharing a more complete picture of trainee competence emerged during the meeting.  It was generally felt that members should share concrete, specific examples that can be interpreted by the group in a relevant context.  Individual participation is important for this to occur, and discussion amongst the faculty members led to additional insights and comments about each trainee.  The discussion uncovered information not written on global or other evaluation forms.  One faculty member noted that written evaluations are most useful for the highest and lowest performing trainees.  He stated that “97% of the time my evaluation is the same for trainees in between those two extremes and in the middle of the competency spectrum.”  He also felt that Likert scales are not useful and that group discussion is a much more effective way to share information about group members’ experiences with individual trainees.  Some members reported feeling that it was easier to identify troubling trends with trainees based upon conversations with faculty members who have worked with these individuals.  Oftentimes they said that it is hard to write negative comments on an evaluation form, especially if that form goes back to the trainee.  As a result, these “neutral” evaluations may not identify trends in a way that conversation does.  For example, if more concerning faculty observations arise with trainees and there has been no record of borderline performance or other concerns, it may be harder to justify things like program level remediation, etc.

 

To facilitate a shared mental model, some CCCs had additional resources available to the group.  In most meetings, the group had access to a copy of the Milestones.  In others the CCC chair had complete evaluation packets for each faculty member and asked the group to read the entire packet prior to discussing trainees.  The latter led to notable periods of silence as committee members read the evaluation summaries, as well as specific Milestone criteria. 

 

Standing in a Group

 

In all of the CCCs, participation in the meeting did not seem to be influenced by military rank, seniority, specialty, or whether one was a voting member of the group.  In two CCCs chief residents attended as non-voting members of the committee, and they freely provided input from a near peer perspective about their experiences with most of the residents.  These individuals had unique insight into most of the trainees that other faculty members may not be in a position to observe, and their input on the CCC provided information the committees considered very valuable.  One CCC also solicited input from two program administrators, and their comments were valuable regarding trainee accomplishment of administrative tasks. 

 

Environmental Factors   

      

Task Complexity/Uncertainty  

      

We identified the following themes under this heading: specific role of the CCC (e.g. two members in one meeting actively debated what regulative power the committee had when a trainee may need additional supervision); faculty member roles (e.g. should the members be more of a coach/mentor versus an evaluator or should trainees be assigned to faculty members based upon aligned interests?); developing remediation plans; lack of faculty member experience with certain trainees; and the Milestones themselves.  In regards to the Milestones, members at times expressed frustration with the Milestone definitions, especially those with lengthier anchors and what the milestones don’t capture (e.g., burnout).  In several instances groups noted the difficulty in determining numerical milestone assignments given how the Milestones were written/phrased and commented on the difficulty in fitting individuals into the specifically defined categories.  Some groups acknowledged the difficulty and/or worry in selecting only one demonstrative experience as a justification for a number assignment.  For programs with a larger number of trainees (i.e. greater than 15 individuals), information and evaluation gaps complicated the ability of the CCC to make informed assignments of some individual trainees’ Milestone levels.  Additionally, a large number of trainees coupled with the number of Milestones typically led to longer meetings, which seemed to truncate the discussion and the competency evaluation process compared to CCCs with fewer trainees.

 

Threats 

         

One of the biggest environmental threats identified was decision making fatigue.  While members of one CCC admitted that they come to the meeting knowing what needs to be accomplished, almost all acknowledged that they get more fatigued as the meeting continues and tend to make more brusque decisions.  Observations near the end of the meetings included members increasingly stepping out for short intermittent periods, multiple members checking their cell phones and fidgeting, group members openly asking how many trainees they had left to evaluate, and/or comments such as “I’m fading,” “homestretch for milestones!” and “last one!,” to CCC cheers.  Other threats identified during the observations included scheduling conflicts, technical difficulties, and institutional structures/policies (e.g. on several occasions one CCC considered not just how the trainee did/did not perform, but how the institutional structures may/may not be at fault).

Discussion

Based upon a synthesis of the information uncovered in our previous scoping review, we proposed a model (Figure 1) utilizing situated cognition to help better understand trust within a group and the various factors that may influence trust in group decisions from individuals either within or outside of the group.  In our current study, we observed 15 hours of CCC meetings from multiple different GME training programs using a structured observation guide developed from a review of the literature and this proposed model.  We explored how members characterize trust in the setting of a CCC and observed how these committees make decisions about trainee clinical competence and Milestone-based performance in the context of previously identified group trust elements: individual and group level factors, environmental factors, and the interactions between all of these elements.  Our proposed model captured the themes that emerged at the observed CCC meetings.       

 

Our results suggest that diversity of opinion promotes a more complete collective understanding of competence for each trainee.  Varied faculty experiences provide a diversity of information available to the CCC, and interactions among faculty members help to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones.  Further, CCC chairs were instrumental in facilitating a shared mental model by communicating an agenda, determining a process to assign the Milestones, moving along the conversation, engaging individual faculty members as needed, etc.  These findings were consistent with our expectations based on the proposed model.

 

Additionally, CCC meetings generally had similar structure and agendas based upon the size of the training program.  Faculty in smaller training programs tended to have more familiarity with all of the trainees and discussed them in more detail compared to larger programs.  Larger programs also assigned faculty members to individual trainees, and it was their responsibility to complete standardized pre-work on their trainees prior to the meeting.  Groupthink was more of a risk for larger training programs, especially when faculty members did not know their individual trainees well and when assigning individual Milestones.

 

Finally, environmental factors we identified that can compromise trust were task complexity, uncertainty, and threats to the process of group decision making.  One of the most important threats was decision making fatigue.  Almost universally, increasing distractions and observed lapses of concentration seemed to occur the longer the meetings ran.

 

Our research was limited to CCC meeting observations and brief group question sessions at the end of the meetings.  While we were able to collect data about easily observable components of our proposed model such as group size, diversity, and communication patterns, we were unable to obtain data on individual faculty member thoughts and feelings.  As a result, we were unable to test all components of the model, such as an individual’s propensity to trust, vulnerability, relational identity orientation, need for affiliation, etc.  We also did not collect data about what types of assessments each CCC used and had access to.  Given that observers were present in the same room as CCCs, the results may have been influenced by the Hawthorne effect (i.e. CCC members knew they were being watched, which may have impacted their behavior and/or group discussions).  Observations were also limited to meetings occurring during one Milestone reporting cycle at the end of the academic year at one institution. 

 

Future research might gather more validity evidence and explore faculty member thoughts and feelings regarding trust in a group setting.  We also suggest additional research looking at trust in group decisions from a non-CCC member stakeholder perspective.  This might involve examining how department leaders, program directors, and individual faculty members who are not on a CCC define trust in this context.

 

Developing a deeper understanding of our trust in CCCs may help these committees implement a more effective and meaningful process to make collective decisions.  Our model attempts to provide leaders within GME training programs a way to view and attend to the various factors that may impact trust.  The model might also serve as a way to “diagnose” why a CCC goes awry by looking at the CCC discussions through various components of the model (e.g. if environmental or leadership factors may be perceived as “ineffective,” what can be done to change or correct this?).

Conclusion

This observational study should allow program leaders and educators to better understand what factors may contribute to trust in group settings (e.g. CCCs).  As programs develop policies, procedures, and determine the membership of these committees, utilizing our findings may help improve the design and execution of groups like CCCs, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions.

Take Home Messages

  • Little work has been done in health professions education to look at trust and group decisions.
  • Group trust can be broken down into individual and group level factors, interpersonal interactions, and environmental factors.
  • A diversity of opinion promotes a more complete understanding of trainee competence.
  • Varied faculty experiences, interactions, and actions by committee chairs helped to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones.
  • Groupthink and decision making fatigue are two potential threats to competency committee group trust.

Notes On Contributors

JE Sapp is an assistant professor, Department of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD.

 

DM Torre is an associate professor, Department of Medicine, Division of Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD.

 

KL Larsen is an assistant professor, Department of Medicine, Division of Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD.

 

ES Holmboe is Senior Vice President, Milestones Development and Evaluation, Accreditation Council for Graduate Medical Education; professor adjunct of medicine, Yale University; adjunct professor, Uniformed Services University of the Health Sciences; and adjunct professor, Northwestern University Feinberg School of Medicine, Chicago, IL.

 

SJ Durning is a professor, Department of Medicine, Division of Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD.

Acknowledgements

None.

Bibliography/References

Cate, O. T. (2013) ‘Nuts and Bolts of Entrustable Professional Activities’, Journal of Graduate Medical Education, 5(1), pp.157–158. https://doi.org/10.4300/JGME-D-12-00380.1

 

Cate, O. T., Hart, D., Ankel, F., Busari, J., Englander, R., et al. (2016) ‘Entrustment Decision Making in Clinical Training’, Academic Medicine, 91(2), pp. 191-198. https://doi.org/10.1097/ACM.0000000000001044

 

Chahine, S., Cristancho, S., Padgett, J., and Lingard, L. (2017) ‘How do small groups make decisions? : A theoretical framework to inform the implementation and study of clinical competency committees’,  Perspectives on Medical Education, 6(3), pp. 192-198. https://doi.org/10.1007/s40037-017-0357-x

 

Choe, J. H., Knight, C. L., Stiling, R., Corning, K., Lock, K., et al. (2016) ‘Shortening the Miles to the Milestones: Connecting EPA-Based Evaluations to ACGME Milestone Reports for Internal Medicine Residency Programs’, Academic Medicine, 91(7), pp. 943-950. https://doi.org/10.1097/ACM.0000000000001161

 

Damodaran, A., Shulruf, B., and Jones, P. (2017) ‘Trust and risk: a model for medical education’, Medical Education, 51(9), pp. 892-902. https://doi.org/10.1111/medu.13339

 

Durning, S. J. and Artino, A. R. (2011) ‘Situativity theory: A perspective on how participants and the environment can interact: AMEE Guide no. 52’, Medical Teacher, 33(3), pp. 188-199. https://doi.org/10.3109/0142159X.2011.550965

 

Fram, S. M. (2013), ‘The Constant Comparative Analysis Method Outside of Grounded Theory’, The Qualitative Report, 18(1), pp. 1-25. https://nsuworks.nova.edu/tqr/vol18/iss1/1 


Hauer, K. E. (2015) Evaluating Clinical Trainees in the Workplace. On Supervision, Trust and the Role of Competency Committees [dissertation]. Ridderprint BV, the Netherlands: Universiteit Utrecht/Utrecht University.

 

Hauer, K. E., Cate, O. T., Boscardin, C. K., Iobst, W., Holmboe, E. S., et al. (2016) ‘Ensuring Resident Competence: A Narrative Review of the Literature on Group Decision Making to Inform the Work of Clinical Competency Committees’, Journal of Graduate Medical Education, 8(2), pp. 156-164. https://doi.org/10.4300/JGME-D-15-00144.1

 

Hemmer, P. A. and Kelly, W. F. (2017) ‘We need to talk: clinical competency committees in the key of c(onversation)’, Perspectives on Medical Education, 6(3), pp. 141-143. https://doi.org/10.1007/s40037-017-0360-2

 

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., and Cook, D. A. (2014) ‘Standards for reporting qualitative research: a synthesis of recommendations’, Academic Medicine, 89(9), pp.1245-1251. https://doi.org/10.1097/ACM.0000000000000388

 

Sapp, J. E., Torre, D. M., Larsen, K. L., Holmboe, E. S., Durning, S. J. (2019) ‘Trust in Group Decisions: A Scoping Review’, BMC Medical Education, (manuscript under consideration).

 

Shumway, N. M., Dacus, J. J., Lathrop, K. I., Hernandez, E. P., Miller, M., et al. (2015) ‘Use of Milestones and Development of Entrustable Professional Activities in 2 Hematology/Oncology Training Programs’, Journal of Graduate Medical Education, 7(1), pp.101-104. https://doi.org/10.4300/JGME-D-14-00283.1

Appendices

Appendix 1: CCC Observation Themes

 

Model Component

Themes

Evidence

Individual Trustor Factors

Propensity to Trust

N/A

Not observed

Vulnerability

N/A

Not observed

Individual Trustee Factors

Ability

Faculty member experience with trainees

“P10 and P11 consistently volunteered specific, one-on-one examples of experiences with specific trainees along with guidance/recommendations to [trainees].” (KL)

 

“At other times, individuals explicitly acknowledge that they did not have experience with a specific [trainee] and did not feel they had the capacity to make an assessment (P18: ‘I haven’t been here so I don’t think I have that much to add’.)” (KL)

 

“On the whole, the CCC relied frequently upon specific individual stories/experiences with [trainees] as a means of evaluation, with members typically describing one specific personal interaction for one or more [trainees].” (KL)

 

“It was clear that one CCC member (P13) was relied upon to volunteer/respond with commentary specific to the [trainees’] ‘personal style,’ offering input on the relationships between [trainees] and patients.” (KL)

Faculty knowledge requirements

CCC faculty member comment: “Members need to have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance.”  (JS)

 

“Prior to the start of the meeting, the Chair (P32) explicitly indicated that it was his/her first time chairing the CCC meeting and that he/she was unsure how long it would go (or how best to execute the meeting).” (KL)

 

“[The Chair] described prior meetings’ procedural organization, noting that in the past the CCC possibly ‘arbitrarily threw out numbers’ and that P32 wanted this specific CCC meeting to be ‘thoughtful.’” (KL)

 

“There seemed some hesitancy or confusion about next steps (e.g., P44: ‘how do we do this?’ and P46 restating the goals of the CCC).” (KL)

Benevolence

Open mind

CCC faculty member comment: “Members should come to the meeting with no preconceived notions or ‘labels’ for specific trainees.  They should keep an open mind when discussing trainee performance.” (JS)

 

“Other committee members consistently interjected the [trainee] evaluation discussions with questions for other committee members, either about (a) the [trainees’] performance or (b) the CCC process. As typical examples of (a), members would often follow-up with clarifying questions regarding a specific member’s experience with any [trainee] (e.g., when P35 said he/she thinks a [trainee] gets frustrated at times, P32 immediately inquired ‘how so?’).” (KL)

Advocacy

CCC faculty member comments: “The group felt that they were advocates for the [trainees] and the group setting allowed them to identify mentors for individual [trainees] who may be struggling in one area or another.” (JS)

 

CCC faculty member comment: “One member brought up the idea of CCC faculty serving as advocates for their trainees on the CCC.  If trainees have a bad experience or interaction, they should feel comfortable approaching their assigned CCC faculty member to get ahead of this prior to the CCC meeting.” (JS)

Integrity

Buy-in

CCC faculty member comment: “Faculty should recognize that they bought into training these residents/fellows, and the overall agenda of the group should be to make the learners successful.” (JS)

Perceived Commitment to the Group

N/A

Not observed

Group Structure

Diversity

Composition

33.3% to 100% female; 36.4% to 83.3% active duty military

Additional members present (non-physicians)

Diabetic nurse educator, nurse practitioner, 1-2 chief residents, 1-2 administrative personnel

Diversity of opinion

“The committee’s different perspectives mean there is a well-rounded portrait of the [trainee] being painted; the group agreed and described it as a cascade of references for the [trainee’s] performance.” (KL)

 

CCC faculty member comments: “The group felt that a variety of opinions from different faculty levels was important in the process (i.e. including the chief residents, administrators, and program director).” (JS)

Group Member Stability

New faculty selections

 “[The Chair] didn’t think that many faculty members would be good additions, specifically because she didn’t think they understood the mMlestones and the [trainee] education process.” (KL)

Turnover

“Due to turnover of two staff members and the deployment of another, some [trainees] had to be picked up by a new faculty member to complete their end-of-year evaluation and Milestones.” (JS)

Group Size

 

5 to 11 members

Group Processes

 

Cooperation/ Monitoring

Familiarity

“P41 added that the degree of familiarity/comfort with each other (‘we know each other really well’) between both the CCC members and the [trainees] was essential.” (KL)

 

“The CCC included a chief resident. [His/her] input on the CCC provided valuable information about each trainee.” (JS)

Help trainees

“Even in the case of a trainee described as the ‘lowest [first-year trainee] scores we’ve ever had,’ the full group conversation turned to what would be offered in the future in order to aide his/her progress (e.g. P01 stated ‘we could do X to make [the trainee] aware.’).” (KL)

 

“It seemed like the group wanted to generate summative feedback comments (recorded by the Chair) to be provided at a later counseling date with the trainees.” (JS)

Conflict

Task conflict

“Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; inquiries were collegial, open-ended, and seemed to be interpreted as clarifications rather than challenges.” (KL)

 

“No one clearly seemed to disagree with other CCC faculty members, and the group discussion seemed to focus more on brainstorming ideas for how best to support the trainees.” (JS)

Dominant voice/ personality

“P06 volunteered that it is difficult to achieve a confident consensus when there is a forceful personality in the group; other members agreed, and all commented on an example of a colleague with a forceful personality with whom they would not want to serve on such a committee.” (KL)

 

“At this point, P16 would consistently emerge as a dominant personality in both volume and quantity of commentary; unless other CCC members were specifically addressed, P16 volunteered—and at times interrupted— conversation on average four times per each [trainee] discussion (a contrast to other members generally volunteering thoughts zero to two times per [trainee]).” (KL)

 

“At times, P15 would ask general questions about changing the Milestones (e.g., “has the [trainee] moved up to a 3.5?”) and would get no responses at all, to which P15 would point out (e.g., say “Crickets”) and members (most often P16) would volunteer thoughts.” (KL)

Groupthink

“Other CCC faculty members either concurred with the assigned faculty member [Milestone] recommendations (most of the time) or they didn’t (rarely).  There was often minimal group discussion/disagreement on the Milestones as the group seemed reluctant to make changes to assigned faculty recommendations.  Only on a handful of occasions did the discussion about an individual [trainee] spark a discussion about adjusting respective Milestones from the faculty member recommendations” (JS)

 

“If faculty members had no evaluation data to change the existing Milestone level, they left the level from earlier in the year alone.” (JS)

 

“For certain Milestones, there seemed to be less discussion, especially for areas where the faculty members may not have any observational experience with the trainees.” (JS)

Procedural Fairness

Voting

“Though it was clear that P39 was in charge of the group, over time P46 kept formally requesting a vote and appeared to try to institute more formal order to the process.” (KL)

 

“The group followed with a procedural hand-raise vote, with all except P31 voting in favor of ‘graduation with reservations.’” (KL)

 

“At several points CCC members successfully altered the initial numerical assessment and/or P32 would casually ask for a vote from the group.” (KL)

Length of discussion

“During the meeting there was an obvious difference in the amount of time that some [trainees] were discussed as opposed to others.  Struggling [trainees] received significantly more committee discussion than highly performing [trainees] or those with average performance.” (JS)

 

During the CCC meetings observed, the average length of discussion for each trainee varied from 1-3 minutes all the way up to 24 minutes with discussion about one trainee lasting 56 minutes. 

Meeting structure

“As the meeting continued a clear structure emerged, divided according to (Phase 1) a general recounting and discussing of [trainees’] performance and (Phase 2) a specific Milestone assessment of each [trainee].” (KL)

 

“P02 began by assessing each [trainee] according to specific evaluation data, including numerical rankings and formally reading evaluation comments (word-for-word, even when lengthy) while other participants attentively listened.” (KL)

 

“In order of years (second year [trainees], then first year [trainees]), the following pattern was generally employed: P15 would summarize the [trainee’s] schedule, test scores, and evaluation scores (very detailed summaries, but only rarely involving word-for-word reading of specific comments), offer some personal opinions on the strengths/weaknesses of the [trainee], open the discussion to commentary from the CCC members, and then most often end discussion with a summary consensus question (e.g., ‘do we agree?’ ‘other comments?’).” (KL)

 

“After the CCC faculty presented one of their individual interns, a group discussion followed.  During this discussion, the end-of-year Milestones spreadsheet was populated.” (JS)

 

“The order in which each CCC member introduced their [trainees] to the room was determined according to two central criteria: personal time constraints of the member, and whether certain [trainees] had particularly pressing/significant issues to address (indeed P44 began the discussion in respect of personal time constraints). Such cases were introduced earlier in the meeting, whereas satisfactory/typical [trainees] were introduced later in the meeting.” (KL)

Ability to make decisions

“P38 wondered ‘if our only options are to graduate or not…’ and shared that ‘I think we’re at least obligated to share this information’ and ‘I don’t think we have enough information to not graduate [the trainee].’” (KL)

 

“There seemed to be an explicit recognition that the program would be under increased scrutiny for failing to promote someone (e.g. P44 indicated it would look bad for the program, and P46 noted ‘this is going to draw attention when the [trainee] has never failed a rotation’).” (KL)

Milestone assignment

“P02 began each Milestone by explicitly stating the past scores of each of the [trainees], followed by silence while the committee members read the qualifications. Often, the first comment volunteered from any of the committee members would be a ‘new’ score, either the same or marginally higher than prior scores, at which point the group would consider whether they found that suggested score appropriate (e.g. for one Milestone P02 offered ‘I think the [trainee] has reached a 3.’ P03 specifically noted the importance that they all have the same ‘outlook’—an implicit understanding that they don’t want to progress a [trainee] just for the sake of progressing them and that each CCC member buys into that mentality.” (KL)

 

“At this stage there was little to no explicit discussion of numerical Milestone assignments; nearly all the discussion was general about [trainees’] behavior and test performance (with a seemingly implicit assumption that P32 would take those comments and independently assign Milestones as appropriate).  In cases where [trainees] had previously achieved high scores (e.g., 4) on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered (P32: ‘let’s talk globally… I don’t want to go line by line…’) and at times would explicitly state that those Milestones did not need to be considered (P33: ‘If it’s mostly 4’s then we don’t really have to do it [as in depth as for others]’).” (KL)

 

“In regards to the Milestones (the latter part of the meeting), the group went through each individual Milestone with each individual [trainee].  Each Milestone was projected on the screen (which included the title of the Milestone and anchors).” (JS)

 

“Updating the spreadsheet was done in real-time for the first few [trainees] and then was more inconsistent for the duration of the meeting. Most individual Milestone levels for each [trainee] received very little to no discussion. Individual faculty members recommended new/updated Milestones based upon their experience with the [trainee], evaluations they reviewed on the [trainee], and the automated average for each Milestone provided by the electronic evaluation system.” (JS)

Swift Trust

N/A

Not observed

Task Interdependence

N/A

Not observed

Climate

General

“The atmosphere was collegial and positive, and members dove into the discussion quickly.” (KL)

 

“The setting felt laid back and comfortable.” (JS)

Group efficacy

“The group came prepared for a lengthy CCC process; in addition to warning the study team that this would be an open-ended meeting, they brought significant amounts of food for the group to consume throughout the meeting period.” (KL)

 

“It was clear during the group discussion that all of the faculty members knew the [trainees] well and had worked with them frequently.” (JS)

 

“The CCC faculty members knew all of the [trainees] well, so it was easy for them to articulate their opinions about each Milestone without any specific evaluation data.” (JS)

 

“Based on his/her presentation, it seemed clear that this new CCC faculty member does not know or has not worked closely with his/her assigned trainee.” (JS)

Common group identity

“Throughout, the discussion repeatedly made use of the term ‘we’ and referenced what the team could do as a collective to improve the [trainees’] performance.” (KL)

 

“The group felt that redundancy of opinion was useful, especially when making decisions about whether [trainees] are ready for independent practice.  They felt that group consensus in this setting helped to support “[specialty] community standards.”  The group felt that weighing in on when [trainees] are ready for independent practice provides reassurance to society that this group is ensuring that [trainees] graduate with a basic level of skills to practice [specialty] in any setting.” (JS)

Psychological safety

CCC faculty member comment: “Respect for each other is important to build trust, which includes listening to other members when they are talking and sharing.  A collegial atmosphere is important to build trust.” (JS)

 

“P01 referenced the collegial atmosphere and basic competence (with a clear agenda) that makes the CCC a success.” (KL)

Psychological ownership

CCC faculty member comment: “Most of the [trainees] have a longitudinal relationship with their CCC faculty members.” (JS)

Leadership

Conduct of meeting agenda

“The pace was immediately established by the Chair as quick; he/she explicitly stated the order in which [trainees] would be evaluated (third years first, ‘anticipat[ing] it will go quickly’); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones’ meaning (deficiencies, ongoing progress, and aspirational targets.” (KL)

 

“As members showed up and there was a quorum, P32 began by discussing the guidelines and aims of the meeting, emphasizing that he/she would discuss the [trainees] according to their Milestones and overall progress.  P32 distributed information packets on each of the Milestones at the start of the meeting and laid out the organization of the meeting (intending to start from 3rd year [trainees] and progress to 1st year [trainees].” (KL)

Agility

“While the first few [trainees] were initially being evaluated, the Chair was actively entering specific information into the digital Milestone spreadsheet; however, over time he/she appeared to recognize that it was not an effective use of time and that CCC members could simply contact him/her directly with the data to be entered.” (KL)

Facilitating a shared mental model

“As the Chair would transition from his/her own introduction to the discussion, he/she would typically open the conversation by asking, ‘anyone have specific concerns?’  He/she summarized the process of ‘how we do Milestones,’ telling the group the plan to progress according to each individual Milestone and assess all [trainees] within that Milestone at that time (at which point he/she asked for, and received, the group’s consensus agreement on this plan). At times, he/she also directly asked specific CCC members for their own evaluations/experiences with certain [trainees].” (KL)

 

“For the first part of the meeting, the Chair let the group discussion play out without much intervention, and all CCC faculty members seemed to be involved equally in the discussion.  He/she did not really need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion.” (JS)

Interpersonal Interactions

Imported Information

Pre-work assignments

“The discussion centered around the CCC Evaluation Form that was supposed to be filled out by assigned faculty members in advance of the meeting.” (JS)

 

“For each [trainee], faculty members gave some brief comments from the evaluations that they reviewed prior to the meeting along with a general synopsis of the electronically generated numerical values from the evaluation system that was prepopulated on the middle column of the CCC Evaluation Form.” (JS)

 

“There was no set format for how CCC faculty members presented his/her interns.  Some utilized the program’s CCC data collection sheet and some had information on other documents.  However, all of the faculty members covered mostly the same information.” (JS)

Experiences with trainees

“During the general discussion of each [trainee], CCC members would most frequently use specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding [trainee] performance.” (KL)

 

“Each time P12 raised concerns over the course of this specific conversation, other members (P07, P08, P10, P11) volunteered small, specific instances of praise of the same [trainee], and in some instances (P10) explicitly disagreed with P12’s critique.  For another [trainee], P08 raised a personality-related critique; other CCC members (P07, P10, P11) offered specific experiences and praise for the [trainee’s] growth.” (KL)

 

“Members also noted that they sometimes could not think of multiple examples to invoke beyond singular memories, with P32 at one point noting ‘I should have taken more notes during cases I staffed with the [trainee].’” (KL)

 

“P16 added that it was the level of interaction with each of the [trainees] that mattered most in building an effective CCC.” (KL)

Shared Mental Model

Discussion patterns

“Most of the information shared at the meeting was via informal group discussion. Faculty members discussed areas such as medical knowledge, in-training exam scores, procedural competence, mentorship ability (for more senior [trainees]), ability to complete administrative tasks, teaching skills, scholarly activity, medical decision making, patient care, response to feedback, intrinsic motivation, patient presentation skills, and interpersonal communication skills and ability to interact with other [trainees], staff, patients, etc.” (JS)

 

“These discussion patterns largely considered the [trainee] according to global skills/traits (e.g., professionalism) the [trainee] currently exhibits that were then supported with singular experiences of interaction the CCC members had with the [trainees].” (KL)

 

“As the meeting progressed, P33 began to assert a more dominant and organized tone in the meeting, at times ending the initial reading silence faster by offering opening comments (e.g. ‘I can speak to clinic experience with the [trainee]…’) and by explicitly wrapping up conversations (e.g. ‘Who’s next?’ when discussions had devolved into unrelated chatter).” (KL)

Development of a shared mental model

CCC faculty member comments: “Members felt that group input from a personal perspective and discussion is a very important part of this process.  They felt that written evaluations don’t provide nearly the information about an individual [trainee’s] performance than conversation with someone who has worked with them and has a personal opinion about their performance.” (JS)

 

“It was clear that P32 was the final arbiter of the number chosen—per the use of numerous “I” statements (‘I’m fine with 3 or 4;’ ‘I say leave the [trainee] at 3.5,’ etc.).” (KL)

Resources available to the group

“When formally beginning the evaluation of [trainees], P32 asked that all members of the CCC read the entire evaluation packets and comments. This consistently led to notable periods of silence as committee members read the evaluation summaries, as well as when committee members read each individual Milestone criteria silently to themselves during the Milestone assignments.” (KL)

Disparate opinions

“Disparate opinions seemed to be constructive and productive although the degree of disagreement or dissention in the group was low.  Different CCC faculty members brought up additional/supplementary information, and for the most part consensus occurred fairly easily within the group.” (JS)

 

“After generally agreeing with a hybrid solution—graduating the [trainee] with noted reservations—P31 asked if they could formally record P31’s dissent from that decision (with P31 explaining that they recommended graduation without reservations, noting the number of cases the [trainee] had been a part of and emphasizing that ‘if the [trainee] has made it this far without remediation, the CCC’s role should be to graduate the [trainee].’” (KL)

Potential for Future Interactions and Relationships

N/A

Not observed

Political Skill

N/A

Not observed

Relational Identity Orientation

N/A

Not observed

Need for Affiliation

N/A

Not observed

Standing in a Group

Non-voting members

“The chief residents freely provided input about their experiences with most of the [trainees]. They were included in the discussion to provide input from the near peer perspective.  These two faculty members seemed to have a unique perspective and insight into most of the trainees that other faculty members may not in a position to observe.” (JS)

Seniority

“P40 seemed like the most junior faculty member on the CCC, but she freely participated in the discussion for multiple [trainees] and had specific examples of patient care and faculty member interactions with a number of [trainees] from the clinic.” (JS)

Environmental Factors

Reporting Relationships

N/A

Not observed

Task Complexity/ Uncertainty

CCC role

“During the CCC discussion it became clear that group members were unsure of what the process was should they not want to recommend the [trainee] for independent practice; P36: ‘the alternative is what?’” (KL)

 

“There seemed some hesitancy or confusion about next steps (e.g., P44: ‘how do we do this?’ and P46 restating the goals of the CCC).” (KL)

 

“At this point, P33 and P36 actively debated how the CCC assessment information would be shared and what regulative power the committee actually had in an instance in which the [trainee] may need additional supervision.” (KL)

 

“During the meeting, it was unclear from an observer standpoint exactly what decisions the group was making.” (JS)

CCC faculty member role

“Various members discussed the role of the individual CCC faculty member.  Should members be more of a coach/mentor versus an evaluator?  Should trainees be aligned to faculty members based upon aligned interests, specialties, etc.?” (JS)

Milestones

“On several occasions, the group noted difficulty in determining numerical Milestone assignments given how the Milestones were written/phrased, and commented on the difficulty in fitting individuals into the specifically defined categories (e.g., at one point P16 noted the ‘insanity’ of anyone being able to achieve the highest ranking for one specific Milestone).” (KL)

 

“At times, the CCC members expressed frustration with the Milestone definitions, specifically to what the Milestones are unable to capture (e.g., burnout).” (KL)

 

“It seemed to me that there are so many Milestones and so many [trainees] that there were a number of gaps in information to make informed changes to the existing Milestone levels, and faculty members were comfortable leaving Milestones alone in these circumstances or going along with the assigned faculty member recommendations when there was no data to change the Milestone level.” (JS)

Developing remediation plans

“Rarely, comments were made to the effect of not knowing what to do (e.g., P02 stated for one [trainee] that they ‘don’t know how to help.’” (KL)

Outside Group Memberships/ Social Ties

N/A

Not observed

Threats

Decision-making fatigue

“The meeting ran several minutes past its scheduled time, and there was some small indication that time pressure and decision-making fatigue were setting in among the group—for instance, the group was quicker to both deliver evaluations and come to consensus on those evaluations over time.” (KL)

 

“[Faculty members] nearly unanimously acknowledged that they get more fatigued as the meeting continued and often end up brusquely assessing [trainees] on the final Milestones.” (KL)

Schedule conflicts

“At one point P08 took a phone call and left the room, and by 8:30 P07 had to leave to attend another meeting.” (KL)

 

“In one instance a CCC member (P36) needed to complete outside grading work—but rather than leave the CCC meeting, he/she simply asked the committee if it was ok for him/her to focus on the computer and submitting grades while the committee continued its discussion and that P36 would chime in when necessary (the group agreed; P36 did redirect focus away from the meeting but occasionally volunteered comments in line with [trainee] discussions).” (KL)

Institutional structures/ policies

“Several times the CCC considered not just how the [trainee] did/did not perform, but how the institutional structures were/were not at fault.” (KL)

Technical difficulties

“At one point, P02 sought to locate additional electronic materials to show the group using the available screen, but due to technical difficulties he/she was unable to have the materials at their disposal.” (KL)

Staffing Levels

N/A

Not observed

Declarations

There are some conflicts of interest:
The authors declare that they have no conflicts of interest. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the Department of Defense, or the U.S. Government.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

Our research was approved by the Walter Reed National Military Medical Center Institutional Review Board (IRB; WRNMMC-2018-0137). All participating training program directors and individual CCC faculty members provided written consent prior to their participation in the study.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Ken Masters - (07/07/2019) Panel Member Icon
/
This is an interesting qualitative observational study of group trust in the setting of clinical competency decisions by committee. The authors have clearly explained the central issue of the need for trust (within the group, and, by extension, in the processes) by which the committee establishes clinical competencies required, and against the theoretical framework of situated cognition. The process involved observation of committees.

Some issues with the paper
• I am concerned with the methodology and terminology. The authors say that they used the constant comparison method in their data recording and coding, but had an initial proposed framework (“Before collecting data we developed a structured observation guide (included as a supplemental file) based on our literature review and proposed model”. Then the guide was thoroughly piloted. My understanding of constant comparison (Glaser 1965) is that it should be used precisely when there is no framework, no pre-supposition, no model or theory of any sort. Constant comparison aims at generating ideas and theory from scratch, with no limitations. But the authors appear to have started with a proposed model (in effect, a hypothesis), and then coded and gathered their data, and then matched that data to the model. The Methods speaks of data gathering and analysis, (“Our proposed model served as an organizing framework for reporting our findings"), and there was no modelling or theory construction, which is THE central feature of constant comparison. The process followed by the researchers is perfectly respectable, but it appears to a process of testing and refining the model, not constant comparison. I would like the authors to address this seeming contradiction, or to perhaps rephrase the terminology used to describe their method.

• I think that the authors should make it more explicit that they are at a military institution, otherwise the statement that people were not “influenced by military rank” might seem strange. (Yes, the researchers do point out that ethics approval by the “Walter Reed National Military Medical Center Institutional Review Board”, but that is only single mention, and could slip by unnoticed.) So, in the line “program CCCs at our institution” they may wish to mention the institution’s name explicitly.

• Given the point above, the authors may wish to explore, or at least touch upon, and mention in the limitations, the extent to which being at a military institution may impact on the group dynamics in ways in which would not be applicable in non-military institutions. While the researchers have commented that individual ranks did not appear to influence the work, a reader might wonder if other military attitudes and behaviour, discipline, punctuality, etc., might have impacted in some way on the committees.

• Given that the researchers began with a proposed model and then gathered data, it would be useful to see some discussion about whether or not the new data caused the model to be modified, and how and why, perhaps with a before-and-after image of the model.

• The Introduction speaks of the use of situation cognition, as does the opening line of the Discussion. Apart from that, however, there appears to be little direct and overt reference to the theory. Perhaps the authors could make the connections a little clearer.

So, a well-planned and detailed study, but clarity is needed on the terminology of the methodology and some other contextual issues.
Possible Conflict of Interest:

For Transparency: I am an Associate Editor of MedEdPublish

Richard Hays - (25/04/2019) Panel Member Icon
/
While the title was attractive, I found this paper difficult to read and to digest. The topic is interesting, as so many educators spend time at meetings like those described here, where judgements are combined about learning progress of students or trainee doctors. I may well have observed most variants of normal and abnormal behaviour among fellow committee members, and also may well have gone through most of the possible phases of elation, frustration and even anger. However, after reading this paper I am not sure that I have a better understanding of group dynamics, nor a sense of how such meetings should be managed. I wonder if the term 'trust' is too broad for the concept at the heart of the study? Trust is such a complex construct, based on shared experiences, reliability and mutual respect. At first, I thought that the trust was about the performance of the students/trainees, but I do not think it is, even though individual judgements of a leaner's progress may include an element of trust in a junior colleague to do the right thing, and therefore be 'safe'. This may well be a contributor to the judgments expressed at meetings. Greater clarification of what the authors mean and where it is applied would be helpful. I like the emergence of 'group think' as a real risk, because the medical profession is sometimes seen as a group that inappropriately 'sticks together' on issues of competence/negligence. Are adverse judgements based on a single, major error or several minor flaws? Is there evidence from direct observation or the powerful 'rumour mills'? What role does recognising 'people like us" play? How can the strong voices (either way) be countered in a situation where diversity of judgement is healthy? Overall, a paper of interest to all involved in committee meetings, but potentially of more valued if it provided more concrete strategies for chairing/facilitating/participating in committee meetings.
Possible Conflict of Interest:

I am the Editor of MedEdPublish