We observed six CCC meetings in the following specialties from May to June 2018: general pediatrics, pediatric endocrinology, neonatal-perinatal medicine, internal medicine, and adult cardiology and hematology-oncology. During this time, we observed a total of 15 hours’ worth of meetings amounting to 36 pages of data, and we reached thematic saturation before the sixth CCC meeting. Our proposed model served as an organizing framework for reporting our findings. See Appendix 1 for themes that were observed during these meetings, with representative quotes, which are organized under various components of the model (Figure 1) and discussed below. We aggregated data from the observations and the debrief discussions into the sections below and contend that the different components of the model may affect group trust.
Figure 1. Proposed Group Trust Model

Group Level Factors
Group Structure
The size of the committees varied from 5 to 11 individuals (average 8, standard deviation 2.8). All assigned faculty members were present for the meeting with two exceptions. An absent faculty member on one CCC provided her scoring sheet to the CCC chair in advance of the meeting. On another CCC, the committee had to table the discussion of an absent faculty member’s trainees because he had not provided recommendations about the updated Milestone levels for these individuals. Besides physician faculty members, some of the CCCs also included a diabetic nurse educator, nurse practitioner, chief residents who had completed a core residency program, and training program administrative personnel. CCC members commonly reported that a diversity of opinions from different faculty members contributes to a more complete collective understanding/picture of competence for each trainee.
Group Processes
Group Conflict
In all of our observations, we did not witness any instances of affective (relationship) conflict. Rather, during each meeting we observed at least one episode of task (cognitive) conflict, which tends to arise from individual differences in viewpoints, ideas, or opinions when group members participate in a shared undertaking. Observers did not specify how frequently these episodes occurred during a single meeting, and CCC faculty members in general seemed to offer personal experiences that were largely confirmatory and agreeable relative to points about learners raised by the CCC chair or other members. However, “individuals at times volunteered differing comments, but [these experiences] were usually treated as apples-and-oranges differences rather than conflicting experiences (KL).” When assigning individual Milestones, some differences of opinion occurred. “Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; the majority of inquiries were collegial, open-ended, and were interpreted as clarifications rather than challenges (KL).”
Some faculty members felt that dominant voices/personalities may negatively impact a CCC. One member remarked, “It is difficult to achieve a confident consensus when there is a forceful personality in the group, especially if other members of the committee may be more conflict averse.” In one CCC, however, a faculty member consistently emerged as a dominant personality in both volume and quantity of commentary. Unless other CCC members were specifically addressed, this individual volunteered, and at times interrupted, conversation an average of four times per each trainee discussion (a contrast to other members generally volunteering thoughts zero to two times per trainee). These comments consisted of generally confirming/restating the point of others, injected with personal experiences from working with various trainees. The other CCC members never attempted to prevent this individual from speaking, nor did they give any indication of annoyance or frustration (rather, there was an indication that they simply were comfortable and did not mind having the forceful personality present).
Group Procedural Fairness
Regarding the assignment of individual Milestones, procedures varied with each training program observed. For smaller programs, such as pediatric endocrinology (3 trainees), the group generally went through each individual Milestone in greater detail with each trainee, and individual trainees were not assigned to faculty members. In cases where trainees had previously achieved high scores on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered (“let’s talk globally… I don’t want to go line by line…”) and at times would explicitly state that those Milestones did not need to be considered (“if it’s mostly 4’s then we don’t really have to do it [as in depth as for others]”). However, this was not universally true; on another CCC, several trainees had their Milestone levels decreased given some faculty concerns about professionalism and completing administrative tasks on time.
For training programs with a larger number of trainees (i.e. internal medicine and pediatrics with 74 and 29 trainees respectively), faculty members were assigned individual trainees and utilized a standardized CCC data collection form that they filled out in advance of the meeting, which included proposing updated Milestone levels for each trainee. During the meeting, faculty members took turns presenting the information off of these forms, which seemed to help standardize the process of discussing trainees and assigning Milestones. Faculty presentations were followed by varying degrees of discussion. In some circumstances, “the group seemed reluctant to make changes to the assigned faculty recommendations (JS),” and the discussion was more likely to involve groupthink (i.e. group members refrained from expressing doubts and judgments or disagreeing with the consensus).
In other CCCs, there also seemed to be less discussion for certain Milestones, especially in areas where the faculty members may not have any observational experience or evaluation data for specific trainees. In some committees, “very rarely were specific Milestone numbers discussed; rather the discussions focused almost entirely on [trainee] skill-sets and general scoring (KL).” Multiple CCC members also indicated that they felt they should not progress trainees in the context of the Milestones just for the sake of progression. Members stated that they should be comfortable keeping trainees at their same level if no clear evidence exists to suggest that the learner has progressed in that specific Milestone.
Group Climate
The atmosphere of the meetings was “collegial and positive (KL)” and “comfortable (JS).” Each meeting seemed to be a safe environment where all opinions were welcome and openly discussed. One member stated that “respect for each other is important to build trust, which includes listening to other members when they are talking and sharing.” In one CCC, the discussion repeatedly made use of the term “we” and referenced what the team could do as a collective to improve the trainees’ performance, which is the hallmark of a common group identity. Further, the cardiology CCC chair stated that consensus in this setting helped to support “[cardiology] community standards.” The group agreed that determining when fellows are ready for independent practice provides reassurance to society that these trainees graduate with a basic level of skills to practice cardiology in any setting. Finally, it was clear during some of the group discussions (especially in smaller programs with fewer than 15 trainees) that all of the faculty members knew the trainees well and had worked with them frequently.
Group Leadership
The two most important leader behaviors that we observed were the abilities of the CCC chair to conduct the meeting and to facilitate a shared mental model. In most of the meetings we observed, the chair provided an overview of the Milestones to the group and the structure of the session. “The pace was immediately established by the chair as quick; he explicitly stated the order in which [trainees] would be evaluated (third years first, ‘anticipat[ing] it will go quickly’); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones’ meaning (deficiencies, ongoing progress, and aspirations) (KL).”
Two different meeting formats emerged during our observations. Some CCCs went through all of the trainees in order (e.g. by post-graduate year, alphabetically, or based upon other external factors) discussing the trainee in general terms and assigning the Milestones at the same time. “It seemed like the faculty who had other competing schedule requirements were given priority to present their [trainees] early (JS).” For other CCCs, meetings generally had two distinct phases. In phase 1 the chair led a discussion to generate formative comments for each trainee, and in phase 2 the group assigned the Milestones for each trainee.
Generating a shared mental model seemed to be one of the most important parts of making group decisions and assigning Milestones, and the CCC chair appeared critical in facilitating this. In CCCs of smaller programs (i.e. fewer than 15 trainees), the chair usually introduced and provided some information about each trainee. “[The CCC chair] began by summarizing the past activities and future plans of each of the [trainees], in addition to comments about exam performances and most recent one-on-one experiences with the [trainee] (KL).” In larger programs, assigned faculty members generally provided this information as discussed above. In all of the meetings, group members tended to freely share their opinions; for instance, in one meeting “the CCC chair did not need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion (JS).” When one of the members commented about a trainee’s performance, other members tended to jump into the conversation to provide their individual experience with the trainee.
Individual Level Factors
Several faculty members felt that individuals on a CCC should have some baseline knowledge and experience in medical education. One CCC member noted that “individuals who serve on the committee should have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance.” Additional areas identified by the observers include buy in (“faculty should recognize that they bought into training these residents/fellows, and the overall agenda of the group should be to make the learners successful, to the maximum extent possible”); having an open mind (“members should come to the meeting with no preconceived notions or ‘labels’ for specific trainees”); and advocacy for the trainees (“the group felt that they were advocates for the trainees, and the group setting allowed them to identify mentors for individual trainees who may be struggling (JS)”). We did not observe any specific comments related to patient advocacy and if any tension may result from advocacy for patients versus trainees.
Interpersonal Interactions
Imported Information
Information brought to the meeting by individual faculty members is important to helping build a shared mental model. Two main themes that emerged from the observations include information obtained from faculty pre-work assignments and individual experiences with trainees. For CCC decisions, the latter seemed to be the most frequently relied upon source of information. Especially for smaller programs, CCCs frequently used specific individual stories and/or experiences with trainees as a means of evaluation. During the general discussion of each trainee, CCC members would most frequently offer specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding trainee performance.
Shared Mental Model
During our observations we noted three general themes whereby CCCs build a shared mental model: assignment of the Milestones, perceptions regarding the developmental level and overall clinical competence of various trainees, and the work and processes of the CCC. When building a shared mental model, CCCs discussed a number of different topics to include trainee global strengths/weaknesses (e.g. praises of bedside manner, maturity and concerns of professionalism, and personality issues), instances in which feedback was given to trainees (and whether they did/did not respond to it), and general discussions of what the CCC members could do and actions they could take in the future in order to aid the trainees and foster additional growth. Indeed, when discussing solutions for questions and/or concerns about certain trainees, CCC members either described specific instances in which they attempted to correct behavior one-on-one or discussed team actions in abstract terms (e.g., “we have to somehow make it clear to [the trainee]…” or “we might need to consider program-level remediation…”). A sample of other topics discussed are listed in Appendix 1.
When there was quick group consensus regarding the Milestones, CCC chairs would generally move the discussion forward to the next trainee. In instances when a member disagreed with any of the proposed numbers, he or she would usually either implicitly ask the CCC member who offered the number why (“really?”) or offer a contradictory opinion (“I’d say [the trainee] is still probably at a x…”), at which point CCC members would offer specific examples, explanations, or evaluations that explain their points. Additionally, there was an occasional voiced reluctance to offer “perfect” scores, as the committee generally agreed that trainees may never achieve some aspirational competencies.
Conversations seemed to be key to the process of discussing trainee competence and building a shared mental model. Members used the analogy of a puzzle in which each faculty member had pieces of data about experience with trainees, and together through discussion and information sharing a more complete picture of trainee competence emerged during the meeting. It was generally felt that members should share concrete, specific examples that can be interpreted by the group in a relevant context. Individual participation is important for this to occur, and discussion amongst the faculty members led to additional insights and comments about each trainee. The discussion uncovered information not written on global or other evaluation forms. One faculty member noted that written evaluations are most useful for the highest and lowest performing trainees. He stated that “97% of the time my evaluation is the same for trainees in between those two extremes and in the middle of the competency spectrum.” He also felt that Likert scales are not useful and that group discussion is a much more effective way to share information about group members’ experiences with individual trainees. Some members reported feeling that it was easier to identify troubling trends with trainees based upon conversations with faculty members who have worked with these individuals. Oftentimes they said that it is hard to write negative comments on an evaluation form, especially if that form goes back to the trainee. As a result, these “neutral” evaluations may not identify trends in a way that conversation does. For example, if more concerning faculty observations arise with trainees and there has been no record of borderline performance or other concerns, it may be harder to justify things like program level remediation, etc.
To facilitate a shared mental model, some CCCs had additional resources available to the group. In most meetings, the group had access to a copy of the Milestones. In others the CCC chair had complete evaluation packets for each faculty member and asked the group to read the entire packet prior to discussing trainees. The latter led to notable periods of silence as committee members read the evaluation summaries, as well as specific Milestone criteria.
Standing in a Group
In all of the CCCs, participation in the meeting did not seem to be influenced by military rank, seniority, specialty, or whether one was a voting member of the group. In two CCCs chief residents attended as non-voting members of the committee, and they freely provided input from a near peer perspective about their experiences with most of the residents. These individuals had unique insight into most of the trainees that other faculty members may not be in a position to observe, and their input on the CCC provided information the committees considered very valuable. One CCC also solicited input from two program administrators, and their comments were valuable regarding trainee accomplishment of administrative tasks.
Environmental Factors
Task Complexity/Uncertainty
We identified the following themes under this heading: specific role of the CCC (e.g. two members in one meeting actively debated what regulative power the committee had when a trainee may need additional supervision); faculty member roles (e.g. should the members be more of a coach/mentor versus an evaluator or should trainees be assigned to faculty members based upon aligned interests?); developing remediation plans; lack of faculty member experience with certain trainees; and the Milestones themselves. In regards to the Milestones, members at times expressed frustration with the Milestone definitions, especially those with lengthier anchors and what the milestones don’t capture (e.g., burnout). In several instances groups noted the difficulty in determining numerical milestone assignments given how the Milestones were written/phrased and commented on the difficulty in fitting individuals into the specifically defined categories. Some groups acknowledged the difficulty and/or worry in selecting only one demonstrative experience as a justification for a number assignment. For programs with a larger number of trainees (i.e. greater than 15 individuals), information and evaluation gaps complicated the ability of the CCC to make informed assignments of some individual trainees’ Milestone levels. Additionally, a large number of trainees coupled with the number of Milestones typically led to longer meetings, which seemed to truncate the discussion and the competency evaluation process compared to CCCs with fewer trainees.
Threats
One of the biggest environmental threats identified was decision making fatigue. While members of one CCC admitted that they come to the meeting knowing what needs to be accomplished, almost all acknowledged that they get more fatigued as the meeting continues and tend to make more brusque decisions. Observations near the end of the meetings included members increasingly stepping out for short intermittent periods, multiple members checking their cell phones and fidgeting, group members openly asking how many trainees they had left to evaluate, and/or comments such as “I’m fading,” “homestretch for milestones!” and “last one!,” to CCC cheers. Other threats identified during the observations included scheduling conflicts, technical difficulties, and institutional structures/policies (e.g. on several occasions one CCC considered not just how the trainee did/did not perform, but how the institutional structures may/may not be at fault).