Developing a process to assess clinical competence is an essential component of all graduate medical education (GME) programs.  Built upon the fundamentals of competency-based medical education, the Accreditation Council for Graduate Medical Education (ACGME) Milestones help GME programs determine which of their trainees are on appropriate paths towards unsupervised clinical practice and which are not.  Within a GME program, the clinical competency committee (CCC) forms a fundamental part of determining trainees’ progress in the context of the Milestones, and some recent studies describe data sources that CCCs use and how they weigh this information in their collective decisions (Choe et al., 2016; Shumway et al., 2015).  Several other studies have also looked at group decision making processes and how these concepts may be utilized by CCCs (Chahine et al., 2017; Hauer et al., 2016; Hemmer and Kelly, 2017). 

Given the important decisions shouldered by the CCC (e.g. overall progress, trainee promotion, and remediation), trust is an important foundational component of these committees, enabling open and collaborative discussions of strengths and areas of improvement, to include recommendations of remediation and dismissal.  In the context of individual trainee-supervisor relationships, health professions education research has explored trust in decisions that faculty make about the level of supervision granted for different clinical responsibilities and the factors that influence those decisions (Damodaran, Shulruf, and Jones, 2017; Hauer, 2015; Cate et al., 2016).  Entrustable Professional Activities (EPAs) are an example of these decisions and address tasks that trainees are permitted to execute once they attain a sufficient level of competence; the level of supervision for each EPA depends upon how much a supervisor trusts a trainee to complete that specific task (Cate, 2013).  While individual decisions of trust are important, little work has been done in health professions education to look at trust as it relates to group decisions, such as trust within a CCC, trust in the context of how characteristics and processes of these committees might impact collective decisions, and group member trust in these decisions. 

In a recent scoping review, we defined group trust (which encompasses both trust within a group as an aggregate or combination of individual trust and trust in the context of group decisions) as a group-directed willingness to accept vulnerability to actions of the members based on the expectation that members will perform a particular action important to the group, encompassing social exchange, collective perceptions, and interpersonal trust.  We also examined literature outside of health professions education to determine what factors may influence trust from the perspective of those who rely on decision making groups and proposed a model (Figure 1) to illustrate the relationship between trust at the individual and group levels (Sapp et al., 2019).  In this model, we delineate individual level factors, group level factors (i.e. group structure and processes), interpersonal interactions, and environmental factors.  As empirical data in a health professions education context has not been collected for this literature-informed model, we conducted the current study with two primary purposes: a) test our proposed model in the context of CCC collective decisions and assignment of the Milestones in primary care (i.e. internal medicine and pediatrics) and subspecialty settings and b) assess what factors are important to individual faculty members regarding trust in the setting of CCC decisions. 

Theoretical Framework

For decision making groups to reach a consensus, communication and interactions between and among people, objects (those items that extend human capabilities in a learning or task-oriented setting, such as learning management systems), and the environment are believed to be essential.  We therefore used situated cognition as the underlying theoretical framework for our model (Sapp et al., 2019).  Situated cognition contends that thinking is viewed as situated (or located) within the larger social and physical context of the environment (Durning and Artino, 2011).  This theory suggests that a group setting is highly complex with multiple components (i.e. social, cultural, and physical) and opportunities for interactions between these components.  It recognizes the complex interplay between participants, objects that augment individuals’ cognitive capabilities (i.e. artifacts), and the environment.  Situated cognition places emphasis on how these various interactions lead to thinking and decisions, and this framework informed our model.

We conducted a qualitative observational study involving GME program CCCs at our institution to determine how collective decisions about trainee progress within these programs are made in the context of the Milestones.  We focused on internal medicine and pediatrics residency and subspecialty fellowship programs within the National Capital Consortium because they represent the largest population of GME trainees within our institution.

Given the social and group dynamics underpinning CCCs and their decisions, along with the multiple inputs to our model, we believe that a qualitative approach was optimal for this study.  Our qualitative approach also allows for iterative construction of the results based upon the emergence of conceptual categories and descriptive themes.

Data Collection

Before collecting data we developed a structured observation guide (included as a supplemental file) based on our literature review and proposed model (Figure 1) to measure various components of trust and to focus data collection on a CCC’s structure and processes.  We also held conversations with subject matter experts who informed revisions to the guide.  The structured guide included the following headings along with a sample of items within those sensitizing concepts: individual-level factors (i.e. observed behaviors that may affect interactions amongst group members, group decisions, etc.), group structure (e.g. group size and diversity), group processes (e.g. group conflict, procedures utilized by the CCC, overall climate of the group, and leadership), information sharing/interpersonal interactions, and environmental factors (e.g. task complexity, uncertainty, and threats).  Next we pilot tested the guide by observing four CCC meetings in internal medicine and adult endocrinology (amounting to 10.5 hours of observation) from September 2017 to May 2018 and made revisions based upon observations in these meetings.  Observers were part of the research team, had the opportunity to practice observing during these four CCC meetings, and helped to revise the observation guide.

We sent out requests to a purposive sample of programs to observe their CCC meetings during the June 2018 end-of-year Milestone reporting period.  For programs that elected to participate, we conducted structured observations of the CCC meetings using the revised observation guide.  We had two observers in the room, and they observed as spectators (as opposed to a participant observer).  To minimize interference in the CCC meeting, observers sat outside the group circle along the perimeter of the room.  Observations were from an outside perspective, and the inquiry of the observation was directed entirely by the observers (i.e. there was no collaboration with those being studied).  Observers provided full disclosure of their role to those being observed and independently recorded field notes utilizing the structured observation guide.  At the end of each CCC, the lead observer asked the committee, “what do you think contributes to trust in CCC decisions?”  Observers collected information provided by individual committee members on their field notes.   

Data Analysis

Field notes were independently read and coded by the two CCC meeting observers (JS, KL).  To code and analyze the data, we used a constant comparison method (Fram, 2013) to identify and confirm themes and/or topics that emerged from the observations and discussions.  We looked at published approaches to group decision making (i.e. individual and group level factors, environmental factors, the interactions between all of these elements, and measurable components of trust), how CCCs operationalize this information, components of trust in the setting of decisions made by these groups, and the emergence of new concepts or themes.  We sought counter-examples to determine how new concepts or categories may vary from the published literature and how this impacts the process of group decision making.  Data analysis occurred simultaneously with the collection of data in order to monitor for thematic saturation and the need for any new directions in data collection.  The research team (JS, KL, DT, SD) met regularly to discuss coding and resolve all differences.  During analysis, memoranda and diagrams were used to keep written records of the process.  Techniques to enhance trustworthiness of the qualitative analysis process per the Standards of Reporting Qualitative Research included member checking, maintaining an audit trail, and collection techniques to allow triangulation of the data (O’Brien et al., 2014).

Ethics

Our research was approved by the Walter Reed National Military Medical Center Institutional Review Board (IRB; WRNMMC-2018-0137).  All participating training program directors and individual CCC faculty members provided written consent prior to their participation in the study.

We observed six CCC meetings in the following specialties from May to June 2018: general pediatrics, pediatric endocrinology, neonatal-perinatal medicine, internal medicine, and adult cardiology and hematology-oncology.  During this time, we observed a total of 15 hours’ worth of meetings amounting to 36 pages of data, and we reached thematic saturation before the sixth CCC meeting.  Our proposed model served as an organizing framework for reporting our findings.  See Appendix 1 for themes that were observed during these meetings, with representative quotes, which are organized under various components of the model (Figure 1) and discussed below.  We aggregated data from the observations and the debrief discussions into the sections below and contend that the different components of the model may affect group trust.

Figure 1.  Proposed Group Trust Model        

Group Level Factors

Group Structure

The size of the committees varied from 5 to 11 individuals (average 8, standard deviation 2.8).  All assigned faculty members were present for the meeting with two exceptions.  An absent faculty member on one CCC provided her scoring sheet to the CCC chair in advance of the meeting.  On another CCC, the committee had to table the discussion of an absent faculty member’s trainees because he had not provided recommendations about the updated Milestone levels for these individuals.  Besides physician faculty members, some of the CCCs also included a diabetic nurse educator, nurse practitioner, chief residents who had completed a core residency program, and training program administrative personnel.  CCC members commonly reported that a diversity of opinions from different faculty members contributes to a more complete collective understanding/picture of competence for each trainee.

Group Processes

Group Conflict

In all of our observations, we did not witness any instances of affective (relationship) conflict.  Rather, during each meeting we observed at least one episode of task (cognitive) conflict, which tends to arise from individual differences in viewpoints, ideas, or opinions when group members participate in a shared undertaking.  Observers did not specify how frequently these episodes occurred during a single meeting, and CCC faculty members in general seemed to offer personal experiences that were largely confirmatory and agreeable relative to points about learners raised by the CCC chair or other members.  However, “individuals at times volunteered differing comments, but [these experiences] were usually treated as apples-and-oranges differences rather than conflicting experiences (KL).”  When assigning individual Milestones, some differences of opinion occurred.  “Occasionally, after committee members suggested scores, other committee members would inquire how they saw the [trainee] as satisfying that score; the majority of inquiries were collegial, open-ended, and were interpreted as clarifications rather than challenges (KL).”

Some faculty members felt that dominant voices/personalities may negatively impact a CCC.  One member remarked, “It is difficult to achieve a confident consensus when there is a forceful personality in the group, especially if other members of the committee may be more conflict averse.”  In one CCC, however, a faculty member consistently emerged as a dominant personality in both volume and quantity of commentary.  Unless other CCC members were specifically addressed, this individual volunteered, and at times interrupted, conversation an average of four times per each trainee discussion (a contrast to other members generally volunteering thoughts zero to two times per trainee).  These comments consisted of generally confirming/restating the point of others, injected with personal experiences from working with various trainees.  The other CCC members never attempted to prevent this individual from speaking, nor did they give any indication of annoyance or frustration (rather, there was an indication that they simply were comfortable and did not mind having the forceful personality present).

Group Procedural Fairness

Regarding the assignment of individual Milestones, procedures varied with each training program observed.  For smaller programs, such as pediatric endocrinology (3 trainees), the group generally went through each individual Milestone in greater detail with each trainee, and individual trainees were not assigned to faculty members.  In cases where trainees had previously achieved high scores on their prior Milestone assignments, committee members would both implicitly assume that those Milestones did not need to be considered (“let’s talk globally… I don’t want to go line by line…”) and at times would explicitly state that those Milestones did not need to be considered (“if it’s mostly 4’s then we don’t really have to do it [as in depth as for others]”).  However, this was not universally true; on another CCC, several trainees had their Milestone levels decreased given some faculty concerns about professionalism and completing administrative tasks on time.   

For training programs with a larger number of trainees (i.e. internal medicine and pediatrics with 74 and 29 trainees respectively), faculty members were assigned individual trainees and utilized a standardized CCC data collection form that they filled out in advance of the meeting, which included proposing updated Milestone levels for each trainee.  During the meeting, faculty members took turns presenting the information off of these forms, which seemed to help standardize the process of discussing trainees and assigning Milestones.  Faculty presentations were followed by varying degrees of discussion.  In some circumstances, “the group seemed reluctant to make changes to the assigned faculty recommendations (JS),” and the discussion was more likely to involve groupthink (i.e. group members refrained from expressing doubts and judgments or disagreeing with the consensus).

In other CCCs, there also seemed to be less discussion for certain Milestones, especially in areas where the faculty members may not have any observational experience or evaluation data for specific trainees.  In some committees, “very rarely were specific Milestone numbers discussed; rather the discussions focused almost entirely on [trainee] skill-sets and general scoring (KL).”  Multiple CCC members also indicated that they felt they should not progress trainees in the context of the Milestones just for the sake of progression.  Members stated that they should be comfortable keeping trainees at their same level if no clear evidence exists to suggest that the learner has progressed in that specific Milestone.

Group Climate

The atmosphere of the meetings was “collegial and positive (KL)” and “comfortable (JS).”  Each meeting seemed to be a safe environment where all opinions were welcome and openly discussed. One member stated that “respect for each other is important to build trust, which includes listening to other members when they are talking and sharing.”  In one CCC, the discussion repeatedly made use of the term “we” and referenced what the team could do as a collective to improve the trainees’ performance, which is the hallmark of a common group identity.  Further, the cardiology CCC chair stated that consensus in this setting helped to support “[cardiology] community standards.”  The group agreed that determining when fellows are ready for independent practice provides reassurance to society that these trainees graduate with a basic level of skills to practice cardiology in any setting.  Finally, it was clear during some of the group discussions (especially in smaller programs with fewer than 15 trainees) that all of the faculty members knew the trainees well and had worked with them frequently.

Group Leadership

The two most important leader behaviors that we observed were the abilities of the CCC chair to conduct the meeting and to facilitate a shared mental model.  In most of the meetings we observed, the chair provided an overview of the Milestones to the group and the structure of the session.  “The pace was immediately established by the chair as quick; he explicitly stated the order in which [trainees] would be evaluated (third years first, ‘anticipat[ing] it will go quickly’); the pertinent information (timeline details, when Milestones are due, etc), and a brief reminder of the core of the Milestones’ meaning (deficiencies, ongoing progress, and aspirations) (KL).” 

Two different meeting formats emerged during our observations.  Some CCCs went through all of the trainees in order (e.g. by post-graduate year, alphabetically, or based upon other external factors) discussing the trainee in general terms and assigning the Milestones at the same time.  “It seemed like the faculty who had other competing schedule requirements were given priority to present their [trainees] early (JS).”  For other CCCs, meetings generally had two distinct phases.  In phase 1 the chair led a discussion to generate formative comments for each trainee, and in phase 2 the group assigned the Milestones for each trainee.

Generating a shared mental model seemed to be one of the most important parts of making group decisions and assigning Milestones, and the CCC chair appeared critical in facilitating this.  In CCCs of smaller programs (i.e. fewer than 15 trainees), the chair usually introduced and provided some information about each trainee.  “[The CCC chair] began by summarizing the past activities and future plans of each of the [trainees], in addition to comments about exam performances and most recent one-on-one experiences with the [trainee] (KL).”  In larger programs, assigned faculty members generally provided this information as discussed above.  In all of the meetings, group members tended to freely share their opinions; for instance, in one meeting “the CCC chair did not need to engage individual faculty members because all freely participated when it seemed like they had something to contribute to the discussion (JS).”  When one of the members commented about a trainee’s performance, other members tended to jump into the conversation to provide their individual experience with the trainee.

Individual Level Factors

Several faculty members felt that individuals on a CCC should have some baseline knowledge and experience in medical education.  One CCC member noted that “individuals who serve on the committee should have a basic level of competence as a physician within their specialty and a basic understanding of evaluating and discussing trainee performance.”  Additional areas identified by the observers include buy in (“faculty should recognize that they bought into training these residents/fellows, and the overall agenda of the group should be to make the learners successful, to the maximum extent possible”); having an open mind (“members should come to the meeting with no preconceived notions or ‘labels’ for specific trainees”); and advocacy for the trainees (“the group felt that they were advocates for the trainees, and the group setting allowed them to identify mentors for individual trainees who may be struggling (JS)”).  We did not observe any specific comments related to patient advocacy and if any tension may result from advocacy for patients versus trainees.  

Interpersonal Interactions

Imported Information

Information brought to the meeting by individual faculty members is important to helping build a shared mental model.  Two main themes that emerged from the observations include information obtained from faculty pre-work assignments and individual experiences with trainees.  For CCC decisions, the latter seemed to be the most frequently relied upon source of information.  Especially for smaller programs, CCCs frequently used specific individual stories and/or experiences with trainees as a means of evaluation.  During the general discussion of each trainee, CCC members would most frequently offer specific experiential anecdotes to illustrate their opinions, both to offer positive and negative evidence regarding trainee performance. 

Shared Mental Model

During our observations we noted three general themes whereby CCCs build a shared mental model: assignment of the Milestones, perceptions regarding the developmental level and overall clinical competence of various trainees, and the work and processes of the CCC.  When building a shared mental model, CCCs discussed a number of different topics to include trainee global strengths/weaknesses (e.g. praises of bedside manner, maturity and concerns of professionalism, and personality issues), instances in which feedback was given to trainees (and whether they did/did not respond to it), and general discussions of what the CCC members could do and actions they could take in the future in order to aid the trainees and foster additional growth.  Indeed, when discussing solutions for questions and/or concerns about certain trainees, CCC members either described specific instances in which they attempted to correct behavior one-on-one or discussed team actions in abstract terms (e.g., “we have to somehow make it clear to [the trainee]…” or “we might need to consider program-level remediation…”).  A sample of other topics discussed are listed in Appendix 1.

When there was quick group consensus regarding the Milestones, CCC chairs would generally move the discussion forward to the next trainee.  In instances when a member disagreed with any of the proposed numbers, he or she would usually either implicitly ask the CCC member who offered the number why (“really?”) or offer a contradictory opinion (“I’d say [the trainee] is still probably at a x…”), at which point CCC members would offer specific examples, explanations, or evaluations that explain their points.  Additionally, there was an occasional voiced reluctance to offer “perfect” scores, as the committee generally agreed that trainees may never achieve some aspirational competencies.

Conversations seemed to be key to the process of discussing trainee competence and building a shared mental model.  Members used the analogy of a puzzle in which each faculty member had pieces of data about experience with trainees, and together through discussion and information sharing a more complete picture of trainee competence emerged during the meeting.  It was generally felt that members should share concrete, specific examples that can be interpreted by the group in a relevant context.  Individual participation is important for this to occur, and discussion amongst the faculty members led to additional insights and comments about each trainee.  The discussion uncovered information not written on global or other evaluation forms.  One faculty member noted that written evaluations are most useful for the highest and lowest performing trainees.  He stated that “97% of the time my evaluation is the same for trainees in between those two extremes and in the middle of the competency spectrum.”  He also felt that Likert scales are not useful and that group discussion is a much more effective way to share information about group members’ experiences with individual trainees.  Some members reported feeling that it was easier to identify troubling trends with trainees based upon conversations with faculty members who have worked with these individuals.  Oftentimes they said that it is hard to write negative comments on an evaluation form, especially if that form goes back to the trainee.  As a result, these “neutral” evaluations may not identify trends in a way that conversation does.  For example, if more concerning faculty observations arise with trainees and there has been no record of borderline performance or other concerns, it may be harder to justify things like program level remediation, etc.

To facilitate a shared mental model, some CCCs had additional resources available to the group.  In most meetings, the group had access to a copy of the Milestones.  In others the CCC chair had complete evaluation packets for each faculty member and asked the group to read the entire packet prior to discussing trainees.  The latter led to notable periods of silence as committee members read the evaluation summaries, as well as specific Milestone criteria. 

Standing in a Group

In all of the CCCs, participation in the meeting did not seem to be influenced by military rank, seniority, specialty, or whether one was a voting member of the group.  In two CCCs chief residents attended as non-voting members of the committee, and they freely provided input from a near peer perspective about their experiences with most of the residents.  These individuals had unique insight into most of the trainees that other faculty members may not be in a position to observe, and their input on the CCC provided information the committees considered very valuable.  One CCC also solicited input from two program administrators, and their comments were valuable regarding trainee accomplishment of administrative tasks. 

Environmental Factors   

Task Complexity/Uncertainty  

We identified the following themes under this heading: specific role of the CCC (e.g. two members in one meeting actively debated what regulative power the committee had when a trainee may need additional supervision); faculty member roles (e.g. should the members be more of a coach/mentor versus an evaluator or should trainees be assigned to faculty members based upon aligned interests?); developing remediation plans; lack of faculty member experience with certain trainees; and the Milestones themselves.  In regards to the Milestones, members at times expressed frustration with the Milestone definitions, especially those with lengthier anchors and what the milestones don’t capture (e.g., burnout).  In several instances groups noted the difficulty in determining numerical milestone assignments given how the Milestones were written/phrased and commented on the difficulty in fitting individuals into the specifically defined categories.  Some groups acknowledged the difficulty and/or worry in selecting only one demonstrative experience as a justification for a number assignment.  For programs with a larger number of trainees (i.e. greater than 15 individuals), information and evaluation gaps complicated the ability of the CCC to make informed assignments of some individual trainees’ Milestone levels.  Additionally, a large number of trainees coupled with the number of Milestones typically led to longer meetings, which seemed to truncate the discussion and the competency evaluation process compared to CCCs with fewer trainees.

Threats 

One of the biggest environmental threats identified was decision making fatigue.  While members of one CCC admitted that they come to the meeting knowing what needs to be accomplished, almost all acknowledged that they get more fatigued as the meeting continues and tend to make more brusque decisions.  Observations near the end of the meetings included members increasingly stepping out for short intermittent periods, multiple members checking their cell phones and fidgeting, group members openly asking how many trainees they had left to evaluate, and/or comments such as “I’m fading,” “homestretch for milestones!” and “last one!,” to CCC cheers.  Other threats identified during the observations included scheduling conflicts, technical difficulties, and institutional structures/policies (e.g. on several occasions one CCC considered not just how the trainee did/did not perform, but how the institutional structures may/may not be at fault).

Based upon a synthesis of the information uncovered in our previous scoping review, we proposed a model (Figure 1) utilizing situated cognition to help better understand trust within a group and the various factors that may influence trust in group decisions from individuals either within or outside of the group.  In our current study, we observed 15 hours of CCC meetings from multiple different GME training programs using a structured observation guide developed from a review of the literature and this proposed model.  We explored how members characterize trust in the setting of a CCC and observed how these committees make decisions about trainee clinical competence and Milestone-based performance in the context of previously identified group trust elements: individual and group level factors, environmental factors, and the interactions between all of these elements.  Our proposed model captured the themes that emerged at the observed CCC meetings.       

Our results suggest that diversity of opinion promotes a more complete collective understanding of competence for each trainee.  Varied faculty experiences provide a diversity of information available to the CCC, and interactions among faculty members help to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones.  Further, CCC chairs were instrumental in facilitating a shared mental model by communicating an agenda, determining a process to assign the Milestones, moving along the conversation, engaging individual faculty members as needed, etc.  These findings were consistent with our expectations based on the proposed model.

Additionally, CCC meetings generally had similar structure and agendas based upon the size of the training program.  Faculty in smaller training programs tended to have more familiarity with all of the trainees and discussed them in more detail compared to larger programs.  Larger programs also assigned faculty members to individual trainees, and it was their responsibility to complete standardized pre-work on their trainees prior to the meeting.  Groupthink was more of a risk for larger training programs, especially when faculty members did not know their individual trainees well and when assigning individual Milestones.

Finally, environmental factors we identified that can compromise trust were task complexity, uncertainty, and threats to the process of group decision making.  One of the most important threats was decision making fatigue.  Almost universally, increasing distractions and observed lapses of concentration seemed to occur the longer the meetings ran.

Our research was limited to CCC meeting observations and brief group question sessions at the end of the meetings.  While we were able to collect data about easily observable components of our proposed model such as group size, diversity, and communication patterns, we were unable to obtain data on individual faculty member thoughts and feelings.  As a result, we were unable to test all components of the model, such as an individual’s propensity to trust, vulnerability, relational identity orientation, need for affiliation, etc.  We also did not collect data about what types of assessments each CCC used and had access to.  Given that observers were present in the same room as CCCs, the results may have been influenced by the Hawthorne effect (i.e. CCC members knew they were being watched, which may have impacted their behavior and/or group discussions).  Observations were also limited to meetings occurring during one Milestone reporting cycle at the end of the academic year at one institution. 

Future research might gather more validity evidence and explore faculty member thoughts and feelings regarding trust in a group setting.  We also suggest additional research looking at trust in group decisions from a non-CCC member stakeholder perspective.  This might involve examining how department leaders, program directors, and individual faculty members who are not on a CCC define trust in this context.

Developing a deeper understanding of our trust in CCCs may help these committees implement a more effective and meaningful process to make collective decisions.  Our model attempts to provide leaders within GME training programs a way to view and attend to the various factors that may impact trust.  The model might also serve as a way to “diagnose” why a CCC goes awry by looking at the CCC discussions through various components of the model (e.g. if environmental or leadership factors may be perceived as “ineffective,” what can be done to change or correct this?).

This observational study should allow program leaders and educators to better understand what factors may contribute to trust in group settings (e.g. CCCs).  As programs develop policies, procedures, and determine the membership of these committees, utilizing our findings may help improve the design and execution of groups like CCCs, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions.