A needs assessment survey for clinical competency committee (CCC) faculty development

Background. The Accreditation Council for Graduate Medical Education (ACGME) requires training programs to have a clinical competency committee (CCC) that regularly reviews each house officer’s performance. Objective. To identify areas for CCC faculty development via a survey of CCC faculty. Methods. An anonymous online survey was sent to CCC faculty of ACGME-accredited residency and fellowship programs at our large academic institution. Results. Response rate was 42% (177 of 417 identified CCC members). Respondents were from residency (51%) and fellowship (49%) programs. Fifty-nine percent (104 of 177) reported that more time during CCC meetings was spent discussing low performers whereas 41% (73 of 177) responded that equal time was spent discussing high and low performers. Fifty-four percent of respondents (95 of 177) reported being unaware of CCC resources available through the ACGME and 59% (104 of 177) indicated that faculty development sessions on CCC resources and best practices would be helpful. Seventy-two percent of respondents (128 of 177) thought that incorporation of a CCC had a positive impact on house officer assessment. Conclusions. A survey of CCC members at our institution indicates that faculty believe CCC review is a valuable component of house officer assessment. Further faculty development is needed to increase familiarity with available CCC resources and best practices.


Introduction
In 1998 the Accreditation Council for Graduate Medical Education (ACGME) began to shift from process-based accreditation toward an outcome-based system. Clinical training program accreditation requires evaluation of house Shelgikar A, Gelb D, McDermott M, Hathaway R, Biermann J, London Z MedEdPublish https://doi.org/10.15694/mep.2019.000079.1 Page | 2 officer competence in six core domains (patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice). (Swing, 2007) In 2012, Nasca and colleagues proposed the Next Accreditation System, which includes milestone-based assessment of each house officer as determined by a clinical competency committee (CCC). (Nasca et al., 2012) Each ACGMEaccredited program must have a CCC that meets at least semi-annually to perform a milestone-based assessment of each house officer.(Accreditaton Council for Graduate Medical Education, 2017b) CCCs are composed of program faculty who use multiple assessment tools to assign competency-based milestone ratings to each house officer.(Accreditaton Council for Graduate Medical Education, 2017a) CCCs are given considerable latitude to decide the weight given to each assessment tool, the definition of competence, and the procedures to follow in ensuring competence and remediating deficiencies.
A first step in evaluating the effectiveness of CCCs is to determine how much variability exists among CCCs with respect to how they have approached these decisions and how much variability exists among CCC faculty members with respect to how they perceive their CCC responsibilities. The literature offers some guidance to CCC faculty about their roles, (French, Dannefer and Colbert, 2014) and one cardiovascular fellowship program director has described how his faculty spent substantial time on CCC meeting preparation and attendance, (Hong, 2015) but a broader survey of CCC members from a variety of medical specialties is needed. Prior work has suggested that CCCs tend to use a problem identification approach rather than a developmental approach to performance reviews. (Hauer et al., 2015) Further study is required to help identify the specific types of tools and data inputs used during CCC reviews.
The goals of this study were to (1) conduct a single-site survey of CCC members to understand responsibilities involved with CCC service, (2) review the number and type of assessment tools used by CCCs across a broad range of medical specialties, and (3) assess the need for ongoing faculty development for CCC members to better understand CCC roles, responsibilities, and activities.

Survey.
We developed a self-administered survey to collect information from CCC faculty at the University of Michigan, a large academic medical center, about their CCC experience. Because no preexisting survey specific to CCC faculty perspectives was available, this survey was created with input from program directors (AS and ZL), clinical faculty (DG and MM), and our designated institutional official (JB). The survey included a combination of multiple choice questions and free-text answers. Responses were collected anonymously. The survey instrument is provided as supplemental content.

Data Collection and
Analysis. An online inquiry form was sent to all program directors and program coordinators (n=106) of the ACGME-accredited graduate medical education programs at Michigan Medicine, asking for their program name, program type (core residency vs. fellowship) and number of CCC members. One week later, a link to the anonymous online survey instrument was sent to the same group of program directors and program coordinators with a request to share this link with their respective CCC members. Two mailings of the survey link were sent over two weeks to maximize response. Data were stored on a password-protected network drive and in the secure survey account. Descriptive analyses were performed to understand respondents' characteristics and their responses to the survey items.  (Figure 1). Evaluation forms with ACGME milestone terminology and faculty evaluations of house officers were the most commonly used data for CCC review. respondents (72 of 177) reported spending equal CCC discussion time on high and low performing house officers, while 59% (105 of 177) reported that more CCC discussion time was devoted to discussion of house officers with low performance. With respect to the amount of CCC meeting time that was spent discussing house officer remediation, 75% of respondents (132 of 177) indicated <25% of CCC meeting time, 19% (33 of 177) indicated 25-50% of CCC meeting time, and 3% (6 of 177) indicated > 50% of CCC meeting time; the remaining 3% (6 of 177) of respondents did not reply to this question.
The majority of CCC members (76%, 135 of 177) reported they did not have protected time for their CCC involvement. Program directors, including assistant and associate program directors, were more likely to receive protected time for CCC responsibilities (41%, 18 of 44) compared to non-program director CCC members (19%, 25 of 133).

Impressions of CCC review.
Overall, CCC members reported a favorable impression of how CCCs have affected the review of house officer performance (Figure 2). Fifty-four percent of respondents (95 of 177) reported being unaware of CCC resources available through the ACGME and 59% (104 of 177) indicated that faculty development sessions on CCC resources and best practices would be helpful.

Discussion
Our survey of CCC members at our large academic medical center showed an overall favorable impression of CCC activities. Respondents identified a multitude of assessment tools (for example, chart review, simulation center performance, and evaluation forms) used to inform CCC assignment of competency-based milestone ratings to house officers. Future surveys of CCC members should include more detailed inquiry on how CCCs select assessment tools, how often these tools are used, and how the resultant data are incorporated into CCC review of house officer performance. Most respondents found that incorporation of CCCs was a valuable component of house officer assessment, although some found that CCC review made the overall process more time-consuming and complicated. Identification and dissemination of best practices may facilitate more streamlined workflow for CCC members, particularly those who do not receive protected time for CCC responsibilities.
This survey identified three areas of need. First, the majority of respondents were not aware of ACGME resources available for CCCs. Second, many CCC members would like more faculty development sessions related to CCC activities. The third need is for protected time for CCC members to accomplish their CCC responsibilities. We have shared these results during a program directors' meeting at our institution as part of our effort to increase awareness of national and local resources available for CCCs. At this meeting, program directors shared their experiences and challenges related to CCC service. Our institution is now working to share clinical assessment tools across programs. Program directors and other CCC members are encouraged to engage with medical education groups within their specialties to advance their understanding of CCC resources and best practices.
Our program response rate of 62% is similar to other needs assessment surveys in medical education. (Curran et al., 2012) The survey response rate of 42% was lower than the program response. Despite this limitations, we were able to collect information from CCC members to inform our needs assessment and guide next steps for faculty development. A potential opportunity identified through this work is the creation of a list of CCC members across programs, perhaps even across specialties and/or institutions, to facilitate future communication and research related to CCC responsibilities. CCCs could work to develop procedures to maintain the benefits indicated in our results (e.g., process standardization and thoroughness of reviews) that would also mitigate the complexity in using the Next Accreditation System.

Conclusion
This survey of CCC members at one institution showed that CCCs are perceived to be helpful in assessing house officer performance. Ongoing faculty development is required to enhance awareness of CCC-related resources. CCC members desire protected time to fulfill their responsibilities in this often unfunded mandate for house officer assessment and program accreditation. Opportunities for future study include efforts to facilitate communication and share best practices among institutional, regional, and national CCCs.

Take Home Messages
1. Faculty find implementation of clinical competency committees (CCCs) to be valuable in the assessment of house officer performance.