The Accreditation Council for Graduate Medical Education (ACGME) requires that pediatric residents achieve competency in 13 procedures (Supplemental File 1), (Accreditation Council for Graduate Medical Education, 2017). Discussions about which procedures remain within the scope of pediatric residency training and general pediatric practice are ongoing (Ben-Isaac et al., 2013). It is unclear if pediatric residents are meeting this requirement. Individual residents’ procedural experience is influenced by program schedule, presence of fellows, hospital size/acuity, location, seasonal variations in patient illness (Hicks et al., 2000; Chaytors, Szafran and Crutcher, 2001; Goertzen, 2006; Lee et al., 2010; Schumacher, Frintner and Cull, 2016; Schumacher, Frintner and Cull, 2018), and clinical practice changes (Moote et al., 2011). These effects may be exacerbated by dedicated procedure teams and increased use of advanced practice providers (Moote et al., 2011). Previous studies reported that pediatric residents failed to meet ACGME’s procedural requirements, but most were limited surveys of residents prior to graduation, and do not include pediatric residents’ assessment of procedural training adequacy (Gaies et al., 2007; Schumacher, Frintner and Cull, 2018).

The current study assessed procedural experience of pediatric residents and recent graduates, and examined their perceptions regarding adequacy of procedural training. We hypothesize that procedural experience will be influenced by post-graduate year of training, program size and career choice.

Study overview

The study followed a cross-sectional observational study design. Subjects included pediatric residents and recent graduates of ACGME-accredited programs between September 2017-July 2018. The primary outcome was to quantify the number of procedures performed by pediatric residents, while assessing impact of program size, location and career choice. Secondary outcomes include perceived sufficiency of procedural volume. 

Study subjects

Current 2^nd and 3^rd year pediatric residents in nine ACGME-accredited programs in the United States, and recent graduates, were invited to participate. Programs varied in size and location (Figure 1). 

Figure 1: Distribution of Selected Residency Programs 

Survey tool 

An electronic survey, developed using online software (Qualtrics, Provo, UT), was distributed via email. A questionnaire previously utilized to assess procedural experience in neonatology fellows (Sawyer et al., 2016) guided survey development. Questions on demographics, numbers of specific procedures performed, cumulative procedural performance, and use/perceived accuracy of procedure logs were included (Supplemental File 2). Feedback was solicited from pediatric faculty/trainees (three fellowship program directors, three faculty educators, one fellow, and three residents). Residents from one program completed pilot testing to optimize clarity and functionality of the 32-item tool prior to distribution. An email survey link was distributed to eligible subjects in July 2018, with a reminder one month later.  

Variables of Interest

Post-graduate year of training was classified as PGY2, PGY3 or resident graduate. Program size was either small (<15 residents/class), medium (16-25 residents/class), or large (>25 residents/class). Career choice was either fellowship, hospitalist or primary care.  Procedural proficiency was defined as >5 performed procedures (DeMeo et al., 2015).

Statistical Analysis

We aimed at reaching at least 30% of 622 eligible residents, to get a representative number of participants. Each of our variables of interest had 3 levels, therefore, with 30-50 subjects per level, we were powered at 80% with two-sided alpha of 0.05 to find effect sizes in the range of 0.3-1.0, using a two-sided Mann-Whitney test. Descriptive statistics summarized continuous and categorical variables. Between-group comparisons in the number of procedures were performed using Student’s t-test (data summarized as means with standard deviations) and Wilcoxon Rank-sum test (data summarized as medians with range). Since the primary study focus was to provide associative information for future studies, we did not perform adjusted comparisons and did not adjust for multiple comparisons. Statistical significance was established at alpha=0.05. Analyses were performed using SAS 9.4 (Cary, NC). Yale’s Institutional Review Board approved this study.

Demographics

Study response rate was 28% (n=173/622), including recent graduates (28%, 48/173), and residents in the 3^rd (32%, 55/173) and 2^nd (36%, 63/173) years of training. Subject demographics are presented in Table 1. Career plans varied, with many (58%, 95/173) planning fellowship.  Others contemplated private practice (23%, 38/173) or pediatric hospitalist positions (7%), with 12% uncertain. Of those pursuing fellowship, 48% (46/96) intended to focus in a critical care specialty.

Use of Procedural Logs

Procedure log use was reported by 98% (169/173). Only 44% (76/173) reported their documentation as “accurate”/ “very accurate”. Twelve percent (21/173) only reported successful attempts.  

Competency Assessment 

Existing programmatic requirements for achievement of “competency” were reported by 67% (116/173). Ten percent (17/173) reported a standardized process for determining procedural competency prior to performing a procedure independently. The most cited practice was completing >5 supervised procedures, which was reported by 47% (8/17).   

Procedure Experience

Senior residents had significantly increased procedural experience generally (p<0.001) except for bladder catheterization and administering immunizations (Table 2). Individual procedural experiences varied significantly (p<0.001). Only 6% (11/173) of respondents had successfully completed at least one of each of the ACGME required procedures (Table 2). Of recent graduates, 81% (39/48) had not performed at least one each of the required procedures during residency. 

Simple laceration repair (average 8/resident) and bag mask ventilation (BVM) (average 7.5/resident) were performed most commonly. Bladder catheterizations (average 0.7/resident; 113/173 never performed) and reduction of dislocation (average 1.2/resident; 70/173 never performed) were infrequently performed. Potentially life-saving procedures were not universally encountered in training. Intubation, BVM, and peripheral intravenous insertion (PIV) were not completed by 59 (35.5%), 18 (10.8%), and 79 (47.6%) of respondents, respectively.  

Most procedural opportunities occurred in the emergency department (83%, 123/173), with the fewest opportunities reported in PICU (33%, 49/173).

Program Size

Significant differences in procedural experience were noted in programs of varying size. Residents in large programs performed fewer procedures than those in medium and small programs(p=0.04). Residents in small programs performed more bladder catheterizations (p=0.004) and neonatal endotracheal intubations (p=0.026) (Table 3). Medium and small programs had greater exposure to BVM, lumbar puncture, and umbilical catheter placement. 

Career Choice

Anticipated career path did not impact overall procedure experience (p=0.50) (Table 4). Significant differences were noted amongst specific procedures. Residents intending to enter primary care practice reported performing more simple laceration repairs(p=0.05) and splinting(p=0.019). Future hospitalists reported higher numbers of lumbar punctures (p=0.012) and umbilical catheter insertions (p=0.005). Residents planning to pursue critical care or EM fellowships (p=0.17) reported more venipunctures (p=0.006) and PIV insertions (p=0.004). No difference was noted in numbers of other life-saving procedures, such as intubation and BMV, based on anticipated career path.

Resident perception of procedure experience 

Most residents did not feel they had sufficient procedural experience for their level of training (62%, 103/173), and 90% (149/173) desired more experience, most commonly for PIV (39/149, 26%) and intubation (59/149, 40%). Insufficient procedural experience was reported by 54% (26/48) of graduates with 79% (38/48) desiring additional procedural training, most commonly for intubation (45%, 20/44) and PIV (32%, 14/44).

Suggestions to improve exposure were procedural electives (143/173, 83%) and simulation-based training (87/173, 50%) (Table 1). 

This cross-sectional cohort study investigated procedural training experience amongst a diverse group of pediatric residents and graduates of ACGME-accredited training programs, and affirmed concerns that present-day pediatric residents have fewer opportunities to attain competency in ACGME-required procedures (Gaies et al., 2007; Schumacher, Frintner and Cull, 2018). Although it was anticipated that future career choice and geographic location may influence procedural exposure, only program size influenced overall procedural numbers. A majority of residents deemed their current procedural training as insufficient, and desired supplemental training in at least one procedure.  

Residents reported increasing procedural experience throughout training. Deficiencies exist with regards to specific procedures, as noted by cumulative numbers performed and residents’ reported self-perception of inadequate procedural experience. This is a significant decline from previous eras (Leone, Rich and Finer, 2005; Gabrani et al., 2018) with numerous contributing factors, including changes in medical management (Finer et al., 2004; Wyckoff et al., 2015), staffing (Moote et al., 2011; Smith and Hall, 2011), supervision (Gozzo et al., 2011), and workflow (Accreditation Council for Graduate Medical Education, 2017). Previous authors have reported decreased volume for specific procedures (i.e., intubation), but this study demonstrates a decline across all procedures (Gozzo et al., 2011; Engorn et al., 2018). Residents are now infrequently performing procedures on the floor and intensive care units, which may be due to presence of subspecialty fellows or other staff who need to develop proficiency. 

Similar to previous reports, (Schumacher, Frintner and Cull, 2018) program size affected procedural volume. Smaller programs may afford opportunities for procedures given fewer additional providers (fellows, practitioners) or absence of competing centers regionally. However, it is unclear if these differences were clinically meaningful. 

Different from previous descriptions of procedural experience amongst pediatric residents entering primary care positions (Walton and Edwards, 2002; Ben-Isaac et al., 2013; Schumacher, Frintner and Cull, 2018), we assessed for differences in residents entering subspecialties that require technical proficiency. Despite assumptions that trainees would advocate for additional procedural opportunities, their cumulative procedure numbers did not differ, suggesting influence of other factors (competition for procedures by other providers, limited procedural numbers overall, and work-hours restrictions). These residents must rely on fellowship training to master skills, compounding concerns regarding declining procedural opportunities for subspecialty fellows (Gozzo et al., 2011; Mittiga et al., 2013; Engorn et al., 2018).

In our study, residents reported procedural numbers by referencing their procedure log. Although most residents are required to use procedure logs, concern exists regarding accuracy of the reports, and inconsistency about what should be included (i.e., recording unsuccessful attempts). Currently, the ACGME does not require that pediatric residents maintain a procedure log as it does for surgical residents, where concerns about compliance and accuracy have been raised (Accreditation Council for Graduate Medical Education, 2020; Wolf et al., 2017). It would be beneficial for the ACGME to monitor pediatric residents’ procedural experience across programs to inform expectations on feasibility of current procedural training requirements, potential modifications necessary, or ability to supplement procedural training with simulation. Local/ national data collection would allow PDs to identify trainees who are falling behind their peers with regards to procedural volume/performance using training specific metrics or a dashboard (Accreditation Council for Graduate Medical Education, 2017).

A major challenge of assessing resident procedural performance is that valid, standardized methods to define and assess procedural competency have not been established. Of programs having a defined system, most used an arbitrary numeric threshold for procedures completed under direct observation which the majority of residents are not reaching. This method does not evaluate cognitive procedural aspects or performance of discrete psychomotor steps, both of which contribute significantly to procedural success and safety. Further research is needed to define competency in all ACGME-required procedures, similar to previous work on neonatal intubation (Leone, Rich and Finer, 2005; DeMeo et al., 2015). In a retrospective single center observational study, using statistical modeling to estimate outcomes of resident intubations, pediatric residents needed to perform >4 successful intubations to be competent (DeMeo et al., 2015). In our study, most residents did not perform 4 intubations during residency. Standardized expectations for procedural training should exist across programs to protect patient safety, and include assessment of procedural sequence/technique based upon assessment tools with documented evidence of validity. Identification of minimum procedural thresholds for graduation might lead to procedural goal achievement. In addition, if these numbers are unattainable clinically, simulation or other modalities may serve as substitutes. 

As it is unlikely that the majority of pediatric residents are meeting ACGME’s existing expectations for procedural training, it may be time to re-evaluate what procedures are necessary for pediatric residents (Gilhooly, Redden and Leonard, 2015). Previous research has focused on skills required by primary care physicians, and was limited to graduates from a single pediatric residency program (Ben-Isaac et al., 2013) or general pediatricians in a single state (Iyer et al., 2019). Recently, general pediatricians noted 11 of the ACGME-recommended procedures were never performed in practice, yet were perceived to be important (Iyer et al., 2019). National surveys of primary care pediatricians and subspecialty PDs could inform discussions on procedures necessary for current practice, including which should be mastered during residency. Individual residents may benefit from specialized pathways based on career goals, with differing procedural expectations. 

Novel strategies are needed to supplement procedural training for pediatric residents if the current 13 procedures remain “essential,” presenting a challenge for pediatric educators. Residents desire alternative procedural training methods, especially simulation, where trainee competency has been demonstrated in numerous procedures (Augustine and Kahana, 2012; Barsuk et al., 2012; McMillan et al., 2016). Programs should consider implementing structured experiences combining simulation and clinical attempts using an established framework (Sawyer et al., 2015). Other strategies include electives which may improve exposure to particular procedures (ex., intubation during ENT/anesthesia rotation). 

Our study has some limitations. It is unclear if residents utilized procedural logs to answer survey questions, which may lead to recall and response bias. Logs may contain inaccurate information. Due to the cross-sectional nature of the study, we were unable to ascertain individual subject’s procedural volume over years of training. We surveyed a selection of institutions across a spectrum of size and geographical locations, but study enrollment was below target. Therefore, findings may not be generalizable to the population of residents as a whole.

In a cross-sectional sample of residents from programs of varying sizes and locations, we demonstrated low procedural volumes that likely challenge program’s ability to effectively establish trainee procedural competency during training. ACGME may need to reassess what procedures are necessary in current pediatric practice and how to feasibly establish competency within training constraints, including supplementation with simulation-based training and development of individual career-based learning pathways.