Research article
Open Access

Cut Down, Continue, Commence: Resident and Attending Perceptions of a Standardized, Face-to-Face Feedback System

Emily Liang[1], Surbhi Singhal[1], Jingkun Yang[1], Andre Kumar[1]

Institution: 1. Stanford University
Corresponding Author: Dr Andre Kumar ([email protected])
Categories: Postgraduate (including Speciality Training)
Published Date: 11/01/2021

Abstract

Introduction: Standardized feedback systems, such as “Stop, Start, and Continue” can improve written feedback quality, but their effect on verbal feedback is not well known.

 

Objective: To evaluate resident and attending perceptions of feedback before and after implementation of a standardized feedback model for inpatient rotations.

 

Methods: N=65 internal medicine residents and N=16 attendings rotating on a general wards teaching service between 9/2018-12/2018 at a single academic center participated in this study. We evaluated the implementation of a standardized feedback system called the 3 C’s (“Continue, Cut-Down, Commence”). Participants were sent surveys regarding feedback pre- and post- intervention. Survey items included perceptions of feedback frequency, quality, and duration.

 

Results: Survey response rates were 52.3% for pre-intervention/post-intervention residents, 93.8% for pre-intervention attendings, and 81.3% for post-intervention attendings. Only 36% of attendings reported using a formalized system for in-person feedback before the intervention, while 76% of attendings reported using the 3 C’s method after the intervention. Residents reported higher satisfaction with verbal feedback (χ2=4.3, df=1, p=0.03) and feedback quality related to medical knowledge (χ2=4.31, df=1, p=0.037) following the intervention. Residents reported participating in verbal feedback less often than attendings in both the pre-intervention (p=0.003) and post-intervention periods (p<0.001). Attendings did not report a perceived improvement in feedback frequency or quality following the intervention.

 

Discussion: Standardized feedback systems may improve resident satisfaction with verbal feedback without significantly impacting attending perceptions of feedback quality. Even after the implementation of standardized feedback, residents may perceive they participate in verbal feedback less frequently than attendings.

 

Keywords: Feedback; Graduate Medical Education; Residents; Attendings; Quality

Introduction

Feedback is critical for effective learning and professional development (Ende, 1983; Ericsson, 2008). Consistent and systematic feedback improves clinical performance across multiple learning environments (Veloski et al., 2006). In recognition of the importance of feedback, the Accreditation Council for Graduate Medical Education (ACGME) requires formative feedback in addition summative evaluations for residency training (Common Program Requirements, no date). Unfortunately, formative feedback may be underutilized in graduate medical education (GME; Sender Liberman et al., 2005; Yarris et al., 2009; Hajar, Wanat and Fett, 2020).

Previous approaches to address formative feedback during residency training have utilized the implementation of standardized feedback systems (Peccoralo et al., 2012; Wilkinson et al., 2013). These systems can include faculty development (Wilkinson et al., 2013), conversation guides (Peccoralo et al., 2012), or standardized templates (Peccoralo et al., 2012; Hoon et al., 2015; Albano et al., 2019). There is growing evidence that these methods may improve trainee performance, but their implementation is not well studied (Veloski et al., 2006). In this brief report, we discuss the implementation of a standardized feedback system called the “3 C’s: Cut-Down, Continue, and Commence” and compare its effectiveness to our pre-existing system.

Methods

Study Participants & Setting
This was a prospective cohort study that evaluated internal medicine residents’ and attendings’ perceptions of feedback at a single academic medical center. Residents and attending hospitalists on our inpatient wards teaching service between 9/2018-12/2018 were included. Our Institutional Review Board reviewed and approved the study.
 
Prior to the intervention, our institution did not utilize a standardized face-to-face feedback system. Resident and attending evaluations were completed online (www.medhub.com) at the end of the four-week rotation. It was at the attendings’ discretion to provide additional face-to-face feedback.
 
Resident Feedback System
This study evaluated the implementation of an standardized feedback system called “Stop, Start, and Continue,”(Hoon et al., 2015). This model has been shown to improve the depth and quality of written feedback, but it has been less studied for face-to-face feedback (Hoon et al., 2015). We adapted this model into the “3 C’s” system for resident feedback: “Cut-Down, Commence, and Continue” (Figure 1). The term “stop” was adjusted to “cut-down” as this follows the best-practices of feedback by making it non-threatening and more actionable (Schwartz, Tsang and Blair, 2016). To make the system easier to memorize, we used the letter “C” to describe the other feedback components.

At the beginning of the intervention, all attendings received an in-person orientation to this feedback system. Due to scheduling constraints, the residents received an orientation email as opposed to in-person orientation. Both groups were given pocket feedback cards to serve as a reminder (Figure 1) and were sent weekly email reminders to use this standardized system.

Figure 1. The 3 C’s of Feedback.

Figure Footnote: This study utilized an abbreviated feedback system akin to the “Stop, Start and Continue” method called the 3 C’s (above). Stop was modified to Cut-Down to allow the feedback to be more revisable and non-threatening.

Measurements
Both groups were sent electronic surveys pre- and post-intervention to assess their perceptions of feedback (Supplementary File 1). These surveys included perceptions of the frequency and quality of feedback as related to the ACGME core competencies. All post-intervention surveys were sent at the conclusion of the intervention.
 
Statistical Analysis
All surveys utilized a symmetric 5-point Likert scale. Categorical variables were dichotomized by Likert values of 4 & 5 vs. all others and were analyzed via Chi-Square Analysis. Continuous variables (e.g. frequency of feedback) were analyzed using Mann Whitney test. All analyses were performed using R (Vienna, Austria).

Results/Analysis

Baseline Characteristics
During the intervention period, 65 residents and 16 attendings rotated on the general wards teaching service. Survey response rates were 52.3% (n=34) for pre-intervention and post-intervention residents, 93.8% (n=15) for pre-intervention attendings, and 81.3% (n=13) for post-intervention attendings. Pre-intervention residents included the following post-graduate years (PGY): PGY-1 (33.3%), PGY-2 (15.2%), and PGY-3 (52.5%). Post-intervention residents included: PGY-1 (32.4%), PGY-2 (29.4%), PGY-3 (38.2%). There were no significant PGY differences between the groups (p=0.34).

The 3C’s System and Feedback Quality
Only 36% of pre-intervention attendings reported using a formalized system for in-person feedback. The attendings reported using the 3C’s method a median 76% [IQR: 42-92%] of the time during the intervention. The proportion of residents who reported feeling “satisfied” or “very satisfied” with the face-to-face feedback they received significantly increased from pre-intervention (56%) to post-intervention (79%; χ2=4.3, df=1, p=0.03). In contrast, attending satisfaction with providing face-to-face feedback did not significantly change (pre-intervention: 57%; post-intervention: 62%; χ2=0.05, df=1, p=0.82).

Despite feeling more satisfied with face-to-face feedback, the post-intervention residents did not report higher quality of positive feedback (χ2=0.44, df=1, p=0.51) or constructive feedback (χ2=1.06, df=1, p=0.30). Post-intervention residents reported significantly improved quality of feedback related to the ACGME Milestone for Medical Knowledge (χ2=4.31; df=1; p=0.037). Post-intervention resident perceptions of feedback quality did not improve for the other ACGME milestones (Table 1). Attendings did not feel they gave higher-quality feedback across any sub-domains following the intervention period (Table 1). Interestingly, only 21% of residents preferred the 3 C’s method (46% unsure), while 31% of attendings preferred the 3 C’s (62% unsure) to their usual method of feedback.

Table 1: Survey Results

Attending

Pre-

Intervention

Post-

Intervention

χ2

p=

  Satisfaction with Verbal Feedback

57.1%

61.5%

0.05

0.82

  Quality of Positive Feedback

85.7%

69.2%

1.06

0.30

  Quality of Constructive Feedback

50.0%

53.8%

0.04

0.84

  Feedback: Direct Patient Care

73.3%

46.2%

0.90

0.34

  Feedback: Medical Knowledge

53.3%

41.6%

0.18

0.67

  Feedback: Professionalism

80.0%

92.3%

1.01

0.32

  Feedback: Communication

66.6%

66.6%

0.52

0.47

  Feedback: Systems Based Practice

60.0%

53.8%

1.18

0.28

  Feedback: Practiced Based Learning

60.0%

41.6%

 0.62

0.43

Residents

       

  Satisfaction with Verbal Feedback

56.6%

79.4%

4.30

0.03*

  Quality of Positive Feedback

75.7%

82.4%

0.44

0.51

  Quality of Constructive Feedback

42.5%

47.1%

1.06

0.30

  Feedback: Direct Patient Care

50.0%

64.7%

1.20

0.27

  Feedback: Medical Knowledge

44.2%

67.6%

4.31

0.04*

  Feedback: Professionalism

73.5%

76.5%

0.12

0.72

  Feedback: Communication

73.5%

61.8%

0.91

0.34

  Feedback: Systems Based Practice

31.2%

34.4%

0.03

0.86

  Feedback: Practiced Based Learning

33.4%

37.5%

0.28

0.60

Footnote: The surveys were based on 5-point Likert scales. Likert values of 4 & 5 on the scale were combined and compared against other values. The percentages shown above represent the proportion of respondents who selected 4 or 5 on the survey scales. Chi-Square (χ2) analysis was performed to compare the dichotomized variables (Likert values 4+5 vs. all others). Note: * denotes p <0.05.

Perceptions of Feedback
Residents perceived they participated in face-to-face feedback sessions less often than attendings. Pre-intervention residents reported participating in verbal feedback a median 60% [IQR 50-80%] of the time, compared to 81% [IQR: 71-98%] of attendings (p=0.003). Post-intervention residents reported participating in verbal feedback a median 70% [IQR: 50-82] of the time, compared to 91%  [IQR: 75-100%] of attendings (p<0.001). The majority of respondents in both groups agreed the average face-to-face feedback session was 5-10 minutes in length before and after the intervention (pre-intervention residents: 54.5%, post-intervention residents: 50.0%, pre-intervention attendings: 71.4%, post-intervention attendings: 76.9%).

Discussion

In this study, we assessed a standardized feedback system called the 3 C’s (“Cut Down, Commence and Continue”) in terms of resident and attending perceptions of feedback quality, frequency and duration. Residents reported higher satisfaction with face-to-face feedback after this system was implemented, while attendings reported no difference. Despite this improvement, residents did not strongly prefer the 3 C’s method. Furthermore, residents and attendings did not report improved quality of feedback across several domains, including the ACGME milestones. Finally, residents and attendings differed significantly in their perception of how often feedback was given.
 
Previous authors have shown that higher trainee satisfaction with feedback may not relate to the actual quality of feedback (Boehler et al., 2006).  Additionally, there are well-reported discrepancies between resident and attending perceptions of feedback in terms of quality, frequency, or duration (Sender Liberman et al., 2005; Carr et al., 2018; Albano et al., 2019). As suggested by this study, the implementation of a standardized feedback system like the 3 C’s may not immediately improve these issues. More abbreviated feedback methods like “Stop, Start, and Continue” can improve written feedback quality, but it is unclear if they can improve verbal feedback quality (Hoon et al., 2015). Feedback should be specific, timely, unambiguous, non-threatening, and revisable (Schwartz, Tsang and Blair, 2016), but formal feedback systems may be employed at the end of the rotation where the feedback is no longer timely or immediately revisable. Our intervention encouraged frequent feedback, which may have improved resident satisfaction. However, its abbreviated nature may have made it challenging to improve feedback quality around more complex topics such as ACGME milestones.
 
Limitations of this study include its small sample size, potential selection bias from surveys, and lack of randomization. The post-intervention surveys were sent at the end of the intervention period, and the results may be subject to recall bias. Finally, trainee or attending perceptions of feedback quality are imperfect measures. Future studies should use standardized methodologies to evaluate feedback quality.

Conclusion

Standardized feedback systems like “Cut-Down, Commence, and Continue” may improve resident satisfaction with feedback without improving the perceived quality of feedback. Residents may have differing perceptions of feedback frequency compared to their attendings. Together, these results suggest potential caveats to more abbreviated feedback systems for verbal feedback and add to the growing literature on the nature of feedback in GME.

Take Home Messages

  • Residents and attending physicians may have differing perceptions of feedback frequency

  • Standardized verbal feedback systems like may improve resident satisfaction with feedback, but the overall quality of feedback may not improve.

Notes On Contributors

Dr. Emily Liang MD, is a third year medical resident at the Stanford University Internal Medicine Residency Program. She is interested in improving feedback for trainees on clinical rotations.

Dr. Surbhi Singhal MD, is a chief medical resident at the Stanford Unviersity Internal Medicine Residency Program. She is interested in graduate medical education and improving feedback among trainees.

Dr. Jingkun Yang MD, is a clinical assistant professor of medicine within the division of hospital medicine at Stanford University. She is interested in medical education interventional research.

Dr. Andre Kumar MD MEd, is a clinical assistant professor of medicine within the division of of hospital medicine at Stanford University and is the director of the hospitalist training program.

Acknowledgements

Figure 1 Source: Andre Kumar, MD, MEd.

Bibliography/References

Albano, S., Quadri, S. A., Farooqui, M., Arangua, C. et al. (2019) ‘Resident Perspective on Feedback and Barriers for Use as an Educational Tool’, Cureus, 11(5), p. E4633. https://doi.org/10.7759/cureus.4633

Boehler, M., Rogers, D., Schwind, C. J., Mayforth, R., et al. (2006) ‘An investigation of medical student reactions to feedback: a randomised controlled trial’, Medical Education, 40(8), pp. 746–749. https://doi.org/10.1111/j.1365-2929.2006.02503.x

Carr, B. M., O'Neil, A., Lohse, C., Heller, S., et al. (2018) ‘Bridging the gap to effective feedback in residency training: perceptions of trainees and teachers’, BMC Medical Education, 18(1), p. 225. https://doi.org/10.1186/s12909-018-1333-9

Accredidation Council for Graduate Medical Education. Common Program Requirements (no date). Available at: https://www.acgme.org/What-We-Do/Accreditation/Common-Program-Requirements (Accessed: 29 July 2020).

Ende, J. (1983) ‘Feedback in clinical medical education’, JAMA: the Journal of the American Medical Association, 250(6), pp. 777–781. https://doi.org/10.1001/jama.1983.03340060055026

Ericsson, K. A. (2008) ‘Deliberate practice and acquisition of expert performance: a general overview’, Academic Emergency Medicine, 15(11), pp. 988–994. https://doi.org/10.1111/j.1553-2712.2008.00227.x

Hajar, T., Wanat, K. A. and Fett, N. (2020) ‘Survey of resident physician and attending physician feedback perceptions: There is still work to be done’, Dermatology Online Journal, 25(12), pp 112-115.

Hoon, A., Oliver, E., Szpakowska, K., and Newton, P. (2015) ‘Use of the “Stop, Start, Continue” method is associated with the production of constructive qualitative feedback by students in higher education’, Assessment & Evaluation in Higher Education. Routledge, 40(5), pp. 755–767. https://doi.org/10.1080/02602938.2014.956282

Peccoralo, L., Karani, R., Coplit, L., and Korenstein, D. (2012) ‘Pocket card and dedicated feedback session to improve feedback to ward residents: a randomized trial’, Journal of Hospital Medicine, 7(1), pp. 35–40. https://doi.org/10.1002/jhm.934

Schwartz, D., Tsang, J. and Blair, K. (2016) The ABCs of How We Learn. New York, NY: W.W. Norton & Company.

Sender Liberman, A., Liberman, M., Steinert, Y., McLeod, P. et al. (2005) ‘Surgery residents and attending surgeons have different perceptions of feedback’, Medical Teacher, 27(5), pp. 470–472. https://doi.org/10.1080/0142590500129183

Veloski, J., Boex, J. R., Grasberger, M. J., McLeod, P., et al. (2006) ‘Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7’, Medical Teacher, 28(2), pp. 117–128. https://doi.org/10.1080/01421590600622665

Wilkinson, S. T., Couldry, R., Phillips, H., Buck, B., et al. (2013) ‘Preceptor development: providing effective feedback’, Hospital Pharmacy, 48(1), pp. 26–32. https://doi.org/10.1310/hpj4801-26

Yarris, L. M., Linden J. A., Hern, H., Lefebevre, J., et al. (2009) ‘Attending and resident satisfaction with feedback in the emergency department’, Academic Emergency Medicine, 16 Suppl 2, pp. S76–81. https://doi.org/10.1111/j.1553-2712.2009.00592.x

Appendices

None.

Declarations

There are no conflicts of interest.
This has been published under Creative Commons "CC BY-SA 4.0" (https://creativecommons.org/licenses/by-sa/4.0/)

Ethics Statement

The Stanford University Institutional Review Board (IRB) approved this expedited protocol (ID: 54278) on 12/15/2019.

External Funding

This article has not had any External Funding

Reviews

Please Login or Register an Account before submitting a Review

Anita Samuel - (26/05/2021) Panel Member Icon
/
This article presents a model for standardized verbal feedback in an internal medicine residency training program. The 3C’s model is memorable and could be a useful strategy to guide feedback sessions.
Below are some suggestions to strengthen this article:

- In the introduction you state that the report will discuss the “implementation” of the feedback tool. However, there is insufficient information on how the tool was implemented. For example, were attendings supposed to use this tool for all their verbal feedback sessions – formal and informal?
- Figure 1 is unclear. The items in gray – “CUT DOWN use of medical jargon” – is this an example or is this what Cut Down refers to?
- How long was the in-person orientation that the attendings received and what was covered in this session? Providing good feedback is complex. Were the attendings given tips on how to provide better feedback or just given an introduction to this model?
- If this 3C model has not improved quality of feedback, what are your next steps? Will you continue with this model? Why should another institution adopt this model?
Possible Conflict of Interest:

For transparency, I am an Editorial Board Member of MedEdPublish.

Haipeng Xiao - (19/03/2021)
/
Thank you for submitting this manuscript. The perception and quality of a standardized feedback system called 3 C's ("Cut Down, Continue, Commence") was evaluated among internal medicine residents and attendings in a single academic center in the US, in which both groups of participated reported different perceptions on the feedback system, and the quality of the feedback did not seem to improve. This is an interesting topic, worthy of further investigation. The authors used an electronic survey to gather data regarding the perception and quality of the new feedback system. Apart from the limitation mentioned by the authors themselves (including the possible inclusion of recall bias, etc.), it would be better to further explore, preferably with a qualitative approach, why the perceptions among residents and attendings differed, as well as why the quality of the feedback remained relatively unchanged. I believe these are more relevant research questions to be answered before the findings of this study could be rendered transferrable to other contexts.

Other minor issue includes: There are different use of the 3C's in terms of word sequence throughout the manuscript. It would be better to use a unified description.
Possible Conflict of Interest:

For transparency, I am an Editorial Board Member of MedEdPublish.

BALAJI ARUMUGAM - (12/01/2021) Panel Member Icon
/
Thanks for opportunity to review this research article.
First of all, congrats to the authors for publishing an article on this feedback system intervention among residents and the attendings simultaneously using likert scale questionnaire focusing on the aspects like Satisfaction with Verbal Feedback, Quality of Positive Feedback, Quality of Constructive Feedback, Feedback: Direct Patient Care, Feedback: Medical Knowledge, Feedback: Professionalism, Feedback: Communication, Feedback: Systems Based Practice, Feedback: Practiced Based Learning.
The results showed that the pre and post intervention among the attendings groups were not showing significant results statistically but there was improvement in all the aspects.
BUT THE SPECIAL MENTION ON
1. Standardized feedback systems, such as “Stop, Start, and Continue” can improve written feedback quality TO the implementation of a standardized feedback system called the 3 C’s (“Continue, Cut-Down, Commence”).
2. Cut down – use of medical jargons with patients
3. Continue – daily check in with all members of the team
4. Commence – putting a physical exam in your discharge summary

Note: The major concern or the limitation would be to train the residents and the attendings on feedback system. There was no clarity on that aspect and the same was not mentioned anywhere in the study.
Possible Conflict of Interest:

NONE