A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones: Clinical Competency Committee vs. Resident Self-Assessment

Ryan S. Watson, Andrew J. Borgert, Colette T. O'Heron, Kara J. Kallies, Richard A. Sidwell, John D. Mellinger, Amit R. Joshi, Joseph M Galante, Lowell W. Chambers, Jon B. Morris, Robert K. Josloff, Marc L. Melcher, George M. Fuhrman, Kyla P. Terhune, Lily Chang, Elizabeth M. Ferguson, Edward D. Auyang, Kevin R. Patel, Benjamin T. Jarman

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Objective: The Accreditation Council for Graduate Medical Education requires accredited residency programs to implement competency-based assessments of medical trainees based upon nationally established Milestones. Clinical competency committees (CCC) are required to prepare biannual reports using the Milestones and ensure reporting to the Accreditation Council for Graduate Medical Education. Previous research demonstrated a strong correlation between CCC and resident scores on the Milestones at 1 institution. We sought to evaluate a national sampling of general surgery residency programs and hypothesized that CCC and resident assessments are similar. Design: Details regarding the makeup and process of each CCC were obtained. Major disparities were defined as an absolute mean difference of ≥0.5 on the 4-point scale. A negative assessment disparity indicated that the residents evaluated themselves at a lower level than did the CCC. Statistical analysis included Wilcoxon rank sum and Sign tests. Setting: CCCs and categorical general surgery residents from 15 residency programs completed the Milestones document independently during the spring of 2016. Results: Overall, 334 residents were included; 44 (13%) and 43 (13%) residents scored themselves ≥0.5 points higher and lower than the CCC, respectively. Female residents scored themselves a mean of 0.08 points lower, and male residents scored themselves a mean of 0.03 points higher than the CCC. Median assessment differences for postgraduate year (PGY) 1-5 were 0.03 (range: -0.94 to 1.28), -0.11 (range: -1.22 to 1.22), -0.08 (range: -1.28 to 0.81), 0.02 (range: -0.91 to 1.00), and -0.19 (range: -1.16 to 0.50), respectively. Residents in university vs. independent programs had higher rates of negative assessment differences in medical knowledge (15% vs. 6%; P = 0.015), patient care (17% vs. 5%; P = 0.002), professionalism (23% vs. 14%; P = 0.013), and system-based practice (18% vs. 9%; P = 0.031) competencies. Major assessment disparities by sex or PGY were similar among individual competencies. Conclusions: Surgery residents in this national cohort demonstrated self-awareness when compared to assessments by their respective CCCs. This was independent of program type, sex, or level of training. PGY 5 residents, female residents, and those from university programs consistently rated themselves lower than the CCC, but these were not major disparities and the significance of this is unclear.

Original languageEnglish (US)
JournalJournal of Surgical Education
DOIs
StateAccepted/In press - 2017

Fingerprint

Graduate Medical Education
Clinical Competence
Accreditation
self-assessment
accreditation
graduate
resident
education
Internship and Residency
surgery
Self-Assessment
Nonparametric Statistics
Patient Care
university
self awareness
trainee
patient care
statistical analysis

Keywords

  • Clinical competency committee
  • Competencies
  • Domains
  • Interpersonal and Communication Skills
  • Milestones
  • Practice-Based Learning and Improvement
  • Professionalism
  • Resident evaluation
  • Systems-Based Practice

ASJC Scopus subject areas

  • Surgery
  • Education

Cite this

A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones : Clinical Competency Committee vs. Resident Self-Assessment. / Watson, Ryan S.; Borgert, Andrew J.; O'Heron, Colette T.; Kallies, Kara J.; Sidwell, Richard A.; Mellinger, John D.; Joshi, Amit R.; Galante, Joseph M; Chambers, Lowell W.; Morris, Jon B.; Josloff, Robert K.; Melcher, Marc L.; Fuhrman, George M.; Terhune, Kyla P.; Chang, Lily; Ferguson, Elizabeth M.; Auyang, Edward D.; Patel, Kevin R.; Jarman, Benjamin T.

In: Journal of Surgical Education, 2017.

Research output: Contribution to journalArticle

Watson, RS, Borgert, AJ, O'Heron, CT, Kallies, KJ, Sidwell, RA, Mellinger, JD, Joshi, AR, Galante, JM, Chambers, LW, Morris, JB, Josloff, RK, Melcher, ML, Fuhrman, GM, Terhune, KP, Chang, L, Ferguson, EM, Auyang, ED, Patel, KR & Jarman, BT 2017, 'A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones: Clinical Competency Committee vs. Resident Self-Assessment', Journal of Surgical Education. https://doi.org/10.1016/j.jsurg.2017.06.009
Watson, Ryan S. ; Borgert, Andrew J. ; O'Heron, Colette T. ; Kallies, Kara J. ; Sidwell, Richard A. ; Mellinger, John D. ; Joshi, Amit R. ; Galante, Joseph M ; Chambers, Lowell W. ; Morris, Jon B. ; Josloff, Robert K. ; Melcher, Marc L. ; Fuhrman, George M. ; Terhune, Kyla P. ; Chang, Lily ; Ferguson, Elizabeth M. ; Auyang, Edward D. ; Patel, Kevin R. ; Jarman, Benjamin T. / A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones : Clinical Competency Committee vs. Resident Self-Assessment. In: Journal of Surgical Education. 2017.
@article{0bd5a9bd82364123b6a50dc9e58d446c,
title = "A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones: Clinical Competency Committee vs. Resident Self-Assessment",
abstract = "Objective: The Accreditation Council for Graduate Medical Education requires accredited residency programs to implement competency-based assessments of medical trainees based upon nationally established Milestones. Clinical competency committees (CCC) are required to prepare biannual reports using the Milestones and ensure reporting to the Accreditation Council for Graduate Medical Education. Previous research demonstrated a strong correlation between CCC and resident scores on the Milestones at 1 institution. We sought to evaluate a national sampling of general surgery residency programs and hypothesized that CCC and resident assessments are similar. Design: Details regarding the makeup and process of each CCC were obtained. Major disparities were defined as an absolute mean difference of ≥0.5 on the 4-point scale. A negative assessment disparity indicated that the residents evaluated themselves at a lower level than did the CCC. Statistical analysis included Wilcoxon rank sum and Sign tests. Setting: CCCs and categorical general surgery residents from 15 residency programs completed the Milestones document independently during the spring of 2016. Results: Overall, 334 residents were included; 44 (13{\%}) and 43 (13{\%}) residents scored themselves ≥0.5 points higher and lower than the CCC, respectively. Female residents scored themselves a mean of 0.08 points lower, and male residents scored themselves a mean of 0.03 points higher than the CCC. Median assessment differences for postgraduate year (PGY) 1-5 were 0.03 (range: -0.94 to 1.28), -0.11 (range: -1.22 to 1.22), -0.08 (range: -1.28 to 0.81), 0.02 (range: -0.91 to 1.00), and -0.19 (range: -1.16 to 0.50), respectively. Residents in university vs. independent programs had higher rates of negative assessment differences in medical knowledge (15{\%} vs. 6{\%}; P = 0.015), patient care (17{\%} vs. 5{\%}; P = 0.002), professionalism (23{\%} vs. 14{\%}; P = 0.013), and system-based practice (18{\%} vs. 9{\%}; P = 0.031) competencies. Major assessment disparities by sex or PGY were similar among individual competencies. Conclusions: Surgery residents in this national cohort demonstrated self-awareness when compared to assessments by their respective CCCs. This was independent of program type, sex, or level of training. PGY 5 residents, female residents, and those from university programs consistently rated themselves lower than the CCC, but these were not major disparities and the significance of this is unclear.",
keywords = "Clinical competency committee, Competencies, Domains, Interpersonal and Communication Skills, Milestones, Practice-Based Learning and Improvement, Professionalism, Resident evaluation, Systems-Based Practice",
author = "Watson, {Ryan S.} and Borgert, {Andrew J.} and O'Heron, {Colette T.} and Kallies, {Kara J.} and Sidwell, {Richard A.} and Mellinger, {John D.} and Joshi, {Amit R.} and Galante, {Joseph M} and Chambers, {Lowell W.} and Morris, {Jon B.} and Josloff, {Robert K.} and Melcher, {Marc L.} and Fuhrman, {George M.} and Terhune, {Kyla P.} and Lily Chang and Ferguson, {Elizabeth M.} and Auyang, {Edward D.} and Patel, {Kevin R.} and Jarman, {Benjamin T.}",
year = "2017",
doi = "10.1016/j.jsurg.2017.06.009",
language = "English (US)",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - A Multicenter Prospective Comparison of the Accreditation Council for Graduate Medical Education Milestones

T2 - Clinical Competency Committee vs. Resident Self-Assessment

AU - Watson, Ryan S.

AU - Borgert, Andrew J.

AU - O'Heron, Colette T.

AU - Kallies, Kara J.

AU - Sidwell, Richard A.

AU - Mellinger, John D.

AU - Joshi, Amit R.

AU - Galante, Joseph M

AU - Chambers, Lowell W.

AU - Morris, Jon B.

AU - Josloff, Robert K.

AU - Melcher, Marc L.

AU - Fuhrman, George M.

AU - Terhune, Kyla P.

AU - Chang, Lily

AU - Ferguson, Elizabeth M.

AU - Auyang, Edward D.

AU - Patel, Kevin R.

AU - Jarman, Benjamin T.

PY - 2017

Y1 - 2017

N2 - Objective: The Accreditation Council for Graduate Medical Education requires accredited residency programs to implement competency-based assessments of medical trainees based upon nationally established Milestones. Clinical competency committees (CCC) are required to prepare biannual reports using the Milestones and ensure reporting to the Accreditation Council for Graduate Medical Education. Previous research demonstrated a strong correlation between CCC and resident scores on the Milestones at 1 institution. We sought to evaluate a national sampling of general surgery residency programs and hypothesized that CCC and resident assessments are similar. Design: Details regarding the makeup and process of each CCC were obtained. Major disparities were defined as an absolute mean difference of ≥0.5 on the 4-point scale. A negative assessment disparity indicated that the residents evaluated themselves at a lower level than did the CCC. Statistical analysis included Wilcoxon rank sum and Sign tests. Setting: CCCs and categorical general surgery residents from 15 residency programs completed the Milestones document independently during the spring of 2016. Results: Overall, 334 residents were included; 44 (13%) and 43 (13%) residents scored themselves ≥0.5 points higher and lower than the CCC, respectively. Female residents scored themselves a mean of 0.08 points lower, and male residents scored themselves a mean of 0.03 points higher than the CCC. Median assessment differences for postgraduate year (PGY) 1-5 were 0.03 (range: -0.94 to 1.28), -0.11 (range: -1.22 to 1.22), -0.08 (range: -1.28 to 0.81), 0.02 (range: -0.91 to 1.00), and -0.19 (range: -1.16 to 0.50), respectively. Residents in university vs. independent programs had higher rates of negative assessment differences in medical knowledge (15% vs. 6%; P = 0.015), patient care (17% vs. 5%; P = 0.002), professionalism (23% vs. 14%; P = 0.013), and system-based practice (18% vs. 9%; P = 0.031) competencies. Major assessment disparities by sex or PGY were similar among individual competencies. Conclusions: Surgery residents in this national cohort demonstrated self-awareness when compared to assessments by their respective CCCs. This was independent of program type, sex, or level of training. PGY 5 residents, female residents, and those from university programs consistently rated themselves lower than the CCC, but these were not major disparities and the significance of this is unclear.

AB - Objective: The Accreditation Council for Graduate Medical Education requires accredited residency programs to implement competency-based assessments of medical trainees based upon nationally established Milestones. Clinical competency committees (CCC) are required to prepare biannual reports using the Milestones and ensure reporting to the Accreditation Council for Graduate Medical Education. Previous research demonstrated a strong correlation between CCC and resident scores on the Milestones at 1 institution. We sought to evaluate a national sampling of general surgery residency programs and hypothesized that CCC and resident assessments are similar. Design: Details regarding the makeup and process of each CCC were obtained. Major disparities were defined as an absolute mean difference of ≥0.5 on the 4-point scale. A negative assessment disparity indicated that the residents evaluated themselves at a lower level than did the CCC. Statistical analysis included Wilcoxon rank sum and Sign tests. Setting: CCCs and categorical general surgery residents from 15 residency programs completed the Milestones document independently during the spring of 2016. Results: Overall, 334 residents were included; 44 (13%) and 43 (13%) residents scored themselves ≥0.5 points higher and lower than the CCC, respectively. Female residents scored themselves a mean of 0.08 points lower, and male residents scored themselves a mean of 0.03 points higher than the CCC. Median assessment differences for postgraduate year (PGY) 1-5 were 0.03 (range: -0.94 to 1.28), -0.11 (range: -1.22 to 1.22), -0.08 (range: -1.28 to 0.81), 0.02 (range: -0.91 to 1.00), and -0.19 (range: -1.16 to 0.50), respectively. Residents in university vs. independent programs had higher rates of negative assessment differences in medical knowledge (15% vs. 6%; P = 0.015), patient care (17% vs. 5%; P = 0.002), professionalism (23% vs. 14%; P = 0.013), and system-based practice (18% vs. 9%; P = 0.031) competencies. Major assessment disparities by sex or PGY were similar among individual competencies. Conclusions: Surgery residents in this national cohort demonstrated self-awareness when compared to assessments by their respective CCCs. This was independent of program type, sex, or level of training. PGY 5 residents, female residents, and those from university programs consistently rated themselves lower than the CCC, but these were not major disparities and the significance of this is unclear.

KW - Clinical competency committee

KW - Competencies

KW - Domains

KW - Interpersonal and Communication Skills

KW - Milestones

KW - Practice-Based Learning and Improvement

KW - Professionalism

KW - Resident evaluation

KW - Systems-Based Practice

UR - http://www.scopus.com/inward/record.url?scp=85021334673&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021334673&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2017.06.009

DO - 10.1016/j.jsurg.2017.06.009

M3 - Article

C2 - 28666959

AN - SCOPUS:85021334673

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

ER -