Short-term stability and spread of the U.S. News & world report primary care medical school rankings

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

Purpose: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. Method: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. Results: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations ≥ 98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P < .001, Kruskal-Wallis test). Conclusions: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.

Original languageEnglish (US)
Pages (from-to)1107-1115
Number of pages9
JournalAcademic Medicine
Volume88
Issue number8
DOIs
StatePublished - Aug 2013

Fingerprint

Medical Schools
ranking
Primary Health Care
news
school

ASJC Scopus subject areas

  • Medicine(all)
  • Education

Cite this

@article{a169a07aa248430498d2bd51b46225a8,
title = "Short-term stability and spread of the U.S. News & world report primary care medical school rankings",
abstract = "Purpose: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. Method: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. Results: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations ≥ 98.9{\%}). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94{\%} and 90{\%}. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P < .001, Kruskal-Wallis test). Conclusions: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.",
author = "Tancredi, {Daniel J} and Bertakis, {Klea D} and Jerant, {Anthony F}",
year = "2013",
month = "8",
doi = "10.1097/ACM.0b013e31829a249a",
language = "English (US)",
volume = "88",
pages = "1107--1115",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "8",

}

TY - JOUR

T1 - Short-term stability and spread of the U.S. News & world report primary care medical school rankings

AU - Tancredi, Daniel J

AU - Bertakis, Klea D

AU - Jerant, Anthony F

PY - 2013/8

Y1 - 2013/8

N2 - Purpose: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. Method: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. Results: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations ≥ 98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P < .001, Kruskal-Wallis test). Conclusions: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.

AB - Purpose: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools' rankings. Method: The authors employed published data and methods to reconstruct the 2009-2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period. Results: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations ≥ 98.9%). Most schools' mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P < .001, Kruskal-Wallis test). Conclusions: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking's validity and usefulness.

UR - http://www.scopus.com/inward/record.url?scp=84882454146&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84882454146&partnerID=8YFLogxK

U2 - 10.1097/ACM.0b013e31829a249a

DO - 10.1097/ACM.0b013e31829a249a

M3 - Article

C2 - 23807103

AN - SCOPUS:84882454146

VL - 88

SP - 1107

EP - 1115

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 8

ER -