Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

Diana L Miglioretti, Laura Ichikawa, Robert A. Smith, Diana S.M. Buist, Patricia A. Carney, Berta Geller, Barbara Monsees, Tracy Onega, Robert Rosenberg, Edward A. Sickles, Bonnie C. Yankaskas, Karla Kerlikowske

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Rationale and Objectives: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. Materials and Methods: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. Results: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. Conclusions: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

Original languageEnglish (US)
JournalAcademic Radiology
DOIs
StateAccepted/In press - Nov 21 2016

Fingerprint

Mammography
Neoplasms
Confidence Intervals
Breast
Research Ethics Committees
Expert Testimony
Registries
Breast Neoplasms
Sensitivity and Specificity
Radiologists

Keywords

  • Interpretive performance
  • Screening mammography
  • Test sets

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice. / Miglioretti, Diana L; Ichikawa, Laura; Smith, Robert A.; Buist, Diana S.M.; Carney, Patricia A.; Geller, Berta; Monsees, Barbara; Onega, Tracy; Rosenberg, Robert; Sickles, Edward A.; Yankaskas, Bonnie C.; Kerlikowske, Karla.

In: Academic Radiology, 21.11.2016.

Research output: Contribution to journalArticle

Miglioretti, DL, Ichikawa, L, Smith, RA, Buist, DSM, Carney, PA, Geller, B, Monsees, B, Onega, T, Rosenberg, R, Sickles, EA, Yankaskas, BC & Kerlikowske, K 2016, 'Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice', Academic Radiology. https://doi.org/10.1016/j.acra.2017.03.016
Miglioretti, Diana L ; Ichikawa, Laura ; Smith, Robert A. ; Buist, Diana S.M. ; Carney, Patricia A. ; Geller, Berta ; Monsees, Barbara ; Onega, Tracy ; Rosenberg, Robert ; Sickles, Edward A. ; Yankaskas, Bonnie C. ; Kerlikowske, Karla. / Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice. In: Academic Radiology. 2016.
@article{3a2cc03793bd4248bc076d2aa260b985,
title = "Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice",
abstract = "Rationale and Objectives: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. Materials and Methods: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95{\%} confidence intervals (CI) were estimated. Results: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95{\%} CI = 0.16, 0.69; breast level = 0.35, 95{\%} CI = 0.03, 0.61) and weak for specificity (0.24, 95{\%} CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. Conclusions: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.",
keywords = "Interpretive performance, Screening mammography, Test sets",
author = "Miglioretti, {Diana L} and Laura Ichikawa and Smith, {Robert A.} and Buist, {Diana S.M.} and Carney, {Patricia A.} and Berta Geller and Barbara Monsees and Tracy Onega and Robert Rosenberg and Sickles, {Edward A.} and Yankaskas, {Bonnie C.} and Karla Kerlikowske",
year = "2016",
month = "11",
day = "21",
doi = "10.1016/j.acra.2017.03.016",
language = "English (US)",
journal = "Academic Radiology",
issn = "1076-6332",
publisher = "Elsevier USA",

}

TY - JOUR

T1 - Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

AU - Miglioretti, Diana L

AU - Ichikawa, Laura

AU - Smith, Robert A.

AU - Buist, Diana S.M.

AU - Carney, Patricia A.

AU - Geller, Berta

AU - Monsees, Barbara

AU - Onega, Tracy

AU - Rosenberg, Robert

AU - Sickles, Edward A.

AU - Yankaskas, Bonnie C.

AU - Kerlikowske, Karla

PY - 2016/11/21

Y1 - 2016/11/21

N2 - Rationale and Objectives: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. Materials and Methods: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. Results: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. Conclusions: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

AB - Rationale and Objectives: Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. Materials and Methods: This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. Results: For test sets with fewer cancers (N = 15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level = 0.46, 95% CI = 0.16, 0.69; breast level = 0.35, 95% CI = 0.03, 0.61) and weak for specificity (0.24, 95% CI = 0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N = 30) were close to 0 and not statistically significant. Conclusions: Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

KW - Interpretive performance

KW - Screening mammography

KW - Test sets

UR - http://www.scopus.com/inward/record.url?scp=85019586734&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85019586734&partnerID=8YFLogxK

U2 - 10.1016/j.acra.2017.03.016

DO - 10.1016/j.acra.2017.03.016

M3 - Article

C2 - 28551400

AN - SCOPUS:85019586734

JO - Academic Radiology

JF - Academic Radiology

SN - 1076-6332

ER -