Diagnostic mammography

Identifying minimally acceptable interpretive performance criteria

Patricia A. Carney, Jay Parikh, Edward A. Sickles, Stephen A. Feig, Barbara Monsees, Lawrence W. Bassett, Robert A. Smith, Robert Rosenberg, Laura Ichikawa, James Wallace, Khai Tran, Diana L Miglioretti

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

Purpose: To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies. Materials and Methods: In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds. Results: Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100 000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100 000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100 000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100 000 women undergoing this workup. Conclusion: Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

Original languageEnglish (US)
Pages (from-to)359-367
Number of pages9
JournalRadiology
Volume267
Issue number2
DOIs
StatePublished - May 2013
Externally publishedYes

Fingerprint

Mammography
Breast
Physicians
Neoplasms
Breast Neoplasms
Health Insurance Portability and Accountability Act
Biopsy
Research Ethics Committees

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Cite this

Carney, P. A., Parikh, J., Sickles, E. A., Feig, S. A., Monsees, B., Bassett, L. W., ... Miglioretti, D. L. (2013). Diagnostic mammography: Identifying minimally acceptable interpretive performance criteria. Radiology, 267(2), 359-367. https://doi.org/10.1148/radiol.12121216

Diagnostic mammography : Identifying minimally acceptable interpretive performance criteria. / Carney, Patricia A.; Parikh, Jay; Sickles, Edward A.; Feig, Stephen A.; Monsees, Barbara; Bassett, Lawrence W.; Smith, Robert A.; Rosenberg, Robert; Ichikawa, Laura; Wallace, James; Tran, Khai; Miglioretti, Diana L.

In: Radiology, Vol. 267, No. 2, 05.2013, p. 359-367.

Research output: Contribution to journalArticle

Carney, PA, Parikh, J, Sickles, EA, Feig, SA, Monsees, B, Bassett, LW, Smith, RA, Rosenberg, R, Ichikawa, L, Wallace, J, Tran, K & Miglioretti, DL 2013, 'Diagnostic mammography: Identifying minimally acceptable interpretive performance criteria', Radiology, vol. 267, no. 2, pp. 359-367. https://doi.org/10.1148/radiol.12121216
Carney PA, Parikh J, Sickles EA, Feig SA, Monsees B, Bassett LW et al. Diagnostic mammography: Identifying minimally acceptable interpretive performance criteria. Radiology. 2013 May;267(2):359-367. https://doi.org/10.1148/radiol.12121216
Carney, Patricia A. ; Parikh, Jay ; Sickles, Edward A. ; Feig, Stephen A. ; Monsees, Barbara ; Bassett, Lawrence W. ; Smith, Robert A. ; Rosenberg, Robert ; Ichikawa, Laura ; Wallace, James ; Tran, Khai ; Miglioretti, Diana L. / Diagnostic mammography : Identifying minimally acceptable interpretive performance criteria. In: Radiology. 2013 ; Vol. 267, No. 2. pp. 359-367.
@article{de0c4e6f040b4aae8311d63fc21782ea,
title = "Diagnostic mammography: Identifying minimally acceptable interpretive performance criteria",
abstract = "Purpose: To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies. Materials and Methods: In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds. Results: Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80{\%}; specificity, less than 80{\%} or greater than 95{\%}; abnormal interpretation rate, less than 8{\%} or greater than 25{\%}; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15{\%} or greater than 40{\%}; PPV of biopsy performed (PPV3), less than 20{\%} or greater than 45{\%}; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85{\%}; specificity, less than 83{\%} or greater than 95{\%}; abnormal interpretation rate, less than 10{\%} or greater than 25{\%}; PPV2, less than 25{\%} or greater than 50{\%}; PPV3, less than 30{\%} or greater than 55{\%}; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100 000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100 000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100 000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100 000 women undergoing this workup. Conclusion: Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.",
author = "Carney, {Patricia A.} and Jay Parikh and Sickles, {Edward A.} and Feig, {Stephen A.} and Barbara Monsees and Bassett, {Lawrence W.} and Smith, {Robert A.} and Robert Rosenberg and Laura Ichikawa and James Wallace and Khai Tran and Miglioretti, {Diana L}",
year = "2013",
month = "5",
doi = "10.1148/radiol.12121216",
language = "English (US)",
volume = "267",
pages = "359--367",
journal = "Radiology",
issn = "0033-8419",
publisher = "Radiological Society of North America Inc.",
number = "2",

}

TY - JOUR

T1 - Diagnostic mammography

T2 - Identifying minimally acceptable interpretive performance criteria

AU - Carney, Patricia A.

AU - Parikh, Jay

AU - Sickles, Edward A.

AU - Feig, Stephen A.

AU - Monsees, Barbara

AU - Bassett, Lawrence W.

AU - Smith, Robert A.

AU - Rosenberg, Robert

AU - Ichikawa, Laura

AU - Wallace, James

AU - Tran, Khai

AU - Miglioretti, Diana L

PY - 2013/5

Y1 - 2013/5

N2 - Purpose: To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies. Materials and Methods: In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds. Results: Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100 000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100 000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100 000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100 000 women undergoing this workup. Conclusion: Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

AB - Purpose: To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies. Materials and Methods: In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds. Results: Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100 000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100 000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100 000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100 000 women undergoing this workup. Conclusion: Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

UR - http://www.scopus.com/inward/record.url?scp=84876910333&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84876910333&partnerID=8YFLogxK

U2 - 10.1148/radiol.12121216

DO - 10.1148/radiol.12121216

M3 - Article

VL - 267

SP - 359

EP - 367

JO - Radiology

JF - Radiology

SN - 0033-8419

IS - 2

ER -