Statistical properties of a utility measure of observer performance compared to area under the ROC curve

Craig K. Abbey, Frank W. Samuelson, Brandon D. Gallas, John M Boone, Loren T. Niklason

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36% fewer readers on average to achieve 80% statistical power compared to AUC.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
Volume8673
DOIs
StatePublished - 2013
EventSPIE Medical Imaging Symposium 2013: Image Perception, Observer Performance, and Technology Assessment - Lake Buena Vista, FL, United States
Duration: Feb 10 2013Feb 11 2013

Other

OtherSPIE Medical Imaging Symposium 2013: Image Perception, Observer Performance, and Technology Assessment
CountryUnited States
CityLake Buena Vista, FL
Period2/10/132/11/13

Fingerprint

Expected Utility
Receiver Operating Characteristic Curve
Statistical property
Observer
receivers
readers
Imaging techniques
curves
Curve
Modality
Effect Size
Paired Comparisons
Statistical Power
Power Analysis
Operating Characteristics
False Positive
Standard deviation
Slope
Figure
Diagnostics

Keywords

  • Area under the curve
  • Expected utility
  • Observer performance
  • ROC analysis

ASJC Scopus subject areas

  • Applied Mathematics
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics

Cite this

Abbey, C. K., Samuelson, F. W., Gallas, B. D., Boone, J. M., & Niklason, L. T. (2013). Statistical properties of a utility measure of observer performance compared to area under the ROC curve. In Proceedings of SPIE - The International Society for Optical Engineering (Vol. 8673). [86730D] https://doi.org/10.1117/12.2007708

Statistical properties of a utility measure of observer performance compared to area under the ROC curve. / Abbey, Craig K.; Samuelson, Frank W.; Gallas, Brandon D.; Boone, John M; Niklason, Loren T.

Proceedings of SPIE - The International Society for Optical Engineering. Vol. 8673 2013. 86730D.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abbey, CK, Samuelson, FW, Gallas, BD, Boone, JM & Niklason, LT 2013, Statistical properties of a utility measure of observer performance compared to area under the ROC curve. in Proceedings of SPIE - The International Society for Optical Engineering. vol. 8673, 86730D, SPIE Medical Imaging Symposium 2013: Image Perception, Observer Performance, and Technology Assessment, Lake Buena Vista, FL, United States, 2/10/13. https://doi.org/10.1117/12.2007708
Abbey CK, Samuelson FW, Gallas BD, Boone JM, Niklason LT. Statistical properties of a utility measure of observer performance compared to area under the ROC curve. In Proceedings of SPIE - The International Society for Optical Engineering. Vol. 8673. 2013. 86730D https://doi.org/10.1117/12.2007708
Abbey, Craig K. ; Samuelson, Frank W. ; Gallas, Brandon D. ; Boone, John M ; Niklason, Loren T. / Statistical properties of a utility measure of observer performance compared to area under the ROC curve. Proceedings of SPIE - The International Society for Optical Engineering. Vol. 8673 2013.
@inproceedings{b6b73e1c41004ccfa80db0a61ddbfd79,
title = "Statistical properties of a utility measure of observer performance compared to area under the ROC curve",
abstract = "The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36{\%} fewer readers on average to achieve 80{\%} statistical power compared to AUC.",
keywords = "Area under the curve, Expected utility, Observer performance, ROC analysis",
author = "Abbey, {Craig K.} and Samuelson, {Frank W.} and Gallas, {Brandon D.} and Boone, {John M} and Niklason, {Loren T.}",
year = "2013",
doi = "10.1117/12.2007708",
language = "English (US)",
isbn = "9780819494474",
volume = "8673",
booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",

}

TY - GEN

T1 - Statistical properties of a utility measure of observer performance compared to area under the ROC curve

AU - Abbey, Craig K.

AU - Samuelson, Frank W.

AU - Gallas, Brandon D.

AU - Boone, John M

AU - Niklason, Loren T.

PY - 2013

Y1 - 2013

N2 - The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36% fewer readers on average to achieve 80% statistical power compared to AUC.

AB - The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36% fewer readers on average to achieve 80% statistical power compared to AUC.

KW - Area under the curve

KW - Expected utility

KW - Observer performance

KW - ROC analysis

UR - http://www.scopus.com/inward/record.url?scp=84878771134&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84878771134&partnerID=8YFLogxK

U2 - 10.1117/12.2007708

DO - 10.1117/12.2007708

M3 - Conference contribution

SN - 9780819494474

VL - 8673

BT - Proceedings of SPIE - The International Society for Optical Engineering

ER -