Statistical properties of a utility measure of observer performance compared to area under the ROC curve

Craig K. Abbey, Frank W. Samuelson, Brandon D. Gallas, John M Boone, Loren T. Niklason

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

The receiver operating characteristic (ROC) curve has become a common tool for evaluating diagnostic imaging technologies, and the primary endpoint of such evaluations is the area under the curve (AUC), which integrates sensitivity over the entire false positive range. An alternative figure of merit for ROC studies is expected utility (EU), which focuses on the relevant region of the ROC curve as defined by disease prevalence and the relative utility of the task. However if this measure is to be used, it must also have desirable statistical properties keep the burden of observer performance studies as low as possible. Here, we evaluate effect size and variability for EU and AUC. We use two observer performance studies recently submitted to the FDA to compare the EU and AUC endpoints. The studies were conducted using the multi-reader multi-case methodology in which all readers score all cases in all modalities. ROC curves from the study were used to generate both the AUC and EU values for each reader and modality. The EU measure was computed assuming an iso-utility slope of 1.03. We find mean effect sizes, the reader averaged difference between modalities, to be roughly 2.0 times as big for EU as AUC. The standard deviation across readers is roughly 1.4 times as large, suggesting better statistical properties for the EU endpoint. In a simple power analysis of paired comparison across readers, the utility measure required 36% fewer readers on average to achieve 80% statistical power compared to AUC.

Original languageEnglish (US)
Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
Volume8673
DOIs
StatePublished - 2013
EventSPIE Medical Imaging Symposium 2013: Image Perception, Observer Performance, and Technology Assessment - Lake Buena Vista, FL, United States
Duration: Feb 10 2013Feb 11 2013

Other

OtherSPIE Medical Imaging Symposium 2013: Image Perception, Observer Performance, and Technology Assessment
CountryUnited States
CityLake Buena Vista, FL
Period2/10/132/11/13

Keywords

  • Area under the curve
  • Expected utility
  • Observer performance
  • ROC analysis

ASJC Scopus subject areas

  • Applied Mathematics
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics

Fingerprint Dive into the research topics of 'Statistical properties of a utility measure of observer performance compared to area under the ROC curve'. Together they form a unique fingerprint.

  • Cite this

    Abbey, C. K., Samuelson, F. W., Gallas, B. D., Boone, J. M., & Niklason, L. T. (2013). Statistical properties of a utility measure of observer performance compared to area under the ROC curve. In Proceedings of SPIE - The International Society for Optical Engineering (Vol. 8673). [86730D] https://doi.org/10.1117/12.2007708