Deep learning and radiomics

the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT

Heidi Coy, Kevin Hsieh, Willie Wu, Mahesh B. Nagarajan, Jonathan R Young, Michael L. Douek, Matthew S. Brown, Fabien Scalzo, Steven S. Raman

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Purpose: Currently, all solid enhancing renal masses without microscopic fat are considered malignant until proven otherwise and there is substantial overlap in the imaging findings of benign and malignant renal masses, particularly between clear cell RCC (ccRCC) and benign oncocytoma (ONC). Radiomics has attracted increased attention for its utility in pre-operative work-up on routine clinical images. Radiomics based approaches have converted medical images into mineable data and identified prognostic imaging signatures that machine learning algorithms can use to construct predictive models by learning the decision boundaries of the underlying data distribution. The TensorFlow™ framework from Google is a state-of-the-art open-source software library that can be used for training deep learning neural networks for performing machine learning tasks. The purpose of this study was to investigate the diagnostic value and feasibility of a deep learning-based renal lesion classifier using open-source Google TensorFlow™ Inception in differentiating ccRCC from ONC on routine four-phase MDCT in patients with pathologically confirmed renal masses. Methods: With institutional review board approval for this 1996 Health Insurance Portability and Accountability Act compliant retrospective study and a waiver of informed consent, we queried our institution’s pathology, clinical, and radiology databases for histologically proven cases of ccRCC and ONC obtained between January 2000 and January 2016 scanned with a an intravenous contrast-enhanced four-phase renal mass protocol (unenhanced (UN), corticomedullary (CM), nephrographic (NP), and excretory (EX) phases). To extract features to be used for the machine learning model, the entire renal mass was contoured in the axial plane in each of the four phases, resulting in a 3D volume of interest (VOI) representative of the entire renal mass. We investigated thirteen different approaches to convert the acquired VOI data into a set of images that adequately represented each tumor which was used to train the final layer of the neural network model. Training was performed over 4000 iterations. In each iteration, 90% of the data were designated as training data and the remaining 10% served as validation data and a leave-one-out cross-validation scheme was implemented. Accuracy, sensitivity, specificity, positive (PPV) and negative predictive (NPV) values, and CIs were calculated for the classification of the thirteen processing modes. Results: We analyzed 179 consecutive patients with 179 lesions (128 ccRCC and 51 ONC). The ccRCC cohort had a mean size of 3.8 cm (range 0.8–14.6 cm) and the ONC cohort had a mean lesion size of 3.9 cm (range 1.0–13.1 cm). The highest specificity and PPV (52.9% and 80.3%, respectively) were achieved in the EX phase when we analyzed the single mid-slice of the tumor in the axial, coronal and sagittal plane, and when we increased the number of mid-slices of the tumor to three, with an accuracy of 75.4%, which also increased the sensitivity to 88.3% and the PPV to 79.6%. Using the entire tumor volume also showed that classification performance was best in the EX phase with an accuracy of 74.4%, a sensitivity of 85.8% and a PPV of 80.1%. When the entire tumor volume, plus mid-slices from all phases and all planes presented as tiled images, were submitted to the final layer of the neural network we achieved a PPV of 82.5%. Conclusions: The best classification result was obtained in the EX phase among the thirteen classification methods tested. Our proof of concept study is the first step towards understanding the utility of machine learning in the differentiation of ccRCC from ONC on routine CT images. We hope this could lead to future investigation into the development of a multivariate machine learning model which may augment our ability to accurately predict renal lesion histology on imaging.

Original languageEnglish (US)
JournalAbdominal Radiology
DOIs
StatePublished - Jan 1 2019
Externally publishedYes

Fingerprint

Oxyphilic Adenoma
Renal Cell Carcinoma
Learning
Kidney
Tumor Burden
Health Insurance Portability and Accountability Act
Neoplasms
Clinical Pathology
Neural Networks (Computer)
Renal oncocytoma
Research Ethics Committees
Informed Consent
Radiology
Clear-cell metastatic renal cell carcinoma
Histology
Software
Retrospective Studies
Fats
Machine Learning
Databases

Keywords

  • Clear cell renal cell carcinoma
  • Deep learning
  • Multiphasic CT
  • Oncocytoma
  • Radiogenomics
  • Radiomics

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology
  • Radiology Nuclear Medicine and imaging
  • Gastroenterology
  • Urology

Cite this

Deep learning and radiomics : the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT. / Coy, Heidi; Hsieh, Kevin; Wu, Willie; Nagarajan, Mahesh B.; Young, Jonathan R; Douek, Michael L.; Brown, Matthew S.; Scalzo, Fabien; Raman, Steven S.

In: Abdominal Radiology, 01.01.2019.

Research output: Contribution to journalArticle

Coy, Heidi ; Hsieh, Kevin ; Wu, Willie ; Nagarajan, Mahesh B. ; Young, Jonathan R ; Douek, Michael L. ; Brown, Matthew S. ; Scalzo, Fabien ; Raman, Steven S. / Deep learning and radiomics : the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT. In: Abdominal Radiology. 2019.
@article{86587df96c8c496e9971c1a9d4cc1a4c,
title = "Deep learning and radiomics: the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT",
abstract = "Purpose: Currently, all solid enhancing renal masses without microscopic fat are considered malignant until proven otherwise and there is substantial overlap in the imaging findings of benign and malignant renal masses, particularly between clear cell RCC (ccRCC) and benign oncocytoma (ONC). Radiomics has attracted increased attention for its utility in pre-operative work-up on routine clinical images. Radiomics based approaches have converted medical images into mineable data and identified prognostic imaging signatures that machine learning algorithms can use to construct predictive models by learning the decision boundaries of the underlying data distribution. The TensorFlow™ framework from Google is a state-of-the-art open-source software library that can be used for training deep learning neural networks for performing machine learning tasks. The purpose of this study was to investigate the diagnostic value and feasibility of a deep learning-based renal lesion classifier using open-source Google TensorFlow™ Inception in differentiating ccRCC from ONC on routine four-phase MDCT in patients with pathologically confirmed renal masses. Methods: With institutional review board approval for this 1996 Health Insurance Portability and Accountability Act compliant retrospective study and a waiver of informed consent, we queried our institution’s pathology, clinical, and radiology databases for histologically proven cases of ccRCC and ONC obtained between January 2000 and January 2016 scanned with a an intravenous contrast-enhanced four-phase renal mass protocol (unenhanced (UN), corticomedullary (CM), nephrographic (NP), and excretory (EX) phases). To extract features to be used for the machine learning model, the entire renal mass was contoured in the axial plane in each of the four phases, resulting in a 3D volume of interest (VOI) representative of the entire renal mass. We investigated thirteen different approaches to convert the acquired VOI data into a set of images that adequately represented each tumor which was used to train the final layer of the neural network model. Training was performed over 4000 iterations. In each iteration, 90{\%} of the data were designated as training data and the remaining 10{\%} served as validation data and a leave-one-out cross-validation scheme was implemented. Accuracy, sensitivity, specificity, positive (PPV) and negative predictive (NPV) values, and CIs were calculated for the classification of the thirteen processing modes. Results: We analyzed 179 consecutive patients with 179 lesions (128 ccRCC and 51 ONC). The ccRCC cohort had a mean size of 3.8 cm (range 0.8–14.6 cm) and the ONC cohort had a mean lesion size of 3.9 cm (range 1.0–13.1 cm). The highest specificity and PPV (52.9{\%} and 80.3{\%}, respectively) were achieved in the EX phase when we analyzed the single mid-slice of the tumor in the axial, coronal and sagittal plane, and when we increased the number of mid-slices of the tumor to three, with an accuracy of 75.4{\%}, which also increased the sensitivity to 88.3{\%} and the PPV to 79.6{\%}. Using the entire tumor volume also showed that classification performance was best in the EX phase with an accuracy of 74.4{\%}, a sensitivity of 85.8{\%} and a PPV of 80.1{\%}. When the entire tumor volume, plus mid-slices from all phases and all planes presented as tiled images, were submitted to the final layer of the neural network we achieved a PPV of 82.5{\%}. Conclusions: The best classification result was obtained in the EX phase among the thirteen classification methods tested. Our proof of concept study is the first step towards understanding the utility of machine learning in the differentiation of ccRCC from ONC on routine CT images. We hope this could lead to future investigation into the development of a multivariate machine learning model which may augment our ability to accurately predict renal lesion histology on imaging.",
keywords = "Clear cell renal cell carcinoma, Deep learning, Multiphasic CT, Oncocytoma, Radiogenomics, Radiomics",
author = "Heidi Coy and Kevin Hsieh and Willie Wu and Nagarajan, {Mahesh B.} and Young, {Jonathan R} and Douek, {Michael L.} and Brown, {Matthew S.} and Fabien Scalzo and Raman, {Steven S.}",
year = "2019",
month = "1",
day = "1",
doi = "10.1007/s00261-019-01929-0",
language = "English (US)",
journal = "Abdominal Radiology",
issn = "2366-004X",
publisher = "Springer New York",

}

TY - JOUR

T1 - Deep learning and radiomics

T2 - the utility of Google TensorFlow™ Inception in classifying clear cell renal cell carcinoma and oncocytoma on multiphasic CT

AU - Coy, Heidi

AU - Hsieh, Kevin

AU - Wu, Willie

AU - Nagarajan, Mahesh B.

AU - Young, Jonathan R

AU - Douek, Michael L.

AU - Brown, Matthew S.

AU - Scalzo, Fabien

AU - Raman, Steven S.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Purpose: Currently, all solid enhancing renal masses without microscopic fat are considered malignant until proven otherwise and there is substantial overlap in the imaging findings of benign and malignant renal masses, particularly between clear cell RCC (ccRCC) and benign oncocytoma (ONC). Radiomics has attracted increased attention for its utility in pre-operative work-up on routine clinical images. Radiomics based approaches have converted medical images into mineable data and identified prognostic imaging signatures that machine learning algorithms can use to construct predictive models by learning the decision boundaries of the underlying data distribution. The TensorFlow™ framework from Google is a state-of-the-art open-source software library that can be used for training deep learning neural networks for performing machine learning tasks. The purpose of this study was to investigate the diagnostic value and feasibility of a deep learning-based renal lesion classifier using open-source Google TensorFlow™ Inception in differentiating ccRCC from ONC on routine four-phase MDCT in patients with pathologically confirmed renal masses. Methods: With institutional review board approval for this 1996 Health Insurance Portability and Accountability Act compliant retrospective study and a waiver of informed consent, we queried our institution’s pathology, clinical, and radiology databases for histologically proven cases of ccRCC and ONC obtained between January 2000 and January 2016 scanned with a an intravenous contrast-enhanced four-phase renal mass protocol (unenhanced (UN), corticomedullary (CM), nephrographic (NP), and excretory (EX) phases). To extract features to be used for the machine learning model, the entire renal mass was contoured in the axial plane in each of the four phases, resulting in a 3D volume of interest (VOI) representative of the entire renal mass. We investigated thirteen different approaches to convert the acquired VOI data into a set of images that adequately represented each tumor which was used to train the final layer of the neural network model. Training was performed over 4000 iterations. In each iteration, 90% of the data were designated as training data and the remaining 10% served as validation data and a leave-one-out cross-validation scheme was implemented. Accuracy, sensitivity, specificity, positive (PPV) and negative predictive (NPV) values, and CIs were calculated for the classification of the thirteen processing modes. Results: We analyzed 179 consecutive patients with 179 lesions (128 ccRCC and 51 ONC). The ccRCC cohort had a mean size of 3.8 cm (range 0.8–14.6 cm) and the ONC cohort had a mean lesion size of 3.9 cm (range 1.0–13.1 cm). The highest specificity and PPV (52.9% and 80.3%, respectively) were achieved in the EX phase when we analyzed the single mid-slice of the tumor in the axial, coronal and sagittal plane, and when we increased the number of mid-slices of the tumor to three, with an accuracy of 75.4%, which also increased the sensitivity to 88.3% and the PPV to 79.6%. Using the entire tumor volume also showed that classification performance was best in the EX phase with an accuracy of 74.4%, a sensitivity of 85.8% and a PPV of 80.1%. When the entire tumor volume, plus mid-slices from all phases and all planes presented as tiled images, were submitted to the final layer of the neural network we achieved a PPV of 82.5%. Conclusions: The best classification result was obtained in the EX phase among the thirteen classification methods tested. Our proof of concept study is the first step towards understanding the utility of machine learning in the differentiation of ccRCC from ONC on routine CT images. We hope this could lead to future investigation into the development of a multivariate machine learning model which may augment our ability to accurately predict renal lesion histology on imaging.

AB - Purpose: Currently, all solid enhancing renal masses without microscopic fat are considered malignant until proven otherwise and there is substantial overlap in the imaging findings of benign and malignant renal masses, particularly between clear cell RCC (ccRCC) and benign oncocytoma (ONC). Radiomics has attracted increased attention for its utility in pre-operative work-up on routine clinical images. Radiomics based approaches have converted medical images into mineable data and identified prognostic imaging signatures that machine learning algorithms can use to construct predictive models by learning the decision boundaries of the underlying data distribution. The TensorFlow™ framework from Google is a state-of-the-art open-source software library that can be used for training deep learning neural networks for performing machine learning tasks. The purpose of this study was to investigate the diagnostic value and feasibility of a deep learning-based renal lesion classifier using open-source Google TensorFlow™ Inception in differentiating ccRCC from ONC on routine four-phase MDCT in patients with pathologically confirmed renal masses. Methods: With institutional review board approval for this 1996 Health Insurance Portability and Accountability Act compliant retrospective study and a waiver of informed consent, we queried our institution’s pathology, clinical, and radiology databases for histologically proven cases of ccRCC and ONC obtained between January 2000 and January 2016 scanned with a an intravenous contrast-enhanced four-phase renal mass protocol (unenhanced (UN), corticomedullary (CM), nephrographic (NP), and excretory (EX) phases). To extract features to be used for the machine learning model, the entire renal mass was contoured in the axial plane in each of the four phases, resulting in a 3D volume of interest (VOI) representative of the entire renal mass. We investigated thirteen different approaches to convert the acquired VOI data into a set of images that adequately represented each tumor which was used to train the final layer of the neural network model. Training was performed over 4000 iterations. In each iteration, 90% of the data were designated as training data and the remaining 10% served as validation data and a leave-one-out cross-validation scheme was implemented. Accuracy, sensitivity, specificity, positive (PPV) and negative predictive (NPV) values, and CIs were calculated for the classification of the thirteen processing modes. Results: We analyzed 179 consecutive patients with 179 lesions (128 ccRCC and 51 ONC). The ccRCC cohort had a mean size of 3.8 cm (range 0.8–14.6 cm) and the ONC cohort had a mean lesion size of 3.9 cm (range 1.0–13.1 cm). The highest specificity and PPV (52.9% and 80.3%, respectively) were achieved in the EX phase when we analyzed the single mid-slice of the tumor in the axial, coronal and sagittal plane, and when we increased the number of mid-slices of the tumor to three, with an accuracy of 75.4%, which also increased the sensitivity to 88.3% and the PPV to 79.6%. Using the entire tumor volume also showed that classification performance was best in the EX phase with an accuracy of 74.4%, a sensitivity of 85.8% and a PPV of 80.1%. When the entire tumor volume, plus mid-slices from all phases and all planes presented as tiled images, were submitted to the final layer of the neural network we achieved a PPV of 82.5%. Conclusions: The best classification result was obtained in the EX phase among the thirteen classification methods tested. Our proof of concept study is the first step towards understanding the utility of machine learning in the differentiation of ccRCC from ONC on routine CT images. We hope this could lead to future investigation into the development of a multivariate machine learning model which may augment our ability to accurately predict renal lesion histology on imaging.

KW - Clear cell renal cell carcinoma

KW - Deep learning

KW - Multiphasic CT

KW - Oncocytoma

KW - Radiogenomics

KW - Radiomics

UR - http://www.scopus.com/inward/record.url?scp=85061714769&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061714769&partnerID=8YFLogxK

U2 - 10.1007/s00261-019-01929-0

DO - 10.1007/s00261-019-01929-0

M3 - Article

JO - Abdominal Radiology

JF - Abdominal Radiology

SN - 2366-004X

ER -