Certainty of mortality risk estimates and its effect on prediction performance in the pediatric ICU

Introduction: Prognostication is central to developing treatment plans and relaying information to patients, family members and other health care providers. The degree of confidence or certainty that a health care provider has with his/her mortality risk assessment is similarly important since a provider may deliver care differently depending upon the assuredness of the risk assessment. Methods: Subjective mortality risk estimates from critical care attendings (5), critical care fellows (9), pediatric residents (34) and nurses (52) were collected on 655 of 668 (98.1%) consecutive admissions to a tertiary PICU. Estimates were collected between 10 and 24 hours after admission following rounds. A measure of certainty (continuous scale from 0 to 5) accompanied each mortality estimate. Estimates were evaluated with 2×2 outcome probabilities, the Kappa statistic and the area under the receiver operating characteristics curve (AUC). The estimates were then re-evaluated after weighting (multiplying) predictions by their respective level of certainty. Results: Overall, there was a significant difference in the predictive accuracy between groups (p<0.001 ). The mean mortality predictions from the attendings (6.09%) more closely approximated the true mortality rate (36 deaths, 5.39%) while fellows (7.87%), residents (10.00%) and nurses (16.29%) overestimated the mean overall PICU mortality. Attendings were more certain of their predictions (4.27) than the fellows (4.01), nurses (3.79), and residents (3.75). If the mortality predictions were weighted with their respective certainties, their performance improved (Table 1). Discussion: The level of medical training correlated with the provider's ability to predict mortality risk. The higher the level of certainty associated with the mortality prediction, the more accurate the prediction; however, high levels of certainty did not guarantee correct predictions. Measures of certainty should be considered when comparing subjective mortality risk estimates or other subjective outcome predictions. Table 1 : Results given as: without certainty weight / with certainty weight Sensitivity Specificity False Pos False Neg Kappa AUC Attendings 98.9 / 99.0 42.9 / 52.9 3.4 / 2.4 30.8 / 26.40.56 / 0.630.93 / 0.95 Fellows 98.9 / 98.8 30.4 / 43.3 4.8 / 3.4 43.3 / 33.3 0.49 / 0.55 0.85 / 0.88 Residents 97.2 / 97.5 34.2 / 40.2 4.3 / 2.9 55.2 / 55.8 0.43 / 0.45 0.92 / 0.94 Nurses 96.8 / 96.5 39.7 / 65.3 5.0 / 2.6 48.9 / 42.2 0.53 / 0.60 0.91 / 0.93.