Examination of Statistical Accuracy of COVID-19 Daily Death Count Predictions
OBJECTIVE: This paper provides a formal evaluation of the predictive performance of a model (and updates) developed by the Institute for Health Metrics and Evaluation (IHME) for predicting daily deaths attributed to COVID-19 for the United States.
STUDY DESIGN: To assess the accuracy of the IHME models, we examine both forecast accuracy, as well as the predictive performance of the 95% prediction intervals (PI).
RESULTS: The initial model underestimates the uncertainty surrounding the number of daily deaths. Specifically, the true number of next day deaths fell outside the IHME prediction intervals as much as 76% of the time, in comparison to the expected value of 5%. Regarding the updated models, our analyses indicate that the April models show little, if any, improvement in the accuracy of the point estimate predictions. Moreover, while we observe a larger percentage of states having actual values lying inside the 95% PI’s, this observation may be attributed to the widening of the PI’s. A major revised model in early May did result in a decrease in the estimated model uncertainty, albeit at the expense of poorer coverage probability.
CONCLUSION: Our analysis calls into question the usefulness of the predictions to drive policy making and resource allocation.