Preliminary Research Finds Ensemble Model Most Consistent for Predicting COVID-19 Deaths

Image

AMHERST, Mass. – An ensemble forecast representing multiple models has generated the most consistent predictions of COVID-19 deaths, according to preliminary research by biostatisticians and epidemiologists at the University of Massachusetts Amherst.

Led by Nicholas Reich, director of the UMass Amherst-based COVID-19 Forecast Hub and the Influenza Forecasting Center of Excellence, the team evaluated about 200,000 forecasts of pandemic deaths from around 25 global teams of infectious-disease forecasters. Their initial findings are available on medRxiv, the health sciences preprint server for research not yet peer-reviewed.

“The ensemble was the only model that ranked in the top half for over 75% of the forecasts it made,” says Reich, associate professor of biostatistics and epidemiology in the School of Public Health and Health Sciences.

The COVID-19 Forecast Hub, which was created in partnership with and is funded by the U.S. Centers for Disease Control and Prevention (CDC), built the ensemble forecast that combines models submitted weekly from top infectious-disease modelers at prominent institutions around the world, as well as those from independent data analysts.

Forecasting is a critical tool for infectious disease response and planning, and the CDC uses the Hub’s weekly forecast in official public communications about the pandemic.

The paper compares the accuracy of different modeling approaches from May through December 2020, adjusting for different U.S. regions and weeks, says lead author Estee Cramer, a Ph.D. epidemiology candidate working in Reich’s lab.

“This paper represents a collaborative modeling effort between government, academic and industry teams,” Cramer says. “Our results demonstrate that the COVID-19 Forecast Hub ensemble model is consistently the most accurate model in predicting the number of new deaths across locations in the United States or across weeks of the pandemic. Though there is variation across individual forecasts, the ensemble is consistently one of the top performers, which makes it a dependable model for government agencies to rely on.”

The research also confirms Reich’s assertion that predictions “degrade” at lengthier looks into the future, which is why the Hub forecasts no more than four weeks out. Other models look as far out as 20 weeks.

“We observed two times the error at four weeks ahead versus one week ahead,” Cramer notes, “and five to six times the error at 20 weeks ahead versus one week.”

The COVID-19 forecasting efforts represent the largest infectious-disease prediction project ever conducted and highlight its importance in battling pandemics.

“This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks,” the paper concludes. “Understanding what leads to more or less accurate and well-calibrated forecasts can inform their development and their use within outbreak science and public policy.”