Evan Ray: Evaluating Infectious Disease Forecasts with Allocation Scoring Rules

Recent years have seen increasing efforts to forecast infectious disease burdens, with a primary goal being to help public health workers make informed policy decisions. However, there has only been limited discussion of how common forecast evaluation metrics might indicate the success of policies based in part on those forecasts. We review a general framework for deriving proper scoring rules that are attuned to a specific decision-making context and apply this framework to develop a novel scoring rule that measures the value of a forecast for informing decisions about how to allocate a limited supply of medical resources. We use probabilistic forecasts of disease burden in each of several regions to determine optimal resource allocations, and then we score forecasts according to how much unmet need their associated allocations would have allowed. We illustrate with forecasts of COVID-19 hospitalizations in the US, and we find that the forecast skill ranking given by this allocation scoring rule can vary substantially from the ranking given by one of the most commonly used scores, the weighted interval score. We see this as evidence that the allocation scoring rule detects forecast value that is missed by traditional accuracy measures and that the general strategy of designing scoring rules that are directly linked to policy performance is a promising direction for epidemic forecast evaluation.