Martin Wainwright: How to tame and validate AI-based predictions?

Ford Professor of Engineering; Professor of Mathematics, [AI+D]

Massachusetts Institute of Technology

ABSTRACT: Statistical prediction is the engine powering dramatic advances in AI: diffusion denoising for generating images, token prediction for large language models, and look-ahead prediction in reinforcement learning.  Complex predictive models can behave remarkably well, but can also fail in mysterious and dangerous ways, including biased and mis-calibrated outputs, poor performance under distribution shift, and high uncertainty associated with their predictive performance.  This talk discusses these challenges, along with some new approaches for tackling them.  We discuss a simple method for combining (potentially misleading) AI-generated data with clean data, and a fast and data-efficient refitting method for assessing predictive performance.