Philippe Rigollet: The Dynamics of Transformers

Massachusetts Institute of Technology

Transformers are a class of neural network architectures behind the model AI models. Yet how signals propagate through their layers remains poorly understood. We study this question by viewing attention as a system of interacting particles, connecting Transformers to ideas from dynamical systems, probability, optimization, and geometry. 

Even simplified models exhibit rich behavior. Tokens can cluster, and distinct groups can persist across many layers before eventually merging. A key insight is that these contractive properties of attention can be understood from a variational perspective, with the token dynamics arising from an underlying energy minimization principle. 

This variational perspective also provides a way to design new architectures. Some of these significantly outperform existing ones, illustrating how mathematics can serve not only to analyze models, but also to guide their design.