Perceptrons and localization of attention's mean-field landscape

In the large-depth, mean-field limit, the evolution of a token distribution in a Transformer can be modeled—under some structural assumptions—as a Wasserstein gradient flow on the sphere. While existing work has focused on the isolated self-attention mechanism, revealing distinct clustering and diffusive regimes, this talk explores the effect of incorporating the perceptron. The central message is that the perceptron induces a localization mechanism: under suitable assumptions, stationary measures are forced to be singular, and in dimension two, they are purely atomic with finite support.

Consequently, even in the descent regime, where attention alone favors diffuse equilibria, the perceptron enforces clustered stationary states. Finally, I will present anti-concentration estimates showing that stable equilibria cannot overly concentrate mass within a small cluster.