The Physics behind Optimization and Generalization: A Symmetry-Irreversibility Framework of Learning

AI has become an empirical science. We are discovering more and more interesting phenomena in these AI models, but the organizing principles of modern AI have been unclear. In this seminar, I will discuss what I call the Symmetry-Irreversibility Framework (SIF), which leverages symmetry and irreversibility, two main concepts and tools from science in general and physics in particular, to analyze and understand the phenomenology of deep learning. I will discuss how interesting phenomena such as implicit sparsity, collapse, the edge of stability, and the more recently discovered Platonic representation hypothesis could be consequences of the model's hidden symmetries and the irreversibility of the training dynamics.