Searching for Small Expressive Neural Networks

The three pillars of machine learning research are expressivity, optimization, and generalization. Expressivity asks what functions a model can possibly represent before training begins. In the first half of the talk, this question will be formulated for polynomial neural networks using nonlinear algebra. In this setting, each architecture determines an algebraic variety whose dimension gives a measure of expressive power. In the second half, I will discuss joint work with Kevin Dao (UW Madison) on the search for small architectures with maximal expressivity, also known as minimal filling architectures. Our main result addresses a conjecture of Kileel, Trager, and Bruna, which predicted that such architectures should have a simple unimodal shape. Time permitting, I will discuss future directions for efficient neural network architectures.