Recent advances in the analysis of the implicit bias of gradient descent on deep networks

The purpose of this talk is to highlight three recent directions in the study of implicit bias --- one of the current promising approaches to trying to develop a tight generalization theory for deep networks, one interwoven with optimization. The first direction is a warm-up with purely linear predictors: here, the implicit bias perspective gives the fastest known hard-margin SVM solver! The second direction is on the early training phase with shallow networks: here, implicit bias leads to good training and testing error, with not just narrow networks but also arbitrarily large ones. The talk concludes with deep networks, providing a variety of structural lemmas which capture foundational aspects of how weights evolve for any width and sufficiently large amounts of training.

Joint work with Ziwei Ji.

Bio:
Matus Telgarsky is an assistant professor at the University of Illinois, Urbana-Champaign, specializing in deep learning theory. He was fortunate to receive a PhD at UCSD under Sanjoy Dasgupta. Other highlights include: co-founding, in 2017, the Midwest ML Symposium (MMLS) with Po-Ling Loh; receiving a 2018 NSF CAREER award; organizing a Simons Insititute summer 2019 program on deep learning with Samy Bengio, Aleskander Madry, and Elchanan Mossel.