The geometry of the loss function of deep neural networks

The mathematical heart of deep learning is gradient descent on a loss function L. If gradient descent converges, it will converge to a critical point of L. Thus the geometry of the locus of critical points is of great interest. We will discuss what is known about the critical points of L, including dimension estimates and connectedness results.