Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks

Natural Gradient Descent (NGD) helps to accelerate the convergence of gradient descent dynamics, but it requires approximations in large-scale deep neural networks because of its high computational cost. Empirical studies have confirmed that some NGD methods with approximate Fisher information converge sufficiently fast in practice. In this talk, we reveal that, under specific conditions, NGD with approximate Fisher information achieves the same fast convergence to global minima as exact NGD. We consider deep neural networks in the infinite-width limit, and analyze the asymptotic training dynamics of NGD in the neural tangent kernel regime. The training dynamics with the approximate Fisher information are identical to those with the exact Fisher information in the function space, and they converge quickly. This result holds in layer-wise, K-FAC, and unit-wise approximations. All of these different approximations have an isotropic gradient in the function space, and this plays a fundamental role in achieving the same fast convergence in training.