On the Spectral Bias of Neural Networks

In this talk, I will introduce a phenomenon called "the spectral bias”, which shows that even though neural networks are quite capable of fitting random target functions with perfect performance, they are biased towards learning the “simpler” components of the target function earlier in the training. Precisely, we inspect the learning process through the lens of Fourier analysis to find that the lower frequencies of the target function are learned first, even when they are underrepresented in the spectrum of the latter. This observation leads to a clearer picture of the kinds of label noise neural networks are particularly vulnerable against, and to why early stopping is as effective as it is. Finally, we also look at the curious interplay between spectral bias and the shape of the data-manifold, which might explain the unreasonable effectiveness of positional embeddings (also known as Fourier features) in the implicit representation learning literature (i.e. Neural Radiance Fields (NeRF) and friends).