Harmful Overfitting in Sobolev Spaces

A growing body of recent work has demonstrated the phenomenon of benign overfitting, in which an overparameterized machine learning model perfectly fits a noisy training dataset while still generalizing well. The typical setting of benign overfitting is where the input dimension is much larger than the size of the training set. In this talk, I will discuss overfitting in the opposite fixed input dimension regime. We consider functions in a Sobolev space $W^{k, p}(\mathbb{R}^d)$ which perfectly fit a noisy training dataset. Under these assumptions, an approximately norm-minimizing interpolator will exhibit harmful overfitting: even as sample size approaches infinity, the generalization error is bounded below by a constant.