Next Courses

This course will introduce into the theory of kernels and associated Hilbert spaces (reproducing kernel Hilbert spaces, RKHS), which play an important role within mathematical learning theory. In Statistical Learning Theory and the theory of Support Vector Machines (SVM), they provide efficient ways to formalise and control the generalisation ability of learning systems, based on the structural risk minimisation principle. A closely related inductive principle comes from regularisation theory. Here, learning is interpreted as an ill-posed inverse problem, where kernels define appropriate regularisers of the problem.

Statistical learning theory (SLT) is a powerful framework to study the generalisation ability of learning machines (their performance in the context of new data). The corresponding bounds depend on certain capacity measures that quantify the expressivity of the underlying architecture (how many different functions it can represent).

Recent case studies have shown that the high performance of state-of-the-art learning machines, particularly deep neural networks, cannot be explained within the framework of classical SLT. On the one hand, such systems work extremely well in over-parametrised regimes where the classical bounds, being strongly dependent on the number of parameters, become uninformative. On the other hand, some evidence suggests that convergence and generalisation should be controlled by bounds that depend on some norm of the optimal solution, rather than the capacity of the architecture. The control of this norm is believed to be strongly dependent on the optimisation algorithm at hand.