Workshop
A Sober Look at Neural Network Initializations
- Ingo Steinwart (Universität Stuttgart)
Abstract
Initializing the weights and the biases is a key part of the training process of a neural network. Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature.
In the first part of the talk, I will discuss some consequences of commonly used initialization strategies for vanilla DNNs with ReLU activations. Based on these insights I will then introduce an alternative initialization strategy, and finally I will present some large scale experiments assessing the quality of the new initialization strategy.
