We have decided to discontinue the publication of preprints on our preprint server as of 1 March 2024. The publication culture within mathematics has changed so much due to the rise of repositories such as ArXiV (www.arxiv.org) that we are encouraging all institute members to make their preprints available there. An institute's repository in its previous form is, therefore, unnecessary. The preprints published to date will remain available here, but we will not add any new preprints here.
Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and when considering the decision boundaries in classification tasks. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity. Finally, we investigate different parameter initialization procedures and show that they can increase the speed of convergence in training.
linear regions of neural networks, maxout units, expected complexity, decision boundary, parameter initialisation
Related publications
inBook
2021
Repository Open Access
Hanna Tseran and Guido Montúfar
On the expected complexity of maxout networks
In: Advances in neural information processing systems 34 : NeurIPS 2021 ; annual conference on neural information processing systems 2021, December 6-14, 2021, virtual / Marc'Aurelio Ranzato (ed.) [S. L.] : NeurIPS, 2021. - pp. 28995-29008
