

Preprint 18/2021
On the Expected Complexity of Maxout Networks
Hanna Tseran and Guido Montúfar
Contact the author: Please use for correspondence this email.
Submission date: 12. Jul. 2021
Pages: 43
Bibtex
MSC-Numbers: 68T07
Keywords and phrases: linear regions of neural networks, maxout units, expected complexity, decision boundary, parameter initialisation
Download full preprint: PDF (11735 kB)
Link to arXiv: See the arXiv entry of this preprint.
Abstract:
Learning with neural networks relies on the complexity of the representable
functions, but more importantly, the particular assignment of typical
parameters to functions of different complexity. Taking the number of
activation regions as a complexity measure, recent works have shown that the
practical complexity of deep ReLU networks is often far from the theoretical
maximum. In this work we show that this phenomenon also occurs in networks with
maxout (multi-argument) activation functions and when considering the decision
boundaries in classification tasks. We also show that the parameter space has a
multitude of full-dimensional regions with widely different complexity, and
obtain nontrivial lower bounds on the expected complexity. Finally, we
investigate different parameter initialization procedures and show that they
can increase the speed of convergence in training.