Search

MiS Preprint Repository

We have decided to discontinue the publication of preprints on our preprint server as of 1 March 2024. The publication culture within mathematics has changed so much due to the rise of repositories such as ArXiV (www.arxiv.org) that we are encouraging all institute members to make their preprints available there. An institute's repository in its previous form is, therefore, unnecessary. The preprints published to date will remain available here, but we will not add any new preprints here.

MiS Preprint
73/2014

On the Number of Linear Regions of Deep Neural Networks

Guido Montúfar, Razvan Pascanu, Kyunghyun Cho and Yoshua Bengio

Abstract

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models compute functions that react equally to complicated patterns of different inputs. The compositional structure of these functions enables them to re-use pieces of computation exponentially often in terms of the network's depth. This paper investigates the complexity of such compositional maps and contributes new theoretical results regarding the advantage of depth for neural networks with piecewise linear activation functions. In particular, our analysis is not specific to a single family of models, and as an example, we employ it for rectifier and maxout networks. We improve complexity bounds from pre-existing work and investigate the behavior of units in higher layers.

Received:
Jul 29, 2014
Published:
Jul 30, 2014
MSC Codes:
82C32, 68R99
Keywords:
Deep learning, neural network, input space partition, rectifier, maxout

Related publications

inBook
2014 Repository Open Access
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho and Yoshua Bengio

On the number of linear regions of deep neural networks

In: NIPS 2014 : Proceedings of the 27th international conference on neural information processing systems - volume 2 ; Montreal, Quebec, Canada, December 8th-13th
Cambridge, MA : MIT Press, 2014. - pp. 2924-2932