Maximal Information Divergence from Statistical Models defined by Neural Networks

Guido Montúfar, Johannes Rauh, and Nihat Ay

Contact the author: Please use for correspondence this email.
Submission date: 04. Mar. 2013
Pages: 10
published in: Geometric science of information : first international conference, GSI 2013, Paris, France, August 28-30, 2013. Proceedings / F. Nielsen ... (eds.)
Berlin [u. a.] : Springer, 2013. - P. 759 - 766
(Lecture notes in computer science ; 8085) 
DOI number (of the published article): 10.1007/978-3-642-40020-9_85
Bibtex
MSC-Numbers: 94A17, 62B10
Keywords and phrases: neural network, exponential family, kullback-leibler divergence, multi-information
Download full preprint: PDF (572 kB)

Abstract:
We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naïve Bayes models, restricted Boltzmann machines, deep belief networks, and various classes of exponential families. We illustrate approaches to compute the maximal divergence from a given model starting from simple sub- or super-models. We give a new result for deep and narrow belief networks with finite-valued units.