Preprint 23/2010

Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines

Guido Montúfar and Nihat Ay

Contact the author: Please use for correspondence this email.
Submission date: 10. May. 2010
Pages: 14
published in: Neural computation, 23 (2011) 5, p. 1306-1319 
DOI number (of the published article): 10.1162/NECO_a_00113
Download full preprint: PDF (122 kB)

We improve recently published results about resources of Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) required to make them Universal Approximators. We show that any distribution p on the set formula7 of binary vectors of length n can be arbitrarily well approximated by an RBM with k-1 hidden units, where k is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of p. In important cases this number is half of the cardinality of the support set of p (given in Le Roux and Bengio, (2008)). We construct a DBN with formula19, formula21, hidden layers of width n that is capable of approximating any distribution on formula7 arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio, (2010).

18.10.2019, 02:14