Delve into the future of research at MiS with our preprint repository. Our scientists are making groundbreaking discoveries and sharing their latest findings before they are published. Explore repository to stay up-to-date on the newest developments and breakthroughs.
MiS Preprint
23/2010
Refinements of Universal Approximation Results for Deep Belief Networks and Restricted Boltzmann Machines
Guido Montúfar and Nihat Ay
Abstract
We improve recently published results about resources of Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) required to make them Universal Approximators. We show that any distribution $p$ on the set $\{0,1\}^n$ of binary vectors of length $n$ can be arbitrarily well approximated by an RBM with $k-1$ hidden units, where $k$ is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of $p$. In important cases this number is half of the cardinality of the support set of $p$ (given in Le Roux and Bengio, (2008)).
We construct a DBN with ${2^n}/{2(n-b)}$, $b\sim \log n$, hidden layers of width $n$ that is capable of approximating any distribution on $\{0,1\}^n$ arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio, (2010).
