MiS Preprint Repository

We consider the set of join probability distributions of $N$ binary random variables which can be written as a sum of $m$ distributions in the following form $p(x_1,\dots ,x_N)=\sum _{i=1}^m{\alpha }_i{f}_i(x_1,\dots ,x_N)$, where ${\alpha }_i\ge 0$, $\sum _{i=1}^m{\alpha }_i=1$, and the $f_i(x_1,\dots ,x_N)$ belong to some exponential family.

For our analysis we decompose the sample space into portions on which the mixture components $f_i$ can be chosen arbitrarily. We derive lower bounds on the number of mixture components from a given exponential family necessary to represent distributions with arbitrary correlations up to a certain order or to represent any distribution.

For instance, in the case where $f_i$ are independent distributions we show that every distribution $p$ on $\{0,1{\}}^N$ is contained in the mixture model whenever $m\ge 2^{N-1}$, and furthermore, that there are distributions which are not contained in the mixture model whenever $m<2^{N-1}$.