On the Convergence and Transformation of the ELBO Objective to Entropy Sums

The variational lower bound (a.k.a. ELBO or free energy) is the central objective for many established as well as many novel algorithms for unsupervised learning. Different reformulations and generalizations of the ELBO have been used for different models including standard variational autoencoders (VAEs), beta-VAEs, diffusion models and many other probabilistic data models. Here we study a novel approach that allows for rewriting the ELBO as a sum of entropies. Given common probabilistic graphical models, we first show that the ELBO converges to an expression summing (positive and negative) entropy terms. The result applies under mild conditions, e.g., distributions defining the graphical model have to be in the exponential family. Secondly, we discuss entropy sums as a tool for the analysis of ELBO-based learning; and we will show how properties of entropies can be exploited for concrete models such as VAEs. Finally, we will present recent results showing how convergence to entropy sums can be used to transform ELBOs to entropy sum objectives, i.e., we will present example models for which entropy sums can directly serve as learning objectives. Numerical results of entropy-based analysis as well as entropy-based learning will support the presented theoretical results. The talk will close with a discussion of related research of my lab and of future work.

Work together with: Simon Damm, Dmytro Velychko, Jan Warnken, Zhenwen Dai, Dennis Forster, Asja Fischer