Transfer Learning for Diffusion Models

Diffusion models have quickly become a powerful tool in generative modeling, producing remarkably realistic synthetic data across various domains, including images, language, and video. In this talk, I’ll start by introducing the basic formulation of diffusion models and how they generate data through iterative denoising. Then, I’ll focus on a central question: How can we transfer a pre-trained diffusion model to a different data distribution? To address this, I’ll present a theoretical framework that analyzes the transfer process. We show that the optimal diffusion model for a target domain combines the capabilities of a source-domain pre-trained model with additional guidance. This insight leads to a practical and principled approach to transfer learning for diffusion models. Interestingly, our framework connects to ideas from reinforcement learning with human feedback (RLHF) and optimal transport, offering a broader perspective on aligning generative models with new domains.