Zhang, Xinjian, and Xiang, Wei (2025) Diffusion Preference Alignment via Attenuated Kullback–Leibler Regularization. Electronics, 14 (15). 2939.

Preview

PDF (Published Version) - Published Version Available under License Creative Commons Attribution. Download (3MB) | Preview

Abstract

Direct preference optimization (DPO) has been successfully applied to align large language models (LLMs) with human preferences. In recent years, DPO has also been used to improve the generation quality of text-to-image diffusion models. However, existing techniques often rely on a single type of reward model. They are also prone to overfitting to inaccurate reward signals. As a result, model quality cannot be continuously improved. To address these limitations, we propose xDPO. This method introduces a novel regularization approach that implicitly defines reward functions for both preferred and non-preferred samples. This design greatly enhances the flexibility of reward modeling. The experimental results show that, after fine-tuning Stable Diffusion v1.5, xDPO achieves significant improvements in human preference evaluations compared to previous DPO methods. It also improves training efficiency by approximately 1.5 times. Meanwhile, xDPO maintains image–text alignment performance that is comparable to the original model.

Actions (Repository Staff Only)

Item Control Page