Chameleon: Dynamic Format Adapter for Efficient Diffusion

arXiv:2609.33496 · cs.LG, cs.AI, cs.CV · Submitted 2026-09-27 · Read on arXiv

cs.LG, cs.AI, cs.CV

Submitted: 2026-09-27

Updated: 2026-09-27

Comments: Under Review at International Conference on Learning Representations 2027

License: http://creativecommons.org/licenses/by/4.0/

The gist: Post-training quantization (PTQ) is the standard way to run modern diffusion models on memory-constrained accelerators, yet every existing diffusion PTQ scheme fixes the in advance and only tunes the

Terminology

Abstract

Post-training quantization (PTQ) is the standard way to run modern diffusion models on memory-constrained accelerators, yet every existing diffusion PTQ scheme fixes the in advance and only tunes the scale, zero point, or per-layer bit-width. At a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs across weight channels, across layers, and along the diffusion timestep, where activation distributions slide from heavy-tailed and noise-dominated to tightly clustered and structured. We propose Chameleon, a PTQ framework that holds the bit-width fixed and treats the format itself as a discrete variable, chosen per weight channel and per (layer, timestep bucket) activation tensor. Activation formats come from INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8, selected ahead of time from two cheap statistics (empirical kurtosis and the closed-form diffusion SNR) and stored in a lookup table; weight formats come from INT8, MXINT8 at 8 bits or INT4, NF4, FP4 E2M1, MXINT4, MXFP4 at 4 bits, selected offline by reconstruction error. An architectural fork adapts the same selection layer to multi-step UNets, single-step distilled models, and Diffusion Transformers. Across SDXL, SDXL-Turbo, and PixArt- α on COCO-2014, Chameleon achieves the best FID in all six backbone times bit-width settings, with CLIP within 0.24 of the FP16 reference and the best of all quantized methods at W 4A 8.

Sources

Related papers