When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
summary
The gist
RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning The paper shows that policy entropy remains…
In short
RLHF often causes flow-matching models to lose diversity during fine-tuning because policy entropy stays constant while perceptual diversity collapses due to mode-seeking gradients. The authors introduce perceptual entropy, a metric capturing this visual diversity, and propose two strategies—Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space—to preserve both quality and variety.
Key concepts
- Policy Entropy
- This measures the randomness or uncertainty in the model's policy during training. In this study, it remains fixed because of a pre-defined noise schedule. However, this constant entropy does not reflect actual visual diversity in the generated images.
- Perceptual Entropy (Hperc)
- A novel concept that quantifies diversity specifically in the visual (perceptual) space. It is calculated by mapping noised samples and policy means through a VAE decoder and a perceptual encoder. This metric reveals how much visual variety is lost during the RL process.
- Perceptual Entropy Constraint (PEC)
- A strategy that incorporates perceptual entropy into the reward signal during training. PEC encourages the model to explore a wider range of visual possibilities, thereby actively fighting against diversity collapse and improving the quality-diversity balance.
- Perceptual Constraints on Generation Space
- An alternative strategy that attempts to maintain diversity by aligning the collapsed perceptual space with a consistent VAE space. This ensures that the generated images remain diverse while adhering to established structural constraints.
Terminology used across episodes
This episode discusses
- When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy · Paper Radio
- Average-Reward Soft Actor-Critic
- Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation
- Forking Paths in Neural Text Generation
- Training Diffusion Models with Reinforcement Learning
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- Emerging Properties in Self-Supervised Vision Transformers
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Reasoning with Exploration: An Entropy Perspective
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints · Paper Radio
- TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
- GPT-4o System Card
- Towards Efficient Exact Optimization of Language Model Alignment
- Rethinking Entropy Regularization in Large Reasoning Models
- MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
- BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
- Flow Matching for Generative Modeling
- Flow-GRPO: Training Flow Matching Models via Online RL
The paper
When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy · Read on arXiv
Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang
Southeast University · Tencent Youtu Lab
RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning. In RL, diversity is often assumed to correlate with policy entropy, motivating entropy regularization. However, we show this intuition breaks in flow models: policy entropy remains constant, even while perceptual diversity collapses. We explain this mismatch both theoretically and empirically: the constant entropy arises from the fixed, pre-defined noise schedule, while the diversity collapse is driven by the mode-seeking nature of policy gradients. As a result, policy entropy fails to prevent the model from converging to a narrow high-reward region in the perceptual space. To this end, we introduce perceptual entropy that captures diversity in a perceptual space and maintains the property of standard entropy. Building upon this insight, we propose two entropy-regularized strategies, Perceptual Entropy Constraint and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve the quality. Experiments across two base models, neural and rule-based rewards, and three perceptual spaces demonstrate consistent gains in the quality-diversity trade-off; PEC achieves the best overall score of 0.734 (vs. baseline's 0.366); a complementary setting of PEC further reaches a diversity average of 0.989 (vs. baseline's 0.047). Our project page (https://xiaofeng-tan.github.io/projects/PEC) is publicly available.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "When Policy Entropy Constraint Fails".
Jane: Here's the gist of the paper. RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning The paper shows that policy entropy remains constant even while perceptual diversity collapses This mismatch is explained by the fixed noise schedule and the mode-seeking nature of policy gradients To address this, the authors introduce perceptual entropy to capture diversity in a perceptual space while maintaining standard entropy properties They propose two strategies, Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve quality The Paradox of Constant Entropy The study investigates the apparent paradox where policy entropy remains invariant during training while diversity collapses.
Tom: This occurs because the noised samples are generated by injecting noise with a pre-defined and fixed intensity into the model-predicted mean, resulting in a Gaussian distribution with fixed covariance The constant policy entropy is determined by the fixed noise scheduler and is independent of flow model parameters θ.
Jane: Conversely, diversity collapse stems from the mode-seeking nature of online RL, where the policy optimizes to concentrate probability distribution on a high-reward region in perceptual space This leads to the multi-peak pretrained policy distribution collapsing into a single-peak perceptual distribution.
Tom: Introducing Perceptual Entropy The authors introduce a novel entropy concept, perceptual entropy Hperc, which faithfully reflects diversity collapse in the perceptual space while remaining consistent with prior entropy-based methods This is defined by mapping noised samples and the old-policy mean into the pixel space via a VAE decoder and then mapping them into a perceptual space using a perceptual encoder The resulting perceptual entropy Hperc(pθ) quantifies diversity collapse in this new space This concept is shown to be analogous to the relationship observed in LLMs, R = −a exp(Hperc) + b.
Paper summary: Tom: So we're talking about this paper, "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy". Basically, they’re looking at how we use reinforcement learning to tune text to image models using flow matching. The big problem they point out is that when you do this, you often see the diversity collapse pretty fast after you fine-tune the model.
Jane: That sounds like a real headache for anyone trying to get varied results from these image generators, right? So what's the core claim here? What's this paper actually saying about why that happens and what they propose?
Tom: They show that policy entropy stays constant even when the actual visual diversity in the images collapses. That’s the paradox they’re tackling, and it seems to come from two places: first, there is a fixed noise schedule that determines the policy entropy which doesn't change with your model parameters theta, and second, because of how RL works online, the policy gradients are mode-seeking.
Lu: That makes sense from a representation learning standpoint. If the noise injection process itself is fixed by the schedule, then you can't expect that specific entropy measure to track changes in what’s actually happening visually in the output space.
Meng: So if we just rely on policy entropy to keep things diverse, it doesn't work because it’s stuck at a constant value while the images get all similar. What does this mean for us when we're trying to deploy these models?
Lalam: I've looked at the literature, and I can tell you that this paper suggests that we need a new way to measure diversity in the perceptual space itself, something that relates to how humans actually see things.
Jane: That sounds like they’re introducing something new. What is this new concept they use? How do they define this perceptual diversity?
Tom: They introduce something called perceptual entropy, which they call Hperc. It’s not the same as policy entropy; it maps both the noised samples and the old policy mean into a pixel space using a variational autoencoder decoder, and then maps that back into a perceptual space using another encoder.
Paper summary: Lu: So they are creating this new metric to specifically capture diversity in that visual space, rather than just relying on the fixed noise schedule of the flow model. This is analogous to how we see entropy working in other areas like LLMs, where the relationship between entropy and some kind of reward structure is defined by a specific formula.
Jane: It sounds like they are saying that this perceptual entropy Hperc(theta) actually tracks that diversity collapse, and it follows a known mathematical relationship, similar to R equals negative alpha exp of Hperc plus beta. That gives them a solid way to quantify the problem.
Meng: If they can measure this perceptual entropy, then what does that give us in terms of fixing the training process? Is it just an observation or is there something actionable?
Tom: They propose two specific strategies to fix this: Perceptual Entropy Constraint, or PEC, and Perceptual Constraints on Generation Space. The idea behind PEC is to bake this perceptual entropy directly into the reward signal.
Lu: So, by adding a term that encourages exploration of a broader perceptual space based on Hperc, they are trying to steer the policy away from just finding one sharp mode in the reward landscape. That’s like giving the model a better compass for where to look next.
Jane: And what about the second strategy, Perceptual Constraints on Generation Space? How does that fit in with PEC? Does it try to fix the space itself or guide the generation process?
Tom: The second one aims to align that collapsed perceptual space with something more stable, specifically the consistent variational autoencoder space. So they’re trying to keep the output distribution tethered to a consistent structure while allowing for that necessary diversity capture.
Lalam: From my side, I see this as a way to regularize the objective function by augmenting it with this perceptual regularization term, Omega of p perc theta. It's essentially adding a penalty or incentive based on how much perceptual diversity we are losing.
Meng: If they can get these results—they mention achieving an overall score of zero point seven three four when using PEC, compared to a baseline score of zero point three six six—that’s a huge jump in quality and diversity trade-off performance. But what about the diversity average?
Paper summary: Tom: They show that with PEC, they reach a diversity average of zero point nine eight nine, which is much better than the baseline's zero point zero four seven. So the paper suggests this method significantly improves both how good the images are and how varied they are across different prompts and domains.
Jane: It’s important to remember that they tested this across two different base models, one neural rewards and one rule-based rewards, and three different perceptual spaces. That shows it’s not just a fluke on one setup.
Lu: The point is that perceptual entropy gives us a reliable identifier for diversity in these flow-based models, which was missing before. It aligns with how we understand diversity mechanisms in other types of generative AI systems too.
Tom: So, to sum up this paper, "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy", the authors identified a mismatch between fixed policy entropy and actual perceptual diversity collapse in flow models. They then introduced perceptual entropy Hperc as a way to measure that true visual diversity.
Jane: And they proposed two ways to use this measurement—PEC and Perceptual Constraints on Generation Space—to encourage exploration and maintain quality without sacrificing variety. It seems like they’ve given us a better tool for controlling these complex generative systems than just the standard policy entropy term.
Meng: For deployment, if we can achieve that quality-diversity trade-off improvement, it means we can generate outputs that are both high quality and genuinely different from each other across a wide range of inputs. That’s something production systems need.
Lalam: I'm confident in the direction this takes because they built this concept on mapping samples into a pixel space first, which grounds the math in what we actually see. It provides a mechanism for improving quality without losing visual variety across prompts and domains.
Tom: So that’s the gist of it—a new metric, a new constraint strategy, leading to better quality and diversity in flow-based RLHF models by measuring perceptual entropy instead of just policy entropy. That should give you an idea of what they did with their research.
Conclusion: Tom: So we've been talking about how standard reinforcement learning often messes up visual diversity in these flow matching models, and this paper is titled "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy."
Jane: It sounds like they found a problem where the policy entropy stays the same even when the actual visual variety drops. So, what's their main fix?
Tom: They introduce this new idea called perceptual entropy, Hperc, which measures diversity directly in the pixel space using a VAE decoder and an encoder. It’s not just guessing; it’s mapping what we actually see into a metric that tracks the collapse.
Lu: What I find interesting is how they set up this new measurement because it relates to existing entropy methods for other models, like LLMs, where there's an equation linking the two.
Meng: So they have this new way to measure visual variety, and then they use that to build two strategies—PEC and Perceptual Constraints on Generation Space—to try and keep the outputs varied.
Jane: PEC seems to be about adding this perceptual entropy directly into the reward signal, which should encourage the AI policy to explore a wider range of visual possibilities.
Tom: Exactly, it’s like giving the model a better instruction set for what constitutes a good output based on perceptual diversity rather than just reward alone.
Lalam: I'm confident that this approach gives us a way to quantify that loss and build in constraints that actively fight the mode-seeking nature of the online RL process.
Jane: What does this actually mean for someone who just uses these image generators every day? Does it make the pictures they see more varied?
Tom: The experiments show a real jump, achieving a quality-diversity trade-off score of zero point seven three four when using PEC, which is much higher than the baseline scores we saw earlier.
Lu: And what they’re showing is that this method preserves visual diversity across different prompts and even different domains without making the image quality worse noticeably.
Meng: If these constraints work, it means we might be able to generate high-quality images that are genuinely different from each other, not just slightly varied versions of the same idea.
Jane: It seems they’ve given us a more reliable tool for controlling this complex area than just relying on the standard policy entropy term.
Tom: The authors show that this perceptual entropy provides an effective identifier of diversity, and it substantially improves the quality-diversity trade-off in flow based reinforcement learning.
Lu: The work demonstrates that PEC preserves visual diversity across prompts and domains without noticeably degrading image quality, which is a tough balance to hit.
Jane: This paper shows how we can move beyond just looking at policy entropy to understanding what’s happening visually in the generation process itself.
Tom: Now, they’ve shown this works on two different base models and three different perceptual spaces, so it looks like a pretty solid technique for flow based RLHF.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound