When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "When Policy Entropy Constraint Fails".
Jane: Here's the gist of the paper. RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning The paper shows that policy entropy remains constant even while perceptual diversity collapses This mismatch is explained by the fixed noise schedule and the mode-seeking nature of policy gradients To address this, the authors introduce perceptual entropy to capture diversity in a perceptual space while maintaining standard entropy properties They propose two strategies, Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve quality The Paradox of Constant Entropy The study investigates the apparent paradox where policy entropy remains invariant during training while diversity collapses.
Tom: This occurs because the noised samples are generated by injecting noise with a pre-defined and fixed intensity into the model-predicted mean, resulting in a Gaussian distribution with fixed covariance The constant policy entropy is determined by the fixed noise scheduler and is independent of flow model parameters θ.
Jane: Conversely, diversity collapse stems from the mode-seeking nature of online RL, where the policy optimizes to concentrate probability distribution on a high-reward region in perceptual space This leads to the multi-peak pretrained policy distribution collapsing into a single-peak perceptual distribution.
Tom: Introducing Perceptual Entropy The authors introduce a novel entropy concept, perceptual entropy Hperc, which faithfully reflects diversity collapse in the perceptual space while remaining consistent with prior entropy-based methods This is defined by mapping noised samples and the old-policy mean into the pixel space via a VAE decoder and then mapping them into a perceptual space using a perceptual encoder The resulting perceptual entropy Hperc(pθ) quantifies diversity collapse in this new space This concept is shown to be analogous to the relationship observed in LLMs, R = −a exp(Hperc) + b.
Paper summary: Tom: So we're talking about this paper, "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy". Basically, they’re looking at how we use reinforcement learning to tune text to image models using flow matching. The big problem they point out is that when you do this, you often see the diversity collapse pretty fast after you fine-tune the model.
Jane: That sounds like a real headache for anyone trying to get varied results from these image generators, right? So what's the core claim here? What's this paper actually saying about why that happens and what they propose?
Tom: They show that policy entropy stays constant even when the actual visual diversity in the images collapses. That’s the paradox they’re tackling, and it seems to come from two places: first, there is a fixed noise schedule that determines the policy entropy which doesn't change with your model parameters theta, and second, because of how RL works online, the policy gradients are mode-seeking.
Lu: That makes sense from a representation learning standpoint. If the noise injection process itself is fixed by the schedule, then you can't expect that specific entropy measure to track changes in what’s actually happening visually in the output space.
Meng: So if we just rely on policy entropy to keep things diverse, it doesn't work because it’s stuck at a constant value while the images get all similar. What does this mean for us when we're trying to deploy these models?
Lalam: I've looked at the literature, and I can tell you that this paper suggests that we need a new way to measure diversity in the perceptual space itself, something that relates to how humans actually see things.
Jane: That sounds like they’re introducing something new. What is this new concept they use? How do they define this perceptual diversity?
Tom: They introduce something called perceptual entropy, which they call Hperc. It’s not the same as policy entropy; it maps both the noised samples and the old policy mean into a pixel space using a variational autoencoder decoder, and then maps that back into a perceptual space using another encoder.
Paper summary: Lu: So they are creating this new metric to specifically capture diversity in that visual space, rather than just relying on the fixed noise schedule of the flow model. This is analogous to how we see entropy working in other areas like LLMs, where the relationship between entropy and some kind of reward structure is defined by a specific formula.
Jane: It sounds like they are saying that this perceptual entropy Hperc(theta) actually tracks that diversity collapse, and it follows a known mathematical relationship, similar to R equals negative alpha exp of Hperc plus beta. That gives them a solid way to quantify the problem.
Meng: If they can measure this perceptual entropy, then what does that give us in terms of fixing the training process? Is it just an observation or is there something actionable?
Tom: They propose two specific strategies to fix this: Perceptual Entropy Constraint, or PEC, and Perceptual Constraints on Generation Space. The idea behind PEC is to bake this perceptual entropy directly into the reward signal.
Lu: So, by adding a term that encourages exploration of a broader perceptual space based on Hperc, they are trying to steer the policy away from just finding one sharp mode in the reward landscape. That’s like giving the model a better compass for where to look next.
Jane: And what about the second strategy, Perceptual Constraints on Generation Space? How does that fit in with PEC? Does it try to fix the space itself or guide the generation process?
Tom: The second one aims to align that collapsed perceptual space with something more stable, specifically the consistent variational autoencoder space. So they’re trying to keep the output distribution tethered to a consistent structure while allowing for that necessary diversity capture.
Lalam: From my side, I see this as a way to regularize the objective function by augmenting it with this perceptual regularization term, Omega of p perc theta. It's essentially adding a penalty or incentive based on how much perceptual diversity we are losing.
Meng: If they can get these results—they mention achieving an overall score of zero point seven three four when using PEC, compared to a baseline score of zero point three six six—that’s a huge jump in quality and diversity trade-off performance. But what about the diversity average?
Paper summary: Tom: They show that with PEC, they reach a diversity average of zero point nine eight nine, which is much better than the baseline's zero point zero four seven. So the paper suggests this method significantly improves both how good the images are and how varied they are across different prompts and domains.
Jane: It’s important to remember that they tested this across two different base models, one neural rewards and one rule-based rewards, and three different perceptual spaces. That shows it’s not just a fluke on one setup.
Lu: The point is that perceptual entropy gives us a reliable identifier for diversity in these flow-based models, which was missing before. It aligns with how we understand diversity mechanisms in other types of generative AI systems too.
Tom: So, to sum up this paper, "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy", the authors identified a mismatch between fixed policy entropy and actual perceptual diversity collapse in flow models. They then introduced perceptual entropy Hperc as a way to measure that true visual diversity.
Jane: And they proposed two ways to use this measurement—PEC and Perceptual Constraints on Generation Space—to encourage exploration and maintain quality without sacrificing variety. It seems like they’ve given us a better tool for controlling these complex generative systems than just the standard policy entropy term.
Meng: For deployment, if we can achieve that quality-diversity trade-off improvement, it means we can generate outputs that are both high quality and genuinely different from each other across a wide range of inputs. That’s something production systems need.
Lalam: I'm confident in the direction this takes because they built this concept on mapping samples into a pixel space first, which grounds the math in what we actually see. It provides a mechanism for improving quality without losing visual variety across prompts and domains.
Tom: So that’s the gist of it—a new metric, a new constraint strategy, leading to better quality and diversity in flow-based RLHF models by measuring perceptual entropy instead of just policy entropy. That should give you an idea of what they did with their research.
Conclusion: Tom: So we've been talking about how standard reinforcement learning often messes up visual diversity in these flow matching models, and this paper is titled "When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy."
Jane: It sounds like they found a problem where the policy entropy stays the same even when the actual visual variety drops. So, what's their main fix?
Tom: They introduce this new idea called perceptual entropy, Hperc, which measures diversity directly in the pixel space using a VAE decoder and an encoder. It’s not just guessing; it’s mapping what we actually see into a metric that tracks the collapse.
Lu: What I find interesting is how they set up this new measurement because it relates to existing entropy methods for other models, like LLMs, where there's an equation linking the two.
Meng: So they have this new way to measure visual variety, and then they use that to build two strategies—PEC and Perceptual Constraints on Generation Space—to try and keep the outputs varied.
Jane: PEC seems to be about adding this perceptual entropy directly into the reward signal, which should encourage the AI policy to explore a wider range of visual possibilities.
Tom: Exactly, it’s like giving the model a better instruction set for what constitutes a good output based on perceptual diversity rather than just reward alone.
Lalam: I'm confident that this approach gives us a way to quantify that loss and build in constraints that actively fight the mode-seeking nature of the online RL process.
Jane: What does this actually mean for someone who just uses these image generators every day? Does it make the pictures they see more varied?
Tom: The experiments show a real jump, achieving a quality-diversity trade-off score of zero point seven three four when using PEC, which is much higher than the baseline scores we saw earlier.
Lu: And what they’re showing is that this method preserves visual diversity across different prompts and even different domains without making the image quality worse noticeably.
Meng: If these constraints work, it means we might be able to generate high-quality images that are genuinely different from each other, not just slightly varied versions of the same idea.
Jane: It seems they’ve given us a more reliable tool for controlling this complex area than just relying on the standard policy entropy term.
Tom: The authors show that this perceptual entropy provides an effective identifier of diversity, and it substantially improves the quality-diversity trade-off in flow based reinforcement learning.
Lu: The work demonstrates that PEC preserves visual diversity across prompts and domains without noticeably degrading image quality, which is a tough balance to hit.
Jane: This paper shows how we can move beyond just looking at policy entropy to understanding what’s happening visually in the generation process itself.
Tom: Now, they’ve shown this works on two different base models and three different perceptual spaces, so it looks like a pretty solid technique for flow based RLHF.
Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang
Southeast University · Tencent Youtu Lab
cs.CV
Submitted: 2026-05-12
Updated: 2026-11-04
Code: https://github.com/discus0434/aesthetic-predictor-v2-5
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning The paper shows that policy entropy remains…
Key concepts
- Policy Entropy
- This measures the randomness or uncertainty in the model's policy during training. In this study, it remains fixed because of a pre-defined noise schedule. However, this constant entropy does not reflect actual visual diversity in the generated images.
- Perceptual Entropy (Hperc)
- A novel concept that quantifies diversity specifically in the visual (perceptual) space. It is calculated by mapping noised samples and policy means through a VAE decoder and a perceptual encoder. This metric reveals how much visual variety is lost during the RL process.
- Perceptual Entropy Constraint (PEC)
- A strategy that incorporates perceptual entropy into the reward signal during training. PEC encourages the model to explore a wider range of visual possibilities, thereby actively fighting against diversity collapse and improving the quality-diversity balance.
- Perceptual Constraints on Generation Space
- An alternative strategy that attempts to maintain diversity by aligning the collapsed perceptual space with a consistent VAE space. This ensures that the generated images remain diverse while adhering to established structural constraints.
Terminology
Summary
RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning The paper shows that policy entropy remains constant even while perceptual diversity collapses This mismatch is explained by the fixed noise schedule and the mode-seeking nature of policy gradients To address this, the authors introduce perceptual entropy to capture diversity in a perceptual space while maintaining standard entropy properties They propose two strategies, Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve quality
The Paradox of Constant Entropy
The study investigates the apparent paradox where policy entropy remains invariant during training while diversity collapses. This occurs because the noised samples are generated by injecting noise with a pre-defined and fixed intensity into the model-predicted mean, resulting in a Gaussian distribution with fixed covariance The constant policy entropy is determined by the fixed noise scheduler and is independent of flow model parameters θ. p. 1 (1) For the
first question, we show that the noised samples xt are generated by injecting noise with a pre-defined
and fixed intensity into the model-predicted mean µθ(xt), resulting in a Gaussian distribution with
fixed covariance. Hence, both likelihoods and policy entropy are determined by the fixed noise
scheduler and are independent of the flow model parameters θ. (2) For the second question, we find
that this paradox stems from the mode-seeking nature of online RL, where the policy is optimized
to concentrate probability distribution on high-reward region in the reward space. In this case, the
multi-peak pretrained policy distribution may collapse into a single-peak perceptual distribution.Where the paper says it
Where the paper says it
Introducing Perceptual Entropy
The authors introduce a novel entropy concept, perceptual entropy Hperc, which faithfully reflects diversity collapse in the perceptual space while remaining consistent with prior entropy-based methods This is defined by mapping noised samples and the old-policy mean into the pixel space via a VAE decoder and then mapping them into a perceptual space using a perceptual encoder The resulting perceptual entropy Hperc(pθ) quantifies diversity collapse in this new space This concept is shown to be analogous to the relationship observed in LLMs, R = −a exp(Hperc) + b.
Entropy-Regularized Strategies
Building on perceptual entropy, two strategies are proposed: Perceptual Entropy Constraint (PEC) and Perceptual Constraints on Generation Space. PEC incorporates perceptual entropy into the reward signal to encourage exploration of a broader perceptual space The alternative, Perceptual Constraints on Generation Space, aims to align the collapsed perceptual space with the consistent VAE space to maintain diversity. p. 1 Building on this concept, we further reveal an empirical entropy-based reward for flow models,
R = −a exp(Hperc) + b, which is analogous to relationships observed in LLMs [12]. Furthermore,
building on the above analysis, we propose two entropy-regularized strategies to mitigate diversity
collapse: Perceptual Entropy Constraint (PEC), which incorporates perceptual entropy into the
reward signal, and Perceptual Constraints on Generation Space, which aim to align the collapsed
perceptual space with the consistent VAE space to maintain diversity.Where the paper says it
Empirical Results and Gains
Experiments across two base models, neural and rule-based rewards, and three perceptual spaces demonstrate consistent gains in the quality-diversity trade-off PEC achieves the best overall score of 0.734 (vs. baseline’s 0. p. 0 Experiments across two base models, neural
and rule-based rewards, and three perceptual spaces demonstrate consistent gains
in the quality-diversity trade-off; PEC achieves the best overall score of 0. baseline’s 0. Experiments across two base models, neural
and rule-based rewards, and three perceptual spaces demonstrate consistent gains
in the quality-diversity trade-off; PEC achieves the best overall score of 0. baseline’s 0. baseline’s 0. 366); a complementary setting of PEC further reaches a diversity
average of 0. dev [33] and SD3. 5-M [18] using neural and rule-based
rewards, alongside PickScore [32], DINO [8], and CLIP [48] perceptual encoders, demonstrate that
perceptual entropy improves the quality-diversity trade-off: PEC achieves the best overall score of
0.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Conclusion and Impact
In conclusion, perceptual entropy provides an effective identifier of diversity, aligning with existing research for LLMs This framework substantially improves the quality-diversity trade-off in flow-based RLHF The work demonstrates that PEC preserves visual diversity across prompts and domains without noticeably degrading image quality The method is a fine-tuning technique applied on top of existing models, and it is noted that improved image generation quality and diversity may lower the barrier to producing misleading or harmful visual content.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Perceptual Entropy Constraint (PEC) can be used to maximize perceptual entropy directly in reward optimization by setting Omega equal to Hperc(pθ), which encourages exploration of a broader perceptual space while aiming for high quality.
-
Perceptual Constraints on Generation Space (PCVAE) can be used to align the collapsed perceptual space with the consistent VAE space by maximizing the negative KL divergence between pperc and pvae, maintaining diversity.
-
Perceptual Reward Shaping can be employed to stabilize training dynamics by incorporating perceptual constraints directly into the reward signal, ensuring that
perceptual and task rewards are jointly normalized within the same GRPO rollout group.
-
Covariance-Based Sample Identification allows for selective suppression of entropy-reducing updates by identifying transitions with high perceptual covariance, preventing
the most aggressive entropy-reducing updates while preserving normal gradient flow on the majority of the batch.
-
Entropy-Aware Constraint via KL Regularization applies a reverse KL penalty to samples ranked by perceptual covariance, which constrains
the magnitude of entropy-reducing updates
on transitions thatmost strongly contribute to perceptual entropy decrease.
Abstract
RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning. In RL, diversity is often assumed to correlate with policy entropy, motivating entropy regularization. However, we show this intuition breaks in flow models: policy entropy remains constant, even while perceptual diversity collapses. We explain this mismatch both theoretically and empirically: the constant entropy arises from the fixed, pre-defined noise schedule, while the diversity collapse is driven by the mode-seeking nature of policy gradients. As a result, policy entropy fails to prevent the model from converging to a narrow high-reward region in the perceptual space. To this end, we introduce perceptual entropy that captures diversity in a perceptual space and maintains the property of standard entropy. Building upon this insight, we propose two entropy-regularized strategies, Perceptual Entropy Constraint and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve the quality. Experiments across two base models, neural and rule-based rewards, and three perceptual spaces demonstrate consistent gains in the quality-diversity trade-off; PEC achieves the best overall score of 0.734 (vs. baseline's 0.366); a complementary setting of PEC further reaches a diversity average of 0.989 (vs. baseline's 0.047). Our project page (https://xiaofeng-tan.github.io/projects/PEC) is publicly available.
Sources
- Average-Reward Soft Actor-Critic
- Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation
- Forking Paths in Neural Text Generation
- Training Diffusion Models with Reinforcement Learning
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- Emerging Properties in Self-Supervised Vision Transformers
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- Reasoning with Exploration: An Entropy Perspective
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints
- TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
- GPT-4o System Card
- Towards Efficient Exact Optimization of Language Model Alignment
- Rethinking Entropy Regularization in Large Reasoning Models
- MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
- BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
- Flow Matching for Generative Modeling
- Flow-GRPO: Training Flow Matching Models via Online RL
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs