Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL".
Jane: Here's the gist of the paper. Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment but often introduces visual hallucinations like over-optimized details and semantic misalignment.
Tom: This work proposes ConsistentRFT, a general framework to mitigate these hallucinations by balancing global semantics and local details through Dynamic Granularity Rollout (DGR) and Consistent Policy Gradient Optimization (CPGO).
Jane: The Core Problem in RFT Existing RFT methods often fail because they suffer from two core issues stemming from exploration and exploitation: limited exploration during stochastic differential equation (SDE) rollouts, leading to an over-emphasis on local details at the expense of global semantics.
Tom: Furthermore, the trajectory imitation process inherent in policy gradient methods distorts the model’s foundational vector field and its cross-step consistency.
Title and authors: Tom: So we're looking at this paper, "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL," and it looks like they’re tackling a big problem. They’re saying that Reinforcement Fine-Tuning on flow models often introduces visual hallucinations, which are like the model adding details that aren't there or getting stuck in weird optimization loops.
Jane: Exactly. The naive question is, why does this happen in the first place? Is it just a flaw in how we fine-tune these generative models?
Lu: It seems like they’re addressing the exploration and exploitation trade-off that happens during the rollouts, where existing methods tend to focus either too much on fine details or too much on broad concepts, both of which lead to those visual artifacts.
Meng: From a deployment standpoint, if we keep seeing these hallucinations in production models, it’s a headache because the output isn't reliable for our users. What does this paper actually propose to fix that instability?
Tom: Well, they introduce two main ideas: Dynamic Granularity Rollout and Consistent Policy Gradient Optimization. They want to balance the global meaning of the image with the fine details we see in it.
Jane: Dynamic Granularity Rollout sounds complicated. Can you explain how that helps? How does it dynamically schedule different noise sources?
Lu: It uses two parts: an inter-group rollout and an intra-group rollout. The idea is to steer the model toward fine or coarse granularity using different noise schedules depending on where it is in the process, which should reduce those hallucinations we saw in existing methods like DanceGRPO and Flow-GRPO.
Meng: If I’m running a production pipeline, I need to know if this adds significant computational overhead. Does this dynamic scheduling make it too slow for real-time inference?
Tom: It does have a cost, but the point is that they are trying to reduce the need for excessive iteration counts. They show in their experiments that their method can manage things across iterations like zero, forty, eighty, one hundred twenty and one hundred sixty.
Jane: So if we look at what they found in Figure two they see these hallucinations when fine-tuning via reward signals where the reward model is trying to boost the in-domain reward by injecting irrelevant details <ref:2602.03425#pg1>.
Lu: That points to a constrained exploration domain, which is something the paper discusses because it suggests existing methods learn a shortcut by boosting that reward signal with those irrelevant details.
Meng: So they’re suggesting we stop letting the reward function just blindly push for detail enhancement? What's the mechanism behind that suggestion?
Tom: That brings us to Consistent Policy Gradient Optimization, or CPGO. This is meant to fix the problem of cross-step inconsistency that happens when you imitate a trajectory using policy gradients.
Title and authors: Jane: How does this consistency thing actually work on a flow model trajectory? It sounds like it’s about aligning the current prediction with something from the previous step.
Lu: It enforces consistency in two ways: first, they try to imitate a model that already has better consistency at step t minus one. Second, they use temporal self-consistency by aligning what happens at step t with what happened at step t minus one.
Meng: Aligning with the previous state sounds like it’s adding complexity to the training loop. Does this mean more data or more complex loss calculations are needed?
Tom: It involves a specific objective function, JCPGO theta G equals negative E over U of tau and zero h of the current policy ODE theta and the old model's ODE theta old at step t minus one. It’s a way to keep the flow-model consistency through single-step prediction fidelity.
Jane: If we think about what this means for someone just watching, does it translate into a cleaner image for them?
Lu: The theoretical justification connects this CPGO to consistency models, showing that minimizing that CPGO objective encourages the current model to maintain consistency with the previous model across the entire denoising trajectory. When minimized, the prediction error between the current and old models falls with the square of the sampling interval, which gets smaller when you have finer time steps.
Meng: So if I were deploying this, I’d need to be careful about how much precision I set for those time steps because of that scaling law they mentioned. That’s a concrete operational detail we need to worry about at scale.
Tom: Exactly. And when we look at the results, they show that ConsistentRFT significantly reduces visual hallucinations by an average of forty-nine percent for low-level hallucinations and thirty-eight percent for high-level perceptual ones compared to other RFT methods.
Jane: That reduction sounds substantial, especially for those high-level perceptual artifacts that are hard to see but ruin the final quality. What about generalization? Does it just fix the hallucination problem or does it improve how well the model handles prompts it hasn't seen before?
Lu: The paper shows that ConsistentRFT outperforms other RFT methods on out-of-domain metrics by showing an improvement of five point one percent over the baseline’s decrease of negative zero point four percent on FLUXone.dev. That suggests it actually improves generalization across different prompts and domains.
Meng: A five point one percent improvement is modest, but if it means we don't need to spend extra time curating domain-specific datasets to avoid these visual issues, that’s a win for deployment pipelines.
Title and authors: Tom: And the Visual Hallucination Evaluator confirms this. It shows artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO. That gives us solid empirical evidence beyond just the model's internal metrics.
Jane: So, if we put it all together, what’s the big picture for people who just use these models? What does this paper actually change for them when they go to generate an image?
Lu: The dual-grained approach resolves that trade-off between optimizing local details and maintaining global semantics. They hit competitive scores on metrics like HPS and CLIP by finding that balance.
Meng: So, the method isn't just about reducing noise; it’s about getting a better quality image for the user at a lower cost than we might think because it’s more stable during training.
Tom: And CPGO is what ensures that stability carries through the entire generation process, not just in one step. It’s about making sure the model stays on track from start to finish.
Jane: So, to wrap up this paper on "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL," it seems they've managed to introduce a unified framework that addresses both the exploration issues and the consistency issues separately.
Lu: It’s a unified plug-in that consistently improves DPO, GRPO, and DDPO in the online setting. The core idea is balancing global semantics with local details through dynamic scheduling and enforcing trajectory consistency with single-step prediction fidelity.
Meng: I just think we need to keep an eye on how they handle the practical tuning parameters they suggest, like that coarse perception timestep being twelve by default. That’s where the real engineering work will be needed.
Tom: Right, so the reduction in visual hallucinations up to forty-nine percent and that five point one percent gain in out-of-domain performance on FLUXone.dev are the numbers we have for this framework.
Jane: It’s a solid piece of work because it tackles those hallucinations directly and shows measurable gains across different evaluation metrics, not just theoretical ones.
Lu: The paper's main point is that you can get better quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning.
Meng: So, for deployment, it means we move away from relying on one-size-fits-all RFT settings and instead use this dual-grained approach to control the trade-off.
Tom: That’s the operational implication: intentional control over those two competing forces during training is what leads to better final outputs.
Jane: It gives us a clearer path forward for improving flow models, showing that balancing local detail and global semantics is achievable with these specific optimization techniques.
The paper's summary: Tom: So, we just talked about how Reinforcement Fine-Tuning on flow models usually makes them hallucinate visual details or get weirdly stuck in optimization loops, and now we’re looking at this paper called "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL."
Jane: Right. So, what they’re saying is that instead of just trying to fix the whole process at once, they have two different strategies working together. The first strategy is Dynamic Granularity Rollout, which basically lets the model decide when to focus on fine details and when it should look at the big picture.
Lu: Exactly. They use this dynamic scheduling of noise sources, splitting it into an inter-group rollout and an intra-group rollout, so they can steer the optimization toward fine or coarse granularity depending on where they are in the process.
Meng: From my side, I’m thinking about deployment cost here. Does this constant changing of noise schedules make the training time explode?
Tom: It does add computational complexity, but the goal is to reduce how many iterations you need to run for a decent result. They show it can handle rollouts across zero up to one hundred sixty steps.
Jane: And then they pair that with something called Consistent Policy Gradient Optimization, or CPGO. This part is all about fixing the inconsistency that happens when the model imitates a previous trajectory.
Lu: CPGO enforces consistency by aligning the current policy with what the old model predicted at step minus one, and also by making sure step t aligns with step t minus one. It keeps that flow-model consistency through single-step prediction fidelity.
Meng: So it’s like they’re checking its work at every single tiny moment during the generation process to make sure it hasn't drifted too far off course?
Tom: Precisely, and the theory behind this is pretty neat. Minimizing that CPGO objective actually shows that the error between the current model and the old model falls with the square of your sampling interval, which gets smaller when you use finer time steps.
Jane: That scaling law is important because it tells us exactly how much precision we need to set for those time steps if we want to see that benefit in practice.
Lu: And looking at the numbers, they show consistent RFT methods are cutting visual hallucinations by forty-nine percent for low-level stuff, and thirty-eight percent for high-level perceptual issues.
Tom: That’s a big drop compared to what DanceGRPO or MixGRPO were doing on their own. But wait, there’s more. They also showed they improve out-of-domain performance by five point one percent on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.
Jane: So it’s not just about making the generated image look better; it actually helps the model generalize better to prompts it hasn't seen before.
Meng: A five point one percent gain is small, but if you aren’t spending all your time building domain-specific datasets just to avoid these visual artifacts, that adds up in production pipelines.
Tom: And they back up those claims with the Visual Hallucination Evaluator, which shows artifact scores reduced by thirty-eight percent and twenty-four percent compared to those other methods. That’s solid empirical evidence.
Jane: It seems like the big picture here is that you can get a more stable, higher quality output by being intentional about how you balance local detail optimization against maintaining global semantics throughout the entire training run.
Lu: This framework acts as a unified plug-in that improves different reinforcement learning methods like DPO and GRPO in an online setting.
Tom: It’s about using this dual-grained approach to control those two competing forces during training, which leads to better final outputs for the user.
The paper's improvements: Tom: So, we’ve covered how they fix hallucinations using Dynamic Granularity Rollout and Consistent Policy Gradient Optimization; now we look at what these specific methods actually change for the model during training.
Jane: What they suggest is that instead of just tweaking one thing, you get two things working in tandem to control the optimization process. It’s like having a steering wheel that can smoothly shift between fine adjustments and broad strokes as you drive.
Lu: Exactly. The Dynamic Granularity Rollout means the model gets to decide whether it should be focusing on very small details or the overall shape of the image at any given moment during its journey through noise.
Meng: From a deployment angle, that sounds like a lot of math to set up during pre-training, but if it stabilizes the output quality so much, maybe it’s worth the overhead.
Tom: The results show that this balance resolves the trade-off between focusing on local details and keeping track of global semantics. They hit competitive scores on things like HPS and CLIP by finding that sweet spot between those two extremes.
Jane: And then they add CPGO to make sure that whatever the model decides to do, it stays consistent with its previous state along the way. It’s about maintaining a stable trajectory even when it's making big changes in detail focus.
Lu: The theoretical justification confirms that minimizing that CPGO objective forces the current model to stay consistent with the prior model across all those steps in the denoising process.
Tom: And when you look at the hard metrics, they report average reductions of forty-nine percent for low-level hallucinations and thirty-eight percent for high-level perceptual ones compared to existing RFT methods.
Jane: That’s a huge drop in visual noise, which is what people actually see when they look at the final generated image.
Meng: So that means we can get much cleaner results without having to manually filter out those weird artifacts in post-processing, which saves time on our end.
Lu: Plus, they showed an improvement of five point one percent on out-of-domain metrics on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.
Tom: That generalization gain is actually pretty significant for a model that’s supposed to be useful across many different types of prompts and scenes.
Jane: It means the model learns the core concepts better, not just how to add distracting textures that look good in one specific context.
Lu: The paper also confirmed this with the Visual Hallucination Evaluator, which showed artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO.
Tom: So, to sum up these improvements, you get a model that’s not just better at one thing but is more stable across the whole generation process while keeping those hallucinations in check.
Jane: This framework serves as a unified plug-in for other reinforcement learning techniques like DPO and GRPO when you’re working online.
Lu: The real implication is that you can get higher quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning, instead of just letting it happen randomly.
Meng: So the operational takeaway for me is that we move away from using one-size-fits-all settings and start using this dual-grained control to manage those competing forces.
Conclusion: Tom: So we’ve covered how they fix hallucinations using Dynamic Granularity Rollout and Consistent Policy Gradient Optimization; now we look at what these specific methods actually change for the model during training.
Jane: What they suggest is that instead of just tweaking one thing, you get two things working in tandem to control the optimization process. It’s like having a steering wheel that can smoothly shift between fine adjustments and broad strokes as you drive.
Lu: Exactly. The Dynamic Granularity Rollout means the model gets to decide whether it should be focusing on very small details or the overall shape of the image at any given moment during its journey through noise.
Meng: From my side, I’m thinking about deployment cost here. Does this constant changing of noise schedules make the training time explode?
Tom: It does add computational complexity, but the goal is to reduce how many iterations you need to run for a decent result. They show it can handle rollouts across zero up to one hundred sixty steps.
Jane: And then they pair that with something called Consistent Policy Gradient Optimization, or CPGO. This part is all about fixing the inconsistency that happens when the model imitates a previous trajectory.
Lu: CPGO enforces consistency by aligning the current policy with what the old model predicted at step minus one, and also by making sure step t aligns with step t minus one. It keeps that flow-model consistency through single-step prediction fidelity.
Meng: So it’s like they’re checking its work at every single tiny moment during the generation process to make sure it hasn't drifted too far off course?
Tom: Precisely, and the theory behind this is pretty neat. Minimizing that CPGO objective actually shows that the error between the current model and the old model falls with the square of your sampling interval, which gets smaller when you use finer time steps.
Jane: That scaling law is important because it tells us exactly how much precision we need to set for those time steps if we want to see that benefit in practice.
Lu: And looking at the numbers, they show consistent RFT methods are cutting visual hallucinations by forty-nine percent for low-level stuff, and thirty-eight percent for high-level perceptual ones compared to existing RFT methods.
Tom: That’s a big drop compared to what DanceGRPO or MixGRPO were doing on their own. But wait, there’s more. They also showed they improve out-of-domain performance by five point one percent on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.
Jane: So it’s not just about making the generated image look better; it actually helps the model generalize better to prompts it hasn't seen before.
Meng: A five point one percent gain is small, but if you aren’t spending all your time building domain-specific datasets just to avoid these visual artifacts, that adds up in production pipelines.
Lu: The paper also confirmed this with the Visual Hallucination Evaluator, which showed artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO.
Tom: So, to sum up these improvements, you get a model that’s not just better at one thing but is more stable across the whole generation process while keeping those hallucinations in check.
Jane: This framework serves as a unified plug-in for other reinforcement learning techniques like DPO and GRPO when you’re working online.
Lu: The real implication is that you can get higher quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning, instead of just letting it happen randomly.
Meng: So the operational takeaway for me is that we move away from using one-size-fits-all settings and start using this dual-grained control to manage those competing forces.
Tom: Indeed, it’s about intentional control over those two competing forces during training that leads to better final outputs.
Jane: So, in the end, the paper "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL" shows a clear path toward more reliable generative models.
Lu: It gives us a concrete method to control the trade-off between local detail and global semantics effectively in flow model training.
Meng: I'm curious if they explore how this scales when we move from standard images to, say, video generation, because that’s where things get messy fast.
Lalam: I am confident this result is sound; the mechanism for consistency control is very robust across the tested configurations.
Xiaofeng Tan, Jun Liu, Yuanting Fan, Bin-Bin Gao, Xi Jiang, Xiaochen Chen, Jinlong Peng, Chengjie Wang, Hongsong Wang
Southeast University · Southern University of Science and Technology Tencent Youtu Lab
cs.CV
Submitted: 2026-02-03
Updated: 2026-10-31
Code: https://github.com/black-forest-labs/flux
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 100/100
The gist: Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment but often introduces visual hallucinations like over-optimized details and semantic misalignment.
Key concepts
- Dynamic Granularity Rollout (DGR)
- DGR dynamically schedules different noise sources during the model's optimization process. It uses progressive noise for fine details and initial noise for coarse details, allowing the model to explore both global semantics and local specifics simultaneously. This scheduling prevents over-optimization on irrelevant local textures.
- Consistent Policy Gradient Optimization (CPGO)
- CPGO enforces temporal consistency in the model's trajectory imitation process. It aligns the current policy prediction with a more stable prior from previous steps, ensuring that cross-step inconsistencies are minimized. This is achieved by aligning results at step t with those from step t-1, stabilizing the flow model's behavior.
- Visual Hallucination Evaluator (VH-Evaluator)
- The VH-Evaluator is a comprehensive tool used to measure visual artifacts in generated images. It combines low-level metrics like Laplacian Variance and Edge Metric with high-level scoring from a Multimodal Large Language Model (MLLM). This allows researchers to quantify specific issues like detail over-optimization, texture enhancement, and grid patterns.
- Global Semantics vs. Local Details Trade-off
- This refers to the inherent conflict in RFT where focusing too much on local details leads to visual hallucinations, while ignoring them results in poor global coherence. Dual-Grained Policy Optimization (DGR) is designed specifically to resolve this trade-off by balancing the exploration of fine details with the preservation of overall semantic structure.
Terminology
Summary
Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment but often introduces visual hallucinations like over-optimized details and semantic misalignment. p. 0 ”
Abstract
Reinforcement Fine-Tuning (RFT) on flow-based models is
crucial for preference alignment. However, they often introduce visual hallucinations like over-optimized details and
semantic misalignment. We first investigate RFT methods from a unified perspective,
and reveal the core problems stemming from two aspects,
exploration and exploitation: (1) limited exploration during stochastic differential equation (SDE) rollouts, leading
to an over-emphasis on local details at the expense of global
semantics, and (2) trajectory imitation process inherent in
policy gradient methods, distorting the model’s foundational vector field and its cross-step consistency. Building
on this, we propose ConsistentRFT, a general framework
to mitigate these hallucinations.Where the paper says it
Where the paper says it
The Core Problem in RFT
Existing RFT methods often fail because they suffer from two core issues stemming from exploration and exploitation: limited exploration during stochastic differential equation (SDE) rollouts, leading to an over-emphasis on local details at the expense of global semantics. Furthermore, the trajectory imitation process inherent in policy gradient methods distorts the model’s foundational vector field and its cross-step consistency. p. 0 This work preliminarily explores
why visual hallucinations arise and how to reduce them. We first investigate RFT methods from a unified perspective,
and reveal the core problems stemming from two aspects,
exploration and exploitation: (1) limited exploration during stochastic differential equation (SDE) rollouts, leading
to an over-emphasis on local details at the expense of global
semantics, and (2) trajectory imitation process inherent in
policy gradient methods, distorting the model’s foundational vector field and its cross-step consistency. Despite their successes, these methods [17, 20, 30, 31,
57, 58, 74, 81] often struggle to generate consistent, realistic
images due to visual hallucination (see Fig. This manifests as: (1) detail over-optimization [78], including irrelevant details, over-sharpening, and grid artifacts [69, 70]; (2)
semantic misalignment with the prompt; and (3) cross-step
inconsistency, characterized by semantic degradation and
noticeable divergence under few-step sampling (see Fig.Where the paper says it
Where the paper says it
Mitigation via Dynamic Granularity Rollout (DGR)
The Dynamic Granularity Rollout (DGR) mechanism is designed to balance global semantics and local details by dynamically scheduling different noise sources. This involves:
-
Inter-Group Dynamic-Grained Rollout, which steers the model toward fine- or coarsegrained optimization via dynamic exploration using progressive noise for fine-grained rollouts and initial noise for coarsegrained rollouts. p. 1
Where the paper says it
DGR comprises inter-group and intra-group schedules. In the intergroup schedule, we steer the model toward fine- or coarsegrained optimization via dynamic exploration: progressive noise for fine-grained rollouts and initial noise for coarsegrained rollouts.
-
Intra-Group Dynamic-Grained Rollout, which uses a progress-aware rollout with clustering to maintain diversity while reducing computation by performing coarse progress perception at intermediate states. p. 1
Where the paper says it
In the intergroup schedule, we steer the model toward fine- or coarsegrained optimization via dynamic exploration: progressive noise for fine-grained rollouts and initial noise for coarsegrained rollouts. In the intra-group schedule, we adopt a progress-aware rollout with clustering to maintain diversity while reducing computation.
Mitigation via Consistent Policy Gradient Optimization (CPGO)
Consistent Policy Gradient Optimization (CPGO) is introduced to mitigate inconsistencies arising from trajectory imitation by preserving flow-model consistency through single-step prediction fidelity. CPGO enforces this consistency via two complementary ways:
-
Imitation of a model with better consistency, aligning the current policy with the old model's prediction at step (t−1).
-
Temporal self-consistency, aligning results at step t with those of step (t−1). p. 5
This is formulated as JCPGO(θ, G) = −Et∈U(τ,0) h∥πODEθ(x k t, t) − πODEθold(x k t−1, t − 1)∥22i. p. 5Where the paper says it
(2)), and (ii) temporal self-consistency (e. , aligning results at step t with those of step t-1).
Where the paper says it
A naive consistency-style formulation [50] is constrained by (i) target proximity to the current policy and (ii) reliability of teacher predictions. We therefore define: JConsistentRFT(θ, G) = JGRPO(θ, G) + ω · JCPGO(θ, G) JCPGO(θ, G) = −Et∈U(τ,0) h ∥π ODE θ (x k t , t) − π ODE θold (x k t−1 , t − 1)∥ 2 2 i where τ is a threshold to exclude unreliable results from high-noise samples, and ω is the weight parameter.
Evaluation and Results
Extensive experiments demonstrate that ConsistentRFT significantly mitigates visual hallucinations, achieving average reductions of 49% for low-level and 38% for high-level perceptual hallucinations. ConsistentRFT outperforms other RFT methods on out-of-domain metrics by showing an improvement of 5. p. 0 Extensive experiments
demonstrate that ConsistentRFT significantly mitigates visual hallucinations, achieving average reductions of 49%
for low-level and 38% for high-level perceptual hallucinations. Furthermore, ConsistentRFT outperforms other RFT
methods on out-of-domain metrics, showing an improvement of 5. the baseline’s decrease of -0. 4%) over
FLUX1. The analysis confirms two key findings: (i) existing RFT methods often introduce visual hallucinations, and
(ii) our method effectively mitigates these introduced hallucinations. For instance, while DanceGRPO [69] and MixGRPO [30] inflate artifact scores to 2.Where the paper says it
Where the paper says it
Where the paper says it
Key Findings on Optimization Trade-offs
The ablation study reveals a clear trade-off between fine-grained optimization favoring local details and coarsegrained optimization maintaining global semantics. The dynamic-grained approach resolves this trade-off, attaining competitive HPS (0. p. 7 326 HPS, 0. Our DGR resolves
this trade-off, attaining competitive HPS (0. Our DGR resolves
this trade-off, attaining competitive HPS (0. 349) and CLIP
(0. 349) and CLIP
(0. 376) by balancing local details and global semantics. Effectiveness of CPGO. CPGO enhances internal consistency, raising the average out-of-domain score to 2. CPGO enhances internal consistency, raising the average out-of-domain score to 2. 523
and ImageReward by 9.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Theoretical Justification
The theoretical justification for CPGO connects it to consistency models, showing that minimizing the CPGO objective encourages the current model to maintain consistency with the previous model across the denoising trajectory. The proof of Corollary 1 shows that when CPGO is minimized, the prediction error between the current and old models is controlled by ODE discretization error, which decays as O((∆t)p+1) with finer time steps. p. 1 Third, we propose Consistent Policy Gradient Optimization (CPGO) for policy gradient methods (DPO, DDPO,
GRPO) to mitigate inconsistencies arising from trajectory
imitation. CPGO preserves flow-model consistency by
maintaining single-step prediction fidelity: it aligns the current model’s prediction at step t with the old model’s prediction at step (t−1).Where the paper says it
Practical Implementation Guidelines
Key hyperparameters for ConsistentRFT are determined empirically; for instance, the coarse perception timestep ts is set to 12 as a default choice and ts=8 as a faster variant. The CPGO weight ω is recommended to be searched in the range of 10−6 and 10−7 so that the CPGO loss remains significantly smaller than the GRPO loss. p. 0 We then introduce a Consistent Policy Gradient Optimization (CPGO)
that preserves the model’s consistency by aligning the current policy with a more stable prior. Extensive experiments
demonstrate that ConsistentRFT significantly mitigates visual hallucinations, achieving average reductions of 49%
for low-level and 38% for high-level perceptual hallucinations. Classic baselines (DPO [56],
DDPO [6]) are reproduced via the DanceGRPO codebase
under our setup. During rollout, we use Nold=16 steps and
group size 12; Inter-Group DGS period is 40 with coarse
ratio 0. As a toy example in Fig. 4 illustrates, this conflict disrupts the learned
velocity consistency, leading a inconsistent results across
different sampling steps.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
ConsistentRFT achieves state-of-the-art performance by reducing visual hallucinations by up to 49% while improving out-of-domain generalization by +5.1% on FLUX1. p. 0 the baseline’s decrease of -0. 4%) over
FLUX1. In this work, we aim to mitigate visual
hallucinations in flow-based models. To this end, we
first analyze its potential causes and subsequently propose
Semantic-Consistent Reinforcement Fine-Tuning (ConsistentRFT), a general framework designed for online RFT
methods, including DDPO-, DPO-, and GRPO-like methods [6, 56, 69]. investigation into visual hallucinations during post-training
and proposes an initial mitigation strategy, we recognize
that addressing this challenge requires not only a robust
RFT method but also more powerful and scalable reward models [64]. We anticipate future developments of
stronger open-source reward models by the community.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Dynamic Granularity Rollout (DGR) allows models to balance global semantics and local details by dynamically scheduling different noise sources, which reduces visual hallucinations like detail ignore and unstable optimization.
-
Consistent Policy Gradient Optimization (CPGO) preserves model consistency by aligning the current policy with a more stable prior through single-step prediction fidelity, mitigating cross-step inconsistency in trajectory imitation processes.
-
The system can achieve significant reductions in visual hallucination, achieving average reductions of 49% for low-level and 38% for high-level perceptual hallucinations compared to existing RFT methods.
-
The improved system demonstrates superior out-of-domain performance, showing an improvement of 5.1% (v.s. the baseline’s decrease of-0.4%) over FLUX1.dev models like the baseline, enhancing generalization capabilities across different prompts and domains.
-
The Visual Hallucination Evaluator (VH-Evaluator) provides a comprehensive assessment by combining low-level metrics like Laplacian Variance and Edge Metric with high-level MLLM-based scoring to quantify detail over-optimization, texture over-enhancement, and grid pattern artifacts.
Abstract
Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for alignment. However, they often introduce visual hallucinations like over-optimized details and semantic misalignment. This work explains why visual hallucinations arise and shows how to reduce them. We first investigate RFT methods from a unified perspective and trace hallucinations to two mechanisms. Limited exploration from stochastic differential equation (SDE) rollouts yields detail-dominated within-group differences, leading to fine-grained optimization that over-emphasizes local details and neglects global semantics. Trajectory imitation writes reward-selected SDE noise into the velocity field, leading to inconsistent optimization that degrades cross-step consistency. Building on this, we propose DGPO, a general framework to mitigate hallucinations. Specifically, we design Dynamic Granularity Rollout (DGR) to balance global semantics and local details by dynamically scheduling noise sources. We then introduce a Consistent Policy Gradient Optimization that preserves the consistency by aligning the policy with a more stable prior. Experiments show that DGPO reduces low-/high-level hallucinations by 49%/38% and improves the average of out-of-domain (OoD) metrics by +4.9% on FLUX.1.dev, compared with the baseline's-0.7%. Project page: https://xiaofeng-tan.github.io/projects/ConsistentRFT/
Sources
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Training Diffusion Models with Reinforcement Learning
- HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformer
- Human Alignment of Large Language Models through Online Preference Optimisation
- Directly Fine-Tuning Diffusion Models on Differentiable Rewards
- Score-Based Generative Modeling with Critically-Damped Langevin Diffusion
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Counting Hallucinations in Diffusion Models
- Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
- TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
- Bridging Offline and Online Reinforcement Learning for LLMs
- MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE
- BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Models
- Flow Matching for Generative Modeling
- Flow-GRPO: Training Flow Matching Models via Online RL
- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
- Mitigating Hallucinations in Diffusion Models through Adaptive Attention Modulation
- DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Dragging with Geometry: From Pixels to Geometry-Guided Image Editing
Related papers
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs
- From Formulas to Protocols: A Compositional View of Binary Target Segmentation Metrics
- Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment
- Efficient All-in-One Weather Restoration using Spectral Harmonization
- Peak-End-Net: A Peak-End Rule Inspired Framework for Generalizable Video Aesthetic Assessment