Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL

summary

The gist

Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment but often introduces visual hallucinations like over-optimized details and semantic misalignment.

In short

ConsistentRFT addresses visual hallucinations in Reinforcement Fine-Tuning (RFT) of flow-based models by balancing global meaning and local detail. It introduces Dynamic Granularity Rollout to manage noise scheduling and Consistent Policy Gradient Optimization to ensure trajectory consistency. This framework significantly reduces low-level and high-level visual artifacts while improving out-of-domain generalization.

Key concepts

Dynamic Granularity Rollout (DGR)
DGR dynamically schedules different noise sources during the model's optimization process. It uses progressive noise for fine details and initial noise for coarse details, allowing the model to explore both global semantics and local specifics simultaneously. This scheduling prevents over-optimization on irrelevant local textures.
Consistent Policy Gradient Optimization (CPGO)
CPGO enforces temporal consistency in the model's trajectory imitation process. It aligns the current policy prediction with a more stable prior from previous steps, ensuring that cross-step inconsistencies are minimized. This is achieved by aligning results at step t with those from step t-1, stabilizing the flow model's behavior.
Visual Hallucination Evaluator (VH-Evaluator)
The VH-Evaluator is a comprehensive tool used to measure visual artifacts in generated images. It combines low-level metrics like Laplacian Variance and Edge Metric with high-level scoring from a Multimodal Large Language Model (MLLM). This allows researchers to quantify specific issues like detail over-optimization, texture enhancement, and grid patterns.
Global Semantics vs. Local Details Trade-off
This refers to the inherent conflict in RFT where focusing too much on local details leads to visual hallucinations, while ignoring them results in poor global coherence. Dual-Grained Policy Optimization (DGR) is designed specifically to resolve this trade-off by balancing the exploration of fine details with the preservation of overall semantic structure.

Terminology used across episodes

This episode discusses

The paper

Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL · Read on arXiv

Xiaofeng Tan, Jun Liu, Yuanting Fan, Bin-Bin Gao, Xi Jiang, Xiaochen Chen, Jinlong Peng, Chengjie Wang, Hongsong Wang

Southeast University · Southern University of Science and Technology Tencent Youtu Lab

Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for alignment. However, they often introduce visual hallucinations like over-optimized details and semantic misalignment. This work explains why visual hallucinations arise and shows how to reduce them. We first investigate RFT methods from a unified perspective and trace hallucinations to two mechanisms. Limited exploration from stochastic differential equation (SDE) rollouts yields detail-dominated within-group differences, leading to fine-grained optimization that over-emphasizes local details and neglects global semantics. Trajectory imitation writes reward-selected SDE noise into the velocity field, leading to inconsistent optimization that degrades cross-step consistency. Building on this, we propose DGPO, a general framework to mitigate hallucinations. Specifically, we design Dynamic Granularity Rollout (DGR) to balance global semantics and local details by dynamically scheduling noise sources. We then introduce a Consistent Policy Gradient Optimization that preserves the consistency by aligning the policy with a more stable prior. Experiments show that DGPO reduces low-/high-level hallucinations by 49%/38% and improves the average of out-of-domain (OoD) metrics by +4.9% on FLUX.1.dev, compared with the baseline's-0.7%. Project page: https://xiaofeng-tan.github.io/projects/ConsistentRFT/

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL".

Jane: Here's the gist of the paper. Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment but often introduces visual hallucinations like over-optimized details and semantic misalignment.

Tom: This work proposes ConsistentRFT, a general framework to mitigate these hallucinations by balancing global semantics and local details through Dynamic Granularity Rollout (DGR) and Consistent Policy Gradient Optimization (CPGO).

Jane: The Core Problem in RFT Existing RFT methods often fail because they suffer from two core issues stemming from exploration and exploitation: limited exploration during stochastic differential equation (SDE) rollouts, leading to an over-emphasis on local details at the expense of global semantics.

Tom: Furthermore, the trajectory imitation process inherent in policy gradient methods distorts the model’s foundational vector field and its cross-step consistency.

Title and authors: Tom: So we're looking at this paper, "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL," and it looks like they’re tackling a big problem. They’re saying that Reinforcement Fine-Tuning on flow models often introduces visual hallucinations, which are like the model adding details that aren't there or getting stuck in weird optimization loops.

Jane: Exactly. The naive question is, why does this happen in the first place? Is it just a flaw in how we fine-tune these generative models?

Lu: It seems like they’re addressing the exploration and exploitation trade-off that happens during the rollouts, where existing methods tend to focus either too much on fine details or too much on broad concepts, both of which lead to those visual artifacts.

Meng: From a deployment standpoint, if we keep seeing these hallucinations in production models, it’s a headache because the output isn't reliable for our users. What does this paper actually propose to fix that instability?

Tom: Well, they introduce two main ideas: Dynamic Granularity Rollout and Consistent Policy Gradient Optimization. They want to balance the global meaning of the image with the fine details we see in it.

Jane: Dynamic Granularity Rollout sounds complicated. Can you explain how that helps? How does it dynamically schedule different noise sources?

Lu: It uses two parts: an inter-group rollout and an intra-group rollout. The idea is to steer the model toward fine or coarse granularity using different noise schedules depending on where it is in the process, which should reduce those hallucinations we saw in existing methods like DanceGRPO and Flow-GRPO.

Meng: If I’m running a production pipeline, I need to know if this adds significant computational overhead. Does this dynamic scheduling make it too slow for real-time inference?

Tom: It does have a cost, but the point is that they are trying to reduce the need for excessive iteration counts. They show in their experiments that their method can manage things across iterations like zero, forty, eighty, one hundred twenty and one hundred sixty.

Jane: So if we look at what they found in Figure two they see these hallucinations when fine-tuning via reward signals where the reward model is trying to boost the in-domain reward by injecting irrelevant details <ref:2602.03425#pg1>.

Lu: That points to a constrained exploration domain, which is something the paper discusses because it suggests existing methods learn a shortcut by boosting that reward signal with those irrelevant details.

Meng: So they’re suggesting we stop letting the reward function just blindly push for detail enhancement? What's the mechanism behind that suggestion?

Tom: That brings us to Consistent Policy Gradient Optimization, or CPGO. This is meant to fix the problem of cross-step inconsistency that happens when you imitate a trajectory using policy gradients.

Title and authors: Jane: How does this consistency thing actually work on a flow model trajectory? It sounds like it’s about aligning the current prediction with something from the previous step.

Lu: It enforces consistency in two ways: first, they try to imitate a model that already has better consistency at step t minus one. Second, they use temporal self-consistency by aligning what happens at step t with what happened at step t minus one.

Meng: Aligning with the previous state sounds like it’s adding complexity to the training loop. Does this mean more data or more complex loss calculations are needed?

Tom: It involves a specific objective function, JCPGO theta G equals negative E over U of tau and zero h of the current policy ODE theta and the old model's ODE theta old at step t minus one. It’s a way to keep the flow-model consistency through single-step prediction fidelity.

Jane: If we think about what this means for someone just watching, does it translate into a cleaner image for them?

Lu: The theoretical justification connects this CPGO to consistency models, showing that minimizing that CPGO objective encourages the current model to maintain consistency with the previous model across the entire denoising trajectory. When minimized, the prediction error between the current and old models falls with the square of the sampling interval, which gets smaller when you have finer time steps.

Meng: So if I were deploying this, I’d need to be careful about how much precision I set for those time steps because of that scaling law they mentioned. That’s a concrete operational detail we need to worry about at scale.

Tom: Exactly. And when we look at the results, they show that ConsistentRFT significantly reduces visual hallucinations by an average of forty-nine percent for low-level hallucinations and thirty-eight percent for high-level perceptual ones compared to other RFT methods.

Jane: That reduction sounds substantial, especially for those high-level perceptual artifacts that are hard to see but ruin the final quality. What about generalization? Does it just fix the hallucination problem or does it improve how well the model handles prompts it hasn't seen before?

Lu: The paper shows that ConsistentRFT outperforms other RFT methods on out-of-domain metrics by showing an improvement of five point one percent over the baseline’s decrease of negative zero point four percent on FLUXone.dev. That suggests it actually improves generalization across different prompts and domains.

Meng: A five point one percent improvement is modest, but if it means we don't need to spend extra time curating domain-specific datasets to avoid these visual issues, that’s a win for deployment pipelines.

Title and authors: Tom: And the Visual Hallucination Evaluator confirms this. It shows artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO. That gives us solid empirical evidence beyond just the model's internal metrics.

Jane: So, if we put it all together, what’s the big picture for people who just use these models? What does this paper actually change for them when they go to generate an image?

Lu: The dual-grained approach resolves that trade-off between optimizing local details and maintaining global semantics. They hit competitive scores on metrics like HPS and CLIP by finding that balance.

Meng: So, the method isn't just about reducing noise; it’s about getting a better quality image for the user at a lower cost than we might think because it’s more stable during training.

Tom: And CPGO is what ensures that stability carries through the entire generation process, not just in one step. It’s about making sure the model stays on track from start to finish.

Jane: So, to wrap up this paper on "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL," it seems they've managed to introduce a unified framework that addresses both the exploration issues and the consistency issues separately.

Lu: It’s a unified plug-in that consistently improves DPO, GRPO, and DDPO in the online setting. The core idea is balancing global semantics with local details through dynamic scheduling and enforcing trajectory consistency with single-step prediction fidelity.

Meng: I just think we need to keep an eye on how they handle the practical tuning parameters they suggest, like that coarse perception timestep being twelve by default. That’s where the real engineering work will be needed.

Tom: Right, so the reduction in visual hallucinations up to forty-nine percent and that five point one percent gain in out-of-domain performance on FLUXone.dev are the numbers we have for this framework.

Jane: It’s a solid piece of work because it tackles those hallucinations directly and shows measurable gains across different evaluation metrics, not just theoretical ones.

Lu: The paper's main point is that you can get better quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning.

Meng: So, for deployment, it means we move away from relying on one-size-fits-all RFT settings and instead use this dual-grained approach to control the trade-off.

Tom: That’s the operational implication: intentional control over those two competing forces during training is what leads to better final outputs.

Jane: It gives us a clearer path forward for improving flow models, showing that balancing local detail and global semantics is achievable with these specific optimization techniques.

The paper's summary: Tom: So, we just talked about how Reinforcement Fine-Tuning on flow models usually makes them hallucinate visual details or get weirdly stuck in optimization loops, and now we’re looking at this paper called "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL."

Jane: Right. So, what they’re saying is that instead of just trying to fix the whole process at once, they have two different strategies working together. The first strategy is Dynamic Granularity Rollout, which basically lets the model decide when to focus on fine details and when it should look at the big picture.

Lu: Exactly. They use this dynamic scheduling of noise sources, splitting it into an inter-group rollout and an intra-group rollout, so they can steer the optimization toward fine or coarse granularity depending on where they are in the process.

Meng: From my side, I’m thinking about deployment cost here. Does this constant changing of noise schedules make the training time explode?

Tom: It does add computational complexity, but the goal is to reduce how many iterations you need to run for a decent result. They show it can handle rollouts across zero up to one hundred sixty steps.

Jane: And then they pair that with something called Consistent Policy Gradient Optimization, or CPGO. This part is all about fixing the inconsistency that happens when the model imitates a previous trajectory.

Lu: CPGO enforces consistency by aligning the current policy with what the old model predicted at step minus one, and also by making sure step t aligns with step t minus one. It keeps that flow-model consistency through single-step prediction fidelity.

Meng: So it’s like they’re checking its work at every single tiny moment during the generation process to make sure it hasn't drifted too far off course?

Tom: Precisely, and the theory behind this is pretty neat. Minimizing that CPGO objective actually shows that the error between the current model and the old model falls with the square of your sampling interval, which gets smaller when you use finer time steps.

Jane: That scaling law is important because it tells us exactly how much precision we need to set for those time steps if we want to see that benefit in practice.

Lu: And looking at the numbers, they show consistent RFT methods are cutting visual hallucinations by forty-nine percent for low-level stuff, and thirty-eight percent for high-level perceptual issues.

Tom: That’s a big drop compared to what DanceGRPO or MixGRPO were doing on their own. But wait, there’s more. They also showed they improve out-of-domain performance by five point one percent on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.

Jane: So it’s not just about making the generated image look better; it actually helps the model generalize better to prompts it hasn't seen before.

Meng: A five point one percent gain is small, but if you aren’t spending all your time building domain-specific datasets just to avoid these visual artifacts, that adds up in production pipelines.

Tom: And they back up those claims with the Visual Hallucination Evaluator, which shows artifact scores reduced by thirty-eight percent and twenty-four percent compared to those other methods. That’s solid empirical evidence.

Jane: It seems like the big picture here is that you can get a more stable, higher quality output by being intentional about how you balance local detail optimization against maintaining global semantics throughout the entire training run.

Lu: This framework acts as a unified plug-in that improves different reinforcement learning methods like DPO and GRPO in an online setting.

Tom: It’s about using this dual-grained approach to control those two competing forces during training, which leads to better final outputs for the user.

The paper's improvements: Tom: So, we’ve covered how they fix hallucinations using Dynamic Granularity Rollout and Consistent Policy Gradient Optimization; now we look at what these specific methods actually change for the model during training.

Jane: What they suggest is that instead of just tweaking one thing, you get two things working in tandem to control the optimization process. It’s like having a steering wheel that can smoothly shift between fine adjustments and broad strokes as you drive.

Lu: Exactly. The Dynamic Granularity Rollout means the model gets to decide whether it should be focusing on very small details or the overall shape of the image at any given moment during its journey through noise.

Meng: From a deployment angle, that sounds like a lot of math to set up during pre-training, but if it stabilizes the output quality so much, maybe it’s worth the overhead.

Tom: The results show that this balance resolves the trade-off between focusing on local details and keeping track of global semantics. They hit competitive scores on things like HPS and CLIP by finding that sweet spot between those two extremes.

Jane: And then they add CPGO to make sure that whatever the model decides to do, it stays consistent with its previous state along the way. It’s about maintaining a stable trajectory even when it's making big changes in detail focus.

Lu: The theoretical justification confirms that minimizing that CPGO objective forces the current model to stay consistent with the prior model across all those steps in the denoising process.

Tom: And when you look at the hard metrics, they report average reductions of forty-nine percent for low-level hallucinations and thirty-eight percent for high-level perceptual ones compared to existing RFT methods.

Jane: That’s a huge drop in visual noise, which is what people actually see when they look at the final generated image.

Meng: So that means we can get much cleaner results without having to manually filter out those weird artifacts in post-processing, which saves time on our end.

Lu: Plus, they showed an improvement of five point one percent on out-of-domain metrics on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.

Tom: That generalization gain is actually pretty significant for a model that’s supposed to be useful across many different types of prompts and scenes.

Jane: It means the model learns the core concepts better, not just how to add distracting textures that look good in one specific context.

Lu: The paper also confirmed this with the Visual Hallucination Evaluator, which showed artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO.

Tom: So, to sum up these improvements, you get a model that’s not just better at one thing but is more stable across the whole generation process while keeping those hallucinations in check.

Jane: This framework serves as a unified plug-in for other reinforcement learning techniques like DPO and GRPO when you’re working online.

Lu: The real implication is that you can get higher quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning, instead of just letting it happen randomly.

Meng: So the operational takeaway for me is that we move away from using one-size-fits-all settings and start using this dual-grained control to manage those competing forces.

Conclusion: Tom: So we’ve covered how they fix hallucinations using Dynamic Granularity Rollout and Consistent Policy Gradient Optimization; now we look at what these specific methods actually change for the model during training.

Jane: What they suggest is that instead of just tweaking one thing, you get two things working in tandem to control the optimization process. It’s like having a steering wheel that can smoothly shift between fine adjustments and broad strokes as you drive.

Lu: Exactly. The Dynamic Granularity Rollout means the model gets to decide whether it should be focusing on very small details or the overall shape of the image at any given moment during its journey through noise.

Meng: From my side, I’m thinking about deployment cost here. Does this constant changing of noise schedules make the training time explode?

Tom: It does add computational complexity, but the goal is to reduce how many iterations you need to run for a decent result. They show it can handle rollouts across zero up to one hundred sixty steps.

Jane: And then they pair that with something called Consistent Policy Gradient Optimization, or CPGO. This part is all about fixing the inconsistency that happens when the model imitates a previous trajectory.

Lu: CPGO enforces consistency by aligning the current policy with what the old model predicted at step minus one, and also by making sure step t aligns with step t minus one. It keeps that flow-model consistency through single-step prediction fidelity.

Meng: So it’s like they’re checking its work at every single tiny moment during the generation process to make sure it hasn't drifted too far off course?

Tom: Precisely, and the theory behind this is pretty neat. Minimizing that CPGO objective actually shows that the error between the current model and the old model falls with the square of your sampling interval, which gets smaller when you use finer time steps.

Jane: That scaling law is important because it tells us exactly how much precision we need to set for those time steps if we want to see that benefit in practice.

Lu: And looking at the numbers, they show consistent RFT methods are cutting visual hallucinations by forty-nine percent for low-level stuff, and thirty-eight percent for high-level perceptual ones compared to existing RFT methods.

Tom: That’s a big drop compared to what DanceGRPO or MixGRPO were doing on their own. But wait, there’s more. They also showed they improve out-of-domain performance by five point one percent on FLUXone.dev models compared to the baseline's decrease of negative zero point four percent.

Jane: So it’s not just about making the generated image look better; it actually helps the model generalize better to prompts it hasn't seen before.

Meng: A five point one percent gain is small, but if you aren’t spending all your time building domain-specific datasets just to avoid these visual artifacts, that adds up in production pipelines.

Lu: The paper also confirmed this with the Visual Hallucination Evaluator, which showed artifact scores reduced by thirty-eight percent and twenty-four percent compared to methods like DanceGRPO and MixGRPO.

Tom: So, to sum up these improvements, you get a model that’s not just better at one thing but is more stable across the whole generation process while keeping those hallucinations in check.

Jane: This framework serves as a unified plug-in for other reinforcement learning techniques like DPO and GRPO when you’re working online.

Lu: The real implication is that you can get higher quality results by being intentional about how you balance your exploration strategy during reinforcement fine-tuning, instead of just letting it happen randomly.

Meng: So the operational takeaway for me is that we move away from using one-size-fits-all settings and start using this dual-grained control to manage those competing forces.

Tom: Indeed, it’s about intentional control over those two competing forces during training that leads to better final outputs.

Jane: So, in the end, the paper "Dual-Grained Policy Optimization for Hallucination Reduction in Flow-Based RL" shows a clear path toward more reliable generative models.

Lu: It gives us a concrete method to control the trade-off between local detail and global semantics effectively in flow model training.

Meng: I'm curious if they explore how this scales when we move from standard images to, say, video generation, because that’s where things get messy fast.

Lalam: I am confident this result is sound; the mechanism for consistency control is very robust across the tested configurations.

More episodes

← Home