Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Heeding the Inner Voice".
Jane: Here's the gist of the paper. Achieving precise spatial control over generated outputs remains challenging in text-to-image diffusion models, and this paper proposes InnerControl as a novel training strategy to enforce spatial consistency across all diffusion steps Problem Statement Recent work on ControlNet++ focuses on applying cycle consistency loss only to the final denoising steps, which suffers from temporal misalignment because the main spatial knowledge appears in the early generation stages This limitation is highlighted because extending reward losses to earlier steps leads to a significant decrease in generated image quality, producing visible artifacts on the generated images The analysis indicates that this poor result is probably caused by inefficient signal extraction in the early sampling steps, producing inaccurate signals for loss calculations InnerControl Training Strategy The proposed InnerControl addresses temporal misalignment by enforcing consistency between input control and extracted signals from intermediate diffusion features across the whole sampling trajectory This is achieved by training lightweight control prediction probes, which are small convolutional networks designed to reconstruct input control signals from intermediate UNet features at every denoising step The core objective is defined as Lalignment = L(cspatial, H
ControlNet(cspatial, ctxt, xT, t): , t), which penalizes discrepancies between the input control and the extracted signal at each step t Training Objective Integration The final training objective for InnerControl uses a weighted combination of standard diffusion loss, rewarding loss at the early denoising stage, and the additional alignment loss The full objective is Ltraining = Ldiffusion + α · Lreward + β · Lalignment, where α and β are specific weights used across experiments This combined loss is shown to improve control correspondence and image fidelity without visible artifacts compared to reward loss at the early steps Experimental Validation Experiments on the MultiGen-20M dataset demonstrate that InnerControl improves control alignment (RMSE) and image quality (FID) across different guidance scales For depth estimation, RMSE is reduced by 7.87% compared to ControlNet++
4: at the guidance scale 7.
Title and authors: Tom: So, to summarize this paper, "Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback," they introduce a training strategy called InnerControl. This isn't just sticking a reward at the end; it’s about actively training small convolutional networks, which we call control prediction probes.
Jane: These probes are trained to reconstruct your input control signals, like an edge map or a depth map, directly from the intermediate features of the UNet model at every single denoising step throughout the whole generation process. They calculate an alignment penalty during this process.
Lu: The core idea is to punish any discrepancy between what you give it as control and what it extracts from those inner layers at each timestep. This forces a much tighter spatial correspondence all along the way, not just at the finish line of generation.
Meng: From an engineering standpoint, this means we're adding a new loss term to the training objective that runs constantly during sampling, which is different from just one big loss at the end. What’s the actual mechanism for these probes?
Tom: They use lightweight convolutional neural networks as these probes, and they are inspired by techniques like Readout Guidance. The goal is to see how well those small networks can reconstruct the input control signals even when looking at features from earlier denoising stages where the structure is starting to emerge.
Jane: It’s interesting because they are focusing on those early stages where spatial structure first appears, which we know is a key area for control. They are essentially trying to give the model better feedback as it builds the image step by step, not just once at the end of everything.
Lalam: Lalam thinks this is powerful because it addresses that temporal misalignment they mentioned earlier directly by enforcing consistency across all diffusion steps. It’s about making sure the spatial understanding stays aligned from start to finish.
The paper's summary: Tom: They show that this InnerControl strategy improves control correspondence and image fidelity without introducing those visible artifacts that happen when you just apply reward losses at the early stages of denoising. That’s a big win for practical application in things like medical imaging or precise scene reconstruction.
Jane: It’s not just about making the final image look better; it's about making the control mapping more robust throughout the entire generation process, which is what they mean by aligning control and extracted signals across the whole trajectory.
Lu: They analyze existing work that looks at UNet layers at different denoising steps for tasks like depth estimation or segmentation. They found that most prior methods just aggregate features across these steps, but this new approach allows them to capture task-specific information right at the individual stages.
Meng: If we think about deployment, this suggests a more stable training process because the alignment is built into every step of the diffusion process rather than being a correction applied post-hoc. That’s something that would reduce training instability when we push our models harder.
Tom: The paper proves that this enhanced reward training approach, which now includes this new alignment loss, actually improves control correspondence and image fidelity compared to just using reward loss at the early steps alone. That means we get better control without sacrificing quality at the end.
Jane: So, they are showing that by using these lightweight probes to estimate signals from intermediate UNet features, you get a more accurate spatial alignment that leads to better results overall for various tasks like depth estimation and segmentation.
The paper's improvements: Tom: One of the key improvements they highlight is in how this method handles depth estimation. They show that for depth estimation, the Root Mean Square Error, or RMSE, is reduced by seven point eight seven percent when compared to a previous method called ControlNet++ at a guidance scale of seven.
Jane: That seven point eight seven percent reduction in RMSE is pretty concrete evidence that this method works better for geometric tasks when you are conditioning on depth maps. It shows tangible improvement over what was previously available in the literature.
Lu: The paper also touches on the architecture itself, noting that while prior research focused heavily on using self-attention outputs from the UNet decoder for structure estimation, they slightly modified it to leverage those self-attention outputs instead of just standard convolutional features.
Meng: From a production standpoint, if this means better RMSE scores, it suggests we can deploy models that are more reliable when conditioning on depth information because the underlying control signal extraction is more accurate even from noisy latents.
Tom: They also note that for tasks like LineArt and HED conditioned images under high guidance scales, their alignment strategy significantly improves both the fidelity to the input controls and the stability under those strong guidance conditions.
Jane: So, when you're pushing a model with a lot of guidance—say, for very clear line art—this method produces outputs that are less noisy and have more accurate structure compared to what we see with other methods.
Conclusion: Tom: So to wrap up the paper "Heeding the Inner Voice: Aligning ControlNet Training via Intermediate Features Feedback," they’ve shown that training lightweight control prediction probes at every step enforces spatial consistency across the entire diffusion trajectory, not just the last few steps.
Jane: This means we get better control alignment and image fidelity because it forces the model to maintain a correct relationship between its input and the features it extracts throughout all of generation. It’s about getting that spatial understanding right from the beginning of denoising.
Lu: The implication here is that by using these intermediate features feedback mechanisms, we can achieve much higher precision in tasks like depth estimation, as evidenced by the seven point eight seven percent RMSE reduction they found over ControlNet++.
Meng: For deployment, this suggests a more robust training regime where the alignment is baked into the process from start to finish, which should translate to better stability when running models on real traffic or complex inputs.
Lalam: I’m just sure this paper will keep pushing us toward more precise spatial control in diffusion models because it solves that temporal misalignment problem by enforcing consistency throughout every single step.
Nina Konovalova, Maxim Nikolaev, Andrey Kuznetsov, Aibek Alanov
AIRI Institute of Artificial Intelligence Research in Russia · HSE University of Russia
cs.CV
Submitted: 2025-07-03
Updated: 2026-10-31
Comments: The project page is available at \url{https://controlgenai.github.io/InnerControl/}
Code: https://github.com/ControlGenAI/InnerControl
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 79/100
The gist: Achieving precise spatial control over generated outputs remains challenging in text-to-image diffusion models…
Key concepts
- ControlNet++ Limitation
- Previous ControlNet++ methods applied cycle consistency loss only to the final denoising steps. This caused temporal misalignment because the main spatial knowledge resides in earlier generation stages, resulting in poor results and visible artifacts when reward losses were extended to earlier steps.
- InnerControl Training Strategy
- This strategy enforces consistency between input control and extracted signals from intermediate diffusion features across the entire sampling trajectory. It involves training lightweight control prediction probes to reconstruct the input control signals from intermediate UNet features at every denoising step.
- Alignment Loss (Lalignment)
- The core objective is a new loss function, Lalignment, which penalizes discrepancies between the original input control and the signal extracted from intermediate diffusion features at each timestep 't'. This loss is integrated into the total training objective to guide the model toward better spatial correspondence throughout generation.
- Weighted Training Objective
- The final training objective combines standard diffusion loss, a reward loss at early stages, and the new alignment loss (Ltraining = Ldiffusion + α · Lreward + β · Lalignment). This weighted combination ensures that the model improves control correspondence and image fidelity while avoiding artifacts associated with applying reward losses only to late steps.
Terminology
Summary
Achieving precise spatial control over generated outputs remains challenging in text-to-image diffusion models, and this paper proposes InnerControl as a novel training strategy to enforce spatial consistency across all diffusion steps
Problem Statement
Recent work on ControlNet++ focuses on applying cycle consistency loss only to the final denoising steps, which suffers from temporal misalignment because the main spatial knowledge appears in the early generation stages This limitation is highlighted because extending reward losses to earlier steps leads to a significant decrease in generated image quality, producing visible artifacts on the generated images The analysis indicates that this poor result is probably caused by inefficient signal extraction in the early sampling steps, producing inaccurate signals for loss calculations
InnerControl Training Strategy
The proposed InnerControl addresses temporal misalignment by enforcing consistency between input control and extracted signals from intermediate diffusion features across the whole sampling trajectory This is achieved by training lightweight control prediction probes, which are small convolutional networks designed to reconstruct input control signals from intermediate UNet features at every denoising step The core objective is defined as Lalignment = L(cspatial, H [ControlNet(cspatial, ctxt, xT, t)], t), which penalizes discrepancies between the input control and the extracted signal at each step t
Training Objective Integration
The final training objective for InnerControl uses a weighted combination of standard diffusion loss, rewarding loss at the early denoising stage, and the additional alignment loss The full objective is Ltraining = Ldiffusion + α · Lreward + β · Lalignment, where α and β are specific weights used across experiments This combined loss is shown to improve control correspondence and image fidelity without visible artifacts compared to reward loss at the early steps
Experimental Validation
Experiments on the MultiGen-20M dataset demonstrate that InnerControl improves control alignment (RMSE) and image quality (FID) across different guidance scales For depth estimation, RMSE is reduced by 7.87% compared to ControlNet++ [4] at the guidance scale 7. p. 6 At the same time, for a higher guidance scale (7. 5) our
method shows improvements for HED and LineArt compared to ControlNet++ [4] training strategy,
while being more efficient than the Ctrl-U [5] method for depth control task.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Precision spatial control in image generation is improved by enforcing consistency between input controls and extracted signals throughout every diffusion step using a novel training objective called InnerControl. This allows the AI system to generate images with superior alignment to complex spatial conditions like depth maps and edges across the entire sampling trajectory, rather than just the final steps.
-
The AI system gains enhanced controllability when generating images conditioned on depth maps or edge maps because it learns to
reconstruct input control signals from intermediate UNet features at every denoising step.
This capability enables accurate signal estimation even fromvery noisy latents,
leading to better fidelity and stability in generated outputs compared to previous methods. -
The system benefits from a more robust training objective that combines standard diffusion loss, reward loss at early stages, and an additional alignment loss called Lalignment. This weighted combination ensures that the AI model maintains high control alignment (RMSE) while simultaneously preserving perceptual quality (FID), avoiding the
visible artifacts
seen when applying reward losses only to late denoising steps. -
The improved system can achieve higher fidelity and better stability in depth estimation tasks by utilizing a lightweight convolutional network H(cdot, t) that estimates control signals directly from intermediate features at every timestep. This approach is shown to
predict results more aligned with the final depth prediction for SD1.5, especially at the early stage of generation,
leading to significant RMSE reductions over baselines like ControlNet++. -
The AI system demonstrates superior performance in generating LineArt and HED conditioned images under high guidance scales, showing that its alignment strategy
significantly improves both the fidelity to input controls and stability under high guidance scales.
This means the system produces less noisy, more accurately structured outputs when operating with strong conditioning signals.
Abstract
Despite significant progress in text-to-image diffusion models, achieving precise spatial control over generated outputs remains challenging. One of the popular approaches for this task is ControlNet, which introduces an auxiliary conditioning module into the architecture. To improve alignment of the generation image and control, ControlNet++ proposes a cycle consistency loss to refine correspondence between controls and outputs, but restricts its application to the final denoising steps, while the main structure is introduced at an early stage of generation. To address this issue, we suggest InnerControl -- a training strategy that enforces spatial consistency across a broad range of diffusion steps, including early high-noise ones. Specifically, we train lightweight control prediction probes -- small convolutional networks -- to reconstruct input control signals (e.g., edges, depth) from intermediate diffusion features across denoising steps. We show empirically that such models can extract signals even from very noisy latents and utilize these models to generate pseudo ground truth controls during training. Suggested approach enables alignment loss that minimizes the difference between predicted and target condition over most of the diffusion trajectory. Our experiments demonstrate that our method improves control alignment, with perceptual fidelity preserved or improved in most settings and small FID trade-offs in others. Our approach can be applied to both UNet- and Transformer-based diffusion architectures and be integrated with established training techniques (e.g., ControlNet++) to achieve high performance across different condition methods such as edge, segmentation and depth conditions.
Sources
- IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
- Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling
- Denoising Diffusion Implicit Models
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Composer: Creative and Controllable Image Synthesis with Composable Conditions
- RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
- UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
- Label-Efficient Semantic Segmentation with Diffusion Models
- CleanDIFT: Diffusion Features without Noise
- EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
- What the DAAM: Interpreting Stable Diffusion Using Cross Attention
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
- Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs