SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers".
Jane: Here's the gist of the paper. Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent’s spatial-frequency structure at each denoising step.
Tom: This adaptive scaling improves both structural coherence and fine-detail fidelity.
Jane: Pruning redundant reasoning tokens Diffusion transformers (DiTs) have become the dominant approach to text-to-image (T2I) generation, producing images with a level of quality that would have been hard to imagine just a few years ago.
Tom: Existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability.
Paper summary: Tom: So we're looking at a paper called SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers, and the main idea is that diffusion transformers struggle when you try to generate images much bigger than what they were trained on.
Jane: Exactly. The authors point out that existing training-free methods usually apply a uniform scaling across the RoPE components, but that doesn't work well because different parts of an image need different scaling for structure versus fine detail, right?
Lu: Right. This paper proposes a training-free method called SEGA that dynamically scales attention based on the latent's spatial frequency structure at every single denoising step. The core idea is to make the scaling content-aware instead of fixed and uniform, which should improve both structural coherence and fine detail fidelity.
Meng: But how do they actually figure out what the spatial frequency structure is? That sounds like a heavy computation just to get the scaling right for inference.
Tom: Well, they use a spectral analysis stage to transform that latent from a spatial domain into the frequency domain using two complementary views from one two dimensional fast Fourier transform. This gives them axis wise profiles and radial profiles, which summarize how energy is distributed across different spatial scales.
Jane: So after they get these spectral profiles, what happens next? They use those summaries to create a modulator that dictates the scaling for each RoPE dimension at that moment during denoising.
Lu: The modulator is built from a reference scale and two components: a per dimension correction that adjusts the scaling based on spatial frequency, and a global amplitude factor that sets how strong that adjustment is. This whole setup defines the final per dimension scaling as m of alpha d equals the reference scale multiplied by this modulator.
Tom: So what's actually interesting about this dynamic allocation? The paper shows it works in early steps where the latent is just noise and the spectrum is flat, keeping the scaling near a reference value, but as distinct structural energy emerges later, SEGA selectively sharpens attention at under resolved frequencies while softening it at over emphasized ones.
Jane: That sounds like it’s actively managing how much attention goes to different parts of the image based on what's happening in the current noise level. So, what does this mean for someone who just wants to generate a high resolution image?
Lu: It means that instead of relying on fixed rules for scaling, you get continuous adaptation guided by the latent's spectral content. This directly impacts attention mechanism stability by preserving semantic locality and entity consistency in a way uniform scaling methods don't.
Meng: If this works across architectures, like Flux or Qwen, that’s promising for deployment because we don't need to retrain the entire model just to push resolution higher than the original training range of one thousand twenty-four by two thousand forty-eight. What about the computational cost of that spectral analysis?
Tom: The paper shows that this process is simple enough, requiring no retraining or architectural changes, but it does require this spectral analysis stage at inference time. The experiment shows SEGA consistently improves structural coherence and fine detail fidelity across baselines and resolution settings, even reaching ultra high resolutions exceeding thirty-six million pixels.
Jane: That result with those four thousand ninety-six by two thousand forty-eight images is quite a big claim for extrapolation beyond the native training range <ref:2605.22668#pg1>. So, what's the final word on SEGA in this paper?
Lu: The conclusion is that SEGA enables stable high-resolution synthesis well beyond the native training regime by adapting RoPE components scaling to the spectral structure of the current latent. It shows that the latent’s spectral structure can serve as a useful signal for guiding RoPE scaling at inference time.
Tom: So, it’s a simple modification that requires no retraining or architectural changes, yet it consistently improves structure, semantics, and visual quality across resolutions and model architectures. It really demonstrates how coupling RoPE dimensions with spatial frequencies is useful for inference-time adaptation of pre-trained generative models.
Conclusion: Tom: So SEGA is this new technique for diffusion transformers that lets them make images much bigger than they were trained on without retraining them.
Jane: Right, and instead of just guessing how to scale the model, it uses the image's own internal structure—its spectral content—to figure out exactly how to adjust those attention components at every step of generation.
Lu: The authors show this works by looking at the latent in a frequency domain and using that information to guide the RoPE scaling, which is what controls how much detail you get.
Meng: But what does this mean for me when I'm trying to deploy these models on my production servers? Does it make running them faster or just give me better output quality?
Tom: It means they can push resolution way past four thousand ninety-six by two thousand forty-eight with structural coherence that keeps making sense, which is a big jump for quality.
Jane: So the paper is really about figuring out how the latent's spectral structure—that frequency information—can be used as a guide for scaling attention in these massive generative models during inference.
Lu: The core idea they are pushing is that this dynamic, content-aware scaling stops being a fixed formula and starts reacting to what's actually emerging in the image at each denoising step.
Meng: I still have to ask about the cost of that spectral analysis stage you mentioned earlier, Tom. Does running this analysis slow down my inference time enough to be a real production tool?
Tom: The authors show it requires no retraining or any new architecture changes, just this scaling modification, and they say it consistently improves structure and detail across different models like Flux and Qwen.
Jane: So the implication is that we don't need massive amounts of extra training data just to get high resolution; we can adapt the existing model on the fly using its own data representation.
Lu: It definitely opens up a new direction for how we think about inference in these pre-trained generative models, showing that latent spectral structure is a useful signal for guiding attention scaling at inference time.
Javad Rajabi, Kimia Shaban Koorosh, Roohi David B., Lindell Babak, Taati
University of Toronto Vector Institute
cs.CV
Submitted: 2026-05-21
Updated: 2026-11-15
Comments: NeurIPS 2026. Project page: https://rajabi2001.github.io/sega/
Code: https://github.com/black-forest-labs/flux
Project page: https://rajabi2001.github.io/sega
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 94/100
The gist: Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors…
Key concepts
- Diffusion Transformers (DiTs)
- These are the dominant models for text-to-image generation. They work by iteratively refining random noise into a coherent image guided by text prompts. However, they often fail when asked to create images at resolutions higher than what they were originally trained on.
- RoPE Scaling
- Rotary Positional Embedding (RoPE) is a mechanism used in DiTs to encode positional information, which helps the model understand where different parts of an image are located. SEGA modifies how this scaling happens during generation, making it adaptive instead of fixed.
- Spectral Analysis
- This technique converts the latent image from its spatial representation (pixels) into a frequency domain. It analyzes how energy is distributed across different spatial scales and frequencies within the image at each step, providing a map of the image's structural content.
Terminology
Summary
Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent’s spatial-frequency structure at each denoising step. p. 0 Abstract
Diffusion transformers (DiTs) have emerged as a dominant architecture for textto-image generation, yet their performance drops when generating at resolutions
beyond their training range. Existing training-free approaches mitigate this by
modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. We introduce SEGA, a training-free
method that dynamically scales attention across RoPE components according to
the latent’s spatial-frequency structure at each denoising step. This adaptive scaling
improves both structural coherence and fine-detail fidelity.Where the paper says it
Where the paper says it
Pruning redundant reasoning tokens
Diffusion transformers (DiTs) have become the dominant approach to text-to-image (T2I) generation, producing images with a level of quality that would have been hard to imagine just a few years ago. Existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability. p. 1 1 Introduction
Diffusion transformers (DiTs) [3, 4] have become the dominant approach to text-to-image (T2I)
generation, producing images with a level of quality that would have been hard to imagine just a few
years ago. Despite considerable improvements, existing T2I models remain largely constrained by
the resolution ranges used during training, typically between 10242
and 20482
resolutions, limiting
their practical applicability [5, 6, 7]. Despite considerable improvements, existing T2I models remain largely constrained by
the resolution ranges used during training, typically between 10242
and 20482
resolutions, limiting
their practical applicability [5, 6, 7]. Consequently, extrapolating beyond this training resolution
at inference time often leads to notable quality degradation and even structural breakdown. These bottlenecks have motivated
growing interest in training-free high-resolution synthesis from pre-trained models [10, 11, 12, 13]. Existing training-free methods for high-resolution image generation generally fall into two categories:
(i) direct inference [14, 15, 16, 17] and (ii) multi-stage guidance-based approaches [18, 19, 20, 5, 6]. Existing training-free methods for high-resolution image generation generally fall into two categories:
(i) direct inference [14, 15, 16, 17] and (ii) multi-stage guidance-based approaches [18, 19, 20, 5, 6]. Direct inference methods attempt to extend pretrained models to higher resolutions by modifying
the denoising process or adjusting components such as positional encoding and attention without
additional training. Direct inference methods attempt to extend pretrained models to higher resolutions by modifying
the denoising process or adjusting components such as positional encoding and attention without
additional training. In contrast, multi-stage approaches first generate a base-resolution image and then
use it to guide high-resolution synthesis.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
SEGA introduces content-aware dynamic scaling into DiTs by coupling lightweight spectral analysis with RoPE components. The core insight is that RoPE scaling for high-resolution extrapolation should be content-aware rather than fixed and uniform. p. 3 4 Method
Spectral-Energy Guided Attention (SEGA) introduces content-aware dynamic scaling into DiTs by
coupling lightweight spectral analysis with RoPE components. Our key insight is that RoPE scaling
for high-resolution extrapolation should be content-aware rather than fixed and uniform. Our key insight is that RoPE scaling
for high-resolution extrapolation should be content-aware rather than fixed and uniform. SEGA
achieves this by deriving per-dimension scaling from the latent’s spectral structure at each denoising
step. Formulation Overview. SEGA applies attention scaling through RoPE using a dimension-wise
scaling term md. Here, M(a)
d
(Z) is our novel dynamic
modulator derived from the spectral structure of the current intermediate latent Z. It consists of two
complementary components: s
(a)
d
(Z), a per-dimension correction that determines the distribution
of scaling across RoPE dimensions, and σ(Z), a global amplitude factor that sets the strength of
that adjustment.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The spectral analysis stage transforms the current latent from the spatial domain to the frequency domain to characterize its spatial frequency content. This involves extracting two complementary spectral views from a single 2D Fast Fourier Transform F2D: (1) Axis-wise profiles, which map spectral energy to spatial frequencies along each axis, and (2) Radial profile, which provides a rotation-invariant summary of how energy is distributed across spatial scales. p. 4 Given the latent hidden states Z ∈ R
N×C with N =
H · W tokens,1 we reshape them back to their native 2D layout, average across channels, and subtract
the average value across the spatial dimensions to obtain a zero-centered 2D map M˜ ∈ R
H×W that
summarizes the spatial structure of the latent. From M˜ we extract two complementary spectral views
from a single 2D Fast Fourier Transform F2D:
• Axis-wise profiles. 2 From Spectrum to Per-Dimension RoPE Scaling
The second stage converts the spectral profiles into the modulator M(Z), which defines the perdimension scaling applied to the rotary embeddings. This formulation consists of three components:
a reference scale that anchors the scaling, a per-dimension term that scales individual dimensions,
and a global gate that controls the strength of that scaling. Final scaling formula. Combining the three components, we define the modulator and the resulting
per-dimension scaling m
(a)
d
along each spatial axis a ∈ {H, W} as
M(a)
d
(Z) = 1 − σ · s
(a)
d
, m
(a)
d = mref · M(a)
d
(Z).Where the paper says it
Where the paper says it
Where the paper says it
Analysis of Spectral-Energy Guided Attention
The method's effectiveness is demonstrated by its ability to adapt scaling continuously to the latent’s spectral content at each denoising step. In early steps where the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the reference scale mref. p. 5 This occurs because SEGA is
content-aware, dynamically adapting scaling to the latent’s spatial frequencies. In early steps where
the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the
reference scale mref. In early steps where
the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the
reference scale mref. However, as distinct structural energy emerges in later steps, SEGA selectively
6 SEGA mitigates this
failure mode by shaping the attention grid much earlier in the denoising process. By dynamically
modulating the magnitude of rotary embeddings, our method preserves semantic locality and entity
consistency that uniform scaling methods fail to maintain.Where the paper says it
Where the paper says it
Where the paper says it
Experiments
Extensive experiments show that SEGA consistently improves structural coherence and fine-detail fidelity and achieves superior performance across baselines and resolution settings, including ultrahigh resolutions exceeding 36 million pixels. The comparison against state-of-the-art baselines on both Flux [1] and Qwen [2] establishes SEGA as a new state-of-the-art for high-resolution image generation across both architectures. p. 7 Quantitative comparison. As shown in Table 1 and Table 2, SEGA establishes a new state-ofthe-art for high-resolution image generation across both the Flux and Qwen architectures. 80
DyPE + SEGA 1. 05
40962
resolution, SEGA outperforms all baseline models across every evaluated metric, setting a new
benchmark for high-resolution generation.Where the paper says it
Where the paper says it
Conclusion
SEGA enables stable high-resolution synthesis well beyond the native training regime by adapting RoPE components scaling to the spectral structure of the current latent. This simple modification requires no retraining or architectural changes, yet consistently improves structure, semantics, and visual quality across resolutions and model architectures. p. 8 By making attention scaling
frequency-aware across RoPE components, SEGA addresses a key limitation of existing uniform
scaling strategies, which often trade off global coherence against fine-detail fidelity. This simple
modification requires no retraining or architectural changes, yet consistently improves structure,
semantics, and visual quality across resolutions and model architectures. Consequently,
a static, uniform scaling at inference time cannot accommodate these variations. Building on this view, we introduce SEGA (Spectral-Energy Guided Attention), a training-free,
content-aware method that dynamically adapts attention scaling to the latent’s spectral structure by
deriving per-component scaling magnitudes at each denoising step. More broadly, frequencyaware attention scaling may also benefit video and other modalities where resolution extrapolation
remains challenging. We hope the spectral perspective guidance introduced here motivates further
research on modifying attention behavior, particularly for resolution extrapolation, to better unlock
the capacity of pretrained generative models.Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Content-aware dynamic scaling for Rotary Position Embeddings (RoPE) allows SEGA to
dynamically adapt attention scaling to the latent’s spectral structure by deriving per-component scaling magnitudes at each denoising step.
This improves bothstructural coherence and fine-detail fidelity
compared to static scaling, resolving the trade-off between preserving global structure and recovering fine detail. -
Spectral analysis of the latent involves extracting axis-wise profiles and a radial profile from the 2D power spectrum to characterize spatial frequency content. These profiles determine scaling, enabling SEGA to
sharpen attention at under-resolved frequencies and soften it at over-emphasized ones
based on the latent's actual spectral energy. -
SEGA enables stable high-resolution synthesis across a wide range of extrapolated resolutions, including ultrahigh resolutions exceeding 36 million pixels. It achieves this by
preserving local texture and object-level detail while maintaining the broader structure of the image,
as shown in Figure 14. -
The method's performance is superior to many existing training-free baselines, consistently achieving
the highest semantic alignment and image quality across different settings
on both Flux and Qwen architectures at high resolutions. -
SEGA provides robustness and consistency across diverse high resolutions, maintaining
highly stable results
even when other models experiencesignificant performance drops as the resolution increases.
Abstract
Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. However, these strategies apply a uniform and content-agnostic scaling across RoPE components with distinct frequency characteristics, inducing a trade-off between preserving global structure and recovering fine detail. We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent's spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity. Experiments show that SEGA consistently improves high-resolution synthesis across multiple target resolutions, outperforming state-of-the-art training-free baselines.
Sources
- Qwen-Image Technical Report
- HiFlow: Training-free High-Resolution Image Generation with Flow-Aligned Guidance
- Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesis
- UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformers
- DyPE: Dynamic Position Extrapolation for Ultra High Resolution Diffusion
- FiT: Flexible Vision Transformer for Diffusion Model
- FreCaS: Efficient Higher-Resolution Image Generation via Frequency-aware Cascaded Sampling
- YaRN: Efficient Context Window Extension of Large Language Models
- Improving Progressive Generation with Decomposable Flow Matching
- ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
- Extending Context Window of Large Language Models via Positional Interpolation
- UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformers
- RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers
- Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Gemini: A Family of Highly Capable Multimodal Models
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs