SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers

arXiv:2605.22668 · cs.CV · Submitted 2026-05-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers".

Jane: Here's the gist of the paper. Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent’s spatial-frequency structure at each denoising step.

Tom: This adaptive scaling improves both structural coherence and fine-detail fidelity.

Jane: Pruning redundant reasoning tokens Diffusion transformers (DiTs) have become the dominant approach to text-to-image (T2I) generation, producing images with a level of quality that would have been hard to imagine just a few years ago.

Tom: Existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability.

Paper summary: Tom: So we're looking at a paper called SEGA: Spectral-Energy Guided Attention for Resolution Extrapolation in Diffusion Transformers, and the main idea is that diffusion transformers struggle when you try to generate images much bigger than what they were trained on.

Jane: Exactly. The authors point out that existing training-free methods usually apply a uniform scaling across the RoPE components, but that doesn't work well because different parts of an image need different scaling for structure versus fine detail, right?

Lu: Right. This paper proposes a training-free method called SEGA that dynamically scales attention based on the latent's spatial frequency structure at every single denoising step. The core idea is to make the scaling content-aware instead of fixed and uniform, which should improve both structural coherence and fine detail fidelity.

Meng: But how do they actually figure out what the spatial frequency structure is? That sounds like a heavy computation just to get the scaling right for inference.

Tom: Well, they use a spectral analysis stage to transform that latent from a spatial domain into the frequency domain using two complementary views from one two dimensional fast Fourier transform. This gives them axis wise profiles and radial profiles, which summarize how energy is distributed across different spatial scales.

Jane: So after they get these spectral profiles, what happens next? They use those summaries to create a modulator that dictates the scaling for each RoPE dimension at that moment during denoising.

Lu: The modulator is built from a reference scale and two components: a per dimension correction that adjusts the scaling based on spatial frequency, and a global amplitude factor that sets how strong that adjustment is. This whole setup defines the final per dimension scaling as m of alpha d equals the reference scale multiplied by this modulator.

Tom: So what's actually interesting about this dynamic allocation? The paper shows it works in early steps where the latent is just noise and the spectrum is flat, keeping the scaling near a reference value, but as distinct structural energy emerges later, SEGA selectively sharpens attention at under resolved frequencies while softening it at over emphasized ones.

Jane: That sounds like it’s actively managing how much attention goes to different parts of the image based on what's happening in the current noise level. So, what does this mean for someone who just wants to generate a high resolution image?

Lu: It means that instead of relying on fixed rules for scaling, you get continuous adaptation guided by the latent's spectral content. This directly impacts attention mechanism stability by preserving semantic locality and entity consistency in a way uniform scaling methods don't.

Meng: If this works across architectures, like Flux or Qwen, that’s promising for deployment because we don't need to retrain the entire model just to push resolution higher than the original training range of one thousand twenty-four by two thousand forty-eight. What about the computational cost of that spectral analysis?

Tom: The paper shows that this process is simple enough, requiring no retraining or architectural changes, but it does require this spectral analysis stage at inference time. The experiment shows SEGA consistently improves structural coherence and fine detail fidelity across baselines and resolution settings, even reaching ultra high resolutions exceeding thirty-six million pixels.

Jane: That result with those four thousand ninety-six by two thousand forty-eight images is quite a big claim for extrapolation beyond the native training range <ref:2605.22668#pg1>. So, what's the final word on SEGA in this paper?

Lu: The conclusion is that SEGA enables stable high-resolution synthesis well beyond the native training regime by adapting RoPE components scaling to the spectral structure of the current latent. It shows that the latent’s spectral structure can serve as a useful signal for guiding RoPE scaling at inference time.

Tom: So, it’s a simple modification that requires no retraining or architectural changes, yet it consistently improves structure, semantics, and visual quality across resolutions and model architectures. It really demonstrates how coupling RoPE dimensions with spatial frequencies is useful for inference-time adaptation of pre-trained generative models.

Conclusion: Tom: So SEGA is this new technique for diffusion transformers that lets them make images much bigger than they were trained on without retraining them.

Jane: Right, and instead of just guessing how to scale the model, it uses the image's own internal structure—its spectral content—to figure out exactly how to adjust those attention components at every step of generation.

Lu: The authors show this works by looking at the latent in a frequency domain and using that information to guide the RoPE scaling, which is what controls how much detail you get.

Meng: But what does this mean for me when I'm trying to deploy these models on my production servers? Does it make running them faster or just give me better output quality?

Tom: It means they can push resolution way past four thousand ninety-six by two thousand forty-eight with structural coherence that keeps making sense, which is a big jump for quality.

Jane: So the paper is really about figuring out how the latent's spectral structure—that frequency information—can be used as a guide for scaling attention in these massive generative models during inference.

Lu: The core idea they are pushing is that this dynamic, content-aware scaling stops being a fixed formula and starts reacting to what's actually emerging in the image at each denoising step.

Meng: I still have to ask about the cost of that spectral analysis stage you mentioned earlier, Tom. Does running this analysis slow down my inference time enough to be a real production tool?

Tom: The authors show it requires no retraining or any new architecture changes, just this scaling modification, and they say it consistently improves structure and detail across different models like Flux and Qwen.

Jane: So the implication is that we don't need massive amounts of extra training data just to get high resolution; we can adapt the existing model on the fly using its own data representation.

Lu: It definitely opens up a new direction for how we think about inference in these pre-trained generative models, showing that latent spectral structure is a useful signal for guiding attention scaling at inference time.

Javad Rajabi, Kimia Shaban Koorosh, Roohi David B., Lindell Babak, Taati

University of Toronto Vector Institute

cs.CV

Submitted: 2026-05-21

Updated: 2026-11-15

Comments: NeurIPS 2026. Project page: https://rajabi2001.github.io/sega/

Code: https://github.com/black-forest-labs/flux

Project page: https://rajabi2001.github.io/sega

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 94/100

The gist: Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors…

Key concepts

Diffusion Transformers (DiTs)
These are the dominant models for text-to-image generation. They work by iteratively refining random noise into a coherent image guided by text prompts. However, they often fail when asked to create images at resolutions higher than what they were originally trained on.
RoPE Scaling
Rotary Positional Embedding (RoPE) is a mechanism used in DiTs to encode positional information, which helps the model understand where different parts of an image are located. SEGA modifies how this scaling happens during generation, making it adaptive instead of fixed.
Spectral Analysis
This technique converts the latent image from its spatial representation (pixels) into a frequency domain. It analyzes how energy is distributed across different spatial scales and frequencies within the image at each step, providing a map of the image's structural content.

Terminology

Summary

Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range The authors introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent’s spatial-frequency structure at each denoising step. p. 0

Where the paper says it

Abstract Diffusion transformers (DiTs) have emerged as a dominant architecture for textto-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling.

This adaptive scaling improves both structural coherence and fine-detail fidelity. p. 0
Where the paper says it

We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent’s spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity.

Pruning redundant reasoning tokens

Diffusion transformers (DiTs) have become the dominant approach to text-to-image (T2I) generation, producing images with a level of quality that would have been hard to imagine just a few years ago. Existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability. p. 1

Where the paper says it

1 Introduction Diffusion transformers (DiTs) [3, 4] have become the dominant approach to text-to-image (T2I) generation, producing images with a level of quality that would have been hard to imagine just a few years ago. Despite considerable improvements, existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability [5, 6, 7].

Extrapolating beyond this training resolution at inference time often leads to notable quality degradation and even structural breakdown. p. 1
Where the paper says it

Despite considerable improvements, existing T2I models remain largely constrained by the resolution ranges used during training, typically between 10242 and 20482 resolutions, limiting their practical applicability [5, 6, 7]. Consequently, extrapolating beyond this training resolution at inference time often leads to notable quality degradation and even structural breakdown.

Existing training-free methods for high-resolution image generation generally fall into two categories: (i) direct inference [14, 15, 16, 17] and (ii) multi-stage guidance-based approaches [18, 19, 20, 5, 6]. p. 1
Where the paper says it

These bottlenecks have motivated growing interest in training-free high-resolution synthesis from pre-trained models [10, 11, 12, 13]. Existing training-free methods for high-resolution image generation generally fall into two categories: (i) direct inference [14, 15, 16, 17] and (ii) multi-stage guidance-based approaches [18, 19, 20, 5, 6].

Direct inference methods attempt to extend pretrained models to higher resolutions by modifying the denoising process or adjusting components such as positional encoding and attention without additional training. p. 1
Where the paper says it

Existing training-free methods for high-resolution image generation generally fall into two categories: (i) direct inference [14, 15, 16, 17] and (ii) multi-stage guidance-based approaches [18, 19, 20, 5, 6]. Direct inference methods attempt to extend pretrained models to higher resolutions by modifying the denoising process or adjusting components such as positional encoding and attention without additional training.

In contrast, multi-stage approaches first generate a base-resolution image and then use it to guide high-resolution synthesis. p. 1
Where the paper says it

Direct inference methods attempt to extend pretrained models to higher resolutions by modifying the denoising process or adjusting components such as positional encoding and attention without additional training. In contrast, multi-stage approaches first generate a base-resolution image and then use it to guide high-resolution synthesis.

How the training signal is built

SEGA introduces content-aware dynamic scaling into DiTs by coupling lightweight spectral analysis with RoPE components. The core insight is that RoPE scaling for high-resolution extrapolation should be content-aware rather than fixed and uniform. p. 3

Where the paper says it

4 Method Spectral-Energy Guided Attention (SEGA) introduces content-aware dynamic scaling into DiTs by coupling lightweight spectral analysis with RoPE components. Our key insight is that RoPE scaling for high-resolution extrapolation should be content-aware rather than fixed and uniform.

SEGA achieves this by deriving per-dimension scaling from the latent’s spectral structure at each denoising step. p. 3
Where the paper says it

Our key insight is that RoPE scaling for high-resolution extrapolation should be content-aware rather than fixed and uniform. SEGA achieves this by deriving per-dimension scaling from the latent’s spectral structure at each denoising step.

This formulation involves a dimension-wise scaling term md, defined as fSEGA(x, n, d) = m(a)d · fRoPE(x, n, d), where m(a)d = mref · M(a)d (Z). p. 3
Where the paper says it

Formulation Overview. SEGA applies attention scaling through RoPE using a dimension-wise scaling term md.

The modulator M(a)d (Z) consists of two complementary components: s(a)d (Z), a per-dimension correction that determines the distribution of scaling across RoPE dimensions, and σ(Z), a global amplitude factor that sets the strength of that adjustment. p. 3
Where the paper says it

Here, M(a) d (Z) is our novel dynamic modulator derived from the spectral structure of the current intermediate latent Z. It consists of two complementary components: s (a) d (Z), a per-dimension correction that determines the distribution of scaling across RoPE dimensions, and σ(Z), a global amplitude factor that sets the strength of that adjustment.

Where the gains come from

The spectral analysis stage transforms the current latent from the spatial domain to the frequency domain to characterize its spatial frequency content. This involves extracting two complementary spectral views from a single 2D Fast Fourier Transform F2D: (1) Axis-wise profiles, which map spectral energy to spatial frequencies along each axis, and (2) Radial profile, which provides a rotation-invariant summary of how energy is distributed across spatial scales. p. 4

Where the paper says it

Given the latent hidden states Z ∈ R N×C with N = H · W tokens,1 we reshape them back to their native 2D layout, average across channels, and subtract the average value across the spatial dimensions to obtain a zero-centered 2D map M˜ ∈ R H×W that summarizes the spatial structure of the latent. From M˜ we extract two complementary spectral views from a single 2D Fast Fourier Transform F2D: • Axis-wise profiles.

The second stage converts these spectral profiles into the modulator M(Z), consisting of a reference scale mref, a per-dimension correction s(a)d, and a global gate σ(Z). p. 4
Where the paper says it

2 From Spectrum to Per-Dimension RoPE Scaling The second stage converts the spectral profiles into the modulator M(Z), which defines the perdimension scaling applied to the rotary embeddings. This formulation consists of three components: a reference scale that anchors the scaling, a per-dimension term that scales individual dimensions, and a global gate that controls the strength of that scaling.

The final scaling formula combines these components to define the modulator and the resulting per-dimension scaling m(a)d = mref · M(a)d (Z). p. 5
Where the paper says it

Final scaling formula. Combining the three components, we define the modulator and the resulting per-dimension scaling m (a) d along each spatial axis a ∈ {H, W} as M(a) d (Z) = 1 − σ · s (a) d , m (a) d = mref · M(a) d (Z).

Analysis of Spectral-Energy Guided Attention

The method's effectiveness is demonstrated by its ability to adapt scaling continuously to the latent’s spectral content at each denoising step. In early steps where the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the reference scale mref. p. 5

Where the paper says it

This occurs because SEGA is content-aware, dynamically adapting scaling to the latent’s spatial frequencies. In early steps where the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the reference scale mref.

As distinct structural energy emerges in later steps, SEGA selectively sharpens attention at under-resolved frequencies while softening it at over-emphasized ones. p. 5
Where the paper says it

In early steps where the latent is dominated by noise and the spectrum is relatively flat, the scaling remains near the reference scale mref. However, as distinct structural energy emerges in later steps, SEGA selectively 6

This content-aware spectral redistribution directly impacts the attention mechanism’s stability by preserving semantic locality and entity consistency that uniform scaling methods fail to maintain. p. 6
Where the paper says it

SEGA mitigates this failure mode by shaping the attention grid much earlier in the denoising process. By dynamically modulating the magnitude of rotary embeddings, our method preserves semantic locality and entity consistency that uniform scaling methods fail to maintain.

Experiments

Extensive experiments show that SEGA consistently improves structural coherence and fine-detail fidelity and achieves superior performance across baselines and resolution settings, including ultrahigh resolutions exceeding 36 million pixels. The comparison against state-of-the-art baselines on both Flux [1] and Qwen [2] establishes SEGA as a new state-of-the-art for high-resolution image generation across both architectures. p. 7

Where the paper says it

Quantitative comparison. As shown in Table 1 and Table 2, SEGA establishes a new state-ofthe-art for high-resolution image generation across both the Flux and Qwen architectures.

SEGA outperforms all baseline models across every evaluated metric, setting a new benchmark for high-resolution generation at 40962 resolution. p. 8
Where the paper says it

80 DyPE + SEGA 1. 05 40962 resolution, SEGA outperforms all baseline models across every evaluated metric, setting a new benchmark for high-resolution generation.

Conclusion

SEGA enables stable high-resolution synthesis well beyond the native training regime by adapting RoPE components scaling to the spectral structure of the current latent. This simple modification requires no retraining or architectural changes, yet consistently improves structure, semantics, and visual quality across resolutions and model architectures. p. 8

Where the paper says it

By making attention scaling frequency-aware across RoPE components, SEGA addresses a key limitation of existing uniform scaling strategies, which often trade off global coherence against fine-detail fidelity. This simple modification requires no retraining or architectural changes, yet consistently improves structure, semantics, and visual quality across resolutions and model architectures.

SEGA shows that the latent’s spectral structure can serve as a useful signal for guiding RoPE scaling at inference time. p. 1
Where the paper says it

Consequently, a static, uniform scaling at inference time cannot accommodate these variations. Building on this view, we introduce SEGA (Spectral-Energy Guided Attention), a training-free, content-aware method that dynamically adapts attention scaling to the latent’s spectral structure by deriving per-component scaling magnitudes at each denoising step.

The coupling it reveals between RoPE dimensions and spatial frequencies inspires further research on inference-time adaptation of pretrained generative models. p. 8
Where the paper says it

More broadly, frequencyaware attention scaling may also benefit video and other modalities where resolution extrapolation remains challenging. We hope the spectral perspective guidance introduced here motivates further research on modifying attention behavior, particularly for resolution extrapolation, to better unlock the capacity of pretrained generative models.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Content-aware dynamic scaling for Rotary Position Embeddings (RoPE) allows SEGA to dynamically adapt attention scaling to the latent’s spectral structure by deriving per-component scaling magnitudes at each denoising step. This improves both structural coherence and fine-detail fidelity compared to static scaling, resolving the trade-off between preserving global structure and recovering fine detail.

  2. Spectral analysis of the latent involves extracting axis-wise profiles and a radial profile from the 2D power spectrum to characterize spatial frequency content. These profiles determine scaling, enabling SEGA to sharpen attention at under-resolved frequencies and soften it at over-emphasized ones based on the latent's actual spectral energy.

  3. SEGA enables stable high-resolution synthesis across a wide range of extrapolated resolutions, including ultrahigh resolutions exceeding 36 million pixels. It achieves this by preserving local texture and object-level detail while maintaining the broader structure of the image, as shown in Figure 14.

  4. The method's performance is superior to many existing training-free baselines, consistently achieving the highest semantic alignment and image quality across different settings on both Flux and Qwen architectures at high resolutions.

  5. SEGA provides robustness and consistency across diverse high resolutions, maintaining highly stable results even when other models experience significant performance drops as the resolution increases.

Abstract

Diffusion transformers (DiTs) have emerged as a dominant architecture for text-to-image generation, yet their performance drops when generating at resolutions beyond their training range. Existing training-free approaches mitigate this by modifying inference-time attention behavior, often through Rotary Position Embeddings (RoPE) extrapolation combined with attention scaling. However, these strategies apply a uniform and content-agnostic scaling across RoPE components with distinct frequency characteristics, inducing a trade-off between preserving global structure and recovering fine detail. We introduce SEGA, a training-free method that dynamically scales attention across RoPE components according to the latent's spatial-frequency structure at each denoising step. This adaptive scaling improves both structural coherence and fine-detail fidelity. Experiments show that SEGA consistently improves high-resolution synthesis across multiple target resolutions, outperforming state-of-the-art training-free baselines.

Sources

Related papers