SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting

arXiv:2610.11170 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting".

Tom: The gist: SACQ, a plug-in structured prediction head, replaces flatten readout in long-horizon time series forecasting by adopting a two-stage decoding pipeline involving coarse scaffold generation,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Hey Jane, so we're looking at this paper called SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting. It sounds like they're tackling the problem of how long-horizon time series forecasting models handle those long stretches of future steps by moving away from a simple flat readout.

Jane: Right, Tom, so basically, instead of one single projection mapping all historical memory to all future points at once, they're proposing a way to decode it in stages. It seems like they are trying to make the connection between what happened in the past and what's coming in a more explicit way.

Lu: I think the core idea here is that this implicit coupling of a single shared projection obscures how individual future positions relate back to specific historical data points, which makes them really sensitive to bad inputs or noisy labels, as they found on page two of THIS PAPER SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting <ref:2610.11170#pg1,SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting>.

Meng: So you're saying the current setup is a bit too opaque? What exactly does this two-stage pipeline look like in practice, beyond just saying it's not a flat readout?

Tom: Exactly, Meng. They break it down into building a coarse patchgrid scaffold first, which gives them that initial structure. Then they use cross-attention to refine each future position by querying the historical memory directly.

Jane: That sounds like they are giving the model a chance to look back at relevant parts of the history for every single step it's trying to predict in the future. It’s like having personalized lookups instead of one giant lookup table for everything at once.

Lalam: From an LLM perspective, this structured decoding sounds really helpful because it allows the model to apply different logic depending on where it is in the prediction sequence, which should lead to much more nuanced outputs.

Title and authors: Lu: And they formalize that refinement using a gated fusion step, merging that attention correction with the initial coarse scaffold through a learned perpatch gate, which is detailed in section III.C of THIS PAPER SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting.

Tom: That gating mechanism is key, right? It lets the model decide how much to trust the coarse structure versus the fine-tuned correction from the attention layers.

Meng: From an engineering standpoint, that means we’re not just throwing a bunch of attention layers at it; they are combining those signals in a structured way before it hits the final output, which should stabilize optimization under long horizons.

Jane: And to handle the training side, they use this batch-adaptive scaled log-cosh loss to keep things stable when dealing with noisy labels or extreme supervision.

Lalam: That loss function calibrates its own scale based on the current minibatch median residual, which is smart because it means it adjusts its sensitivity automatically instead of relying on a fixed setting.

Tom: So they're addressing two big issues: the structural problem of how long sequences are read and the training problem of instability with noisy data.

Jane: And they show that SACQ performs well across different encoders, like PatchTST and patch-Mamba, without needing to change the core encoding part.

Lu: The results on page one of THIS PAPER SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting show that under patch-Linear settings, the benefit is largest on difficult ETTh2 longhorizon settings where flatten errors are highest <ref:2610.11170#pg1,SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting>.

Tom: So it’s not just better in one area; it shows a real improvement across different backbone architectures when you look at those tough long-term scenarios.

Meng: What about robustness? They tested this against inference-time input corruption using the TSRBench protocol, and SACQ actually retains the best or tied-best cells as the corruption severity rises.

Jane: That’s a significant finding because it means this structured approach is more resilient when you introduce real-world noise during testing than just relying on a flat readout does.

Title and authors: Lalam: The structural alignment with causal Readout, or SAR, is another part of this work that helps preserve the SACQ pipeline by changing only the self-attention layers that operate on future-patch queries.

Lu: SAR applies a causal mask Mij equals zero if j less than i and Mij equals negative infinity if j greater than or equal to i so that information flows monotonically along increasing future-patch index, without forcing an explicit chain factorization on the outputs.

Tom: So they’re ensuring that the model processes information in a sensible temporal order for these future queries, which is important for long sequences.

Jane: And they also look at how it handles target spikes during training using that scaled log-cosh objective and found that S+slc is strongest on Weather and on MAE at long horizons.

Meng: The efficiency analysis shows that while the total parameters and latency grow with backbone width, SACQ-only curves rise more slowly compared to the full model, suggesting it’s a plug-in head with bounded deployment overhead.

Lalam: It seems like this paper is really focused on making long-horizon forecasting more reliable by making the connection between past and future explicit through computation.

Tom: So, to wrap up this discussion of SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting, it’s about moving from implicit coupling to an explicit two-stage decoding process with memory-conditioned refinement.

Jane: It’s about building a system that can handle long sequences more reliably by separating the coarse structure from the fine refinement and fusing them intelligently.

Lu: The implication is that we can get better performance under label noise and input perturbations because the connections are made computationally rather than hidden in parameters.

Meng: For us engineers, it means this head is a modular component you can plug into existing encoders, like PatchTST or patch-Mamba, and it should give us a solid boost in accuracy without adding excessive computational weight.

Lalam: And for the broader culture of AI development, this suggests that structured prediction methods are a really strong direction for tackling complex sequence modeling challenges in real-world forecasting.

The paper's summary: Tom: So, we’re looking at SACQ today, and to recap: this paper is about replacing that simple flat readout in long-term time series forecasting with a whole two-stage decoding pipeline.

Jane: Exactly. It's not just changing one part; it's building a system where the model first gets a rough sketch of the future steps, then it refines those steps by looking back at the historical memory for each specific future point.

Lu: What’s really interesting is how they do that refinement using cross-attention, which means every single future position queries the entire history to get a correction residual.

Meng: And they combine that attention-derived correction with the initial coarse scaffold using this learned perpatch gate, which lets the model decide how much to trust each signal.

Tom: That gating mechanism is what keeps things stable when you’re predicting far into the future, especially when those input labels are a bit noisy.

Jane: And to handle that instability during training, they use this batch-adaptive scaled log-cosh loss, which adjusts its own sensitivity based on the current batch of data.

Lu: That loss function calibres itself by setting a scale parameter beta based on the batch median residual, which means it treats small errors like squared error but dampens larger ones.

Meng: It’s smart because it avoids using a fixed calibration value that might not fit the whole dataset perfectly.

Tom: And they showed this works well across different models, even keeping the core patch encoders exactly the same.

Jane: That’s important because it means you don't have to rebuild your entire forecasting backbone just to use this readout head.

Lu: They found that SACQ improves performance over flatten readouts in PatchTST and patch-Mamba, especially when things get really long, like at a time horizon of seven hundred steps.

Meng: That kind of consistent improvement across different architectures suggests the underlying structural idea is quite robust.

Tom: And they even looked at how it handles input corruption during testing with this causal readout alignment, which they call SAR.

Jane: SAR ensures that as the model looks ahead into the future, the information flows in a sensible way—monotonically increasing along those future steps.

Lu: It applies a causal mask where each future position only sees information from earlier ones, without forcing a chain factorization on every single output step.

Tom: So we’re moving away from just relying on a massive shared projection to something much more explicit and structured.

Jane: It’s about making the historical-to-future correspondence an explicit computation, rather than a hidden pattern in the parameters.

Lu: This shifts the focus from implicit coupling to staged decoding, which is a fundamental change in how we think about these prediction heads.

Meng: From an engineering standpoint, it means this head is a modular component you can plug into existing encoders without needing to overhaul your entire system structure.

Tom: It’s definitely a step toward making long-term forecasting more reliable, even when the data gets messy or the horizon gets huge.

Jane: And that robustness we talked about earlier? It means better performance under both training noise and real-world input perturbations.

Lu: So, this paper is really focused on creating a structured prediction head that handles long sequences by separating the coarse structure from the fine refinement and fusing them smartly.

Meng: The efficiency analysis also suggests that while there’s some overhead, SACQ-only curves rise slower than the full model, so it's not adding massive latency for deployment.

Tom: It’s a solid plug-in head that brings accuracy gains without a huge hit to runtime complexity.

Jane: So, this work suggests that by making those connections explicit and structuring the decoding process, we can get more reliable results for long-horizon time series forecasting.

Lu: We’ve seen how this impacts different models and how it handles various data stresses during testing.

Meng: Next up, we’re going to look at the SAR mechanism in more detail on page eight.

The paper's improvements: Tom: So, we’re talking about what makes SACQ actually better than just some standard readout head for long sequences.

Jane: Well, the main improvement is that it gets rid of that single shared projection from a flat readout and replaces it with this two-stage decoding process.

Lu: It starts by building a coarse patchgrid scaffold first, which gives you that initial global structure before you even think about fine details.

Meng: Then, for every future position, it applies cross-attention to query the historical memory directly for a position-specific correction residual.

Tom: And that’s where the power comes in: you merge that attention refinement with the coarse scaffold using a learned perpatch gate.

Jane: That gate is crucial because it lets each future step decide how much to trust the rough structure versus the detailed historical correction.

Lu: This approach fundamentally changes how we think about making connections between what happened in the past and where we are going in time.

Meng: It’s like instead of one giant lookup table for everything, you get a set of personalized lookups tailored to each specific point in the future sequence.

Tom: And this makes it much more stable when the input data is noisy or if you’re predicting really far out into the horizon.

Jane: They achieved that stability by using a batch-adaptive scaled log-cosh loss, which recalibrates its error sensitivity based on what’s in the current batch.

Lu: That loss function effectively manages those noisy labels and extreme supervision by damping the penalties for large errors while keeping small errors MSE-like.

Meng: And they proved this works well across different model types like PatchTST and patch-Mamba, which shows it isn't tied to one specific architecture.

Tom: The SAR mechanism they introduced, Structural Alignment with Causal Readout, is also a big part of the improvement for robustness.

Jane: SAR changes only the self-attention layers that look at future patches by applying a causal mask so information flows in a sensible temporal order.

Lu: This means it helps prevent bad information from leaking across time when we’re refining those future steps.

Meng: So, what does this mean for practical deployment? It’s a plug-in head that improves accuracy without needing to change the entire model architecture for inference or training.

Tom: For someone just listening in, it means you get a more reliable forecast when dealing with long sequences and noisy data because the model is structured to handle those challenges explicitly.

Jane: It moves us closer to having forecasting models that can handle really complex, long-term predictions with better confidence.

Lu: Future work they mentioned is looking at lower latency versions of this SACQ pipeline and testing it under even more severe distribution shifts or irregular time horizons.

Meng: So the next thing we need to watch is how fast they can make this run in a real-world production environment where speed matters.

Conclusion: Tom: So we’re wrapping up on SACQ, which is really about using structured decoding to replace that simple flat readout in long time series forecasting.

Jane: It’s essentially about making the connection between history and future an explicit, two-stage process involving a coarse scaffold and memory-conditioned refinement.

Lu: The key takeaway here is that this method stabilizes optimization under long horizons and noisy labels by separating the structure building from the correction step.

Meng: And it shows that you don't need to change the core encoder; you just plug in this head, which is pretty practical for engineers.

Tom: Right, so we’re talking about a structured prediction head that handles long sequences with better stability and accuracy under tough conditions.

Jane: It changes how we build these forecasting models by moving from implicit coupling to an explicit pipeline where every future step gets its own tailored correction.

Lu: This concept opens up possibilities for creating truly adaptive forecasting systems that can react differently depending on the specific position in the sequence.

Meng: From a practical standpoint, it means less guesswork when we’re deploying these models because the training objective is much more stable.

Tom: Exactly, so SACQ is about making those long-term predictions more reliable by being transparent about how they are being calculated.

Jane: And the work on SAR shows that this structure actually helps preserve that reliability when you introduce input corruption during testing.

Lu: The potential here is in creating models that can maintain high fidelity even when the input data itself is messy or shifted from what it was trained on.

Meng: It’s a significant step toward building AI systems that are more robust to real-world data imperfections, which is always going to be necessary.

Tom: So this paper, SACQ: Structured Decoding with Memory-Conditioned Refinement for Long-Horizon Forecasting, gives us a solid method for tackling those long sequence challenges.

Jane: It really emphasizes the power of explicit computation over hidden parameter patterns for better performance.

Lu: Next up, we’re going to look at how they handle those extreme distribution shifts in more detail on page eight.

Guo Cheng, Zhengzhuo Xu, Chenchen Jing, Jingyi Hou

University of Science and Technology Beijing, China · Zhejiang University of Technology, Hangzhou, China

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 10 pages, 7 figures

Code: https://github.com/PLACEHOLDER/SACQ

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: SACQ, a plug-in structured prediction head, replaces flatten readout in long-horizon time series forecasting by adopting a two-stage decoding pipeline involving coarse scaffold generation,

Key concepts

Two-Stage Decoding
This method breaks down long-horizon prediction into two explicit steps instead of one. The first stage builds a rough structural map (scaffold) of the future sequence. The second stage uses this scaffold to query historical memory for specific corrections, leading to a more precise final forecast.
Memory-Conditioned Refinement
Each future prediction step queries the model's historical memory using cross-attention. This allows the model to generate position-specific correction residuals based on relevant past information, refining the initial coarse scaffold into a detailed forecast.
Gated Fusion
A learned gate is used to merge the coarse patch forecast and the attention-derived prediction residual. This mechanism selectively combines structural information from the scaffold with fine-grained corrections from attention, producing a stable final output.
Batch-Adaptive Scaled Log-cosh Loss
This robust loss function stabilizes optimization during training on long sequences and noisy data. It automatically adjusts the penalty based on the current batch's residual scale, treating very small errors differently than larger ones to ensure stable learning.

Summary

The gist: SACQ, a plug-in structured prediction head, replaces flatten readout in long-horizon time series forecasting by adopting a two-stage decoding pipeline involving coarse scaffold generation, memory-conditioned refinement via cross-attention, and gated fusion to stabilize optimization under long horizons and noisy labels.

How it works

SACQ recasts long-horizon forecasting as explicit staged decoding rather than relying on the implicit coupling of a single shared projection in flatten readouts. The pipeline involves three main stages: first, building a coarse patch-grid scaffold through a lightweight projection that preserves global structure. Second, applying cross-attention so each future position queries historical memory to produce a position-specific correction residual. Third, merging the attention-derived correction with the coarse scaffold via a learned perpatch gate.

SACQ Head Design Principles

The design principles for SACQ focus on making historical-to-future correspondence explicit through computation rather than hidden parameter patterns. Key design principles include:

  1. P1: two-stage decoding reduces oneshot regression difficulty by separating patch-grid coarse structure from memory-conditioned refinement.

  2. P2: explicit query–key alignment restores historical-to-future correspondence as computation, not hidden parameter patterns.

  3. P3: joint reencoding of coarse priors and cross-attention outputs combines these signals through feature concatenation followed by linear projection, keeping coarse scaffolding and high-order interactions in one space before self-attention and mitigating brittle propagation when horizons are long.

  4. P4: gated residual readout fuses the coarse patch forecast and the attentionproduced prediction residual under a learned gate, so the final output is a gated mixture of coarse structure and refinement.

Robust Training Objective

To stabilize optimization under long horizons and noisy labels, SACQ adopts a batch-adaptive scaled log-cosh loss. This loss function calibrates robustness to the current residual scale by rescaling residuals before applying log-cosh, setting the width of the robust transition region based on the batch median residual. Specifically, it uses a formula where errors much smaller than β are treated like MSE, while larger errors receive damped penalties. The scale β is estimated automatically from the current minibatch to avoid dataset-specific calibration.

Structural Alignment with Causal Readout (SAR)

The Structural Alignment with causal Readout (SAR) module preserves the SACQ pipeline by changing only the self-attention sublayers that operate on future-patch queries. SAR applies a causal mask Mij = 0 if j < i and Mij = −∞ if j ≥ i, so that information flows monotonically along increasing future-patch index. This encourages representations to be refined from near to far without imposing an explicit chain factorization on scalar or patch outputs.

Performance and Robustness

SACQ attains top-tier test MSE/MAE across PatchTST, DLinear, and patch-Mamba backbones with only modest incremental overhead in parameters and latency. In robustness tests, SACQ substantially outperforms flatten readouts under inference-time input corruption and trainingset label-noise stress tests. For example, in inference-time input corruptions using the TSRBench protocol, PatchSACQ retains the most best or tied-best cells as severity rises.

Efficiency Analysis

The efficiency analysis shows that while total parameters and end-to-end latency grow with backbone width dmodel, the SACQ-only curves rise more slowly compared to the full model. At dmodel=256, the incremental decoder adds roughly 2.7M parameters and about 2.6 ms per forward pass, versus ∼8.2M parameters and ∼9.2 ms for the full model. This suggests SACQ is a plug-in head with bounded deployment overhead.

Conclusion

SACQ successfully recasts long-horizon forecasting as an explicit prediction mapping problem while keeping patch encoders unchanged and replacing flatten readout with staged decoding, conditional future queries, memory-guided refinement, and gated fusion trained with batch-adaptive scaled log-cosh for stable multistep optimization. SACQ consistently improves accuracy over strong baselines under a unified benchmark protocol. Future work will target lower-latency SACQ variants and broader evaluation under severe distribution shifts and irregular horizons.

--- Page 9 ---

Table VI reports test MSE/MAE averaged over Tf ∈ 96, 192, 336, 720 with N=512. BOLD: BEST MSE/MAE AMONG PATCHSACQ, PATCHLGA, AND PATCHTST AT EACH (dataset, severity).

--- Page 8 ---

Fig. 5 plots mean α across forecast steps. Spiked input histories raise mean α above clean-input levels.

--- Page 7 ---

Fig. 6 varies lookback N ∈ 96, 336, 512, 720, 1440, 1920 on ETTh1 with fixed readout pairings per curve.

--- Page 6 ---

Table II reports Tf ∈ 96, 720 under patch-Linear and patch-Mamba backbones. BOLD MARKS THE BETTER MSE/MAE IN EACH BACKBONE PAIR AT EACH (dataset, Tf).

--- Page 5 ---

Fig. 2 illustrates the cross-self attention module structure.

--- Page 4 ---

Table III ablates individual SACQ modules on ETTh1, ETTm1, Weather, and ECL at Tf=720.

--- Page 3 ---

Fig. 3 shows robustness on ETTh1 under training-set corruption.

--- Page 2 ---

Table I compares SACQ against LSINet [13], PatchLGA [9], PatchTST [4], CI-TSMixer, FiLM [21], DLinear [6], FEDformer [16], and TimesNet [15].

--- Page 1 ---

Abstract—Long-term time series forecasting (LTSF) models predominantly employ patch-based encoders terminated by a flatten readout head that maps the entire encoded historical memory to all future steps through a single shared projection. SACQ, a plug-in structured prediction head that replaces flatten readout while keeping the encoder unchanged.

--- Page 10 ---

Fig. 7 compares end-to-end totals against SACQ-only overhead.

--- Page 9 ---

Table VI reports test MSE/MAE averaged over Tf ∈ 96, 192, 336, 720 with N=512.

--- Page 8 ---

Fig. 4 shows SAR ablation under training-set tail shift.

--- Page 7 ---

Fig. 5 plots mean gate α across clean versus spiked input histories.

--- Page 9 ---

Table V reports clean-test MSE and MAE on ETTh1 and Weather at Tf=720.

--- Page 6 ---

Table IV reports ETTh1, ETTm2, and Weather at Tf ∈ 96, 720.

--- Page 4 ---

Table I compares SACQ against published baselines.

--- Page 3 ---

Table II reports L/M = backbone, F/S = flatten (MSE) vs. SACQ (scaled log-cosh) on ETT.

--- Page 2 ---

Fig. 1 shows Flatten readout vs. SACQ decoding.

--- Page 1 ---

Index Terms—Long-term time series forecasting, Prediction head, Structural decoding, Robust loss, Memory-conditioned attention.

--- Page 4 ---

Section III.C discusses the Gated fusion in detail.

--- Page 5 ---

Eq. (6) defines the scaled log-cosh objective.

--- Page 8 ---

Eq. (7) estimates β from the batch median residual.

--- Page 9 ---

Eq. (8) defines the implemented training loss.

--- Page 2 ---

Section II.B discusses Prediction Heads as Structured Decoders.

--- Page 3 ---

Section III.C details the SACQ Head design principles.

--- Page 5 ---

Eq. (4) defines the Gated Residual Readout.

--- Page 6 ---

Eq. (3) defines the coupled representation at layer L.

--- Page 7 ---

Eq. (5) defines the causal mask Mij.

Improvements for AI systems

  1. No more representation coupling limitations in readout heads by replacing flatten readout with SACQ, which adopts a "two-stage decoding pipeline: it first establishes a coarse patchgrid forecast scaffold, then refines each future position through cross-attention over historical memory and merges the attention-derived correction with the coarse scaffold via a learned perpatch gate. This allows models to apply differentiated readout logic at different future positions, stabilizing performance under inference-time input perturbations."

  2. Robust training against label noise and extreme supervision is achieved by adopting a batch-adaptive scaled logcosh loss defined as l(e; β) = β−2 log cosh(e/β), where the scale is automatically calibrated by setting β = k m with k=0.5, ensuring that errors much smaller than β are treated like MSE, whereas errors much larger than β receive damped, approximately linear penalties.

  3. Improved generalization across different encoder families is realized because SACQ maintains its structure across backbones, as demonstrated by results showing SACQ lowers both metrics versus flatten on every reported cell, specifically highlighting that under patch-Linear, the benefit is largest on difficult ETTh2 longhorizon settings where flatten errors are highest; under patchMamba, SACQ wins uniformly and the gap widens at Tf=720.

  4. Enhanced robustness to input corruption is achieved through the SAR (Structural Alignment with causal Readout) mechanism, which "preserves the SACQ pipeline in Sec. III-C—historical memory H, coarse scaffold F and content embedding C, cross-attention to H, concat–linear fusion with C in Eq. (3), gated residual readout in Eq. (4), and non-autoregressive unpatchfying to Yˆ —and changes only the self-attention sublayers that operate on future-patch queries. This mechanism ensures information flows monotonically along increasing future-patch index, making the model more resilient to input corruption" during testing.

  5. The system can better handle target spikes during training by utilizing the scaled log-cosh objective, as shown in Table IV, where S+slc is strongest on Weather and on MAE at long horizons, demonstrating that the robust loss effectively suppresses gradients from extreme outliers while preserving MSE-like sensitivity for typical errors.

Related papers