Frequency Domain Reservoir Computing
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Frequency Domain Reservoir Computing".
Jane: Here's the gist of the paper. Frequency Domain Reservoir Computing introduces FRESCO, an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear recurrent updates By employing a novel dimensional zero-padding input embedding, a packed frequency-domain readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference Furthermore, FRESCO matches the state-ofthe-art predictive performance on memory benchmarks, sequential classification, and multivariate long-horizon forecasting.
Tom: Pruning redundant reasoning tokens The paper addresses the quadratic sequence-length bottleneck of transformers by introducing Frequency Domain Reservoir Computing (FRESCO), an ESN architecture that operates entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear recurrent updates.
Jane: This approach circumvents the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling a novel dimensional zero-padding embedding.
Tom: By replacing the dense O(N 2) recurrent product with element-wise Hadamard multiplications between frequency-domain state and weight representations, the per-step cost of the recurrent update drops to O(N), even for a fully dense, non-linear reservoir.
Title and authors: Tom: We're looking at "Frequency Domain Reservoir Computing" by Klaus Schertler and Xiomara Runge. It sounds like they are taking the idea of reservoir computing and moving it entirely into the frequency domain to solve scaling issues.
Jane: That’s what it suggests, and the title makes it clear that they are aiming to eliminate some of those domain shift headaches that usually come with these types of models.
Lu: Basically, they are addressing that quadratic sequence-length bottleneck in transformers by introducing FRESCO, which is an ESN architecture operating entirely in the frequency domain >
Meng: So how does moving it to the frequency domain actually help beyond just changing the math? What’s the main practical hurdle they overcame?
Tom: They circumvented it by lifting that one dimensional circular convolution formulation to two dimensions, which allows them to use a novel dimensional zero padding embedding >
Jane: That embedding strategy is key because it means you don't need a full N point FFT anymore for your input transformation; you can use a single O of n log n FFT instead >
Lu: Because they replace the dense O of N squared recurrent product with element wise Hadamard multiplications between frequency domain state and weight representations, which drops the per step cost to O of n >
Meng: So it’s not just about the input transformation being faster, but it's about making sure that every single recurrent update is fast enough for large reservoir sizes >
Tom: Right. And they call this packed frequency-domain readout because they use the Hermitian symmetry of the reservoir state to get a contiguous real valued vector without needing an inverse FFT >
Jane: So that’s a nice win because it means no extra FFT steps after you finish the main computation, which really helps inference speed up >
The paper's summary: Tom: This paper, "Frequency Domain Reservoir Computing" is about taking the standard Echo State Network and reformulating it in the frequency domain to exploit those inherent efficiencies of that processing technique >
Jane: And they systematically avoid the usual slow Fast Fourier Transforms that are usually associated with these approaches, which is a major hurdle in this area >
Lu: They reformulate the standard ESN equation so you can use those frequency domain techniques while circumventing the costly FFTs usually associated with these approaches >
Meng: So what’s the actual mechanism for avoiding those FFT costs in this new formulation? Is it just clever input embedding?
Tom: The input embedding strategy is dimensional zero padding, which lets an input of size N times something be transformed using a single O of n log n FFT instead of a full N point FFT >
Jane: And that’s how they get the initial transformation step to be efficient enough for their needs because it replaces a full N point FFT with just that specific one >
Lu: The key is replacing the dense O of N squared recurrent product with element wise Hadamard multiplications between frequency domain state and weight representations, leading to the state update in the frequency domain given by equation four >
Meng: So they're basically saying you can do all your recurrent calculations just by doing simple element wise products now? That sounds like a huge reduction in complexity >
Tom: It is. And they show that this new formulation results in O of N complexity for dense, non linear recurrent updates, even for a fully dense reservoir >
Jane: So the overall message is that frequency domain processing provides a principled route to overcome the quadratic bottleneck and makes it efficient enough to scale properly >
The paper's improvements: Tom: They lay out several specific improvements they made to make this work better, and these are what really matter for engineering implementation >
Jane: What are the key things they did that move beyond just the basic O of N complexity claim? Are there structural changes in how you build the reservoir?
Lu: One improvement is dimensional zero padding, which we already touched on but it’s a specific strategy mapping low dimensional inputs into the frequency domain, eliminating standard transform bottlenecks >
Meng: And then they have this packed frequency-domain readout which is designed to be redundancy free and memory contiguous and FFT free >
Tom: That readout leverages the Hermitian symmetry of the reservoir state to give you a contiguous real valued vector without requiring inverse FFTs for training or inference >
Jane: So that's a nice practical detail because it means you’re not adding complexity after the main computation when you need to do your final prediction step >
Lu: Plus, they also introduce a structural mixing mechanism via a discrete circular shift to couple frequency components without sacrificing optimal scaling and enabling rich cross frequency harmonic mixing in the reservoir dynamics >
Meng: So that means it’s not just one trick, but they’re actively designing the dynamics for better feature representation during training >
Tom: Right. It sounds like they’s not just optimizing the speed; they's also improving how features are learned through this mixing mechanism >
Conclusion: Jane: So to wrap up, this paper on "Frequency Domain Reservoir Computing" shows that shifting computation to the frequency domain is a very sustainable alternative to more resource intensive sequence modeling approaches >
Tom: It proves that the mathematical elegance of the frequency domain offers a powerful and highly sustainable alternative for dense recurrent layers >
Lu: It achieves O of n complexity for dense, non linear recurrent updates by replacing the dense O of n squared recurrent product with element wise Hadamard multiplications between frequency domain state and weight representations >
Meng: From an engineering standpoint, they’re achieving massive inference speedups compared to standard ESNs and matching predictive accuracy on long horizon forecasting while consuming up to three orders of magnitude less energy >
Lalam: I'm confident that this work is very solid because the mathematical equivalence to spatial training through proposition four means it’s provably correct in that regard >
Jane: It just shows that the mathematical elegance of this approach is a powerful way to bypass those quadratic bottlenecks in sequence modeling >
Airbus Central Research & Technology · University of Pisa · Bielefeld University
cs.LG
Submitted: 2026-06-23
Updated: 2026-11-10
Comments: Accepted to NeurIPS 2026 (main track); camera-ready version
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: Frequency Domain Reservoir Computing introduces FRESCO, an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense…
Key concepts
- Frequency Domain Reservoir Computing (FRESCO)
- FRESCO is a novel ESN architecture that performs all recurrent calculations within the frequency domain. It replaces the dense, quadratic recurrent product with simpler element-wise multiplications between frequency-domain states and weights, drastically reducing computational cost while preserving non-linear dynamics.
- Dimensional Zero-Padding
- This is an input embedding strategy that transforms an input of size Nx into a 2D representation. It allows the input to be processed using a single FFT operation of O(Nx log Nx) instead of a full N-point FFT, significantly lowering the computational cost during the initial transformation step.
- Packed Frequency-Domain Readout
- This readout method exploits the Hermitian symmetry of the reservoir state. It allows for extracting a contiguous, real-valued vector directly from the frequency domain without needing an inverse FFT, making inference faster and more efficient than traditional methods.
- O(N) Complexity
- The goal of FRESCO is to achieve linear complexity for recurrent updates. This is accomplished by using frequency domain techniques to replace the dense O(N^2) matrix-vector product with element-wise Hadamard multiplications, ensuring that the per-step cost scales linearly with the reservoir size N.
Terminology
Summary
Frequency Domain Reservoir Computing introduces FRESCO, an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear recurrent updates By employing a novel dimensional zero-padding input embedding, a packed frequency-domain readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference Furthermore, FRESCO matches the state-ofthe-art predictive performance on memory benchmarks, sequential classification, and multivariate long-horizon forecasting. p. 0 By employing a novel dimensional zero-padding input embedding,
a packed frequency-domain readout, and a natively applied frequency-domain
non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference. Furthermore, FRESCO matches the state-ofthe-art predictive performance on memory benchmarks, sequential classification,
and multivariate long-horizon forecasting, offering a scalable path forward for
dense recurrent architectures.Where the paper says it
Pruning redundant reasoning tokens
The paper addresses the quadratic sequence-length bottleneck of transformers by introducing Frequency Domain Reservoir Computing (FRESCO), an ESN architecture that operates entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear recurrent updates. This approach circumvents the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling a novel dimensional zero-padding embedding. p. 1 In this paper, we introduce frequency domain reservoir computing (FRESCO), an ESN architecture
that operates entirely in the frequency domain and is designed for efficiency. FRESCO circumvents
the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling
a novel dimensional zero-padding embedding. This allows an input of size Nx to be transformed
using a single O(Nx log Nx) FFT rather than a full N-point FFT. By replacing the dense O(N2
)
recurrent product with element-wise Hadamard multiplications between frequency-domain state and
weight representations, the per-step cost of the recurrent update drops to O(N), even for a fully dense,
non-linear reservoir. This allows an input of size Nx to be transformed
using a single O(Nx log Nx) FFT rather than a full N-point FFT. By replacing the dense O(N2
)
recurrent product with element-wise Hadamard multiplications between frequency-domain state and
weight representations, the per-step cost of the recurrent update drops to O(N), even for a fully dense,
non-linear reservoir.Where the paper says it
Where the paper says it
Where the paper says it
How the training signal is built
The core innovation of FRESCO is to i) reformulate the standard ESN of Eq. (1) in the frequency domain to exploit the inherent efficiencies of frequency-domain processing techniques, yet ii) systematically circumvent costly FFTs usually associated with these approaches. p. 3 3 FRESCO
The core innovation of FRESCO is to i) reformulate the standard ESN of Eq. (1) in the frequency
domain to exploit the inherent efficiencies of frequency-domain processing techniques, yet ii) systematically circumvent costly FFTs usually associated with these approaches. 3 FRESCO
The core innovation of FRESCO is to i) reformulate the standard ESN of Eq. (1) in the frequency
domain to exploit the inherent efficiencies of frequency-domain processing techniques, yet ii) systematically circumvent costly FFTs usually associated with these approaches. FRESCO circumvents
the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling
a novel dimensional zero-padding embedding. This allows an input of size Nx to be transformed
using a single O(Nx log Nx) FFT rather than a full N-point FFT. This allows an input of size Nx to be transformed
using a single O(Nx log Nx) FFT rather than a full N-point FFT. By replacing the dense O(N2
)
recurrent product with element-wise Hadamard multiplications between frequency-domain state and
weight representations, the per-step cost of the recurrent update drops to O(N), even for a fully dense,
non-linear reservoir.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The efficiency gains stem from several key architectural advantages detailed in FRESCO. These include:
-
Frequency domain reservoir computing, which executes with O(N) complexity natively in the frequency domain. p. 0
Where the paper says it
To address this limitation, we introduce Frequency Domain Reservoir Computing (FRESCO), an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear recurrent updates. By employing a novel dimensional zero-padding input embedding, a packed frequency-domain readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference.
-
Dimensional zero-padding, which reduces the FFT complexity from O(N log N) to O(Nx log Nx). p. 1
Where the paper says it
FRESCO circumvents the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling a novel dimensional zero-padding embedding. This allows an input of size Nx to be transformed using a single O(Nx log Nx) FFT rather than a full N-point FFT.
-
Packed frequency-domain readout, which exploits the Hermitian symmetry of the reservoir state to yield a contiguous, real-valued vector without requiring inverse FFTs. p. 1
Where the paper says it
Finally, a packed frequency-domain readout exploits the Hermitian symmetry of the reservoir state to yield a contiguous, real-valued vector. On this representation, ridge regression is provably equivalent to spatial-domain training, entirely removing the inverse FFT from both training and inference.
Performance and Stability
FRESCO maintains competitive performance by ensuring mathematical equivalence to spatial training through unitary transformations. Specifically, the packed frequency-domain readout is mathematically identical to training on the spatial states via Proposition 4, which states that performing ridge regression on the transformed features is equivalent to the original domain. p. 5 This
mechanism executes training (via standard ridge regression) and inference (via a real-valued linear
readout) natively on the frequency-domain representation, completely eliminating computational
overhead while maintaining mathematical equivalence to a training process performed explicitly on
the inverse-transformed reservoir states. This equivalence stems from the unitary nature of the normalized DFT: performing ridge regression on
the flattened complex state vec(Rˆ ) is mathematically identical to training on the spatial states (Proposition 4, Appendix A. Proposition 3 (Necessary condition for FRESCO ESP). A necessary condition for FRESCO to
satisfy the Echo State Property is
Y
N1
i=1
Y
N2
j=1
|Wˆ
r
[i, j]|
1/N
≤
1
α
. We use average pooling over the packed reservoir
states to extract a single feature vector for the readout classifier (details in Appendix C. As detailed in Table 1, FRESCO achieves competitive predictive performance alongside substantial
inference speedups over standard ESNs, reaching up to a factor of 26.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
FRESCO demonstrates that the computational bottlenecks of dense recurrent layers can be elegantly bypassed by shifting computation to the frequency domain. It proves that the mathematical elegance of the frequency domain offers a powerful, highly sustainable alternative to more resource-intensive sequence modeling approaches. p. 8 Beyond sheer speed, it matches the predictive accuracy of heavily parameterized
deep sequence models on complex, long-horizon multivariate forecasting while consuming up to
three orders of magnitude less energy. Ultimately, this proves that the mathematical elegance of
the frequency domain offers a powerful, highly sustainable alternative to more resource-intensive
sequence modeling approaches. While achieving strictly linear scaling
requires constraining recurrent connectivities to
circular convolutions, the absence of statistically
significant accuracy differences across our classification benchmarks confirms that this structural
prior does not systematically compromise expressivity. Empirically, FRESCO achieves highly favorable scaling for both input embedding and recurrent updates, surpassing the inference speeds of
standard ESNs. Empirically, FRESCO achieves highly favorable scaling for both input embedding and recurrent updates, surpassing the inference speeds of
standard ESNs. Beyond sheer speed, it matches the predictive accuracy of heavily parameterized
deep sequence models on complex, long-horizon multivariate forecasting while consuming up to
three orders of magnitude less energy.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
Frequency Domain Reservoir Computing
Klaus Schertler
Airbus Central Research & Technology
Taufkirchen, Germany
klaus.schertler@airbus. p. 0 Frequency Domain Reservoir Computing
Klaus Schertler
Airbus Central Research & Technology
Taufkirchen, Germany
klaus. schertler@airbus.Where the paper says it
Xiomara Runge
Airbus Central Research & Technology
Taufkirchen, Germany
xiomara.runge@airbus. p. 0 schertler@airbus. com
Xiomara Runge
Airbus Central Research & Technology
Taufkirchen, Germany
xiomara.Where the paper says it
--- Page 2 ---
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states. Echo State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent weights to circumvent backpropagation through time, enabling a closed-form training solution. p. 0 it
Abstract
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states. Echo
State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent
weights to circumvent backpropagation through time, enabling a closed-form
training solution. Echo
State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent
weights to circumvent backpropagation through time, enabling a closed-form
training solution. However, achieving the expressivity needed for complex tasks
demands large reservoirs, exposing an O(N2
) state-update bottleneck that prevents
ESNs from matching the scale of contemporary recurrent models.Where the paper says it
Where the paper says it
The per-step cost of a dense ESN scales as O(N 2) with the number of reservoir neurons, dominated by a recurrent matrix-vector product. The frequency domain offers a principled route to overcome this bottleneck because the 1D Discrete Fourier Transform (DFT) diagonalizes circulant weight matrices, collapsing the dense matrix-vector product to an element-wise Hadamard product (circular convolution theorem). p. 1 The frequency domain offers a principled route to overcome this bottleneck. Because the 1D Discrete Fourier Transform (DFT) diagonalizes circulant weight matrices, the dense
matrix-vector product collapses to an element-wise Hadamard product (circular convolution theorem).Where the paper says it
FRESCO circumvents the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling a novel dimensional zero-padding embedding. This allows an input of size Nx to be transformed using a single O(Nx log Nx) FFT rather than a full N-point FFT. p. 1 FRESCO circumvents
the input transformation bottleneck by lifting the 1D circular convolution formulation to 2D, enabling
a novel dimensional zero-padding embedding. This allows an input of size Nx to be transformed
using a single O(Nx log Nx) FFT rather than a full N-point FFT.Where the paper says it
The paper establishes stability conditions for ESNs through the Echo State Property (ESP), which imposes constraints on the recurrence weights. The FRESCO variants satisfy the ESP under certain conditions, as shown in Proposition 2 (Sufficient condition for FRESCO ESP). p. 6 Proposition 2 (Sufficient condition for FRESCO ESP). If α · maxi,j |Wˆ
r
[i, j]| < 1, then FRESCO
satisfies the Echo State Property. • Packed frequency-domain readout: We design a redundancy-free, memory-contiguous,
and FFT-free state readout, provably equivalent to spatial-domain training and inference. • High-efficiency: We show that FRESCO drastically reduces computational and energy
footprints while delivering competitive results against established sequence models.Where the paper says it
Where the paper says it
--- Page 3 ---
A dense ESN’s per-step complexity is dominated by these matrix-vector multiplications, scaling at O(N squared + NNx). Overcoming this quadratic bottleneck is the primary objective of FRESCO, while preserving the standard computational profile for both training and inference. p. 2 A dense ESN’s per-step complexity is dominated by these matrix-vector multiplications, scaling at
O(N2 + NNx). Overcoming this quadratic bottleneck is the primary objective of FRESCO, while
preserving the standard computational profile for both training and inference. A dense ESN’s per-step complexity is dominated by these matrix-vector multiplications, scaling at
O(N2 + NNx). Overcoming this quadratic bottleneck is the primary objective of FRESCO, while
preserving the standard computational profile for both training and inference. A comparison of both
architectures is shown in Figure 1. A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint
on the recurrence weights (Proposition 1). A comparison of both
architectures is shown in Figure 1. A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint
on the recurrence weights (Proposition 1). A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint
on the recurrence weights (Proposition 1). In practice, reservoirs are initialized by drawing Wr at
random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr). A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint
on the recurrence weights (Proposition 1). In practice, reservoirs are initialized by drawing Wr at
random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr). A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint
on the recurrence weights (Proposition 1). In practice, reservoirs are initialized by drawing Wr at
random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The core innovation of FRESCO is to i) reformulate the standard ESN of Eq. (1) in the frequency domain to exploit the inherent efficiencies of frequency-domain processing techniques, yet ii) systematically circumvent costly FFTs usually associated with these approaches. The FRESCO concept is outlined and compared to a standard ESN in Figure 1. p. 3 (1) in the frequency
domain to exploit the inherent efficiencies of frequency-domain processing techniques, yet ii) systematically circumvent costly FFTs usually associated with these approaches. The FRESCO concept is
outlined and compared to a standard ESN in Figure 1. The FRESCO concept is
outlined and compared to a standard ESN in Figure 1. It comprises a novel method for efficiently
embedding arbitrary inputs into the frequency domain, executing the recurrent updates in the frequency domain, and performing readouts without requiring subsequent FFTs on the reservoir state.Where the paper says it
Where the paper says it
The FRESCO architecture's advantages are summarized as follows:
• Frequency domain reservoir computing: We introduce FRESCO, a dense, non-linear ESN architecture executed with O(N) complexity natively in the frequency domain.
• Dimensional zero-padding: We propose a novel input embedding strategy mapping lowdimensional inputs into the frequency-domain, eliminating standard transform bottlenecks. p. 1 Our main contributions are summarized as follows:
• Frequency domain reservoir computing: We introduce FRESCO, a dense, non-linear
ESN architecture executed with O(N) complexity natively in the frequency domain. • Dimensional zero-padding: We propose a novel input embedding strategy mapping lowdimensional inputs into the frequency-domain, eliminating standard transform bottlenecks.Where the paper says it
• Packed frequency-domain readout: We design a redundancy-free, memory-contiguous, and FFT-free state readout, provably equivalent to spatial-domain training and inference. p. 1 • Dimensional zero-padding: We propose a novel input embedding strategy mapping lowdimensional inputs into the frequency-domain, eliminating standard transform bottlenecks. • Packed frequency-domain readout: We design a redundancy-free, memory-contiguous,
and FFT-free state readout, provably equivalent to spatial-domain training and inference.Where the paper says it
--- Page 4 ---
A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint on the recurrence weights. In practice, reservoirs are initialized by drawing Wr at random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr). This requires an O(N 3) eigendecomposition, a bottleneck that scales poorly with reservoir size. p. 2 In practice, reservoirs are initialized by drawing Wr at
random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr). This
requires an O(N3
) eigendecomposition, a bottleneck that scales poorly with reservoir size.Where the paper says it
--- Page 5 ---
A dense ESN’s per-step complexity scales at O(N squared + NNx). Overcoming this quadratic bottleneck is the primary objective of FRESCO, while preserving the standard computational profile for both training and inference. A key stability requirement for ESNs is the Echo State Property (Definition 1), imposing a constraint on the recurrence weights. In practice, reservoirs are initialized by drawing Wr at random and then rescaling its elements to enforce a desired value of its spectral radius ρ(Wr).
--- Page 7 ---
--- Page 8 ---
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Dimensional zero-padding allows an input of size Nx to be transformed using a single O(Nx log Nx) FFT rather than a full N-point FFT, drastically reducing computational costs compared to naive frequency-domain ESNs.
-
The packed frequency-domain readout exploits the Hermitian symmetry of the reservoir state to yield a contiguous, real-valued vector, which is then used for ridge regression entirely avoiding inverse FFTs during training and inference.
-
FRESCO (mix) introduces a structural mixing mechanism via a discrete circular shift to couple frequency components without sacrificing optimal scaling, enabling rich cross-frequency harmonic mixing in the reservoir dynamics.
-
The architecture achieves O(N) complexity for dense, non-linear recurrent updates by replacing the dense O(N2) recurrent product with element-wise Hadamard multiplications between frequency-domain state and weight representations.
-
The system can perform multivariate long-horizon forecasting on datasets like ETT and Weather while consuming up to three orders of magnitude less energy than deep sequence models, matching or exceeding their predictive accuracy.
Abstract
While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states. Echo State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent weights to circumvent backpropagation through time, enabling a closed-form training solution. However, achieving the expressivity needed for complex tasks demands large reservoirs, exposing an O(N 2) state-update bottleneck that prevents ESNs from matching the scale of contemporary recurrent models. To address this limitation, we introduce Frequency Domain Reservoir Computing (FRESCO), an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve O(N) complexity for dense, non-linear updates via structured circulant recurrence. By employing a novel dimensional zero-padding input embedding, a packed frequency-domain readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference. Furthermore, FRESCO achieves competitive predictive performance on memory benchmarks, sequential classification, and multivariate long-horizon forecasting, offering a scalable and energy-efficient path forward for dense recurrent architectures.
Sources
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Efficiently Modeling Long Sequences with Structured State Spaces
- Mamba-3: Improved Sequence Modeling using State Space Principles
- Echo State Networks as State-Space Models: A Systems Perspective
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis