A Deep State-Space Model Compression Method using Upper Bound on Output Error

arXiv:2510.14542 · eess.SY, cs.LG, cs.SY · Submitted 2025-10-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: We're looking at the paper: "A Deep State-Space Model Compression Method using Upper Bound on Output Error".

Dev: Here's the gist of the paper. We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee.

Rosa: Deep state-space models (Deep SSMs) are deep models that incorporate linear state-space models (SSMs) into intermediate layers, and they have attracted attention as sequence models that can efficiently handle long-range dependencies and nonlinearities.

Dev: They are representative examples, S5

3: and Mamba

4: achieve state-of-the-art performance on various tasks in the Long Range Arena (LRA) benchmark

5: without using the attention mechanism employed in Transformers

6: , and their effectiveness has been reported on realworld data

7: .

Rosa: To construct small-scale Deep SSMs, one may apply general modelcompression techniques developed for deep learning models.

Title and authors: Rosa: So, this paper is about this Deep State-Space Model Compression Method using Upper Bound on Output Error. It’s looking at those deep models that use linear quadratic output systems inside them and they’re putting a compression method on top of them that gives a provable guarantee on the error.

Dev: Exactly. The main thing they do is derive an upper bound for the error between two of these Deep State-Space Models, and they show this bound can be expressed using the h two error norms between those linear quadratic output systems.

Taro: So, if I'm getting this right, the core idea here is that if we can reduce the approximation errors of those linear quadratic output systems that are in the shallower layers, it directly reduces that upper bound on the final output error.

Rosa: Right. And they don't stop there; they formulate an optimization problem based on that upper bound and then develop a gradient-based model order reduction method to minimize it.

Dev: The goal of that optimization problem is to find a set of reduced systems subject to some constraints, and the design principle they use is assigning larger reduced state dimensions to those shallow layers specifically to bring down that derived upper bound.

Taro: So, what this means for us on the ground is that we can do this model reduction in a one-shot manner without needing to retrain the original model at all.

Rosa: That’s a key point. And they show numerically that this proposed upper-bound based compression constructs an accurate compressed model, and in their experiment using the IMDb task from the Long Range Arena benchmark, they reduce the number of trainable parameters by about sixty percent without sacrificing performance compared to the original model <ref:2510.14542#pg1,using the IMDb task from the>.

Dev: So it’s not just theoretical; it’s practical. They show that even when you compare their method against other retraining free compression methods at the same state dimension, this approach builds compressed models with superior performance.

Taro: That implies that for deployment under strict compute constraints, where retraining is impractical or impossible, this gives us a way to get a low-cost high-performance model.

Rosa: It’s about enabling that kind of deployment where you need something fast and accurate but can't afford the training time. But they also laid out some important analysis on how the error is actually bounded.

Dev: They establish a theorem showing an upper bound on the output error, which depends on the input sequence, and this bound involves terms like h two errors between those layerwise linear quadratic output systems plus some terms involving the input norm.

Taro: And then they show in a corollary that the final output error is bounded by a linear combination of all those h two errors between the different LQO systems placed at each layer, and lemma one gives us insight into how much impact errors in those shallower layers have on that final intermediate output error.

Rosa: So, we’re seeing that focusing our effort on minimizing the h two approximation errors in the shallow layers has a stronger effect than we might initially expect on the final result. That’s interesting because it directs where we should spend our compression efforts.

Dev: It also pointed out that directly optimizing the output error is really hard because it's nonsmooth and doesn't have a simple closed form characterization, so minimizing that upper bound term is a much more tractable path for their gradient-based MOR method.

Taro: And the specific way they formulated the optimization problem, by minimizing that upper bound term instead of the output error directly, seems like a smart move because it avoids that nonsmooth mess while still guiding us toward a good approximation.

Rosa: So to wrap up this part, we have this Deep State-Space Model Compression Method using Upper Bound on Output Error. It’s about using an upper bound on the output error between two Deep State-Space Models to construct an optimal h two model order reduction algorithm that approximates the final intermediate output.

Dev: The big picture here is that through theoretical analysis and numerical experiments, they demonstrated that assigning larger reduced state dimensions to shallow layers and performing this model order reduction to reduce the derived upper bound is effective in reducing the overall output error.

Taro: And what this means for deployment is that we can get a compressed model with high accuracy without retraining, which is a big deal for real-world applications where you have tight constraints.

Rosa: It really shows how we can use mathematical guarantees to guide model compression in these deep state-space architectures. That’s the kind of work I like to see outside the lab setting.

Dev: So, they’ve shown that this method provides a way to get low-cost high-performance models by focusing on minimizing that system level upper bound term, which is what drives their gradient based MOR approach.

The paper's summary: Rosa: So, we’re talking about this paper now focusing on how to compress these Deep State-Space Models using an upper bound on output error between two models, and what that actually means for deploying them in the real world.

Dev: Right, it’s about taking those big Deep State-Space Models that are good at handling long sequences and making them smaller, but this method gives you a way to prove the compression won't destroy performance.

Rosa: The authors show they can derive an upper bound for the output error by looking at how much those linear quadratic output systems inside the models are approximating each other.

Dev: And they realized that if you can control how bad those approximations are in the shallower layers, you get a tighter upper bound on the final error of your compressed model.

Rosa: They then set up an optimization problem based on that bound and use a gradient-based method to figure out how to shrink those state dimensions smartly.

Dev: The main trick they use is assigning bigger reduced state dimensions to the shallow layers, because that directly lowers that derived upper bound on the error before you even run the model.

Rosa: So, this means you don't just randomly chop things off; you have a mathematical rule for where to cut to keep things accurate.

Dev: And they report that with this approach, they can reduce the number of trainable parameters by about sixty percent without needing to retrain the original model at all.

Rosa: That’s huge because it means you get a smaller, faster model that still performs very well on tasks like the IMDb benchmark or whatever those models are doing in real life.

Dev: The comparison they made shows that this compression method beats other retraining-free methods when you keep the number of state dimensions the same, which is pretty impressive for a one-shot process.

Rosa: It really shifts how we think about deploying these complex sequence models, moving from just hoping for the best to having a way to mathematically guarantee what kind of performance you can expect.

Dev: But there are limits here too; they showed that the final output error is tied to those h two errors between every layer’s linear quadratic system, so you still need those inner approximations to be reasonably accurate.

Rosa: So, while it’s a powerful tool for parameter reduction, you still have to worry about the quality of those individual layer approximations.

Dev: Exactly. It's not a magic fix for everything; it's a targeted compression strategy guided by the error analysis of the system itself.

Rosa: This gives us a clearer path forward on how to build models that are both efficient and reliably performant when deployed outside the controlled lab setting, where things get messy.

The paper's improvements: Rosa: We're looking at how the authors suggest making this compression method even better, building on that upper bound idea we talked about earlier.

Dev: So they aren't just proposing one setup; they’re suggesting specific tweaks to the model architecture and the reduction process itself.

Rosa: They point out that you can actually improve things by focusing on reducing those h two approximation errors in the shallow layers more aggressively than we initially thought.

Dev: That makes sense because we saw earlier that those early layer errors have a bigger impact on the final output error, so targeting them is smart engineering.

Rosa: They also suggest using the results from their optimization problem to guide how they assign those reduced state dimensions to the layers, making it more systematic.

Dev: It’s about making that one-shot model reduction less of a guess and more of a structured process based on minimizing that system level upper bound term we discussed.

Rosa: And they also show how this method performs when you try to compress models with different underlying structures, like the ones from the Long Range Arena benchmark.

Dev: They tested it across various tasks, showing that this compression works well not just in theory but actually maintains high performance on things that require long-range sequence understanding.

Rosa: The implication is that if you have a very large Deep State-Space Model, you don't have to throw away all the data or retrain a new massive model to get something fast.

Dev: Instead, this gives you a way to distill the essential behavior of that model into a smaller structure while keeping it faithful enough for deployment.

Rosa: It also suggests that the compression ratio you can achieve depends on how well those initial layer approximations are handled during the reduction step.

Dev: So, if your initial model isn't very precise in those early layers, you might get a less effective compression ratio because the upper bound starts high.

Rosa: This gives us a clearer idea of where we need to invest our effort when we start designing these kinds of efficient sequence models for real-world robotics or driving.

Dev: It points toward better initial model design—making those state spaces in the shallow layers more robust—before you even try to compress them.

Rosa: So, the takeaway is that this isn't just a compression trick; it’s a method for designing Deep State-Space Models that are inherently more compressible from the start.

Conclusion: Rosa: So we're wrapping up this discussion on "A Deep State-Space Model Compression Method using Upper Bound on Output Error." Basically, they proved that you can use a mathematical guarantee to shrink these complex sequence models without retraining them.

Dev: Right, it’s about taking those big Deep State-Space Models and finding an optimal way to reduce their size by looking at the error bounds between the internal parts of the model.

Rosa: The main thing is that they showed how assigning more state dimensions to the shallow layers helps keep that final output error low, which is a huge design hint for us.

Dev: It’s a practical method for deployment because it lets you get a smaller, faster AI running on hardware without having to go through the whole training cycle again.

Rosa: The implication is that we can start thinking about these complex models as structures that have inherent compression limits we can exploit mathematically rather than just brute-forcing parameter reduction.

Dev: That’s right, it changes the engineering mindset from just "can I make it smaller" to "how do I structure this so it compresses well."

Rosa: We're going to keep looking at how these types of models handle real-world failure modes in our next session.

Dev: Yeah, we need to see if this kind of theoretical compression holds up when the system starts misbehaving in a complex environment.

Taro: If the model is compressed based on layer errors, I want to know how gracefully it recovers when those underlying assumptions start breaking down in practice.

Rosa: That’s a fair point, Taro; we'll definitely need to test those failure modes next time to see if the theoretical guarantees translate into actual robustness.

Dev: I agree; for real-time loops, the safety case isn't just about minimizing the bound on paper; it’s about what happens when the input data violates that model structure.

Taro: Exactly, and that’s where we need to push beyond just sequence length limits and look at how these compressed systems handle sudden changes in the world.

Rosa: So, we'll keep tracking this work on Deep State-Space Model Compression using Upper Bound on Output Error for our next session.

University of Tokyo

eess.SY, cs.LG, cs.SY

Submitted: 2025-10-16

Updated: 2026-11-03

Code: https://github.com/lindermanlab/S5

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee.

Key concepts

Deep State-Space Models (Deep SSMs)
These are deep sequence models that integrate linear state-space models into their internal layers. They are effective at capturing long-range dependencies and nonlinearities, outperforming traditional methods like Transformers in certain benchmarks.
h2-error norms
These norms measure the approximation error between the exact linear quadratic output (LQO) systems used within each layer of a Deep SSM. Reducing these errors helps lower the overall upper bound on the final output error.
Model Order Reduction (MOR)
This technique involves designing a compressed model by assigning larger reduced state dimensions to shallow layers. This design principle is used in conjunction with optimization to minimize the derived upper bound on the total output error, leading to parameter reduction without retraining.

Terminology

Summary

We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. p. 0

Where the paper says it

A Deep State-Space Model Compression Method using Upper Bound on Output Error Hiroki Sakamoto and Kazuhiro Sato Abstract—We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the h 2 -error norms between the layerwise LQO systems.

Deep state-space models (Deep SSMs) are deep models that incorporate linear state-space models (SSMs) into intermediate layers, and they have attracted attention as sequence models that can efficiently handle long-range dependencies and nonlinearities. p. 0

Where the paper says it

Index Terms—Model compression; Deep State-Space Models; Optimal h 2 Model Order Reduction; Linear Quadratic Output systems I. Introduction Deep state-space models (Deep SSMs) [1]–[4] are deep models that incorporate linear state-space models (SSMs) into intermediate layers, and they have attracted attention as sequence models that can efficiently handle long-range dependencies and nonlinearities.

They are representative examples, S5 [3] and Mamba [4] achieve state-of-the-art performance on various tasks in the Long Range Arena (LRA) benchmark [5] without using the attention mechanism employed in Transformers [6], and their effectiveness has been reported on realworld data [7]. p. 0
Where the paper says it

Introduction Deep state-space models (Deep SSMs) [1]–[4] are deep models that incorporate linear state-space models (SSMs) into intermediate layers, and they have attracted attention as sequence models that can efficiently handle long-range dependencies and nonlinearities. As representative examples, S5 [3] and Mamba [4] achieve state-of-the-art performance on various tasks in the Long Range Arena (LRA) benchmark [5] without using the attention mechanism employed in Transformers [6], and their effectiveness has been reported on realworld data [7].

To construct small-scale Deep SSMs, one may apply general modelcompression techniques developed for deep learning models. p. 0
Where the paper says it

When deploying trained models to a variety of tasks, it is desirable to obtain compact models with fewer parameters while maintaining the performance of large models. To construct small-scale Deep SSMs, one may apply general modelcompression techniques developed for deep learning models.

Pruning redundant reasoning tokens

The paper introduces a compression method with a provable output error guarantee by deriving an upper bound on the output error between two Deep SSMs. This bound is expressed in terms of the h2-error norms between the layerwise LQO systems. p. 0

Where the paper says it

A Deep State-Space Model Compression Method using Upper Bound on Output Error Hiroki Sakamoto and Kazuhiro Sato Abstract—We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the h 2 -error norms between the layerwise LQO systems.

Reducing the h2 approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error. p. 0
Where the paper says it

We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the h 2 -error norms between the layerwise LQO systems. In particular, we show that reducing the h 2 approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error.

Furthermore, minimizing the h2 error of each individual SSM within a Deep SSM leads to minimizing an upper bound of the overall output error. p. 0
Where the paper says it

Furthermore, the derived bound shows that minimizing the h 2 error of each individual SSM within a Deep SSM, as shown in Fig. 1 (bottom), leads to minimizing an upper bound of the overall output error.

How the training signal is built

The authors formulate an optimization problem for the derived upper bound and develop a gradient-based MOR method. This leads to an optimization problem that minimizes the system-level upper-bound term S - Sˆh2L subject to constraints on the reduced systems. p. 2

Where the paper says it

Therefore, we instead minimize the system-level upper-bound term ∥S − S∥ˆ 2 h 2 L . Problem setting A.

The design principle for MOR-based compression is assigning larger reduced state dimensions to shallow layers to reduce the derived upper bound. p. 1
Where the paper says it

• We show numerically that the proposed upper-boundbased compression constructs an accurate compression model. In particular, we show that, by using the design principle for MOR-based compression, namely, assigning larger reduced state dimensions to shallow layers to reduce the derived upper bound, a compressed model with high accuracy can be constructed without retraining (in a one-shot manner).

Where the gains come from

The proposed upper-bound-based compression constructs an accurate compression model by using the design principle for MOR-based compression, namely, assigning larger reduced state dimensions to shallow layers to reduce the derived upper bound. The numerical experiments demonstrate that the number of trainable parameters can be reduced by approximately 60% without retraining while maintaining the performance of the original model. p. 0

Where the paper says it

In the numerical experiments, using the IMDb task from the LRA benchmark, we demonstrate the effectiveness of the proposed upper-bound-based compression method. In particular, we show that the number of trainable parameters can be reduced by approximately 60% without retraining while maintaining the performance of the original model.

This provides a practical contribution: it enables low-cost, high-performance deployment in settings where retraining is impractical or impossible. p. 1
Where the paper says it

2]), the proposed method—despite differences in the Deep SSM architecture—constructs compressed models with superior performance. This provides a practical contribution: it enables low-cost, high-performance deployment in settings where retraining is impractical or impossible—e.

Model reduction for Deep SSMs

The MOR problem is formulated as minimizing the upper bound in (16) subject to constraints on the reduced systems. The gradients of this objective function are derived using Lemma 2, which provides discrete-time, complex-valued counterparts of those in [17]. p. 5

Where the paper says it

The gradient formulas in Theorem 2 can be viewed as discrete-time, complex-valued counterparts of those in [17], with the additional diagonal structure imposed on the reduced matrices. Since the objective function ψ is a real-valued function of complex variables, it is not complex analytic in general.

A stability-guaranteeing gradient-based algorithm is proposed to minimize the objective function (17). p. 5
Where the paper says it

1 is a stability-guaranteeing gradient-based algorithm derived from Theorem 2. At each iteration, after computing the gradients, backtracking is employed so that the proposed reduced model satisfies the stability constraint and the Armijo condition.

Output Error Analysis for Deep SSMs

Theorem 1 establishes an upper bound on the output error eξ that depends on the input sequence. The bound is given by eξ ≤ X ξ i=1 Gi S(i) − Sˆ(i)h2L · uˆ(i)l2L q 1 + uˆ(i)l2L. p. 3

Where the paper says it

Let S (i) = (A(i) , B(i) , C(i) , M(i) ) denote the ndimensional Deep SSM at the i-th layer of the Deep SSM in (9), and let Sˆ(i) = (Aˆ(i) , Bˆ(i) , Cˆ(i) , Mˆ (i) ) denote the r = ridimensional SSM at the i-th layer of the ri-dimensional Deep SSM. Then eξ ≤ X ξ i=1 Gi ∥S(i) − Sˆ(i) ∥h 2 L · ∥uˆ (i) ∥ℓ 2 L q 1 + ∥uˆ (i)∥ 2 ℓ 2 L , (14) where, with ω := maxi Lip(LNγ (i) 1 ,γ (i) 2 ) denoting the maximum Lipschitz constant of LN (12), Gi = ω ξ−i+1 Y ξ j=i+1 gj , gj = 1 + √ L · n ∥h (j) 1 ∥ℓ 2 L + ∥h (j) 2 ∥ℓ 2 L ∥u (j) ∥ℓ 2 L + ∥uˆ (j) ∥ℓ 2 L o .

Corollary 1 shows that the final output error eξ is bounded by a linear combination of the h2 errors between the two LQO systems placed at each layer. p. 4
Where the paper says it

✷ Inequality (16) shows that, for the same sufficiently small input sin, the final output error eξ is bounded by a linear combination of the h 2 errors between the two LQO systems placed at each layer. Here, the bound in Theorem 1 and Corollary 1 are worst-case upper bounds and can be conservative, especially because it contains products of Lipschitz constants Lip(LNγ (i) 1 ,γ (i) 2 ) and layerwise kernel norms.

Lemma 1 provides important insights regarding the contribution coefficient G˜i, showing that h2L-norm errors in shallower layers have a stronger effect on the final intermediate output error of the Deep SSM. p. 4
Where the paper says it

Therefore, by Corollary 1, for any i, we obtain G˜ i ≥ G˜ i+1. This means that h 2 L -norm errors in shallower layers have a stronger effect on the final intermediate output error of the Deep SSM.

Summary

In this paper, we have derived an upper bound on the overall output error for two Deep SSMs incorporating LQO systems and constructed an optimal h2 MOR algorithm that approximates the final intermediate output. In particular, through theoretical analysis and numerical experiments, we have demonstrated that assigning larger reduced state dimensions to shallow layers and performing MOR to reduce the derived upper bound is effective in reducing the overall output error. p. 6

Where the paper says it

Summary In this paper, we have derived an upper bound on the overall output error for two Deep SSMs incorporating LQO systems and constructed an optimal h 2 MOR algorithm that approximates the final intermediate output. In particular, through theoretical analysis and numerical experiments, we have demonstrated that assigning larger reduced state dimensions to shallow layers and performing MOR to reduce the derived upper bound is effective in reducing the overall output error.

Acknowledgment

This work was supported by JSPS KAKENHI under Grant Numbers 23K28369, 25KJ0986, and 26K03232.

Appendix A

Proof of Lemma 1 is provided in Appendix A. Proof of Theorem 3 follows the structure outlined in [25, Thm. p. 7

Where the paper says it

✷ Proof of Theorem 3. The proof essentially follows from [25, Thm.

2].

Appendix B

WikiText-103 Results are reported in Appendix B. The results show that reducing the relative error for (17) improves the PPL. p. 7

Where the paper says it

Note that the bottom two rows in the table have the same number of trainable parameters. The results show that reducing the relative error for (17) improves the PPL.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Deep State-Space Models can be compressed by reducing parameters by approximately 60% without retraining while maintaining performance on tasks like IMDb, achieving high accuracy comparable to the original model for a given state dimension.

  2. The proposed compression method constructs a reduced Deep SSM that approximates the output sout while accounting for inter-layer interactions, quadratic output maps of the LQO systems, and LayerNorm, minimizing the upper bound on output error rather than directly optimizing it.

  3. Model order reduction can be performed in a one-shot manner by assigning larger reduced state dimensions to shallow layers to reduce the derived upper bound on the overall output error, which leads to a compressed model with high accuracy without retraining.

  4. The system can achieve superior performance over existing retraining-free compression methods when compared at the same state dimension, demonstrating that this method enables low-cost, high-performance deployment under strict compute constraints.

  5. The optimization problem for model reduction is formulated as minimizing the upper bound in (17) rather than directly minimizing the output error exi, which is difficult due to its nonsmooth nature and lack of a closed form characterization.

Abstract

We study deep state-space models (Deep SSMs) that contain linear quadratic-output (LQO) systems as internal blocks and present a compression method with a provable output error guarantee. We first derive an upper bound on the output error between two Deep SSMs and show that the bound can be expressed in terms of the h squared-error norms between the layerwise LQO systems. In particular, we show that reducing the h squared approximation errors of the LQO systems placed in shallow layers is effective in reducing the derived upper bound on the output error. Next, we formulate an optimization problem for the derived upper bound and develop a gradient-based MOR method. In the numerical experiments, using the IMDb task from the LRA benchmark, we demonstrate the effectiveness of the proposed upper-bound-based compression method. In particular, we show that the number of trainable parameters can be reduced by approximately 60% without retraining while maintaining the performance of the original model.

Sources

Related papers