Understanding Mamba-3 Through Oscillatory Dynamics

arXiv:2610.13341 · cs.LG · Submitted 2026-10-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Understanding Mamba-3 Through Oscillatory Dynamics".

Jane: The gist The interpretation of Mamba-3’s data-dependent rotations as underdamped second-order dynamics provides a framework for investigating oscillation's role in sequence modeling,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we’ve been looking at how Mamba-three handles sequence modeling through this idea of underdamped second-order dynamics, and now we need to talk about what this actually means for these architects <ref:2610.13341#pg2>.

Tom: Right, so the authors are calling it "Understanding Mamba-three Through Oscillatory Dynamics," and it basically says that the way the model’s hidden state moves is governed by something that oscillates when it’s underdamped <ref:2610.13341#pg1>.

Jane: Exactly, and they found this oscillation isn't just noise; it turns out this underdamped behavior outperforms other damping regimes on tasks where you have to track a specific state <ref:2610.13341#pg2>.

Tom: It means that for certain memory problems, having a system that can oscillate, rather than just smoothly decay or get critically damped, gives the model a real advantage in accuracy <ref:2610.13341#pg2>.

Jane: The authors show controlled experiments where the underdamped variant hits ninety-five point six eight percent scaled accuracy compared to eighty-one point six zero percent for the overdamped version <ref:2610.13341#pg2>.

Tom: That’s a measurable jump in accuracy we see there <ref:2610.13341#pg2>.

Jane: What I find interesting is that they link this oscillation to something called learned phase regulation, which seems to control how the model manages its history when it gets new data <ref:2610.13341#pg2>.

Tom: Because they found that removing that phase substantially degrades performance on tasks like LAMBADA, which needs you to integrate information across a long context window <ref:2610.13341#pg2>. It’s not just about retaining data; it’s about how you reorient that data as new inputs come in.

Jane: So for someone who only listens, this suggests that the best way to design recurrent memory isn't just deciding how much history to keep, but also figuring out how that history gets transformed when something new arrives <ref:2610.13341#pg2>.

Tom: Precisely. The paper concludes that underdamped dynamics outperform critically damped and overdamped alternatives, and data-dependent rotations beat fixed ones <ref:2610.13341#pg2>. It frames this oscillation as a real design dimension for future sequence architectures <ref:2610.13341#pg2>.

Jane: It’s interesting because it ties these state-space models together with linear attention models through the evolution of their hidden states <ref:2610.13341#pg9>. That’s a big connection to keep thinking about as we look at how these different types of AI learn to remember things.

Lu: This suggests that effective recurrent memory in state-space models depends on both how much information is retained and how that information is transformed as new inputs arrive <ref:2610.13341#pg9>.

Meng: So the main point here is that oscillation, or underdamped dynamics, could be a useful design dimension for sequence architectures because it controls both persistence and transformation <ref:2610.13341#pg9>.

Tom: Yeah, so the core idea is that oscillation provides a mechanism for better recurrent computation, linking state-space models to other architectures through this dynamical perspective <ref:2610.13341#pg9>.

Conclusion: Tom: So we’re wrapping up our look at Mamba-three’s data-dependent rotations and how they behave under these underdamped dynamics <ref:2610.13341#pg9>.

Jane: Right, so the title of the paper is "Understanding Mamba-three Through Oscillatory Dynamics," and it basically says that the way this model's hidden state moves isn't just a simple decay or a straight line; it has this oscillation when it’s underdamped.

Tom: And they prove that this oscillation isn't just noise; it actually gives the model an edge on tasks where you have to track a specific state, beating the overdamped versions <ref:2610.13341#pg9>.

Jane: It means for sequence modeling, instead of just focusing on how much history you keep, you also have to design for how that history gets transformed when new data comes in.

Tom: Exactly. The authors are suggesting that this dynamical perspective links state-space models to linear attention models through their state evolution <ref:2610.13341#pg9>.

Jane: That’s a big connection to keep thinking about as we look at how these different types of AI learn to remember things. Now, this paper opens up a whole new design dimension for future sequence models, so let's see what those next steps look like.

Shutian Liang, Yue Song

Tsinghua University

cs.LG

Submitted: 2026-10-09

Updated: 2026-10-09

Project page: https://shutian-liang.github.io/oscillationmamba

License: http://creativecommons.org/licenses/by/4.0/

The gist: The gist The interpretation of Mamba-3’s data-dependent rotations as underdamped second-order dynamics provides a framework for investigating oscillation's role in sequence modeling, showing that

Key concepts

Underdamped Dynamics: This describes a specific type of second-order system where the system exhibits oscillatory behavior. In this context, it means the hidden state's evolution involves decaying oscillations rather than simple exponential decay or overshooting. This dynamic is superior for tasks requiring accurate state tracking.
Data-Dependent Rotations: These are rotations applied to Mamba-3's data that influence its internal dynamics. The paper shows that these rotations are crucial because they allow the model to adapt its behavior based on the input data, leading to better performance across various tasks compared to models with fixed or removed rotations.
Learned Phase Regulation: This refers to how the model learns to manage the accumulation of information in its recurrent state. By modulating this phase, the model controls constructive accumulation between old history and new updates, which is vital for maintaining accurate long-context predictions.

Summary

The gist The interpretation of Mamba-3’s data-dependent rotations as underdamped second-order dynamics provides a framework for investigating oscillation's role in sequence modeling, showing that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks and benefit long-context prediction through learned phase regulation.

How it works

The paper interprets Mamba-3’s decay–rotation transition as an underdamped second-order system where decay sets damping envelopes and rotation produces oscillatory phase evolution, providing a unified framework for comparing different damping regimes page4. This interpretation is derived by analyzing the discrete rotation to continuous dynamics transition, where the continuous-time generator is found to be equivalent to an underdamped second-order representation with canonical parameters page5. The resulting second-order equation for the hidden state H is H¨ + 2aH˙ + (a squared + ν 2)H = 0, which yields an underdamped regime when 0 < ζ < 1 page5.

State Tracking Comparisons

Controlled state-tracking experiments demonstrate a clear advantage of underdamped dynamics over other regimes, as shown in Table 2 where the Underdamped (Mamba-3) variant achieves 95.68% scaled accuracy compared to 81.60% for the overdamped variant page6. The comparison across damping regimes tests the benefit of oscillatory modes beyond second-order state coupling alone, highlighting that Underdamped dynamics outperform critically damped and overdamped alternatives page5. Furthermore, data-dependent rotations are shown to matter because Mamba-3 substantially outperforms variants with fixed or removed rotations across all three tasks page6.

Long-Context Language Modeling

Oscillation benefits long-context language modeling by improving length generalization, as controlled pretraining shows improved length generalization at both evaluated scales (187M and 444M), with downstream long-context gains emerging at the larger scale page7. Experimental results show that oscillatory models achieve lower PG-19 perplexity beyond the training context length at both scales, with downstream gains on LongBench-E (Bai et al., 2024) at the larger scale page7.

Learned Phase Regulation

Learned phase regulates recurrent state accumulation by moderating constructive accumulation between retained history and incoming updates, as suggested by hidden-state measurements where removing phase increases both history–update alignment and state magnitude at 16K and 64K page8. Phase interventions reveal the importance of learned rotations for long-context prediction, as removing phase substantially degrades performance on LAMBADA, which requires integrating information across the preceding context page6. The analysis shows that Learned phase regulates recurrent state accumulation page8.

Phase Consistency and State Geometry

Phase consistency between writing and readout is important for accurate prediction, as removing rotation from either side alone substantially increases perplexity, whereas removing it from both sides partially restores performance page9. Hidden-state statistics suggest that rotation moderates constructive accumulation by reorienting retained information relative to new updates page8. These findings highlight complementary roles for decay and phase, where decay controls how much information persists, while phase shapes how it combines with new inputs and is accessed over time page9.

Conclusion

These findings suggest that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive, motivating oscillatory dynamics as a design dimension for sequence architectures page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. This dynamical perspective links SSMs, linear-attention models, and other recurrent architectures through their state evolution page9. The work suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models page9. The paper concludes that Underdamped dynamics outperform critically damped and overdamped alternatives, while data-dependent rotations outperform fixed rotations page6. The paper suggests that effective recurrent memory in SSMs depends on both how much information is retained and how that information is transformed as new inputs arrive page9. The paper further motivates oscillatory dynamics as a design dimension for future sequence models <ref:2610.

Improvements for AI systems

Improve state tracking by implementing underdamped dynamics as a useful mechanism for recurrent computation, which allows models to maintain a clear advantage of underdamped dynamics over critically damped and overdamped alternatives on state-tracking tasks.
Enhance long-context prediction and length generalization by incorporating learned phase rotations, as the paper shows that oscillatory models achieve lower PG-19 perplexity (Rae et al., 2019) beyond the training context length at both evaluated scales.
Refine information accumulation during recurrent updates by controlling the interaction between retained history and new inputs; specifically, learned rotations moderate constructive accumulation because removing phase increases both history–update alignment and state magnitude at 16K and 64K in SISO and MIMO models.
Optimize write-read alignment in long contexts by ensuring phase consistency between writing and readout vectors, as removing rotation from either side alone substantially increases perplexity, whereas removing it from both sides partially restores performance.
Develop a generalized framework for recurrent architectures by using oscillatory dynamics as a design dimension for sequence architectures, suggesting extensions to linear-attention and other recurrent models.

Abstract

Mamba-3 introduces input-dependent rotations into recurrent state updates, but their contribution beyond state retention remains unclear. We interpret these rotations as underdamped second-order dynamics, providing a framework for investigating the role of oscillation in sequence modeling. Controlled comparisons show that underdamped dynamics outperform non-oscillatory damping regimes on state-tracking tasks, while language-model pretraining demonstrates benefits for long-context prediction and length generalization. Interventions in pretrained models further establish the importance of learned phase, and state analysis suggests that rotation regulates how retained information combines with incoming updates. Taken together, these findings identify underdamped dynamics as a useful mechanism for recurrent computation and suggest that oscillation supports long-context processing by regulating how information accumulates.