Universal set of Observables for Forecasting Physical Systems through Causal Embedding

arXiv:2105.10759 · math.DS, cs.LG · Submitted 2021-05-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Universal set of Observables for Forecasting Physical Systems through Causal Embedding".

Jane: The gist The authors demonstrate when and how an entire left-infinite orbit of an underlying dynamical system or observations from such left-infinite orbits can be uniquely represented by a pair…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at this paper called Universal set of Observables for Forecasting Physical Systems through Causal Embedding. It sounds like they are trying to figure out a universal way to represent any physical system using just a pair of elements in another space.

Jane: That’s right, Tom. The authors call this causal embedding, and the big idea is that if you can find these pairs, you can learn something that lets you determine what the underlying system actually is and predict its future behavior even if you didn't know the system beforehand.

Lu: It’s fascinating because they are building on something like Takens delay embedding. That theorem essentially says that for a generic set of observations from an orbit, like theta(w i), theta(w i+one),, you can embed that original space W into a space of dimension 2d+one using those delay coordinates 2d, theta(w):= (theta(S-2dw) <ref:2105.10759#pg2>..., theta(w)) #pg2.

Meng: But the authors are going further than just embedding the data; they are proposing a way to learn a function FS between these delay coordinates that can actually lead to forecasting the original system dynamics #pg2. That means you observe those delay coordinates, and you can then compute the map FS which gives you predictions about what happens next in the original space.

Lalam: And one of the key things they highlight is that this embedding 2d, theta acts as a universal observable for any fixed delay d, meaning it doesn't change even if the underlying system W or its map S is different #pg2 <ref:2105.10759#pg2>. That universality is what makes it so powerful for learning.

Tom: So, to put that in simpler terms, they’re proposing this causal embedding as a tool to achieve three main things: first, determining if a system you are looking at is topologically conjugate to the system you started with; second, enabling forecasting because that conjugacy is computable and doesn't depend on the specific system; and third, guaranteeing an attractor for the embedded object even if there are errors in learning the function #pg1.

Title and authors: Jane: That’s a lot of promise. The paper suggests that this approach aims to fix problems with existing reservoir computing methods where you don't have a guarantee of finding a learnable function, which often leads to poor long-term consistency #pg3.

Lu: They are tackling the stability issues found in Takens delay embedding by doing something different; instead of just using the delay coordinates, they are trying to find an embedding f that ensures the image of your system is contained within an attractor of its extension #pg3.

Meng: The paper talks about defining this embedding f using the recursive dynamics of a continuous map g, where you have a space U times X and a function g: U times X to X #pg3. They are looking for a system (U', T') that is conjugate to the original system (U, T), so they can build their model or extension in that new space.

Lalam: And they connect this to forecasting by realizing that if g can causally embed the dynamics, then you can define a relationship GT: H two T b H-one to Y T, and this means GT is conjugate to T b, which implies it’s conjugate to the original system T #pg3 <ref:2105.10759#pg2>.

Tom: So, how do we actually use this for forecasting? The authors realize that you can learn the read-out function by realizing the iterates of GT as an autonomous run of g, which they call reading out un = (x n-one x n) #pg3 <ref:2105.10759#pg2>.

Jane: That’s a really interesting mechanism. So, instead of feeding the RNN sequential data with padding, you can use this autonomous run in a "driven mode" where you collect tuples (x n-one x n) after the dynamics lock onto a solution due to the uniform attraction property #pg3 <ref:2105.10759#pg2>.

Lu: The paper also mentions that this embedding f = H two phi T P 2d, theta qualifies as a generalized synchronization where existential conditions are known for generic linear RNNs #pg3 <ref:2105.10759#pg2>. This is a technical detail showing the theoretical backing for the stability of this method.

Title and authors: Meng: The authors also discuss global dissipativity, which they call GD, and this property allows noisy trajectories to be stabilized back towards an attracting set if forecasting is slightly off #pg3. They show numerically that this happens in systems like the double pendulum #pg1.

Tom: So, the big picture here is that by learning GT through, you’re essentially learning T. They are moving away from standard reservoir computing where invertibility isn't guaranteed and instead focusing on learning the function using a deep learning technique #pg3.

Jane: It seems they are not just trying to make existing methods better; they are proposing a fundamentally different way of linking observations to system dynamics through this causal embedding framework #pg1.

Lu: The implication for the field is that if you can establish causal embedding, you get a universally computable and system-independent conjugacy, which is something we haven't seen in this robust manner before #pg2.

Meng: From an engineering standpoint, it means we can tackle systems like the double pendulum with much more robustness to noise than standard methods allow #pg1.

Lalam: And for culture, if this works reliably across different physical systems, it suggests a more structured way for AI to model complex physical dynamics rather than just pattern matching #pg3.

Tom: So, as we wrap up this discussion on Universal set of Observables for Forecasting Physical Systems through Causal Embedding, the core idea is using causal embedding to learn a function that determines topological conjugacy and enables universal forecasting #pg1.

Jane: The paper suggests that by learning GT through, you can effectively learn the dynamics T, moving beyond standard reservoir computing limitations #pg3.

Lu: It’s about realizing the best of both worlds: using the robustness of reservoir computing and the structural guarantees provided by embedding theorems like Takens #pg2.

Meng: The numerical experiments, especially with systems like the double pendulum, demonstrate that this approach actually delivers on its promise regarding noise stabilization via global dissipativity #pg1.

Lalam: It points toward a future where AI can model complex physical systems not just by fitting data, but by discovering underlying structural relationships through universal observables #pg3.

The paper's summary: Tom: So, we just went over how this paper uses causal embedding to try and find a universal way to represent any physical system through two elements in another space, right?

Jane: Right. And what’s really interesting is that they aren't just looking for any arbitrary representation; they are trying to build something that actually lets you determine if the system you're looking at is topologically conjugate to the original one.

Lu: The authors propose this embedding f as a universal observable, similar to how Takens delay embedding works, but instead of just using those fixed delay coordinates, they are finding a way for that coordinate map itself to be universal regardless of the underlying system's map or state space.

Meng: From an engineering standpoint, that universality is huge because it means the mapping you learn doesn't need to be re-learned every time you switch from one physical system to another; it’s a structural property of the representation itself.

Tom: Exactly. And what they connect this to is forecasting, so if you can find this embedding, then the conjugacy between your original system and the embedded one becomes computable and universal because it doesn't depend on the specific underlying system at all.

Jane: That means they think we could potentially build a forecasting scheme that works even when we don't fully understand the specific physics of what we are looking at.

Lu: The mechanism they use for forecasting involves realizing that you can learn a read-out function,, by treating the iterates of this embedded relationship as an autonomous run of the system dynamics, which is called reading out un = (x n-one x n).

Meng: So, instead of just feeding data into a standard recurrent network sequentially with padding zeros, you use this specific way to read out information from the system's own evolution after it settles into a solution due to some property they call the uniform attraction property.

Tom: Right. And they argue that this approach fixes problems where other methods fail, like poor long-term consistency in reservoir computing because they don't guarantee you can find a learnable function at all.

Jane: They use something called global dissipativity, which is basically a condition that lets noisy trajectories get pulled back towards an attractor if the forecast is slightly off, so it adds this layer of stability that existing methods often lack.

Lu: This entire framework aims to overcome the instability issues you see in standard delay embedding by ensuring the image of your system stays within an attractor of its extension, even if there's some error in learning that function.

Tom: So, to summarize, they’re proposing a way to use this causal embedding to learn a function that determines topological conjugacy and enables universal forecasting without needing prior knowledge of the specific system dynamics.

Jane: And they show it works numerically on systems like the double pendulum, demonstrating this stabilization effect with global dissipativity in those noisy time series.

Lu: The authors are also pointing out that their method substantiates the well-defined structure they're using, unlike some other reservoir computing approaches where the invertibility of the system map isn't guaranteed.

Meng: It’s about learning that read-out function using a deep learning technique rather than trying to simplify the training procedure to make it easier, which is a significant difference.

Tom: It really shifts the focus from just fitting data points to discovering these deeper structural relationships in how systems evolve, and that's something we should definitely keep an eye on.

Jane: What this means for us is that if we can establish causal embedding, we get a universally computable and system-independent link between observations and dynamics, which is a pretty powerful concept.

Lu: The long-term vision here is about moving toward AI that doesn't just memorize patterns but understands the underlying structure of the physical processes it’s modeling.

The paper's improvements: Tom: So, we've talked about what causal embedding is and how they use it to find those universal observables for physical systems, right?

Jane: Right. Now let's talk about the actual improvements they suggest in this approach. They aren't just showing a concept; they’re proposing a whole new way to handle system dynamics.

Lu: They emphasize that by using this embedding f, you get a generalized synchronization, which is basically when the system gets stabilized even if there are errors in the forecasting process itself, because of global dissipativity.

Meng: That’s a big deal for practical AI because it means we don't have to worry as much about the forecast veering off wildly; it stays contained within an attractor, which is what you want when modeling something real.

Tom: So how does that fix the stability problem we talked about earlier? What makes this better than just using standard reservoir computing methods?

Jane: The improvement is moving away from those traditional methods where the existence of a learnable function isn't guaranteed, which often leads to poor long-term consistency in predictions.

Lu: They achieve this by basing their model on a driven system with the unique solution property, which allows them to use an iterative process called an autonomous run of g to learn the dynamics through.

Meng: It means they aren't just trying to fit some curve to data; they are learning the underlying structural relationships of how the system evolves in a way that is more robust.

Tom: And this leads to another point: they show that their approach substantiates the well-defined structure, unlike some other reservoir computing approaches where you can't guarantee the system map is invertible.

Jane: That’s important because invertibility is a big deal for many mathematical tools; if it isn't there, things get messy quickly when you try to work backward or predict accurately.

Lu: So they are using the unique solution property and SI-invertibility conditions to ensure that the learned dynamics are stable enough for them to build this conjugacy between the original system and their extended model.

Tom: It sounds like they're addressing the lack of learnability by providing these specific mathematical guarantees, which is a much stronger foundation than just hoping a deep learning technique finds something useful.

Jane: And they’re also looking ahead with future work; they suggest identifying conditions on the driven system where causal embedding holds even when you feed the input sequentially instead of using those fixed delay coordinates.

Meng: That would be interesting because it removes the requirement to know how long your delay-coordinate vector needs to be, which makes it much more flexible for real-world data.

Lu: If that works, we could potentially apply this framework to a much wider variety of time series without having to pre-define the exact structure of the embedding space.

Tom: So the implication for us is that this is pushing AI toward discovering these deep structural symmetries in physical systems instead of just pattern matching what we’ve already seen.

Conclusion: Tom: So we're wrapping up this look at "Universal set of Observables for Forecasting Physical Systems through Causal Embedding," right?

Jane: Yeah, to recap, the paper proposes a method using causal embedding to learn a universal observable that connects an underlying physical system to its observations in a way that guarantees stability.

Lu: It’s really about establishing this conjugacy between systems without needing prior knowledge of the specific physics involved, which is wild for AI modeling.

Meng: For practical impact, it means we can stabilize trajectories even when forecasting is slightly off because of that global dissipativity property they mention.

Tom: So what does this actually change for someone just listening to the show? Does this mean we can predict things better in our daily lives?

Jane: Well, it means AI systems could model complex physical behaviors with a level of robustness that current methods simply can't achieve, especially when things get noisy.

Lu: It suggests a future where AI moves beyond just fitting data and starts discovering these deep structural relationships in how the physical processes actually evolve over time.

Meng: From an engineering viewpoint, it’s about building more reliable systems because we have a mathematical guarantee that the forecast won't spiral out of control.

Tom: It sounds like they are moving away from those unstable reservoir computing setups where you just hope you find a function that works for long-term predictions.

Jane: Exactly. They’re providing a more structured way to learn, by grounding the learning in these known dynamical properties, which is really helpful for building trust in the AI's output.

Lu: The idea of this universal observable being system-independent is what opens up so many creative possibilities for how we use this framework across different domains.

Tom: So we’ve seen how they use it to forecast things like the double pendulum, and now we know the theoretical foundation supporting that forecasting ability.

Jane: The paper shows the power of combining embedding theorems with concepts like global dissipativity to create a genuinely stable forecasting scheme.

Meng: I'm excited about the potential for this in areas where we deal with unpredictable physical systems, like robotics or fluid dynamics, where stability is everything.

Lu: And eventually, Lalam can take these structural insights and use them to build more sophisticated models that understand the underlying rules of reality better.

Tom: It’s a solid piece of work on how to make AI forecasting more principled and less reliant on pure luck with function learning.

Jane: This paper really shows how foundational mathematical ideas, like Takens embedding, can be combined with modern learning techniques to solve very practical problems in system modeling.

Lu: We should keep watching this because it lays a great groundwork for understanding how we can build these universally useful tools for any dynamical system.

Meng: Yeah, it’s not just theory; it’s showing us a pathway toward more reliable predictive models.

Tom: That's all the time we have for this look at "Universal set of Observables for Forecasting Physical Systems through Causal Embedding."

Jane: It really puts the tools into perspective, and I hope you found this discussion helpful.

G Manjunath, A de Clercq, MJ Steynberg

Department of Mathematics & Applied Mathematics, University of Pretoria · Department of Physics, University of Pretoria

math.DS, cs.LG

Submitted: 2021-05-22

Updated: 2026-10-10

Journal ref: IEEE Transactions on Neural Networks and Learning Systems, Volume: 37 Issue: 5, 2026

DOI: 10.1109/TNNLS.2025.3632965

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: The gist The authors demonstrate when and how an entire left-infinite orbit of an underlying dynamical system or observations from such left-infinite orbits can be uniquely represented by a pair of

Key concepts

Causal Embedding
A technique that maps an entire infinite sequence of observations from a dynamical system into a finite, structured representation using two elements in another space. This embedding allows for the determination of a topologically conjugate system, enabling universal and stable forecasting.
Universal Observable
A feature of Takens delay embedding where the coordinate map remains unchanged even if the underlying system parameters are altered. This makes it a robust way to capture essential dynamics, independent of specific system details.
Global Dissipativity (GD)
A property that ensures noisy trajectories in a forecasting scheme are stabilized towards an attracting set. If predictions are slightly off, the future dynamics do not drift far away, providing remarkable robustness against errors and noise.

Terminology

Summary

The gist The authors demonstrate when and how an entire left-infinite orbit of an underlying dynamical system or observations from such left-infinite orbits can be uniquely represented by a pair of elements in a different space, which they call causal embedding.

Causal Embedding and Universal Observables

The concept of causal embedding is used to learn a function that, together with the driven system, can determine a system that is topologically conjugate to the underlying system (i) determine a system that is topologically conjugate to the underlying system (ii) enable forecasting the underlying system’s dynamics since the conjugacy is computable and universal, i.e., it does not depend on the underlying system (iii) guarantee an attractor containing the image of the causally embedded object even if there is an error made in learning the function. This new forecasting scheme aims to beat existing reservoir computing schemes that often lead to poor long-term consistency as there is no guarantee of the existence of a learnable function, and overcomes the challenges of stability in Takens delay embedding.

The authors build upon the Takens delay embedding theorem, which guarantees that observations can be embedded into a space of dimension 2d+1 using the delay coordinates Φ2d,θ(w):= (θ(S−2dw)..., θ(S−1w), θ(w)). This embedding provides a function FS: (θ(S−2d (w)),..., θ(S−1 (w)), θ(w)) 7→ (θ(S−2d+1(w)),..., θ(w), θ(S(w))) which leads to forecasting. A salient feature of Takens delay embedding is that the delay coordinate map by Φ2d,θ is a universal observable of the data, i.e., for a fixed delay d, it does not change (but only adjusts its domain) even if the underlying W, S or θ are changed.

Overcoming Limitations of Existing Methods

Existing machine learning following reservoir computing techniques are more noise-resistant, have simpler training methods, and often choose to learn a “linear” map with low functional complexity, even though the existence of a learnable map that can precisely predict the dynamics is unknown. A lack of learnability results in poor long-term consistency – such systems employed are listed later here. The authors intend to overcome these challenges by obtaining the best of the above methods in the sense that they can embed W in a new space using observations when Takens embedding holds for the delay coordinates so that f(W) is contained in an attractor of its extension. This allows noisy trajectories to be “stabilized” back as in Fig. 1B instead of veering off as indicated in Fig. 1A and provides remarkable robustness to noise.

Defining Causal Embedding for Dynamical Systems

To define such an embedding f, they consider the recursive dynamics of a continuous map g: U × X → X. A bi-infinite sequence is called a solution of g for an input if xn+1 = g(un, xn) holds for all n ∈ Z. A particularly interesting case is when for each input ¯u there exists exactly one solution, which they call the unique solution property (USP). The USP is equivalent to the echo state property and an asymptotic state contraction property that has been well-studied. A remarkable feature of a driven system with the USP is that the “future states” of a solution need not be solved analytically, but can be computed iteratively without the need for the entire-left infinite history of the input thanks to the “uniform attraction property” (UAP).

A dynamical system in this work is a tuple (U, T) where U is a compact metric space and T: U → U is a surjective function. Finding an equivalent dynamical system to (U, T) means finding another dynamical system (U′, T′) so that there exists a homeomorphism φ: U → U′ with the property that φ ◦ T = T′ ◦ φ. The authors aim to build a model that is either conjugate or a (dynamical) extension of the underlying system so that the underlying system can be forecasted.

Exploitation of Causal Embedding for Forecasting

In the customary RC approach, an RNN is used where observations are fed into the RNN sequentially after padding adequate zeroes to match A’s row number. If the RNN g can causally embed (U, T), then GT: H2 ◦Tb◦H−1 2 on YT is well-defined, and GT is conjugate to Tb and hence to T as seen in the commutativity ladder in Fig 1C. This implies that f = H2 ◦ ϕT ◦ P ◦ Φ2d,θ is the conjugacy between (W, S) and (YT, GT) or embeds W in X × X.

The authors then realize the iterates of GT by: GT (xn−1, xn) = (xn, g(Γ(xn−1, xn), xn)) and call the process of reading out un = Γ(xn−1, xn) as an “autonomous run” of g. This enables them to learn Γ in the “driven mode” of g where they make a sufficient collection of tuples (xn−1, xn) by feeding a finite segment of an orbit of T after the dynamics locks onto a solution due to UAP.

Global Dissipativity and Robustness

The embedding f = H2 ◦ ϕT ◦ P ◦ Φ2d,θ qualifies as a generalized synchronization in [22] where existential conditions are known for generic linear RNNs. Global dissipativity (GD) is the property that permits stabilization of noisy trajectories towards an attracting set. If forecasting is erroneous, the future dynamics of the autonomous run would not drift far away as it gets attracted into the set XU+ × XU+. GD is later numerically illustrated (in Fig 1F and Fig 1G).

Conclusion and Future Work

The authors conclude that by learning GT through Γ as before enables learning T. They also note the difference between the customary reservoir computing where SI-invertibility of g is not guaranteed, and their approach substantiates the well-defined SDD and learn the function Γ (using a deep learning technique) rather than simplifying the training procedure. A future study would identify conditions on the driven system for causal embedding to hold when an input is fed sequentially rather than as delay coordinates, eliminating the requirement to know the length of the delay-coordinate vector.

Numerical Experiments

They consider systems where the limiting behavior of a time-series can be observed, such as the double pendulum (DP) system. They describe the details of learning and training of Γ in [9] while an arbitrarily chosen observable θ is considered for forecasting at discrete-time steps. The results illustrate the remarkable effect of GD which enable RCNs to forecast systems like the double pendulum that has not been achieved before.

References

[1] Yule GU (1927) Vii. on a method of investigating periodicities disturbed series, with special reference to wolfer’s sunspot numbers, Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 226(636-646):267–298.

Improvements for AI systems

  1. To implement causal embedding, AI systems can learn a universal X × X-valued observable for any dynamical system on U just like the delay-coordinate map Φ2d,θ is a universal observable that does not depend on the map S on W. This allows the system to determine if an underlying system is topologically conjugate to the underlying system and enables forecasting since the conjugacy is computable and universal, i.e., it does not depend on the underlying system.

  2. The improved AI can forecast dynamics by utilizing a recurrent conjugate network (RCN) that learns the read-out function Γ, realizing the update as GT = (π2, g) ◦ (Γ, π2) which is conjugate to S or embeds W in X × X."

  3. The system gains remarkable robustness to noise through Global Dissipativity (GD), where the ability of a driven system to causally embed a dynamical system does not depend on the map T on U since H2 is purely determined by g. This allows trajectories to be stabilized back as in Fig. 1B instead of veering off as indicated in Fig. 1A even when the read-out in the autonomous run of g+ yields the iterates of GeT = (π2, g+) ◦ (Γexp, π2), which can be shown to contain an attractor.

  4. The system can achieve accurate modeling of previously intractable systems by overcoming challenges where existing methods have failed due to a lack of learnability or stability. This is achieved by utilizing a driven system with the USP and SI-invertibility conditions, allowing the learned dynamics to be stabilized.

Sources

Related papers