In-context learning as a retraining-free paradigm for constitutive modeling

summary

The gist

Building models that generalize across physical systems without retraining remains a central challenge in computational science <ref:2604.23098#pg2> Here we introduce InContext Modeling (ICM)…

In short

In-Context Modeling (ICM) introduces a retraining-free method for inferring physical relationships directly from observational data. By treating measurements as physical context, it allows a single model to adapt instantly across diverse physical systems without retraining. This paradigm establishes a transferable foundation for scientific learning and scalable modeling across computational science.

Key concepts

InContext Modeling (ICM)
A paradigm where a single AI model infers physical relationships by conditioning on observational data treated as 'physical context.' It works through a single forward pass, allowing the model to adapt to new physical scenarios instantly without needing any parameter updates or retraining.
Physics-Informed Tokenization
This technique breaks down complex strain fields into modular 'deformation subtokens' based on interior equilibrium. These tokens allow the model to aggregate and sample information from various strain fields in a flexible, physics-grounded manner, enabling it to process diverse physical inputs.
Intrinsic Manifold Learning
The model learns an underlying mathematical structure (manifold) determined by how stress relates to strain profiles, rather than relying on external factors like material type or geometry. This intrinsic learning enables robust generalization across unseen materials and loading conditions.

Terminology used across episodes

This episode discusses

The paper

In-context learning as a retraining-free paradigm for constitutive modeling · Read on arXiv

Lingfeng Li, Zhuoyuan Li, Shun Li, Kaixin Zhan, Huajian Gao, Changqing Chen, Liu Yang

Department of Engineering Mechanics, Tsinghua University · Department of Mathematics, National University of Singapore · Mechano-X Institute, Tsinghua University · Institute for Functional Intelligent Materials, National University of Singapore · School of Mathematical Sciences, Peking University

DOI: 10.1073/pnas.2620983123

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "In-context learning as a retraining-free paradigm for constitutive modeling".

Jane: Here's the gist of the paper. Building models that generalize across physical systems without retraining remains a central challenge in computational science <ref:2604.23098#pg2> Here we introduce InContext Modeling (ICM), a retrain-free paradigm that infers physical relationships directly from observational fields <ref:2604.23098#pg2> This work establishes a transferable paradigm for retrain-free scientific learning and a foundation for scalable modeling across computational science <ref:2604.23098#pg2> The Core Paradigm Shift InContext Modeling (ICM) shifts the focus from "modeling-by-optimization" to "modeling-from-context," where a single model infers physical relationships by conditioning on observational data treated as physical context, without parameter updates <ref:2604.23098#pg2> This paradigm enables retrain-free adaptation across diverse physical systems (Fig.

Tom: 1a) <ref:2604.23098#pg2> The model assimilates measurements as physical context and performs inference through a single forward pass <ref:2604.23098#pg2> This approach overcomes bottlenecks such as the prevailing modeling-by-optimization paradigm, inadequate exploitation of governing physics, and an inability to scale with data and computational resources <ref:2604.23098#pg2> Physics-Informed Tokenization ICM is grounded in a common mathematical structure shared across physical disciplines, which is leveraged through physics-informed tokenization <ref:2604.23098#pg2> The strain field is discretized into deformation tokens based on interior equilibrium, while the boundary equilibrium is extracted for the purpose of post-scaling <ref:2604.23098#pg2> Each individual term (An,e, Ie) acts as a “deformation subtoken” when aggregating the set of all interior nodes collected from strain fields <ref:2604.23098#pg2> This modular representation allows for freely sampling and aggregating tokens from an arbitrary number of strain fields <ref:2604.23098#pg2> The core network gθ approximates the scaled strain energy gradient function given the physical context C, up to a scaling factor (Fig.

Jane: 1c,d) <ref:2604.23098#pg2> Scaling and Generalization ICM exhibits favorable scaling behavior with increasing data diversity and computational budget, analogous to foundation models <ref:2604.23098#pg2> Performance improves with increasing data diversity and computational budget <ref:2604.23098#pg2> The model learns an intrinsic manifold primarily determined by the profiles of stress-strain relationships rather than other ad hoc factors, such as material classes, strain energy magnitudes, geometries, and loading conditions <ref:2604.23098#pg2> This intrinsic manifold is visualized in a two-dimensional t-SNE space where adjacent points correspond to nearly identical strain energy density profiles despite highly diverse material classes <ref:2604.23098#pg2> Validation and Post-Scaling The model's generalization across unseen scenarios was verified using four test sets that progressively increase data diversity, including Test-MGL+ <ref:2604.23098#pg2> The relative errors remain in the order of 10−2 for all datasets, showing robust generalization <ref:2604.23098#pg2> A crucial step involves a post-scaling procedure to determine the scaling factor α by comparing unscaled prediction and true external forces across all available boundary conditions <ref:2604.23098#pg2> This scale-agnostic training and post-scaling approach is critical to the model’s generalization across materials <ref:2604.23098#pg2> Comparison with Baselines ICM was compared against the Equilibrium-based Neural Network (ENN)

11: , a representative baseline for unsupervised stress inference <ref:2604.23098#pg2> ICM achieves lower relative stress errors while reducing the overall deployment time by more than three orders of magnitude compared with all ENN variants <ref:2604.23098#pg2> The integration of the trained ICM network into FEniCS confirms that the inferred stress-strain relationship is fully compatible with conventional FEM pipelines <ref:2604.23098#pg2> The experimental validation using Digital Image Correlation (DIC) demonstrates that ICM predicts accurate stress-strain relationships without any material-specific retraining <ref:2604.23098#pg2> Experimental Insights The model's performance is further validated through real-world experiments on two hyperelastic materials, classified as “soft” and “hard” based on their relative stiffness <ref:2604.23098#pg2> The sustained error reduction far beyond the training context length limit demonstrates a remarkable capability for context length extrapolation at inference <ref:2604.23098#pg2> This test-time scaling behavior shows that the prediction error and its interquartile range decrease with more context <ref:2604.23098#pg2> The final prediction of the energy gradient is then given by ∇ψ(I) = α−1gθ(C)(I) <ref:2604.23098#pg2> By shifting from “modeling-by-optimization” to “modeling-from-context”, ICM addresses a central challenge in computational science: how to build models that instantly generalize across physical systems <ref:2604.23098#pg2> The synergy of this in-context modeling paradigm, physicsinformed techniques, and scaling behavior ultimately drives the generalization across unseen physical systems <ref:2604.23098#pg2> ICM is built upon a common mathematical structure shared across disparate physical disciplines, making its methodology readily adaptable to other physically constrained disciplines <ref:2604.23098#pg2> The authors envision the possibility of a unified foundation model where problems from diverse disciplines are treated with a common model under the ICM paradigm <ref:2604.23098#pg2> The derivation of modeling objective in hyperelasticity involves reformulating equilibrium equations into physics-informed tokenization <ref:2604.23098#pg2> The discretized equilibrium equations are recovered in the form of eqs.

Tom: (3) and (8) after aggregating coefficients <ref:2604.23098#pg2> The dimensionless loss function L(θ) is defined to be invariant to the scaling of the network output, forcing the network to focus on the physical equilibrium during training <ref:2604.23098#pg2> Post-scaling determines the final prediction of the energy gradient as ∇ψ(I) = α−1gθ(C)(I) <ref:2604.23098#pg2> The evaluation metrics include the relative prediction error (err), calculated as Serr = averagevuutX squared i,j=1 "S p ij − S t ij max S t ij − min S t ij 2" <ref:2604.23098#pg2> The optimizer used is the Muon algorithm

37: , which performs momentum SGD updates followed by Newton-Schulz orthogonalization on 2D parameter matrices to improve conditioning and generalization <ref:2604.23098#pg2> The code used for data generation, model training and analysis is available via GitHub at https://github.com/scaling-group/icm-hyperelastic with an Apache license <ref:2604.23098#pg2> Data Generation The training set covers 2,000 hyperelastic material models defined by polynomial strain energy functions (Supplementary Section 1) <ref:2604.23098#pg2> The test sets include Test-ID, Test-M, Test-MGL, and Test-MGL+ to comprehensively validate the model performance <ref:2604.23098#pg2> The training set is based on the polynomial model (eq.

Title and authors: Tom: So we're looking at this paper now titled "In-context learning as a retraining-free paradigm for constitutive modeling". It sounds like they're tackling that huge problem where you build a model that works on one material but breaks when you try to use it for something slightly different without having to retrain the whole thing.

Jane: Exactly. The core idea here is moving away from training models by optimizing them for every single specific case, and instead shifting to modeling from context. It suggests that if you treat the observational data—the physical measurements—as the context, a single model can infer the stress-strain relationship directly through just one forward pass without any parameter updates.

Lu: What's interesting here is how they try to address those three bottlenecks they mentioned: the modeling-by-optimization paradigm, not fully using governing physics, and the inability to scale with data and computational resources. They argue that existing approaches encode physical relationships into fixed parameters, which makes adapting to new systems expensive.

Meng: From a deployment standpoint, if you have to retrain every time you change the material or the loading condition, that's a nightmare for real-time applications. I’m curious how much faster this single forward pass is compared to what we currently run on production hardware.

Lalam: My analysis shows that the InContext Modeling paradigm is designed to overcome those limitations by assimilating measurements as physical context and performing inference through a single forward pass, which they describe in the abstract of "In-context learning as a retraining-free paradigm for foundation models in computational science" <ref:2604.23098#pg1>.

Tom: Right. So it's about this shift—modeling from context—where the model takes what you show it and just figures out the physical rule, rather than having to learn that rule from scratch every time you change things. What's the actual mechanism they use to make this happen?

Jane: Well, they ground their approach in a common mathematical structure that exists across different physical disciplines. They use something called physics-informed tokenization where the strain field gets broken down into deformation tokens based on interior equilibrium, and they extract the boundary equilibrium for some post-scaling step.

Lu: That tokenization is key because it allows each individual term, like the strain energy density components, to act as a kind of deformation subtoken when you aggregate all the interior nodes from those strain fields. This lets them freely sample and aggregate tokens from any number of strain fields.

Meng: So it's like they’re creating a modular representation where each piece of physical information is treated separately before being fed into the main network to predict that scaled strain energy gradient function. That sounds complex to manage at scale, though.

Lalam: The core network, denoted by g theta, approximates that scaled strain energy gradient function given the physical context C up to a scaling factor as shown in Fig one c and d of "In-context learning as a retraining-free paradigm for constitutive modeling" <ref:2604.23098#pg3>.

Tom: Okay, so the training phase involves enforcing that interior equilibrium in an unsupervised way. That's interesting because it means they are teaching the model the physics before they ever try to use it to predict new scenarios without retraining.

Jane: And then in the retrain-free part, they recover that stress-strain relationship via a single forward pass, followed by this lightweight post-scaling and analytical reconstruction step. That's where the magic happens for inference.

Lu: The paper points out that the model learns an intrinsic manifold primarily determined by the profiles of stress-strain relationships rather than other ad hoc factors like material classes or loading conditions. This means it’s learning the underlying physics structure itself, not just memorizing examples.

Meng: That sounds much better for generalization, but what about the scaling? If we feed it more diverse data, does it actually get better faster than a traditional optimization approach when we're dealing with huge datasets?

Tom: They claim favorable scaling behavior analogous to foundation models. They show performance improves as you increase both the data diversity and the computational budget you give it. That suggests this isn't just a trick for small problems, but it holds up under larger scales.

Jane: And to prove this, they tested generalization across four test sets that progressively increased data diversity, including Test-MGL+. They found that the relative errors stayed in the order of ten to the minus two for all those datasets, which they say shows robust generalization <ref:2604.23098#pg1>.

Lu: That robustness is important because it means this intrinsic manifold is stable. They visualized this manifold in a two dimensional t-SNE space where adjacent points correspond to nearly identical strain energy density profiles even when the material classes are totally different.

Meng: So they're saying that despite having materials that look nothing alike, if their stress-strain profiles are close in that intrinsic space, the model recognizes it. That’s a powerful idea for handling diverse physical systems without needing a separate model for each one.

Lalam: They also introduce a crucial step called post-scaling to determine the scaling factor alpha by comparing unscaled predictions and true external forces across all available boundary conditions. This scale agnostic training and post-scaling approach is what they say is critical to the model’s generalization across materials.

Tom: That post-scaling step sounds like a necessary correction since they are dealing with different scales in their measurements. So, if we look at how this compares to older methods, what's the real performance gain?

Jane: They compared it against the Equilibrium-based Neural Network or ENN baseline and found that ICM achieves lower relative stress errors while reducing deployment time by more than three orders of magnitude compared to all those ENN variants.

Lu: And they also confirmed that integrating this trained ICM network into FEniCS shows that the inferred stress-strain relationship is fully compatible with conventional finite element methods pipelines. That’s a big deal for practical implementation.

Meng: Three orders of magnitude reduction in deployment time sounds huge, but I need to know how much overhead that training phase actually adds to the initial setup cost. Is it just faster inference, or is the initial physics-informed training process still very demanding?

Tom: They say the training is done in a physics-informed, label-free manner using governing equations. The goal was to enforce interior equilibrium during this phase in an unsupervised way so that they can then do the retrain-free inference later. It’s about making sure it learns the physics first.

Jane: And they validated this with real world experiments on two hyperelastic materials, one soft and one hard, based on their relative stiffness. They found that the sustained error reduction far beyond what you'd expect from just the training context length limit demonstrates a remarkable capability for context length extrapolation at inference.

Lu: That test-time scaling behavior suggests that the prediction error and its interquartile range actually decrease as you give it more context during inference. The final prediction of the energy gradient is given by the formula alpha minus one g theta C I.

Meng: So, if we're at scale in real-world testing, we can just feed it more data and expect better results without having to do a full retraining cycle first? That makes sense for deployment anxiety.

Lalam: The paper also mentions that the model's latent representations across all test sets encompass five material classes, twelve geometries, and seven loading modes. This shows the model isn't just good on one specific setup but has learned a broad range of physical scenarios.

Tom: So it’s not just about solving one specific stress-strain problem; it’s about building a system that can handle this huge variety of inputs because it understands the underlying physics structure. That's what makes me excited about the architecture.

Jane: And the discussion points to three foundational pillars for this success: first, modeling from physical context gives it adaptivity without retraining, second, physics-informed tokenization enables that shared mechanism of projecting complex physical context onto an intrinsic manifold, and third, favorable scaling behaviors at both training and inference stages.

Lu: The synergy between the in-context modeling paradigm and these techniques is what drives the generalization across unseen physical systems. They argue that because it's built on a common mathematical structure shared across disparate physical disciplines, its methodology could be adapted to other physically constrained fields.

Meng: If it really works as they say, we might see a future where we treat problems from totally different scientific areas with one single model under this InContext Modeling paradigm. That would simplify the entire modeling pipeline immensely for production.

Tom: Well, that's a big vision. So to wrap up on "In-context learning as a retraining-free paradigm for constitutive modeling," the authors show that by shifting from modeling-by-optimization to modeling-from-context, you get an AI that infers physical relationships instantly from observational fields.

Jane: They achieve this through physics informed tokenization and demonstrate robust scaling behavior with performance improving with more data diversity and computational budget. This paper lays a foundation for scalable modeling across computational science by showing how to learn intrinsic physical structures.

Lu: The key is that the model learns those stress-strain profiles, which are the intrinsic structure, rather than being dependent on external factors like material classes or loading conditions.

Meng: And for me as someone worried about deployment, it means we can expect lower errors and much faster inference times compared to current methods when we have more data and compute to work with.

Lalam: It's a paradigm shift that uses a common mathematical structure shared across physical disciplines, which makes the methodology readily adaptable to other physically constrained disciplines.

Tom: The paper "In-context learning as a retraining-free paradigm for constitutive modeling" shows us how to build models that instantly generalize across physical systems without retraining.

Jane: It’s about using in context modeling and physics informed techniques to infer physical relationships directly from observational fields instead of relying on static model parameters.

Lu: The synergy between the in-context modeling paradigm, physics informed tokenization, and scaling behavior ultimately drives the generalization across unseen physical systems.

The paper's summary: Tom: So, to recap, this paper is proposing InContext Modeling as a way to build models for physical systems that you don't have to retrain every time you switch materials or conditions.

Jane: Yeah, they’re taking something like an observation of a physical system and using that context to immediately figure out the underlying physics relationship without any parameter updates.

Tom: It’s about shifting from training by optimization to modeling from context, which means the model just infers things through one forward pass instead of needing endless retraining.

Lu: The core idea is that they treat observational data as physical context and use a common mathematical structure shared across different physical disciplines to project that context onto an intrinsic manifold.

Jane: That intrinsic manifold is where the model learns the actual shape of the stress-strain relationship, instead of just memorizing specific examples.

Tom: And they're saying this scaling behavior is good, meaning it gets better as you feed it more diverse data and give it more computational budget, almost like a foundation model.

Lu: That’s what they are claiming. They show performance improves with increased data diversity and computational budget, which is analogous to how foundation models work.

Jane: So what does this mean for someone just listening? It means you can have a single AI system that works on many different materials or setups without needing a brand new training session every time you want it to handle something new.

Tom: Exactly. And the real deal is they’re using physics-informed tokenization to make sure the model focuses on the physical equilibrium during training, which helps it learn that intrinsic structure better.

Lu: They use these tokens, like strain field components, as subtokens so they can freely sample and aggregate them from any number of fields. That modular representation is a big part of how they achieve this generalization across different contexts.

Jane: And then there’s the post-scaling procedure, which is a crucial step where they compare the unscaled prediction against true external forces across all boundary conditions to figure out that scaling factor alpha.

Tom: That scale agnostic approach seems critical because it’s what lets the model generalize across materials, not just within one specific test set.

Lu: They found that by focusing on these stress-strain profiles, the model learns a representation determined primarily by those profiles rather than other ad hoc factors like material classes or loading conditions.

Jane: And they’ve shown this works even when testing on entirely different material formulations, which is what proves that intrinsic manifold idea.

Tom: The comparison to the Equilibrium-based Neural Network shows that ICM achieves lower stress errors and saves a huge amount of deployment time, which is a big win for real applications.

Lu: It also confirms that the inferred relationship works well with conventional finite element methods pipelines, so it’s not just theoretical; it connects to existing engineering workflows.

Jane: So what this means for the world is that we might see a future where one single foundation model can handle modeling problems from totally different scientific areas using this same in-context learning approach.

Tom: It suggests a unified foundation model where problems from diverse disciplines are treated with a common structure, which would simplify the entire modeling pipeline significantly.

Jane: That’s what they are envisioning for future research, treating all physical systems under this one InContext Modeling umbrella.

The paper's improvements: Tom: So, we're talking about how this InContext Modeling approach actually improves upon what we have now in terms of physical modeling for things like materials under stress and strain.

Jane: Well, the main improvement is that it learns an intrinsic manifold based on the stress-strain profiles rather than getting stuck on external factors like material classes or loading conditions.

Tom: That’s a big deal because it means you don't need to manually define those material properties in a huge lookup table for every new scenario.

Lu: They are saying this intrinsic structure is what drives the generalization, meaning if two materials have similar stress-strain profiles, the model recognizes them as related even if they’re completely different substances.

Jane: So it’s like the AI learns the underlying physical law connecting those two materials, instead of just learning that material A has property X and material B has property Y.

Tom: That sounds way more robust for real-world applications because it handles variations in geometry and loading conditions without needing a complete re-training cycle.

Meng: I worry about the cost of building this intrinsic manifold; does defining those tokenization rules actually add a massive amount of overhead to the initial training process?

Lu: The authors claim that they found favorable scaling behavior, meaning performance gets better as you give it more data diversity and more computational budget during training.

Jane: That’s the numbers part—they showed that the model gets better when you increase both the variety of materials and how much compute you use to train it.

Tom: And they even showed this works well on test sets that are totally out-of-distribution, which is a strong validation point for generalization across unseen physical systems.

Meng: If the scaling is good, that’s reassuring for production because it means we can invest more compute and data and expect a better result without having to overhaul our entire training pipeline.

Jane: So what this changes for someone just listening is that you could have one powerful physics-aware system that adapts quickly to new physical problems without needing a massive re-training effort every time things change.

Tom: It’s about moving away from brittle models where you have to fine-tune parameters for every new material and building something that learns the actual physical relationships directly from context.

Lu: And they also showed this method works with different mathematical structures, meaning its methodology could be adapted to other physically constrained fields beyond just hyperelasticity.

Jane: It suggests there’s a common way to approach modeling physics across different areas of science, which is pretty ambitious for a single paper.

Tom: It’s definitely pushing the boundary on how we use foundation models in physical sciences by focusing on these shared mathematical structures.

Conclusion: Tom: So, to wrap up on "In-context learning as a retraining-free paradigm for constitutive modeling," we’ve seen how they moved from training by optimization to modeling from context for physical systems.

Jane: It boils down to using physics-informed tokenization and that scaling behavior to build a single system that infers physical relationships directly from observational data without needing any parameter updates.

Tom: Exactly, it’s about making the model learn the underlying structure itself through those stress-strain profiles instead of just memorizing specific material responses.

Lu: That intrinsic structure is what allows the model to generalize across different physical systems because it's built on a common mathematical foundation shared across physics.

Jane: So, for someone listening, this means we might eventually have one unified foundation model capable of handling modeling problems from totally different scientific areas using this same in-context learning approach.

Tom: It’s a big vision, but the engineering side is getting pretty solid with that test-time scaling and the reduction in deployment time compared to old equilibrium-based methods.

Meng: I still think about what it costs to run that initial physics-informed training phase; does setting up all those tokenization rules and loss functions add a huge computational tax upfront?

Lu: The authors suggest the training is done in an unsupervised way by enforcing interior equilibrium, which they argue is the right way to learn that physical structure.

Jane: And they did show that this model can handle diverse test sets, proving it’s robust enough for real-world generalization across different materials and geometries.

Tom: The key takeaway here is the synergy between in-context modeling and these physics-informed techniques driving the ability to handle unseen physical scenarios.

Lu: They are confident because they demonstrated scaling behavior at both training and inference stages, which gives them a lot of confidence for future work.

Jane: It’s a solid foundation for building scalable modeling tools that don't require massive retraining cycles every time you want to adapt the model to something new.

Tom: So that’s the gist of "In-context learning as a retraining-free paradigm for constitutive modeling," shifting focus from optimization to context inference.

Jane: It’s a really interesting piece because it shows how we can leverage shared mathematical structures across physics to tackle complex, diverse problems in AI.

Meng: I'm still watching how they handle the memory harness aspect; if these models get huge, managing that context window efficiently will be the next big hurdle for deployment.

Lu: They are looking toward a unified foundation model where problems from totally different disciplines are treated with one common model under this ICM paradigm.

More episodes

← Home