Mosaic: A Benchmark Suite for Differentiable Physics Solvers
summary
The gist
Differentiable partial differential equation (PDE) solvers underpin solver-in-the-loop ML training, gradient-based optimal control, and inverse problems…
In short
Mosaic creates a standardized benchmarking framework to test solver gradients, addressing the lack of documentation regarding gradient quality from different PDE solvers. It tests solvers in isolation using finite differences as ground truth, measuring accuracy, cost, and conditioning. Findings show that while pointwise gradient accuracy isn't the main bottleneck, Jacobian conditioning is a better predictor of training stability.
Key concepts
- Solver Gradients
- These are the mathematical derivatives calculated by a PDE solver to determine how small changes in input parameters affect the solution. Mosaic tests how accurately and reliably different solvers can produce these gradients for various physical problems.
- Mosaic Infrastructure
- This is an extensible benchmarking suite that standardizes testing. It packages solvers into containers with a uniform API, allowing researchers to test the solver's gradient mechanism without worrying about the underlying programming language or automatic differentiation strategy.
- Jacobian Conditioning
- This measures how sensitive the solver's output is to small changes in its inputs. Poor conditioning means small errors in gradients can lead to large errors in the final solution, which directly impacts training stability and convergence during machine learning.
- Rollout Length
- This refers to how many steps or iterations a process takes when calculating a gradient or performing an optimization step. The paper shows that Jacobian conditioning often degrades as rollout length increases, meaning longer processes can introduce more numerical instability.
Terminology used across episodes
This episode discusses
- Mosaic: A Benchmark Suite for Differentiable Physics Solvers · Paper Radio
- XLB: A differentiable massively parallel lattice Boltzmann library in Python
- OpenAI Gym
- Don't Unroll Adjoint: Differentiating SSA-Form Programs
- Differentiable Rendering: A Survey
- Adam: A Method for Stochastic Optimization
- APEBench: A Benchmark for Autoregressive Neural Emulators of PDEs · Paper Radio
- NeuralFluid: Neural Fluidic System Design and Control with Differentiable Simulation
- Fourier Neural Operator for Parametric Partial Differential Equations
- Differentiability in Unrolled Training of Neural Physics Simulators on Transient Dynamics
- FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators
- Universal Differential Equations for Scientific Machine Learning
The paper
Mosaic: A Benchmark Suite for Differentiable Physics Solvers · Read on arXiv
Andrin Rehmann, Heiko Zimmermann, Dion Häfner
Pasteur Labs
Differentiable partial differential equation (PDE) solvers underpin solver-in-the-loop MLtraining, gradient-based optimal control, and inverse problems, yet the practical cost of obtaining correct, usable gradients from a given solver on a given problem is largely undocumented. Integration effort, computational cost, memory, gradient accuracy, and numerical stability vary widely across solvers and are discoverable only by trial and error. We introduce Mosaic, an extensible benchmarking framework for differentiable PDE solvers that standardizes access to solver gradients. Each solver is packaged as a containerized component (Tesseract) exposing a uniform gradient API regardless of language or automatic differentiation (AD) strategy, enabling researchers to evaluate, compare, and build on non-trivial physical solvers. Our evaluation of 14 solvers across fluid dynamics, structural mechanics, and heat transfer demonstrates that the benchmark surfaces practically relevant differences: order-of-magnitude variation in computational cost and memory, alongside setup incompatibilities and instabilities that rule out solvers for specific tasks. Despite this variation, all solvers that produce gradients converge to similar optima on our tasks, indicating that the practical barriers are memory limits, numerical stability, and setup compatibility rather than gradient accuracy alone. Mosaic is open-source and available at github.com/pasteurlabs/mosaic.
Transcript
Introduction to the show: ident: Physics. The day in general physics: instruments, methods and results across the field.
Kai: We're looking at the paper: "Mosaic: A Benchmark Suite for Differentiable Physics Solvers".
Mira: Here's the gist of the paper. Differentiable partial differential equation (PDE) solvers underpin solver-in-the-loop ML training, gradient-based optimal control, and inverse problems, yet the practical cost of obtaining correct, usable gradients from a given solver on a given problem is largely undocumented <ref:2606.27895#pg2>.
Kai: Mosaic introduces an extensible benchmarking framework for differentiable PDE solvers that standardizes access to solver gradients <ref:2606.27895#pg4>.
Mira: The Problem of Opacity The practical cost of obtaining correct, usable gradients from a given solver on a given problem is largely undocumented <ref:2606.27895#pg2>.
Kai: Integration effort, computational cost, gradient accuracy, and numerical conditioning vary widely across solvers and are discoverable only by trial and error <ref:2606.27895#pg2>.
Paper summary: Mira: So looking at "Mosaic: A Benchmark Suite for Differentiable Physics Solvers", what it really does is provide this organized way to test different solvers against a standard set of challenges in fluid dynamics, structural mechanics, and heat transfer. It claims that this benchmarking framework surfaces the practical differences between solvers by standardizing access to solver gradients.
Kai: The authors argue that the key finding isn't necessarily about one single solver being perfect, but rather about how different solvers behave when you compare them across these defined metrics like runtime and conditioning.
Lev: And they point out that even if all the solvers produce gradients, they often converge to the same final solution, which means gradient quality might not be the limiting factor for getting a good answer on its own.
Mira: So for us, this suggests we should focus our attention more on how well a solver handles numerical stability and conditioning during long optimization runs rather than just chasing perfect pointwise gradient accuracy at every single step.
Kai: And the paper’s implication is that reverse-mode automatic differentiation seems to offer the best effort to capability tradeoff when working with these complex physics simulations right now.
Lev: So, in simple terms, this work gives us a common testing ground so we know what to expect from different solvers before we commit resources to building the actual ML applications on top of them.
Conclusion: Kai: So we've been looking at Mosaic, and I think the title itself really tells you what this is about—it's a whole system for testing how these math solvers work when you want to use them in machine learning.
Mira: Right. It sets up this standardized way to check different physics solvers against the same set of problems. It’s like setting up a universal lab bench for whatever solver you pick later on.
Kai: Exactly, and it's about making the messy process of getting gradients from these solvers much more predictable for training models with them. The authors are just providing this infrastructure to help people stop guessing how good a solver is going to be.
Lev: From an engineering side, what’s interesting is that they aren't just picking one solver and showing you their results; they’re testing the gradient mechanism itself in isolation first. That means we get a clear view of the fundamental limitations of each approach before we even try to fit it into a larger pipeline.
Mira: And that leads into the core idea—the problem is that right now, figuring out if a solver gives you accurate gradients or if it's numerically stable is totally undocumented. It’s all trial and error for every single solver you look at.
Kai: So what this paper does is create this benchmark suite to standardize those tests—measuring things like how much time it takes, the accuracy of the gradient, and how well the math holds up under pressure. It’s about surfacing these hidden costs.
Lev: And that points toward a crucial detail from their findings—they found that for many problems, if a solver can produce *any* gradient at all, they often end up converging to the same final answer regardless of how much accuracy or conditioning differs between them.
Mira: That’s interesting because it means chasing perfect gradient accuracy isn't always the bottleneck for getting a good result in these physics applications. It suggests that other things are more important.
Kai: So where does this leave us? If we can stop wasting time guessing solver performance and start focusing on stability and conditioning, what kind of real-world systems could we actually build with this standardized testing ground?
More episodes
- 2611.14082-Two atoms with strong light-induced dipolar interactions
- 2611.14040-Characterization of BNLs 20 and 30 micrometer thick LGADs for the Electron Ion Collider
- 2611.13399-Lift-element-based dynamic network of vortex gust-airfoil interactions
- 2611.13476-Hopfion Superlattices in 3D Twisted Moir'e Light Fields
- 2611.13642-Investigation of the timing performance of double-sided p-type 3D silicon sensors produced by CNM
- 2611.13830-SiPM optical readout of a PICOSEC Micromegas precise timing detector
- 2611.14152-Where are the shortest-path nodes in spatial networks?
- 2611.14219-Sub-Ice: semi-automated mapping of basal channels from ice shelf surface elevation data
- 2611.16341-SUMNO: Spatially Unrolled Multigrid Neural Operator for Fluid Flow Surrogates
- 2611.16090-Multi-UTE Myelin Fraction Mapping: Repeatability and Cross-Field-Strength Reproducibility at 1.5T and 3T