Strategies for Pretraining Neural Operators

arXiv:2406.08473 · cs.LG · Submitted 2024-06-12 · Read on arXiv

cs.LG

Submitted: 2024-06-12

Updated: 2024-10-02

Code: https://github.com/anthonyzhou-1/pretraining_pdes

Terminology

• Data Augmentation — 1× in this paper · explained in 7 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes

Sources

• Lie Point Symmetry Data Augmentation for Neural PDE Solvers
• A Simple Framework for Contrastive Learning of Visual Representations
• Towards Multi-spatiotemporal-scale Generalized PDE Modeling
• Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
• Big Transfer (BiT): General Visual Representation Learning
• Unsupervised Representation Learning by Sorting Sequences
• PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers
• PICL: Physics Informed Contrastive Learning for Partial Differential Equations
• Multiple Physics Pretraining for Physical Surrogate Models
• Self-Supervised Learning with Lie Symmetries for Partial Differential Equations
• Shuffle and Learn: Unsupervised Learning using Temporal Order Verification
• Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
• Self-Supervised Learning for Videos: A Survey
• Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior
• A foundational neural operator that continuously learns without forgetting
• Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics
• Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning
• Deciphering and integrating invariants for neural operator learning with various physical mechanisms

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks