Language Models Can Benefit from Domain-Specific Inductive Biases
cs.LG
Submitted: 2026-11-02
Updated: 2026-11-02
Code: https://github.com/KellerJordan/modded-nanogpt
Project page: https://karpathy.github.io/2015/05/21/rnn-effectiveness
Terminology
Sources
- To Code, or Not To Code? Exploring Impact of Code in Pre-training
- Lessons from Studying Two-Hop Latent Reasoning
- From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning
- Scaling Down Deep Learning with MNIST-1D
- TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
- Curvature Tuning: Provable Training-free Model Steering From a Single Parameter
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- Scaling Laws for Neural Language Models
- Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
- Optimizers Qualitatively Alter Solutions And We Should Leverage This
- How to Train Data-Efficient LLMs
- A Sampling Theory Perspective on Activations for Implicit Neural Representations
- Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
- Procedural Core: A Compact Recurrent Initialization for Vision Transformers
- Lasting Effects of Abstract Pretraining Beyond Perplexity
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis