Meta-memorization and memorization scaling laws in transformers
cs.LG, cs.AI
Submitted: 2026-11-01
Updated: 2026-11-01
Terminology
Sources
- The broader spectrum of in-context learning
- Scalable Extraction of Training Data from (Production) Language Models
- Extracting books from production language models
- Extracting memorized pieces of (copyrighted) books from open-weight language models
- Understanding Factual Recall in Transformers via Associative Memories
- A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signatures
- Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
- How much do language models memorize?
- How do language models learn facts? Dynamics, curricula and hallucinations
- The emergence of sparse attention: impact of data distribution and benefits of repetition
- Adam: A Method for Stochastic Optimization
- Hopfield Networks is All You Need
- Representation Learning with Contrastive Predictive Coding
- Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
Related papers
- A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
- FlexMoE: Learning One-for-All Multi-Budget Expert Pruning for Flexible MoE Deployment
- OptiMind: Improving Small Language Models for Optimization Modeling through Formulation Guidance and Multi-Turn Learning
- Staying in the Sweet Spot: Boost Reasoning Evolution via Responsive Rollout Difficulty Manipulation
- Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
- Rethinking Black-Box On-Policy Distillation: A Rubric-based Analysis