Training with Subword Variants Delays Overfitting in Repeated-Data Language Modeling
cs.CL
Submitted: 2026-01-06
Updated: 2026-11-03
Code: https://github.com/tatsu-lab/stanford_alpaca
Terminology
Sources
- Aya 23: Open Weight Releases to Further Multilingual Progress
- Scaling Optimal LR Across Token Horizons
- Byte Pair Encoding is Suboptimal for Language Model Pretraining
- Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunities
- An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
- RegionViT: Regional-to-Local Attention for Vision Transformers
- How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization
- Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Training Compute-Optimal Large Language Models
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
- MorphPiece : A Linguistic Tokenizer for Large Language Models
- Adam: A Method for Stochastic Optimization
- Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Scaling Data-Constrained Language Models
- Speakers Fill Lexical Semantic Gaps with Context
- Diffusion Beats Autoregressive in Data-Constrained Settings
- Toward a Theory of Tokenization in LLMs
- How Much is Enough? The Diminishing Returns of Tokenization Training Data
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data