Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
cs.CL
Submitted: 2026-10-02
Updated: 2026-10-02
Code: https://github.com/EleutherAI/lm-evaluation-harness
Sources
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Measuring Massive Multitask Language Understanding
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- Hash Embeddings for Efficient Word Representations
- Attention Is All You Need
- MaskBit: Embedding-free Image Generation via Bit Tokens
- ByT5: Towards a token-free future with pre-trained byte-to-byte models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering