One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data
cs.CL, cs.LG
Submitted: 2026-11-10
Updated: 2026-11-12
Project page: https://hkunlp.github.io
Terminology
Sources
- Distilling Token-Trained Models into Byte-Level Models
- Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- DeepSeek-V3 Technical Report
- FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
- CUTE: Measuring LLMs' Understanding of Their Tokens
- EXECUTE: A Multilingual Benchmark for LLM Token Understanding
- Better & Faster Large Language Models via Multi-token Prediction
- Sampling from Your Language Model One Byte at a Time
- Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
- TokAlign: Efficient Vocabulary Adaptation via Token Alignment
- Zero-Shot Tokenizer Transfer
- Bolmo: Byteifying the Next Generation of Language Models
- Byte Latent Transformer: Patches Scale Better Than Tokens
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Cross-Tokenizer LLM Distillation through a Byte-Level Interface
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
- MambaByte: Token-free Selective State Space Model
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data