Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs
cs.LG
Submitted: 2026-10-02
Updated: 2026-10-05
Code: https://github.com/RadicalNumerics/RND1
Sources
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Program Synthesis with Large Language Models
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
- The Diffusion Duality, Chapter II: $\Psi$-Samplers
- Discrete Flow Matching
- From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
- Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
- The Diffusion Duality
- Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective
- Fast-dLLM v2: Efficient Block-Diffusion LLM
- DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
- Qwen3 Technical Report
- Dream 7B: Diffusion Large Language Models
- LLaDA-MoE: A Sparse MoE Diffusion Language Model
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks