dMoE: dLLMs with Learnable Block Experts
cs.CL
Submitted: 2026-05-29
Updated: 2026-11-07
Code: https://github.com/fscdc/dMoE
Terminology
Sources
- A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
- Large Language Diffusion Models
- Dream 7B: Diffusion Large Language Models
- Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding
- GPT-4 Technical Report
- Qwen Technical Report
- The Llama 3 Herd of Models
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- Mercury: Ultra-Fast Language Models Based on Diffusion
- LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- LLaDA-MoE: A Sparse MoE Diffusion Language Model
- SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
- Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
- Task-Specific Expert Pruning for Sparse Mixture-of-Experts
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
- Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
- BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
- Learning More Generalized Experts by Merging Experts in Mixture-of-Experts
- MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
- DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data