Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation
cs.CL, cs.AI
Submitted: 2026-10-31
Updated: 2026-10-31
Terminology
Sources
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- Gemma 4 Technical Report
- Distilling the Knowledge in a Neural Network
- Improved Knowledge Distillation via Teacher Assistant
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Qwen3 Technical Report
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
- Pre-trained Summarization Distillation
- When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization
- Understanding and Improving Knowledge Distillation
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Revisiting Knowledge Distillation via Label Smoothing Regularization
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data