BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

arXiv:2608.05104 · cs.LG · Submitted 2026-08-05 · Read on arXiv

Sajib Hossain, Md Kamrus Samad, Anan Ghosh, Labib Imam Chowdhury, Nabeel Mohammed

cs.LG

Submitted: 2026-08-05

Comments: 14 pages, 9 tables, 13 figures. Preprint

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Pruning — 33× in this paper · explained in 4 episodes
• Transformer — 26× in this paper · explained in 2 episodes
• Downstream Tasks — 14× in this paper · explained in 2 episodes
• Perplexity — 12× in this paper · explained in 3 episodes
• Pattern Exploiting Training — 10× in this paper
• F1 Score — 8× in this paper · explained in 4 episodes
• Iterative Pattern Exploiting Training — 8× in this paper
• Transformer Model — 7× in this paper · explained in 2 episodes
• Accuracy — 6× in this paper · explained in 1 episode
• Transformer Models — 6× in this paper · explained in 3 episodes
• Benchmark — 5× in this paper · explained in 2 episodes
• Sparsity — 5× in this paper · explained in 5 episodes
• Few-Shot Learning — 4× in this paper · explained in 6 episodes
• Low-Resource Languages — 4× in this paper · explained in 4 episodes
• Quantization — 4× in this paper · explained in 15 episodes
• Transformers — 4× in this paper · explained in 2 episodes
• BERT Model — 3× in this paper · explained in 2 episodes
• Bangla Natural Language Processing — 3× in this paper
• Distillation — 3× in this paper · explained in 2 episodes
• Knowledge Distillation — 3× in this paper · explained in 15 episodes
• Transfer Learning — 3× in this paper · explained in 16 episodes
• Augmentation — 2× in this paper · explained in 2 episodes
• Model Compression — 2× in this paper · explained in 3 episodes
• Parts of Speech — 2× in this paper
• BERT-based Natural Language Understanding — 1× in this paper
• Benchmarking — 1× in this paper · explained in 4 episodes
• Core — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• International Conference on Computer and Information Technology — 1× in this paper
• Papers — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Task on Sentiment Analysis in Indian Languages — 1× in this paper
• Unsupervised Learning — 1× in this paper · explained in 2 episodes
• Zero-Shot Transfer — 1× in this paper · explained in 5 episodes

Sources

• Neural Machine Translation for Low-Resource Languages: A Survey
• Emotion Classification in a Resource Constrained Language Using Transformer-based Approach
• Bangla Natural Language Processing: A Comprehensive Analysis of Classical, Machine Learning, and Deep Learning Based Methods
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Pruning Filters for Efficient ConvNets
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
• Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Structured Pruning of Large Language Models
• Patient Knowledge Distillation for BERT Model Compression
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Bangla Text Classification using Transformers
• AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages
• A Subword Level Language Model for Bangla Language

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks