Typhoon: Gradient-Based Task-Specific Masking for Masked Language Models
Typhoon is a new masking strategy that automatically chooses which tokens to mask based on how much they affect the model's loss during training. It uses gradients to estimate token importance and calibrates a masking distribution to meet a target budget. Results show that when accounting for random seed variance, Typhoon performs no better than uniform random masking on standard tasks.