Robust Langevin Sampling under State-Dependent Heavy-Tailed Noise with Applications to LLM Fine-Tuning
This research develops Robust Langevin Monte Carlo (RA-LMC) to handle complex optimization problems where gradients are state-dependent and noise is heavy-tailed. The core contribution is a new oracle ($ ext{BG}_p$) that relaxes strict assumptions, allowing the method to provide rigorous nonasymptotic convergence guarantees for sampling in challenging scenarios like fine-tuning Large Language Models.