Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing
The study addresses contextual bandit problems for LLM routing where true rewards are expensive or noisy, allowing access to cheaper surrogate reward signals from machine learning models. The authors propose two methods: a coupled approach that pools true and surrogate rewards, and a decoupled approach that treats them as separate experts. This framework improves sample efficiency and accuracy-cost trade-offs over standard methods.