Context-Aware Precision Scheduling for Small Language Models
The work introduces a method to manage accuracy and speed for small language models as they process longer text contexts. It develops a calibrated predictor to estimate mixed-precision latency based on execution costs, not just weight size. This predictor feeds into an allocator that strategically chooses the best precision for different model parts (tensors) based on whether the goal is minimizing latency or maximizing accuracy.