Attention-Mass Condensation for Sparse Decoding
This research proves that trained transformer models concentrate their attention mass on a tiny subset of positions. By identifying this small 'Condensate Set' and dynamically selecting it, researchers can achieve numerically exact results with full attention complexity at linear time. This method allows for massive speedups in inference while maintaining perfect output matching across various language models.