Compression Is Not Evaluation-Neutral: Fixed RAG Compression Can Distort Reader Comparisons
Fixed compression, applied uniformly to retrieved evidence, distorts how reader capabilities are measured. It simultaneously reduces noise for weaker readers and destroys fine-grained details crucial for stronger readers. This leads to falsely attenuated upgrades and reversed model rankings, meaning compression gains decrease as reader strength increases.