Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

arXiv:2610.11784 · cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Compile the Table".

Jane: The gist The proposed QCOC method exploits exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today called "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning". What is it actually trying to do?

Jane: Basically, the whole idea is that current ways of compressing those in-context examples have a problem. Fixed subsets sacrifice accuracy, and query specific retrieval limits how much you can reuse the cache across different questions.

Lu: This paper proposes QCOC, which shifts the focus from just compressing the examples themselves to compressing what happens when you query those examples. It claims that by using exchangeability and reusing those in-context examples repeatedly, you can compile their full KV cache once into a compact memory shared for later queries.

Meng: So instead of storing all the raw training examples, we're turning them into something smaller—joint-KV prototypes—that we can reuse. But they aren't just throwing away data; they are trying to keep the original example count and how many times each group appears in the clusters.

Lalam: And then they use an anchored closed-form solution to fit the values of these prototypes against attention query vectors generated by those examples, which is how they get their compressed value vectors.

Tom: That sounds pretty complex, so what's the real payoff for us? Is this just a theoretical exercise, or can we actually use this for faster inference?

Jane: The main claim is that QCOC drives the speedup through prototype compression while value fitting helps keep the accuracy close to what you get from using all the context at once. They show that on sixty-four held-out OpenML-CC18 datasets, it achieves high accuracy compared to other methods <ref:2610.11784#pg1,on 64 held-out OpenML-CC18 datasets>.

Meng: I like hearing about performance on those specific datasets, but what's the real constraint here? How much does this compilation cost for us when we start running things?

Lu: They did do analyze the component analysis and found that you need to use both the cluster-count weighting and preserving the original count N together. They said correction alone leaves attention computation inconsistent, and their effects aren't just adding up nicely.

Tom: Okay, so there are distinct roles for those parts—one corrects the way attention is weighted by the query transformation, while value fitting calibrates that resulting output. What does that mean in practice?

Jane: It means the system is doing two things at once: it’s adjusting how much importance each example gets based on your current query, and then it’s fine-tuning what those examples actually predict so the final answer is right.

Meng: From an engineering standpoint, that process of compiling the full cache sounds heavy. What are the specific costs they're talking about for this compilation step?

Paper summary: Lu: They estimated the QCOC compilation cost to be O(N squared d plus INM d plus NM squared plus M3) per head. That's a lot of operations depending on how many examples N you have and the dimension d.

Tom: So they've given us a concrete cost, which is helpful. But what’s the big picture for scaling this up? Can this really handle those long tables we see in practice?

Jane: On long tables, they showed that QCOC remains "close to Full in aggregate while reducing online serving cost across repeated queries." They specifically noted on CC18-Long that QCOC's cache is "five point three eight percentage points (ninety-five percent CI

+four point zero zero, +six point seven eight: ) and three point seven five percentage points (

+two point three one, +five point one six: )" better than the best retrieval baselines at M=thirty-two queries per head on that dataset >

Meng: That kind of improvement over retrieval baselines is significant when you’re talking about repeated queries, which is exactly what we need for efficient serving. But they also flagged a limitation, right? What's the catch?

Lu: Yes, they identified some failure cases in their analysis. They noted that a single shared memory can miss query-local structure, as seen in the Covertype result. So it’s not always perfect if you need hyper-local detail for every single query.

Tom: So we have this method that gets us close to full context accuracy while cutting online costs, but it has these specific structural limitations depending on the query pattern. Where does this leave us in terms of what we can actually use this for right now?

Jane: The paper is really suggesting that reusing a bounded cache provides an advantage over dynamic retrieval when you can represent the full-cache behavior globally. They are also pointing toward future work focusing on reducing compilation cost and selecting among multiple reusable memories.

Meng: So, it’s a step towards a more efficient system where we pre-compile based on expected usage patterns, rather than constantly searching through the whole set every time. It’s about making the initial heavy lift pay off over many subsequent uses.

Lu: And I think what they really highlight is that value fitting reliably improves full-cache fidelity, even if whether that changes the predicted class still depends on each dataset’s decision margins. That's a useful diagnostic tool for us.

Tom: It sounds like this paper gives us a way to treat compression as compiling the attention operator induced by a fixed set of examples, rather than just picking a smaller subset of rows. That's what I wanted to get across about "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning".

Jane: And it really comes down to this: if you reuse that compiled memory enough times, the gains in speedup start to outweigh the initial compilation time, and you get performance very close to what a full context would give you.

Conclusion: Tom: So we've been talking about how this paper, "Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning," tries to squeeze massive context down into something small and fast.

Jane: It’s all about taking those big in-context examples and turning them into a shared memory structure that you can reuse many times for different queries without recomputing everything.

Lu: The authors, they are using exchangeability—meaning the examples behave similarly—to build these joint KV prototypes, which is pretty clever because it’s not just throwing away data randomly.

Meng: And they're fitting the values of those prototypes with some kind of regularized least-squares solve based on attention query vectors produced by the examples. That’s where the real magic happens in terms of accuracy versus size.

Lalam: From my side, I see this as moving from just storing text to storing an operator—a compressed version of the entire attention process defined by those examples. It lets us use a fixed set of knowledge more efficiently across repeated tasks.

Tom: So the main thing is that they’re treating compression like compiling the attention operator itself, rather than just picking a smaller subset of rows from the table.

Jane: Exactly, and they show that when you reuse this compiled memory enough times, you get predictive performance that’s very close to what you'd get from using the entire context all at once.

Lu: They did push back on one thing though, they showed that if you only use a single shared memory structure, it can miss some important local structure specific to a particular query.

Meng: That points toward how we might need multiple reusable memories instead of just one big bucket for this kind of compression to work perfectly.

Lalam: And the paper suggests that reducing the cost of that compilation step and figuring out which memory structures are best to reuse is the next natural direction for this research.

Xu Zhao, *, *Jiaming Zhao, *Bin Zhao, *Yong Yang

JD.com

cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: 24 pages, 7 figures. Includes supplementary material

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist The proposed QCOC method exploits exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries,

Key concepts

Joint KV clustering: This step uses Lloyd K-means on concatenated state vectors (keys and values) to deterministically group example states into prototypes. This organizes the examples into a smaller set of representative clusters, forming the basis for the compressed memory.
Count-based weighting: The paper defines cluster counts based on the number of original in-context examples ($N$) rather than just the number of compressed slots ($M$). This ensures that any length-dependent query transformation is evaluated using the original example count, maintaining accuracy.
Value fitting: This involves using a regularized least-squares solve to fit the compressed value vectors. It optimizes these values by anchoring ridge regression, ensuring they reproduce target outputs and improve full-cache fidelity without increasing residual error on calibration queries.

Terminology

In-Context Learning — 1× in this paper · explained in 7 episodes

Summary

The gist The proposed QCOC method exploits exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries, achieving high accuracy close to full context while significantly reducing online inference cost.

How it works

QCOC shifts the object of compression from in-context examples to the query–output mapping defined by the in-context example set, approximating the function q 7→ AD(q) with a compact memory (Ke, Ve, log w) The process involves several key steps:

** Joint KV clustering**

Within each head, deterministic Lloyd K-means is run on concatenated states zi = [ki/σK; vi/σV] to group states into prototypes j ∈ M The initial compressed states are ekj = 1/Cj X i∈Cj ki, ve0j = 1/Cj X i∈Cj vi.

** Count-based weighting**

The cluster count is defined as wj = |Dj |, where Dj is the resulting group, and P j wj = N. This ensures that Any length-dependent query transformation is evaluated using the original number of in-context examples N rather than the number of compressed slots M.

** Value fitting**

The paper uses a regularized least-squares solve to fit the compressed value vectors to reproduce target outputs, where PcalVe ≈ Ycal. This is achieved by optimizing only values through scale-normalized anchored ridge regression: Ve ∗ = (P⊤ calPcal + λγIM)−1 (P⊤ calYcal + λγVe0).

Key Contributions and Analysis

The authors propose QCOC as a post-hoc KV-compression method for tabular ICL that uses "exchangeability and repeated reuse of labeled in-context examples to aggregate their states into joint-KV prototypes with exact cluster counts, then fits the prototype values using attention query vectors produced by the examples. They derive a prototype approximation bound relating attention-output error to key and value approximation errors, showing that the closed-form value fit cannot increase the residual on its calibration queries".

Performance and Scalability

Empirical results show QCOC achieves the highest mean accuracy among compared compression and retrieval methods at both retained counts on 64 held-out OpenML-CC18 datasets. Across long tables, QCOC remains close to Full in aggregate while reducing online serving cost across repeated queries. Specifically, on CC18-Long, QCOC's cache is 5.38 percentage points (95% CI [+4.00, +6.78]) and 3.75 percentage points ([+2.31, +5.16]) better than the best retrieval baselines at M=32.

Component Analysis and Tradeoffs

A factorial analysis identifies two distinct requirements: Cluster-count weighting and preserving the original count N must be used together: either correction alone leaves the compressed attention computation inconsistent, and their effects are strongly non-additive. The roles of these terms are complementary, as the count terms correct attention weighting and query transformation, while value fitting calibrates the resulting output. Furthermore, QCOC compilation costs O(N 2d + INM d + NM squared + M3) per head.

Conclusion

QCOC treats compression as compilation of the attention operator induced by a fixed example set, rather than selection of a smaller set of rows. It successfully retains predictive performance close to full context and reduces online inference cost across repeated queries on held-out datasets. The gains depend on reusing the same compiled memory across enough queries to justify full-cache construction and one-time compilation. It is noted that a single shared memory can miss query-local structure, as seen in the Covertype result. The paper concludes that Reusing a bounded cache provides an advantage over dynamic retrieval when the full-cache behavior can be represented globally. It is also suggested that Reducing compilation cost and selecting among multiple reusable memories are therefore natural next steps. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic. The work provides a fidelity diagnostic, showing that Value fitting reliably improves full-cache fidelity; whether that changes the predicted class still depends on each dataset’s decision margins. It is also noted that The positive residual–accuracy-loss association in Appendix Figure 7 motivates reconstruction residual as a candidate diagnostic.

Improvements for AI systems

Bold KV cache compression for tabular In-Context Learning (ICL) to achieve significant throughput gains while retaining accuracy close to full context, as QCOC ranks first among compressed methods in ten and averages 0.23 percentage points below full context.
Implement Query-Calibrated Operator Compression (QCOC) which exploits the exchangeability and repeated use of in-context examples by compiling their full KV cache once into compact memory shared across subsequent queries, enabling compact-memory reuse and efficient inference across queries while retaining accuracy close to full context.
Utilize joint-KV clustering, cluster-count weighting, and anchored ridge fitting to construct a reusable operator memory where joint-KV yields 74.28 (Accuracy) at M=32 for CC18 datasets, which is superior to other methods like weighted key centroids in certain configurations.
Employ value fitting via anchored closed-form solution, solving the problem such that the unique solution is Ve∗ = (P⊤calPcal + λγI)−1(P⊤calYcal + λγVe0) to ensure accuracy fidelity while minimizing reconstruction error, as QCOC's mean probability MAE is significantly lower than baselines.
Develop a system capable of handling long-context tabular data by leveraging the reusable operator compilation mechanism, which allows the model to maintain performance on large tables where QCOC is the strongest compressed method on CC18-Long, SUSY, and HIGGS, even though it remains close to Full in aggregate.
Inference systems can be optimized for multi-query serving by leveraging a single shared memory structure that reduces single-core serving time from 3.79 to 1.91 ms per query relative to Full, demonstrating that the online advantage comes from reuse and bounded attention length together.
Integrate component analysis insights by ensuring the system uses both cluster-count weighting and preserving the original count N must be used together because they are complementary corrections: either one alone leaves a substantial mismatch.