Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens

arXiv:2605.24417 · cs.LG · Submitted 2026-05-23 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens".

Tom: The gist The benchmark LLMTabBench evaluates LLMs on tabular classification under low-data conditions to characterize how prior knowledge, in-context information, and data complexity interact in row-wise classification;

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper titled "Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens". Basically, the whole idea here is that relying on massive amounts of labeled data for tabular classification isn't always the only way to go.

Jane: Right, it claims that large language models offer a flexible alternative through zero-shot and few-shot in-context learning from task descriptions, but their performance on tabular data is inconsistent. The authors introduce a benchmark called LLMTabBench to figure out exactly what makes that inconsistency happen.

Lu: What they're trying to do is decompose the in-context performance into specific factors, like how many examples you give it or what kind of task knowledge you provide, so we can isolate those effects.

Meng: So this benchmark evaluates LLMs on binary tabular classification under low-data conditions using ninety-one datasets organized into different suites. It's designed to be a diagnostic tool rather than just a leaderboard for who is the best at classifying these tables with an AI.

Lalam: I see it as trying to map out the limits of how much we can expect from in-context learning when the data is scarce. It’s about seeing where those models actually struggle when they don't have tons of examples to look at.

Tom: Exactly, and they test various serialization formats and different shot counts, ranging from zero-shot all the way up to sixty-four examples for some tasks. They are looking at how data complexity interacts with that in-context information.

Jane: The paper focuses on several key research questions to systematically analyze LLM behavior in low-data settings, such as how competitive zero-shot LLMs are, and whether performance relies more on in-context information or the model's prior knowledge.

Lu: They specifically look at how few-shot examples improve performance and whether combining those examples with external instructions helps beyond just using each source separately. Plus, they create a synthetic complexity grid to see how dataset complexity influences the LLM’s performance on these tabular tasks.

Meng: That data complexity concept is pretty important because it acts as a hard ceiling for what the LLM can actually achieve on tabular data; they found that every model gets a small performance signal from sixteen shots at the simplest level, and not much more beyond that <ref:2605.24417#pg1>.

Lalam: And they also looked at inference efficiency, comparing fast forward scoring against slower generation modes, finding that for mid-to-large open-weight models, forward scoring can be about four point five times faster than generation at zero shots.

Tom: It sounds like a lot of things are being measured here, from the different ways they present the data to how much inference time it takes. The paper also compares LLMs against existing methods like TabPFN and XGBoost for context.

Paper summary: Jane: The results show that zero-shot LLMs can be surprisingly competitive in cold-start scenarios and might even beat models given a few examples, which is an interesting finding. However, they found that row-level leakage isn't actually common when probing for memorization.

Lu: What’s really interesting is the finding about prior knowledge versus in-context information; they found those two channels are complementary rather than redundant, meaning using both together gives a better result than either one alone.

Meng: For mid-to-large models, in-context information seems to be the dominant driver of zero-shot performance, contributing about zero point one to zero point four ROC–AUC over a baseline that has neither column names nor task descriptions.

Lalam: They also found that in terms of few-shot examples, the gains flatten pretty quickly; performance saturates around sixteen shots and then starts shrinking as the model gets bigger.

Tom: That saturation point is pretty telling for how much we can practically expect from in-context learning before we need to invest a lot more labeled data. But they also showed that combining in-context information, few-shot examples, and external instructions gives the strongest results for models like Qwen3-8B and GPT4o-mini <ref:2605.24417#pg1>.

Jane: The paper essentially shows that while LLMs are good at these tasks when they have context, the performance hits a wall related to data complexity. It suggests that we might not need infinitely more labeled data if we understand how to structure the prompts and examples correctly.

Lu: The implication is that instead of just scaling up the labeled dataset, we need to focus on better ways of encoding task knowledge and managing complexity when deploying these models in real-world, low-data scenarios.

Meng: From an engineering standpoint, this tells us that for practical deployment, we should prioritize forward scoring methods over generation mode if we are working with open-weight models where inference speed matters a lot.

Lalam: For me, the big picture is that this gives us a clearer diagnostic tool to understand exactly why an LLM might fail on a tabular task when data is scarce, instead of just guessing.

Tom: So, looking at the full title again, "Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens" it really asks us to question that assumption head-on about how we use these models today.

Jane: The authors are using this benchmark to move past just looking at a single performance metric and instead break down the performance into its component parts. They’ve done that by isolating serialization, example count, task knowledge cues, data complexity, and inference mode.

Paper summary: Lu: It sets up a framework where we can systematically test different ways of deploying LLMs for these kinds of classification tasks under limited labeled data regimes.

Meng: The authors are showing that the behavior is not just random noise; it’s structured by how much context you feed it and how complex the underlying data itself is. That structure is what they want to expose.

Lalam: For culture, this means we can build better systems around these models because now we know the specific levers—like choosing a good serialization format or finding that sweet spot for few-shot examples—that actually move the needle on performance.

Tom: So, to wrap up this part, they’ve shown that zero-shot LLMs are competitive but need context, and that data complexity sets a clear limit on performance regardless of the model size. This leads us into what these findings actually mean for how we deploy these models in the real world.

Jane: Absolutely. The paper is providing concrete insights into the trade-offs between using massive amounts of labeled training data versus relying on in-context learning with LLMs for tabular classification tasks.

Lu: It suggests that when dealing with label-scarce environments, the focus shifts from just collecting more data to intelligently structuring the task information we give the model.

Meng: Practically speaking, if you're deploying an AI system for data classification where you only have a few examples, you need to test those different prompting strategies systematically using this kind of benchmark to find what works fastest and most reliably.

Lalam: It’s about moving from just hoping the LLM gets it right to actually understanding the mechanics of why it gets it right or wrong in these specific low-data situations.

Tom: That's a solid way to put it. The paper by Grushina et al., "Do You Even Need More Data? Rethinking LLM Deployment for Label-Scarce Tabular Classification with TabLens" is a deep look into the limitations and potential of using these large language models for structured data problems without drowning in labeled examples.

Jane: It’s a diagnostic tool that helps us understand how to deploy them more effectively when we can't just throw millions of labels at the problem.

Lu: The real contribution is showing that the interaction between in-context information and task complexity defines the performance landscape for tabular classification with these models.

Meng: For engineers, it means we need to build systems that account for this complexity ceiling when designing our AI pipelines, not just assume scaling up the data solves every problem.

Lalam: I think this work helps make the deployment of these powerful language models more reliable and predictable in real-world settings where data isn't always abundant.

Conclusion: Tom: So we've looked at how this new benchmark, LLMTabBench, breaks down how well these large language models perform when they don't have tons of labeled data for those tabular classification tasks.

Jane: Right, and that whole setup shows us that we can actually dissect exactly what makes an AI succeed or fail in a low-data setting by looking at things like the number of examples or the complexity of the data itself.

Lu: The authors are really pushing back on the idea that just throwing more raw data at an LLM will automatically fix these classification problems.

Meng: I think what they’re showing us is that there are specific levers—like how we structure our prompts or what external knowledge we give them—that actually move performance up in a tangible way.

Lalam: Essentially, the paper suggests that instead of just collecting more labels, the focus needs to shift toward better ways of encoding the task information we feed into these models.

Tom: Exactly, because when you look at how zero-shot performance compares to having a few examples, it’s not always a simple scaling curve.

Jane: And they found that combining in-context information and those few-shot examples with external instructions actually gave the strongest results for the smaller models they tested.

Lu: That suggests we should be designing our AI systems around this triple configuration if we're going to get reliable performance without an endless supply of data.

Meng: From my side, it means when I’m building production systems, I need to account for that complexity ceiling the paper mentioned—that hard limit imposed by the data structure itself.

Lalam: And that ceiling is something we can work around by focusing on how well we manage the input context rather than just increasing the input size indefinitely.

Tom: So, the big question here isn't whether we need more data at all, but rather how intelligently and effectively we use the context we *do* have when data is scarce.

Jane: It really re-frames how you should be thinking about deploying these powerful language models for structured data problems in the real world.

Lu: And it sets up a lot of interesting future work around designing better ways to automate that context encoding process itself, which is where things get really creative.

Tom: Next up, we’re going to look at how this applies directly to building more robust AI pipelines for these kinds of tasks.

Sb AI Lab, HSE University

cs.LG

Submitted: 2026-05-23

Updated: 2026-10-13

Code: https://github.com/sb-ai-lab/llm4tab

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 79/100

The gist: The gist The benchmark LLMTabBench evaluates LLMs on tabular classification under low-data conditions to characterize how prior knowledge, in-context information, and data complexity interact in

Key concepts

LLMTabBench: A diagnostic tool designed to break down LLM performance on tabular classification into separable factors. It tests how prior knowledge, context length (examples), data complexity, and inference modes interact when classifying rows in a table.
Zero-Shot Performance: The ability of an LLM to perform a task without any specific examples or demonstrations provided in the prompt. Results indicate that zero-shot predictions are statistically significant and can be surprisingly competitive, sometimes matching performance achieved with more examples.
Data Complexity: A measure of how difficult or complex the underlying tabular data is. The research found that dataset complexity imposes a hard limit on LLM performance; beyond a certain level of complexity, adding more context or examples yields diminishing returns.
In-Context Information: The knowledge an LLM gains directly from the text provided in the prompt, such as schema descriptions or task instructions. This information is found to be a dominant driver of zero-shot performance for mid-to-large models.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes
In-Context Learning — 2× in this paper · explained in 7 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 8 episodes

Summary

The gist The benchmark LLMTabBench evaluates LLMs on tabular classification under low-data conditions to characterize how prior knowledge, in-context information, and data complexity interact in row-wise classification; this research clarifies the limits of in-context learning for tabular data and informs the deployment of LLMs in low-data regimes.

How it works

The benchmark LLMTabBench is designed as a diagnostic tool to decompose in-context performance into separable factors, isolating the effects of (i) serialization, (ii) example count, (iii) task-knowledge cues, (iv) data complexity, and (v) inference mode The benchmark contains 91 datasets organized into complementary suites including a core real-world suite with 24 publicly available datasets grouped into 8 knowledge domains Evaluation axes include varying the shot count from Zero-shot to k ∈ [4, 8, 16, 32, 64] in-context examples and five tabular-to-text serialization formats Models benchmarked include GPT-4o-mini, Qwen3-1.7B, Qwen3-8B, and Qwen3-14B, alongside baselines such as TabPFN and XGBoost

Key Research Questions

The benchmark supports several research questions to systematically analyze LLM behavior in low-data settings These include RQ1, which asks How competitive are LLMs in zero-shot tabular classification? and RQ2, which probes To what extent does zero-shot performance depend on in-context information versus LLM prior knowledge? The study also investigates how few-shot examples improve performance, as seen in RQ3 and RQ4, and whether combining in-context information, few-shot examples, and external instructions improves performance beyond using each source in isolation Furthermore, RQ6 examines How does dataset complexity influence LLM performance on tabular tasks? by generating a controlled MLP-synthetic complexity grid

Findings on Zero-Shot Performance

The results show that zero-shot LLMs can be highly competitive in cold-start scenarios and may even outperform models given few-shot examples The statistical analysis using a Fixed-Effects Meta-Analysis (FEM) confirms that zero-shot LLM predictions are significantly above a Dirichlet-sampled random baseline across all 32 datasets, with the aggregated p-value being 0 However, probing for memorization reveals that row-level leakage is rare, as only a handful of suspicious cases were flagged and reconstruction tests showed that recognition does not imply row-level memorization The headline finding is that zero-shot LLM predictions are roughly on par with the 16-shot performance of strong conventional baselines, with Qwen3-14B reaching a mean ROC–AUC of 0.714 in the zero-shot regime against 0.753 for 16-shot TabPFN

Role of In-Context Information and Few-Shot Examples

In RQ2, the contribution of in-context information versus prior knowledge is assessed by varying schema indicators and task context, revealing that The two channels are complementary rather than redundant — the joint gain exceeds either channel alone For mid-to-large models, in-context information is the dominant driver of zero-shot performance, accounting for 0.10–0.14 ROC–AUC over a baseline with neither column names nor task description In RQ3, LLMs do exploit in-context demonstrations, but the gains flatten fast, saturating by about 16 shots, and shrinking with model size

Impact of Data Complexity and External Knowledge

RQ6 demonstrates that dataset complexity is a hard ceiling on LLM performance on tabular data; every model extracts a clear +0.09 to +0.15 ROCAUC signal from 16 shots at the simplest level (MLP0), and essentially nothing beyond it The TabPFN-derived ranking lines up with intrinsic-dimension estimates, providing a usable coarse-grained difficulty scale for real-world datasets In RQ5, combining in-context information, few-shot examples, and external instructions yields the strongest results, with the full triple configuration being best for Qwen3-8B and GPT-4o-mini

Inference Efficiency

RQ7 compares forward scoring against generation-based inference modes, showing that forward scoring recovers nearly all of the generation-mode quality at a fraction of the inference cost — a single forward pass with no decoding or output parsing For mid-to-large open-weight models, forward scoring is approximately 4.5× faster than generation at 0-shot and approximately 1.

Improvements for AI systems

Zero-shot Competitiveness in Low-Data Regimes LLMs can be deployed as highly competitive zero-shot tabular classifiers for specific domains and low-complexity datasets, as shown by the finding that zero-shot LLM predictions are roughly on par with the 16-shot performance of strong conventional baselines (Table II).
Inference Mode Optimization For mid-to-large open-weight models, deployment can be optimized by using a low-cost forward scoring inference mode, as it recovers nearly all of the generation-mode quality at a fraction of the inference cost — a single forward pass with no decoding or output parsing.
Contextual Information Tuning Performance in in-context learning is maximized by carefully combining information sources; specifically, the dominant combinations are Expert + prior (best for Qwen3-1.7B at 0.732 and Qwen3-14B at 0.862) and the full triple Few-shot + expert + prior.
Complexity Threshold Identification The intrinsic difficulty of a tabular dataset can be quantified using TabPFN performance as a usable coarse-grained proxy for placing real-world datasets on this complexity scale, allowing deployment strategies to identify the point where few-shot improvement is not statistically distinguishable from zero (Figure 31).
Knowledge Source Interaction Analysis Systems can be designed to leverage expert knowledge alongside prior training, as Expert + prior combination outperforms Expert alone by 0.13–0.22 ROC–AUC across all four models, provided the rules receive contextual grounding to land.

Abstract

Every tabular classification task requires sufficient labeled data. When labels are scarce, practitioners can collect more examples, encode domain expertise into features, or accept lower performance. Large language models offer an alternative by leveraging knowledge acquired during pretraining. This raises three questions: (i) can zero-shot inference reduce the need for labeled data; (ii) can a richer prompt reduce the need for more examples; and (iii) can encoded domain knowledge compensate for missing data? To answer each under explicit, testable conditions, we introduce TabLens, a benchmark of 91 real-world and synthetic datasets across eight domains, plus eight industrial datasets with free-text columns. First, below a complexity threshold estimable from a dataset's intrinsic dimensionality before running an LLM, zero-shot inference matches the accuracy of up to about 16 labeled examples. This holds whether the model generates an answer or, more cheaply, scores candidate labels by log-probabilities, which loses almost no accuracy except for the smallest models. Above the threshold, the advantage disappears. On datasets with free-text columns, however, zero-shot LLMs outperform 16-shot TabPFN-3 Plus without manual feature engineering. Second, a richer prompt reduces the number of required demonstrations only when it includes both real column names and a plain-language task description; providing only one can be worse than neither, particularly at higher shot counts. Third, on LLM-synthetic datasets, explicit if--then expert rules help only when the prompt provides enough task context to interpret them, and even then, they substitute for at most about 16 real examples. Together, these results establish testable conditions under which LLMs can reduce data-collection and feature-engineering requirements for tabular classification.