FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks

arXiv:2606.30336 · cs.LG · Submitted 2026-06-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks".

Tom: The gist A flexible encoder-decoder architecture for in-context learning on tabular data introduces FlexTab, which pairs a single,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper today, "FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks." It sounds like they’re trying to build something that works for a ton of different table tasks without needing a whole new model every time.

Jane: Exactly. The title tells us the main idea is flexibility, pairing one general encoder with several specialized decoders.

Lu: What's interesting here is how they aim to decouple the feature representation from the prediction target, which is a big concept in how we think about these models today.

Meng: So instead of training one model just for regression and another just for classification, this architecture tries to solve it by having one backbone that learns general row information.

Lalam: It suggests that the core representations learned by the encoder should be usable across a wide range of downstream tasks in a table-native setup.

The paper's summary: Tom: So, what does this architecture actually do? FlexTab pairs a single, task-agnostic encoder with a suite of task-specific decoders to produce targetagnostic row embeddings that can be used across many different prediction problems in tables.

Jane: The key difference they highlight is that the encoder builds on the ConTextTab architecture, using standard embedding strategies for numerical, categorical, and time data to embed the input table.

Lu: They use a learnable row token appended to these embeddings along the column axis, similar to how you see CLS tokens in BERT-like models or TabICL.

Meng: After that embedding step, cell tokens and those row tokens get contextualized through alternating cross-column and cross-row attention layers.

Lalam: The decoders are built using the same two dimensional attention blocks as the encoder, but they have a crucial difference in how they do their attention mechanisms.

The paper's improvements: Tom: That difference is what makes it flexible; the cross-column attention in the decoders is realized via cross-attention to those encoder outputs.

Jane: The authors say this separation of feature representation from predictive modeling means you can have different prediction heads for regression or classification, but they share most of the decoder weights.

Lu: For instance, when you're doing classification or regression, they use separate heads but keep the underlying structure shared across those decoders.

Meng: They show this works on six distinct problems: classification, regression, anomaly detection, clustering, entity matching and entity classification in relational databases.

Lalam: It’s important because the paper points out that existing methods like TabPFN or ConTextTab are target-aware, which means they are tightly coupled to a specific prediction goal.

Conclusion: Tom: So, FlexTab achieves state-of-the-art performance on classification, regression, anomaly detection and entity matching while still being competitive with specialized models on entity classification in relational databases.

Jane: The main point is that they propose this flexible architecture because methods that rely on extracting embeddings without a target often use dummy targets which hurt performance.

Lu: They show this flexibility extends to multi-table scenarios, where the FlexTab-Multi decoder can aggregate information from multiple tables and relation paths.

Meng: And in terms of practical application, the paper shows that performance scales favorably with context size, meaning you can get a clearer edge on semantic-heavy tasks by using larger intra-table context.

Lalam: It's a flexible encoder-decoder architecture for in-context learning on tabular data that natively handles classification, regression, anomaly detection, clustering, entity matching and entity classification over relational databases within a single unified framework.

SAP SE

cs.LG

Submitted: 2026-06-29

Updated: 2026-10-12

Comments: Updated, NeurIPS camera-ready version

Code: https://github.com/SAP-samples/flextab

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: The gist A flexible encoder-decoder architecture for in-context learning on tabular data introduces FlexTab, which pairs a single, task-agnostic encoder with a suite of task-specific decoders to

Key concepts

Encoder
The encoder is a general component that processes input tables to create universal row embeddings. It learns features from the data structure itself rather than focusing on a specific prediction goal. These embeddings are task-agnostic, meaning they can be used for many different downstream tasks.
Decoder
Decoders are specialized components paired with the encoder for specific tasks. Each decoder is trained to map the general row embeddings to a particular output format, such as a classification label or a regression value. They use the encoder's representations to perform task-specific predictions.
Target-Agnostic Embeddings
These are row representations generated by the encoder that do not contain any specific target values. Instead, they capture the inherent structural and semantic information of each row in a table. This generality is key, as these embeddings can be leveraged across a wide range of different prediction problems.
In-Context Learning
This refers to the model's ability to perform new tasks based solely on examples provided within the prompt, without explicit fine-tuning for that task. FlexTab enables this by providing a unified framework where the shared encoder and specialized decoders handle diverse problems seamlessly.

Terminology

Summary

The gist A flexible encoder-decoder architecture for in-context learning on tabular data introduces FlexTab, which pairs a single, task-agnostic encoder with a suite of task-specific decoders to produce targetagnostic row embeddings that can be leveraged across a wide range of downstream tasks within a table-native in-context learning setup<ref:2606.30336#pg2>

How it works

FlexTab is based on an encoder-decoder setup that clearly separates internal latent representations from predictive paths by using a variety of problem-specific decoders<ref:2606.30336#pg3>. The core principle is to decouple target-agnostic feature representation from task-specific predictive modeling<ref:2606.30336#pg2>.

The encoder builds on the general setup of the ConTextTab architecture, adopting the same embedding strategy with separate numerical, categorical, and time embedding layers<ref:2606.30336#pg2>. Given input tables with batch size B, R rows, and C columns, the data is embedded into a tensor of shape (B, R, C+1, D), where D denotes the hidden dimension<ref:2606.30336#pg3>. A learnable [ROW] token is appended to the embeddings along the column axis, resulting in a tensor of shape (B, R, C + 1, D), similar to [CLS] tokens in BERT-like models and TabICL<ref:2606.30336#pg3>. After embedding, cell tokens and [ROW] tokens are jointly contextualized through alternating cross-column and cross-row attention layers<ref:2606.30336#pg2>.

The decoders consist of the same 2D-attention blocks as the encoder, with the crucial difference that cross-column attention is realized via cross-attention to the encoder outputs<ref:2606.30336#pg2>. The key distinction is that cross-row attention is applied exclusively to the target column, while the encoder’s [ROW] tokens serve as attention KV inputs to all cross-column attention layers, with the target column acting as the attention Q<ref:2606.30336#pg2>.

Task Specific Implementations

FlexTab is demonstrated on six distinct problems: classification, regression, anomaly detection, clustering, entity matching, and entity classification in relational databases<ref:2606.30336#pg2>.

- Classification and Regression:

The main design follows the description above with separate prediction heads for regression and classification but otherwise shared decoder weights<ref:2606.30336#pg2>. For regression, numerical values are encoded via piecewise linear interpolation over a set of learned reference vectors, and regression is cast as classification over these reference points<ref:2606.30336#pg2>.

- Anomaly Detection:

Anomaly detection is treated with the same architecture as classification but with a decoder trained on binary targets simulating an anomaly detection task<ref:2606.30336#pg2>. This includes unsupervised, one-class (novelty detection), and semi-supervised setups<ref:2606.30336#pg2>.

- Clustering:

For unsupervised clustering, the target column of all rows is masked with [MASK], and the final head layer outputs an embedding vector per row, which is used for clustering<ref:2606.30336#pg2>. The loss computed involves cosine similarity between such vectors and ground truth labels<ref:2606.30336#pg2>.

- Matching:

Cross-table row matching is treated as a pairwise binary classification problem, where row pairs are treated as matching targets<ref:2606.30336#pg2>. For relational entity classification, related rows are identified via breadth-first search (BFS) across the database up to a maximum of n hops<ref:2606.30336#pg2>.

Training and Evaluation

The approach is trained using a large collection of real-world (unlabeled) tables, avoiding reliance on synthetic priors or task-specific pretraining data<ref:2606.30336#pg2>. The encoder is pretrained together with the classification and regression decoder on conventional tabular prediction tasks<ref:2606.30336#pg2>.

The training involves generating classification and regression tasks from randomly sampled real-world tables by selecting context and query rows, then sampling feature and target columns<ref:2606.30336#pg2>. For anomaly detection, targets are generated by transforming a selected column into binary labels based on K-means cuts<ref:2606.30336#pg2>.

Evaluation is conducted on a diverse set of tabular prediction tasks, including classification, regression, anomaly detection, clustering, entity matching, and entity classification across relational databases<ref:2606.30336#pg2>. Performance is compared against state-of-the-art in-context learners like TabPFNv2.6 and ConTextTab<ref:2606.30336#pg2>.

Results and Benchmarks

FlexTab achieves competitive or state-of-the-art performance across all investigated scenarios, highlighting the flexibility of its approach in both single- and multi-table scenarios<ref:2606.30336#pg2>. On semantic benchmarks like CARTE and TextTab, FlexTab ranks best<ref:2606.30336#pg2>.

- Relational Prediction:

FlexTab-Multi's performance scales favorably with both the intra-table context provided to the encoder and the number of auxiliary rows aggregated by the decoder<ref:2606.30336#pg2>. The monotonic improvement with BFS breadth highlights that, despite being pretrained only on chunks derived from single tables, the decoder generalizes to multi-table relational reasoning at inference time<ref:2606.30336#pg2>.

Conclusion and Outlook

FlexTab presents a flexible encoder-decoder architecture for in-context learning on tabular data by decoupling a shared, target-agnostic encoder from a family of task-specific decoders<ref:2606.30336#pg2>. It natively supports classification, regression, anomaly detection, clustering, entity matching, and entity classification over relational databases within a single unified framework<ref:2606.30336#pg2>. The architecture is trained on a large corpus of real-world unlabeled tables<ref:2606.30336#pg2>

Limitations include the necessity to reformulate every (sub)problem as a single-table classification or regression task, and the tight coupling between internal latent representations and predictive modeling<ref:2606.30336#pg2>. Future directions include replacing supervised pretraining with fully self-supervised objectives, jointly training all decoders on top of a shared encoder, or consolidating decoders through a mixture-of-experts design<ref:2606.30336#pg2>. The architecture extends to settings such as domain transfer and few-shot adaptation<ref:2606.30336#pg2>.

How it works

FlexTab's encoder excludes any target values, resulting in general-purpose, target-agnostic row representations that are used as the cross-attention KV in the task-specific decoders<ref:2606.30336#pg2>. The decoder learns to identify patterns over sequences of embeddings corresponding to structurally related rows, enabling relational reasoning across tables<ref:2606.30336#pg2>.

The architecture is not limited to the tasks studied here; it extends to settings such as domain transfer, few-shot adaptation, zero-shot prediction, recommendation, and more<ref:2606.30336#pg2>. The breadth of tasks unified under a single encoder is viewed as a first step toward multi-task training at scale<ref:2606.30336#pg2>.

How it works

The architecture is designed to handle different tabular tasks natively and can easily be extended to additional ones in the future<ref:2606.

Improvements for AI systems

  1. Bold header: Target-Agnostic Feature Representation

FlexTab produces targetagnostic row embeddings that can be leveraged across a wide range of downstream tasks, allowing a single encoder to serve as an effective general-purpose backbone for diverse tabular prediction problems. This capability enables the system to perform classification, regression, anomaly detection, clustering, entity matching, and entity classification without needing separate models for each.

  1. Bold header: Decoupled Task-Specific Decoding

The architecture uses a single shared encoder paired with a suite of taskspecific decoders, which strictly decouples target-agnostic feature representation from task-specific predictive modeling. This allows the model to adapt to different prediction objectives, such as using a regression head for regression or a classification head for classification, while maintaining the same underlying feature encodings.

  1. Bold header: Relational Reasoning Across Tables

The FlexTab-Multi decoder facilitates relational inference by allowing the decoder to attend over a large and diverse set of [ROW] tokens, which enables it to aggregate information from multiple tables and relation paths. This allows for reasoning across tables in relational databases, as shown by its performance on RelBench tasks.

  1. Bold header: Flexible Data Handling in Matching

For entity matching, the model can utilize cross-row attention to the two separate row embeddings and can even use a dedicated matching decoder, trained on matching tasks, allowing it to achieve strong performance where other methods perform quite poorly by recasting cross-table matching as a single-table problem.

  1. Bold header: Scalable Relational Context Aggregation

The FlexTab-Multi setup demonstrates that performance scales favorably with context size, showing a clear and statistically significant edge on semantic-heavy tasks. This confirms that the model can effectively leverage larger intra-table context to produce richer row embeddings and aggregate information from a growing set of [ROW] embeddings up to a saturation point.

  1. Bold header: Robust Anomaly Detection Performance

The FlexTab-AD variant achieves competitive performance in anomaly detection, showing that it outperforms most baselines and is the best-performing method when very few anomalies are labeled, demonstrating effectiveness across unsupervised, one-class, and semi-supervised settings.

  1. Bold header: Efficient Runtime for Large Contexts

The runtime analysis shows that FlexTab can achieve a 2× speedup over ConTextTab for a larger context size of 10k, indicating that the flash attention utilized in FlexTab is more efficient than the memory-efficient attention used by ConTextTab, especially when handling larger tables.

Sources

Related papers