TabFORGE: A Tabular Foundation Model for Generative Modelling

arXiv:2605.09424 · cs.LG · Submitted 2026-05-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "TabFORGE: A Tabular Foundation Model for Generative Modelling".

Jane: Detailed Research Summary: TabFORGE - A Causality-Aware Latent Diffusion Framework for Generative Tabular Modeling This research introduces TabFORGE,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper called "TabFORGE: A Tabular Foundation Model for Generative Modelling." It looks like they're tackling the problem of generating synthetic tabular data.

Jane: Exactly. The title says it’s a foundation model, which is a big deal because these models are supposed to learn general representations.

Lu: They are specifically focused on generative modelling, which means they aren't just trying to predict something; they're trying to create new data that looks like the real stuff.

Meng: So what does that mean for us practically? Is this going to save us time when we need synthetic data for testing our systems?

Tom: It seems they are addressing a gap where existing generative models just don't capture the causal structure of tabular data well. This paper is positioning TabFORGE as a way to fix that misalignment.

Jane: That makes sense. When you’re making synthetic data, if it doesn't respect how the features actually cause each other in the real world, it won't be useful for testing things properly.

Lalam: I see they are building this on top of pre-trained Tabular Foundation Representations for Generation, which is a solid starting point.

Tom: Right. The core idea here seems to be using that foundation knowledge but making it causal and robust through a new architecture called TabFORGE.

The paper's summary: Jane: So what’s the main mechanism they propose? They introduce this causality-aware feature encoder and a latent causal modeling part based on a diffusion transformer.

Tom: The key part I see is that this encoder takes the inter-feature causal interactions from different datasets and maps them into one unified latent space.

Lu: That unified space is important because it’s supposed to capture those general, underlying causal information across various data modalities, which they get by using early-to-middle layers of a trained Prior-data Fitted Network model.

Meng: So they are borrowing some prior knowledge from a PFN model to help their encoder learn these interactions before the diffusion part even starts.

Tom: That’s right. Then, this latent space is modeled by a score-based diffusion transformer that learns the latent distribution through denosing noisy embeddings while preserving the underlying structure of that data.

Jane: And they claim this lets them capture global data structures effectively without being messed up by feature ordering issues, which is a common problem in other generative approaches.

Lalam: The architecture also has this denoising-aligned decoder, which is trained directly on the denoisings embeddings generated by the diffusion part.

Tom: That’s where I think they get their distribution shift robustness; instead of training the decoder on raw feature embeddings, it’s trained on something more stable.

The paper's improvements: Jane: The authors highlight a few key improvements in TabFORGE. One is this unified causal latent space, which I think is crucial for capturing those inter-feature relationships properly.

Tom: They also emphasize the structural fidelity they achieve through the diffusion transformer, saying it preserves causal relationships better than models that only look at feature order or simpler generative priors.

Lu: And they point out this superior distribution shift robustness because of that denoising-aligned decoder, which mitigates the train-inference mismatch that commonly arises in latent diffusion models.

Meng: From a deployment standpoint, I’m interested in how practical this is. They say it offers strong practicability for tabular data generation and avoids imposing feature-order bias.

Tom: And they specifically mention unlimited feature cardinality, which is something we often struggle with when dealing with categorical features in traditional models.

Jane: They also suggest using a lightweight detokeniser for each dataset to handle high-cardinality categorical features without needing complex techniques like hierarchical classification, which keeps the computational overhead down.

Lalam: And they mention a significant efficiency gain: fitting cost is about nine point seven one percent of what it takes to train a strong dataset-specific generator from scratch, and generation cost is around one point six six percent of the fastest in-context tabular foundation model, TabEBM.

Conclusion: Tom: So wrapping up this discussion on "TabFORGE: A Tabular Foundation Model for Generative Modelling," the main implication is that they’ve created a way to generate high-quality synthetic tabular data with better structural fidelity than what we see in existing models.

Jane: It seems like the core idea here is leveraging implicitly learned causal information within a diffusion mechanism to ensure the generated data respects those relationships.

Lu: They argue that by using this framework, TabFORGE is one of the foundation models in their study that surpasses the strongest dataset-specific generator in global utility.

Meng: For me, the efficiency numbers are pretty compelling; if you can get that kind of quality with such a small fraction of training time compared to building something from scratch, that’s a major win for production.

Lalam: I agree. The authors plan to open-source TabFORGE so the community can work on this further, which is a good sign for long-term progress in tabular generation techniques.

Department of Computer Science and Technology, University of Cambridge, UK · SAP SE 3 Télécom Paris, Institut Polytechnique de Paris, France

cs.LG

Submitted: 2026-05-10

Updated: 2026-10-15

Comments: Accepted by the Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS 2026)

Code: https://github.com/SilenceX12138/TabFORGE

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Detailed Research Summary: TabFORGE - A Causality-Aware Latent Diffusion Framework for Generative Tabular Modeling This research introduces TabFORGE, a novel tabular foundation model specifically

Key concepts

TabFORGE: A new tabular foundation model built for generative tasks. It uses a specific two-stage design—a causality-aware encoder followed by a diffusion transformer—to learn general, transferable representations from various tabular datasets, enabling the creation of high-quality synthetic data.
Causality-Aware Feature Encoding: This component maps complex inter-feature causal interactions from different datasets into a unified latent space. It leverages prior knowledge from Prior-data Fitted Network (PFN) models to capture general causal information across various modalities, ensuring the learned representations are structurally meaningful.
Score-Based Diffusion Transformer: A model trained to learn the global data structure within the latent space by modeling score functions. This process allows TabFORGE to capture complex relationships and global structures effectively without being overly constrained by specific feature ordering, leading to better structural fidelity in generated data.

Terminology

Foundation Model — 5× in this paper · explained in 3 episodes
Foundation Models — 3× in this paper · explained in 7 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Generator — 2× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Decoder — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 9 episodes
Distribution Shifts — 1× in this paper · explained in 2 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Latent Space — 1× in this paper · explained in 7 episodes
Representation Learning — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

Detailed Research Summary: TabFORGE - A Causality-Aware Latent Diffusion Framework for Generative Tabular Modeling

This research introduces TabFORGE, a novel tabular foundation model specifically designed for generative modeling of heterogeneous tabular data, addressing a significant gap in existing work. The core motivation stems from the recognition that generative modeling is an exceptionally demanding test for foundation models, requiring robust, holistic representation learning tailored to the specific causal structure inherent in tabular data, rather than merely optimizing for a supervised prediction target.

Core Problem and Motivation

While prior work on tabular foundation models has shown promise in predictive modeling, existing generative tabular foundation models have struggled to consistently match the quality of dataset-specific generators when synthesizing synthetic data. This deficiency is attributed to their misalignment with the distinctive causal structural prior of heterogeneous tabular datasets. TabFORGE was developed precisely to overcome this limitation by explicitly incorporating causal information into its latent representation learning process.

TabFORGE Architecture and Novel Design

TabFORGE is built upon pre-trained Tabular Foundation Representations for Generation and is distinguished by a sophisticated, two-stage design that decouples latent modeling from decoding, thereby enhancing robustness:

Causality-Aware Feature Encoding: The model employs a causality-aware feature encoder to map inter-feature causal interactions from diverse tabular datasets into a unified latent space. Crucially, this encoder utilizes the early-to-middle layers of a trained Prior-data Fitted Network (PFN) model to effectively capture general, underlying causal information across various data modalities.
Latent Causal Modeling via Diffusion Transformer: The model then models these embedded causal interactions within the latent space by pretraining a score-based diffusion transformer across diverse datasets. This diffusion process allows TabFORGE to capture global data structures effectively without being constrained by potentially misaligned feature ordering, which is a common pitfall in other generative approaches.
Robustness via Denoising Alignment: A key innovation for robustness against distribution shifts between training and inference is the decoupling of the components. The decoder is not trained on the raw embeddings produced by the feature encoder; instead, it is pretrained as a denoising-aligned decoder directly on the denoised embeddings generated by the fitted diffusion transformer. This strategy ensures that TabFORGE remains stable and performs reliably even when encountering unseen data distributions at inference time.

Key Distinctions and Advantages

TabFORGE's architecture is defined by three core components:

Unified Causal Latent Space: It successfully encodes complex inter-feature causal interactions from diverse datasets into a single, unified latent space using the causality-aware encoder, leveraging prior knowledge from PFN models.
Structural Fidelity through Diffusion: The use of a score-based diffusion transformer enables the model to capture global data structures effectively, ensuring that generated data preserves the underlying causal relationships better than methods relying solely on feature order or simpler generative priors.
Superior Distribution Shift Robustness: By training the decoder on denoised embeddings, TabFORGE inherently mitigates distribution shifts in the latent space, leading to more reliable inference performance compared to models that rely on clean feature embeddings.

Empirical Results and Performance Metrics

The empirical evaluation of TabFORGE demonstrates its superior capabilities across multiple dimensions:

Generative Quality: TabFORGE is shown to effectively learn and leverage generalizable tabular representations, resulting in the efficient generation of high-quality synthetic tabular data characterized by strong structural fidelity.
Causal Preservation: A major finding is that TabFORGE generates synthetic data that better preserves the causal structures of real data, leading to improved global utility compared to existing models.
Performance Benchmarking: When evaluated comprehensively against 22 benchmark methods across 45 real-world datasets, TabFORGE substantially outperforms previous state-of-the-art tabular foundation models, notably surpassing TabICL and improving its global utility by a clear margin (e.g., from 0.41 to 0.64).
Efficiency: The model exhibits remarkable computational efficiency:
Fitting Cost: Total fitting time is typically only about 9.71% of what is required to train a strong dataset-specific generator (TabSyn) from scratch.
Generation Cost: Generation cost is approximately 1.66% of the fastest in-context tabular foundation model (TabEBM).
Risk Mitigation: TabFORGE significantly reduces the risk of overfitting compared to other high-performing methods, such as SMOTE, while maintaining strong ML efficacy.

Conclusion and Future Direction

In conclusion, TabFORGE introduces a causality-aware latent diffusion framework specifically engineered for heterogeneous tabular data. It successfully generates high-quality synthetic tabular data exhibiting superior structural fidelity by leveraging implicitly learned causal information within a robust latent diffusion mechanism. The authors assert that TabFORGE is the only foundation model in their study to surpass the strongest dataset-specific generator in global utility. Recognizing its potential impact, the authors plan to open-source TabFORGE to foster community collaboration and further advance high-fidelity tabular foundation generation techniques.

Improvements for AI systems

Improve data generation by leveraging implicit causal signals to capture global structures, as TabFORGE is designed to utilise the implicitly learned causal information underlying diverse tabular datasets in a unified latent space induced by a pretrained causality-aware feature encoder. This enables generation that better preserves structural fidelity, which is quantified by global utility.
Enhance robustness against distribution shifts during inference by implementing a denoising-aligned decoder, which the paper states mitigates the train-inference mismatch that commonly arises in latent diffusion models. This ensures that generation occurs on representations more consistent with inference time, unlike previous methods where the decoder is trained on clean embeddings by the encoder.
Increase flexibility for generating high-cardinality categorical features by employing a lightweight detokeniser for each dataset, which allows TabFORGE to naturally generate high-cardinality categorical features without more complex techniques such as hierarchical classification. This reduces computational overhead compared to fixed output heads in prior models.
Reduce training and generation costs significantly by utilizing a pretraining strategy where the frozen latent embeddings are computed only once, avoiding repeated PFN forward passes, which substantially reduces the computational cost of both pretraining and fitting.
Improve generalization across diverse datasets by using an encoder that extracts transferable and causality-aware representations from pretrained PFN models by utilizing their early-to-middle layers, as these layers are hypothesized to implicitly capture inter-feature causal interactions.