MotherTree: Meta-learning on synthetic data improves decision tree training

arXiv:2610.10832 · cs.LG · Submitted 2026-10-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "MotherTree: Meta-learning on synthetic data improves decision tree training".

Tom: The gist: MotherTree, a tabular transformer that meta-learns decision tree induction, generates hard,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Moving on to the details of MotherTree, we're looking at the title and the authors. The paper is called "MotherTree: Meta-learning on synthetic data improves decision tree training."

Jane: That title really tells you exactly what they are focusing on—the idea that using meta-learning from synthetic data can improve how we train these decision trees.

Lu: The authors, Ziyuan Wang and Fredrik D. Johansson, are coming from Chalmers University of Technology in Gothenburg, Sweden, and they've developed this architecture at that institution.

Meng: So having the development come from a university setting gives you an idea about how they approach this kind of research—it’s focused on solving a specific problem within machine learning on tabular data.

Lalam: The authors are presenting a new way to approach decision tree induction, moving away from the traditional methods that require learning every task individually.

Tom: Exactly, and the implication here is that we can generate these decision trees much faster for new problems than if we had to train them completely from scratch every time.

Jane: It’s about getting a strong foundation before you even start training on the actual data set.

Lu: They are showing how meta-learning from a synthetic prior distribution enables strong in-context prediction for previously unseen tasks, especially when dealing with small sample regimes where we usually struggle.

Meng: That's a key area because most of the time, when you have limited data, standard methods really struggle to generalize well.

Lalam: They are introducing MotherTree as a way to tackle that generalization challenge by using the synthetic prior distribution psyn(D) for their training objective Lmeta(phi; D).

The paper's summary: Tom: Now let’s look at the actual summary of the paper. The gist is that MotherTree is a tabular transformer that meta-learns decision tree induction.

Jane: That means it learns how to generate those trees by pre-training on synthetic data, and then uses that knowledge to produce a hard, axis-aligned decision tree in just one forward pass for any new task.

Lu: The mechanism involves a data set encoder Ephienc that first embeds the data D into row-wise representations H, which are then aggregated by class into a single vector-valued data set embedding Ephienc(D).

Meng: This embedding feeds into the tree decoder Dphidec, which maps it to decision tree parameters theta, which are defined as I, T, L—the feature selection indices and thresholds for internal nodes and leaf values.

Lalam: The output of the tree decoder ultimately predicts the node split i sub i tau i for internal nodes or just the leaf value Lj for leaves.

Tom: So it’s a transformer architecture that learns this map Mphi from labeled training set D to decision tree parameters theta, and this process is pre-trained end-to-end without needing reference trees for supervision.

Jane: That means they are creating a reusable generator rather than just another specific classifier for one task.

Lu: They use a tree-structured decoder that applies masked attention with an ancestral attention mask M to encode the root-to-node dependency structure, which is important for capturing that structure.

Meng: So the whole setup is designed to learn how the data set embedding translates into the specific decision tree parameters in one go.

Lalam: And they show that this map Mphi acts exactly like a tree learning algorithm, where it's pre-trained end-to-end with instance labels as supervision.

The paper's improvements: Tom: Let’s talk about the specific improvements they highlight. They emphasize that MotherTree formulates decision tree induction as an amortized meta-learning problem and introduces this transformer model for generating standalone hard axis-aligned decision tree classifiers in a single forward pass.

Jane: That single forward pass is what makes it so efficient for deployment, because you don't need to re-run a whole training process every time.

Lu: They also highlight that the pre-training on synthetic data allows for generalization to real-world tasks with different feature sets, which is a big deal.

Meng: They show that task-specific tuning of the generated tree often outperforms the corresponding from-scratch learner on all benchmarks and sample sizes, which means you get a strong initialization.

Lalam: This strong initialization means that task-specific fine-tuning can be used to outperform from-scratch gradient-based learning and improve performance, especially when training data is limited.

Tom: And they also show robustness against sub-optimal features in parity tasks like XOR benchmarks, where it degrades more gradually and remains substantially more robust across the difficult configurations.

Jane: That robustness against misleading shortcut features means the initialization helps to resist those tricky shortcuts that might ruin a tree's performance when you have limited depth.

Conclusion: Tom: So to wrap up, MotherTree is a tabular transformer that meta-learns decision tree induction using synthetic data to output hard, axis-aligned decision trees in a single forward pass.

Jane: The key implication is that we get a strong initialization from pre-training on synthetic data, which then allows task-specific tuning to beat from-scratch methods.

Lu: It’s a model that provides a task-dependent initialization that consistently outperforms from-scratch optimization when observations are scarce, and it shows the prior affects the tree itself, producing fewer contradictory root-to-leaf paths than GradTree under the same depth and parameterization.

Meng: The runtime is very fast on both GPU and CPU, needing about zero point one five to zero point two seconds on GPU and zero point one seven to zero point three five seconds on CPU across the evaluated training set sizes <ref:2610.10832#pg1>.

Lalam: They also noted that while it's generating complete binary trees with a fixed topology and depth, which prevents adaptation of tree size or topology to individual tasks.

Tom: So the big picture is that meta-learning provides a strong inductive bias when observations are scarce, but task-specific optimization remains complementary for improvement.

Ziyuan Wang, Fredrik D. Johansson

Chalmers University of Technology · University of Gothenburg

cs.LG

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/wOOL/DNDT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 78/100

The gist: The gist: MotherTree, a tabular transformer that meta-learns decision tree induction, generates hard, axis-aligned decision trees in a single forward pass by pre-training on synthetic data to improve

Key concepts

Meta-learning: This is a technique where a model learns how to learn new tasks efficiently. MotherTree uses this by pre-training on synthetic data from various prior distributions, allowing it to quickly generate effective decision trees for completely new, unseen tasks with minimal specific training.
Decision Tree Induction: This refers to the process of automatically learning the structure and parameters of a decision tree—the rules used to classify data. MotherTree meta-learns this process, meaning it learns the optimal way to select features and set thresholds for internal nodes based on prior knowledge.
Tabular Transformer: This is a type of neural network architecture specifically designed to process tabular data (like spreadsheets) using transformer mechanisms. MotherTree uses this structure, combining a data encoder with a tree-structured decoder, to map input data directly to the parameters of a decision tree.
Meta-learning Prior: These are synthetic datasets or structural models used during the pre-training phase. MotherTree is trained using priors like TabICL and TabICLv2. These priors act as a source of inductive bias, guiding the model to learn robust decision tree structures before it ever sees the target task data.

Terminology

Meta-learning — 1× in this paper · explained in 3 episodes

Summary

The gist: MotherTree, a tabular transformer that meta-learns decision tree induction, generates hard, axis-aligned decision trees in a single forward pass by pre-training on synthetic data to improve decision tree training.

How it works

MotherTree is introduced as a tabular transformer that meta-learns decision tree induction, given a training set for a new task, it outputs a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. It is pre-trained on a synthetic prior using stochastic gradient descent without requiring reference trees for supervision. The model combines a tabular data encoder with a tree-structured decoder to learn a map Mϕ from labeled training set D to the decision tree parameters θ, where θ = (I, T, L) represents the feature selection indices and thresholds of internal nodes and leaf values.

The architecture consists of two main components:

Data set encoder Eϕenc: This component first embeds D into row-wise representations H = [h1,..., hn]⊤ ∈ R n×dc, where dc is a hyperparameter, and then aggregates these by class into a single vector-valued data set embedding Eϕenc(D).
Tree decoder Dϕdec: This component maps the data set embedding to decision tree parameters θ = Dϕdec (Eϕenc(D)), utilizing a tree-aware transformer decoder that applies masked attention with an ancestral attention mask M to encode the root-to-node dependency structure.

Meta-Learning and Training

The goal of MotherTree is to amortize learning of hard, axis-aligned decision trees by fitting a reusable tree generator Mϕ: D 7→ θ, whose parameters ϕ are fit on tasks from a synthetic prior. The training objective is formulated as Lmeta(ϕ; D) = −1/n Xn i=1 log pMϕ(D)(yixi), and the optimal parameters are found by minimizing this loss over tasks sampled from a structural causal model (SCM) prior psyn(D). This allows the generator to be trained end-to-end through the tree by minimizing Lmeta(ϕ; D).

The training process involves two stages using different synthetic priors. The first stage utilizes the synthetic prior from TabICL (QU et al., 2025) with 400k training steps, which substantially improves the validation AUC compared to training only on the TabICLv2 prior. The second stage uses the TabICLv2 prior for 125k steps, and incorporating Stage 1 pre-training substantially improves the validation AUC compared to training only on the TabICLv2 prior.

Inference and Tuning

For a new task, MotherTree uses a single forward pass to fit a tree that can be further tuned if desired. The estimated probability of each class is given by the corresponding element of the tree output, pˆMϕ(D)(yixi) = TMϕ(D)(xi)yi. Post-hoc parameter tuning is offered via MotherTreeTune, which fine-tunes all generated parameters θnew = (Inew, Tnew, Lnew) to Dnew using stochastic gradient descent. This task-specific tuning often outperforms the corresponding from-scratch learner on all benchmarks and sample sizes.

Evaluation and Results

MotherTree is evaluated on 64 benchmark classification tasks from TabArena and OpenML CC-18 in controlled small-sample regimes. MotherTree consistently outperforms the corresponding from-scratch learner, GradTree (Marton et al., 2024), e.g., exceeding its AUC by 0.069 on TabArena and 0.077 on OpenML CC-18 for B = 100 training samples. MotherTree-Tune consistently outperforms GradTree and GradTree←TabPFN across training set budgets (Figures 2 & 3), benchmarks, and label types (Table 1).

The study also investigates the utility of meta-learning on synthetic parity (XOR) tasks corrupted with individually-informative distractors. In this setting, MotherTree-Tune degrades more gradually and remains substantially more robust across the difficult configurations, indicating that the meta-learned initialization helps task-specific optimization resist locally predictive but structurally sub-optimal features.

Limitations and Future Directions

MotherTree currently generates complete binary trees with a fixed topology and depth, which simplifies amortized generation but prevents the model from adapting tree size or topology to individual tasks. The study suggests that while meta-learning provides a strong inductive bias when observations are scarce, task-specific optimization remains complementary and can be improved further by continued tuning. Future research should extend MotherTree to generate variable-size or pruned tree structures.

The paper concludes that the study demonstrates the value of meta-learning induction of single-tree classifiers, but tree ensembles universally achieve stronger predictive performance by combining predictions of multiple complementary trees. The work is supported in part by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. The results show that this prior affects the tree itself, as MotherTree uses a larger fraction of leaves and produces fewer contradictory root-to-leaf paths than GradTree under the same depth and parameterization. This shows that the improvement from meta-learning also leads to a better tree structure. The runtime for MotherTree is consistently the fastest method on both GPU and CPU, requiring only about 0.15–0.20 seconds on GPU and 0.17–0.35 seconds on CPU across the evaluated training-set sizes. The runtime for MotherTree-Tune increases with the training-set size, reaching 4.75 seconds on GPU and 5.47 seconds on CPU at B = 1000. The performance gains are strongest when task-specific training sets are limited in size, consistent with the role of meta-learning as a task-conditioned prior. The results suggest that meta-learning and task-specific optimization are complementary, where MotherTree provides a task-dependent initialization that consistently outperforms from-scratch optimization. The paper is available as preprint arXiv:2610.10832v1 [cs.LG] 7 Oct 2026.

--- Page 5 ---

The training process involves two stages using different synthetic priors. The first stage utilizes the synthetic prior from TabICL (QU et al.

Improvements for AI systems

Bold header: Amortized decision tree induction via MotherTree transformer

MotherTree meta-learns decision tree induction, allowing it to output a hard, axis-aligned decision tree, equivalent in form to classically trained trees, in a single forward pass. This enables the system to generate task-specific classifiers rapidly without requiring learning every new task from scratch.

Bold header: Strong initialization for gradient-based learners

The model acts as a strong initializer, as task-specific tuning of the generated tree outperforms the corresponding from-scratch learner on all benchmarks and sample sizes. This means that task-specific fine-tuning can be used to outperform from-scratch gradient-based learning and improve performance, especially when training data is limited.

Bold header: Robustness against sub-optimal features in parity tasks

MotherTree provides a stronger initialization that makes gradient-based tree optimization more robust to misleading shortcut features. This is evident in XOR benchmarks, where it degrades more gradually and remains substantially more robust across the difficult configurations when faced with distractors that consume the limited depth budget.

Bold header: Efficient generation for standalone inference

The system generates a stand-alone model; test-time prediction requires only tree traversal, but no access to the training set or the tree generator. This allows for deployment where in-context learning does not produce a reusable stand-alone model, solving obstacles for interpretation and sharing.

Bold header: Task-specific optimization via MotherTreeTune

The proposed MotherTreeTune variant allows users to fine-tune all generated parameters θnew = (Inew, Tnew, Lnew) to Dnew using stochastic gradient descent, creating models that are competitive with all baseline classes of tree learners.