Thinking in Depth: Retrospective Inference for Tabular Foundation Models

The gist: RETRO introduces a tabular foundation model based on retrospective inference, which allows later stages to explicitly revisit and recombine intermediate information produced earlier in the network to achieve earlier and more distributed prediction refinement across depth

How it works

RETRO is a tabular foundation model that organizes its process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions This approach shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement

Retrospective Inference

The paper posits that predictive refinement is highly uneven across depth and often concentrated in later layers, which motivates the idea that earlier representations can be reconsidered rather than only influencing subsequent stages through the latest accumulated state This organization is called retrospective inference, where intermediate information remains available as the prediction evolves This perspective leads to two coupled design requirements: first, when several intermediate representations remain available, the model must determine which of them are relevant to the current sample; and second, after contextual information is computed from these representations, the model must determine how that update should modify the current query representation

RETRO Components

The RETRO architecture combines retrospective access and selective contextual incorporation through a hierarchical table encoder and a row-level ICL stack The table encoder consists of three column-processing blocks and three row-processing blocks, both with a hidden dimension of 128 and eight attention heads Within the ICL stack, retrospective aggregation constructs the inputs to attention and feed-forward sublayers from available historical contributions Specifically, block Attention Residuals are used for adaptive reuse of intermediate representations These readers allow a stage to provide information that remains useful when selected later, even if that information is insufficient for prediction on its own

Evaluation and Results

RETRO was evaluated on three benchmarks: TabArena, TALENT, and RelArena The results show that RETRO ranks among the top three methods across these benchmarks, lying on the Pareto frontier Specifically, in TabArena classification Elo rankings, RETRO ranks fourth and second for regression In TabArena regression rankings, RETRO ranks second with an Elo of 2082.1 The paper demonstrates that retrospective inference exhibits earlier and more distributed prediction changes, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement

Gated Attention Modulation

The second operation involves query-conditioned Gated Attention, which modulates the contextual updates produced within the resulting stack The gated update enters the current group’s accumulation, shaping the information retained for subsequent reads Analysis of these gates shows that replacing direction reduces mean classification accuracy by 0.616 percentage points, compared with −0.002 for replacing magnitude This suggests that gating affects the geometry of the contextual update rather than only its overall scale

Efficiency and Conclusion

RETRO retains the backbone’s width and depth, adding only historical aggregation modules and gate projections within the ICL stack The performance–runtime trade-off in Figure 1 shows that RETRO improves predictive performance over TabICLv2 with nearly unchanged inference time This positions RETRO as a competitive choice when both predictive quality and inference cost matter Together, these findings support organizing intermediate computation as reusable information rather than only as transient states passed from layer to layer

References

Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Julian J. McAuley. Rezero is all you need: fast convergence at large depth. In UAI, 2021 Amir Rezaei Balef, Mykhailo Koshil, and Katharina Eggensperger. Is one layer enough? understanding inference dynamics in tabular foundation models. In ICML, 2026 Marin Bilos, James T. Wilson, Anderson Schneider, and Yuriy Nevmyvaka. A mechanistic study of tabular foundation models. In ICML, 2026 Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(6):7499–7519, 2024 Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016 Felix den Breejen, Sangmin Bae, Stephen Cha, and Se-Young Yun. Why in-context learning transformers are tabular data classifiers. CoRR, abs/2405.13396, 2024 Moonjung Eo, Min-Kook Suh, Hye-Seung Cho, Jiwon Kim, Seoyoon Kim, Sangjun Nam, and Soonyoung Lee. EXAONE Tabular 1.0: Technical report. CoRR, abs/2608.25774 Nick Erickson, Lennart Purucker, Andrej Tschalzev, David Holzmuller, Prateek Mutalik Desai, David ¨ Salinas, and Frank Hutter. TabArena: A living benchmark for machine learning on tabular data. In NeurIPS, 2025 Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In NeurIPS, 2021 Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. Tabm: Advancing tabular deep learning with parameter-efficient ensembling. In ICLR, 2025 Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev, and Artem Babenko. Tabpack: Efficient hyperparameter ensembles for tabular deep learning. In ICML, 2026 Leo Grinsztajn, Edouard Oyallon, and Ga ´ el Varoquaux. Why do tree-based models still outperform ¨ deep learning on tabular data? In NeurIPS, 2022 Leo Grinsztajn, Klemens Fl ´ oge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Mihir Manium, ¨Shi Bin Hoo, Magnus Buhler, Anurag Garg, Dominik Safaric, Jake Robertson, Benjamin J ¨ager, Simone Alessi. Adrian Hayler. TabPFN-3: Technical ¨ report. CoRR, abs/2605.13986, 2026 Adrian Hayler, Klemens Floge, Alan Arazi, Rishabh Ranjan, Jure Leskovec, Felix Birkel, Brendan ¨Roof, Anurag Garg, Kristina Collins, Lydia Sidhoum. TabPFN-rel and RPI. CoRR, abs/2608.16319, 2026> Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016 Noah Hollmann, Samuel Muller, Katharina Eggensperger, and Frank Hutter.

1. textbfimate predictive refinement distribution across depth using RETRO's mechanism of retrospective inference. This allows for a more nuanced understanding of how information is utilized, as RETRO shifts predictive refinement earlier and more broadly across depth and shows different stages revising different subsets of queries in a pattern suggestive of multi-view refinement.

2. textbf implement query-conditioned modulation to tailor contextual updates at each layer. This is achieved via the Gated Attention mechanism, which modulates the attention output element-wise across representation dimensions, allowing the model to adapt how new contextual information is incorporated for each query.

3. textbf design a selective revisiting mechanism for intermediate representations using Attention Residuals. This allows later stages to explicitly revisit and recombine earlier information by adapting contributions through adaptive reweighting of contributions from different depths, which supports the hypothesis that this reuse favors complementary contributions across depth.

4. textbf leverage multi-view refinement patterns in inference. By observing that different stages revise different subsets of queries, the system can be designed to treat intermediate representations as reusable resources for subsequent inference, leading to a more robust and distributed prediction process across the network depth.

Tabular foundation models (TFMs) are pretrained across diverse tabular tasks and make predictions on a new table at inference time using its labeled examples as context. Most recent TFMs perform such in-context prediction with stacked Transformer layers, repeatedly transforming how examples are represented and compared. By tracing individual queries through several strong TFMs, we find that predictive refinement is highly uneven across depth and is often concentrated in later layers. This uneven refinement motivates us to reconsider how intermediate representations are constructed and reused throughout the network. We introduce Retro, a tabular foundation model based on retrospective inference, where later stages can explicitly revisit and recombine intermediate information produced earlier in the network. Retro organizes this process around two complementary operations: which intermediate information to revisit, and how the resulting contextual update should be shaped for each query. Attention Residuals address the former by adaptively reweighting contributions from different depths, while query-conditioned Gated Attention addresses the latter by modulating the attention output element-wise across representation dimensions. Our analysis shows that Retro shifts predictive refinement earlier and more broadly across depth, with different stages revising different subsets of queries in a pattern suggestive of multi-view refinement. Across TabArena, TALENT, and RelArena, Retro ranks among the top three and lies on the Pareto frontier. These results indicate that directly reusing intermediate representations provides a practical way to better exploit depth in TFMs.