TabReliBench: Evaluating Tabular Foundation Models Beyond Predictive Accuracy

My synthesis will be comprehensive, integrating the benchmark design with the reliability analysis techniques described in the second text to provide a detailed overview of the research.

Here is a long and detailed summary combining both sources:

Comprehensive Research Summary: Evaluating Reliability in Tabular Foundation Models using TABRELIBENCH and Conformal Prediction Analysis

This body of work addresses a critical, yet under-studied, aspect of Tabular Foundation Models (TFMs): the reliability of their predictive outputs, particularly concerning the posterior predictive distributions (PPDs). The research is twofold: first, establishing a rigorous benchmark for comparative analysis using TABRELIBENCH, and second, employing advanced conformal prediction techniques to systematically assess multidimensional reliability metrics across various models and post-processing strategies.

1. TABRELIBENCH: A Multidimensional Benchmark for Reliability

The core contribution of the first text is the introduction of TABRELIBENCH, a large-scale classification and regression benchmark designed specifically to evaluate TFM performance against classical methods (like BART, RealMLP variants, and XGBoost). The benchmark is structured to probe five distinct dimensions of reliability:

1. Point Prediction Accuracy: Standard predictive performance metrics (e.g., Accuracy for classification, Predictive R squared for regression).

2. Distributional Quality: Assessing the overall shape and fidelity of the PPDs generated by the models.

3. Marginal Calibration: Evaluating how well the predicted probabilities/scores align with observed outcomes across all data points (marginal coverage).

4. Classwise or Subgroup Calibration: Examining calibration performance across different classes or subgroups within a dataset, which is crucial for fairness and robustness.

5. Interval Sharpness (or Efficiency): Assessing the width of the prediction intervals relative to the nominal coverage level, indicating how tightly the model constrains uncertainty.

Key Findings from TABRELIBENCH Analysis:

• Metric Divergence: The study reveals that predictive accuracy, marginal calibration, subgroup reliability, and interval efficiency answer fundamentally different questions. A strong TFM optimized for point prediction may fail on distributional quality or subgroup calibration.

• Ranking Instability: The relative rankings of TFMs are not static; they shift significantly between the TabArena benchmark and TABRELIBENCH across different reliability metrics. This suggests that aggregate performance metrics are insufficient for reliable model selection.

• Calibration vs. Predictive Power: A crucial finding is that regression predictive performance (e.g., R squared) is not a reliable indicator of calibration, and classification difficulty often correlates with poor calibration, even when point accuracy is high.

• Post-Processing Limitations: Efforts to improve reliability through ensembling or conformal post-processing show mixed results:

• Ensembling improves predictive performance (median accuracy) but does not consistently enhance the calibration of PPDs.

• Conformal post-processing does not reliably resolve subgroup gaps, indicating that reliable interval evaluation requires jointly examining marginal coverage, subgroup coverage, interval width, and degenerate outputs.

2. Conformal Prediction Analysis: Deconstructing Coverage Reliability

The second text provides the experimental backbone for rigorously testing these reliability claims using conformal prediction (CP) methods applied to the same set of TFMs (including BART and other variants). This analysis focuses intensely on coverage reliability across various nominal levels (alpha = 0.80, 0.85, 0.90, 0.95) and different post-processing techniques:

• Methodology: The experiment utilizes a shared random permutation to create auxiliary (A), calibration (C), and test (T) blocks for conformal procedures.

• Post-Processing Comparison: The study compares several methods:

• Global Split Conformal Methods (SCP-CQR): This method is shown to reduce marginal error throughout, particularly for BART at lower targets, but it can increase Worst-Slab Coverage Error (WSC) at higher coverage levels (80–90%).

• Localized Conformal Methods (CalLCP, RLCP): These methods focus on localized score modeling and calibration.

• Learned Conditional Adaptation (RCP, PCP): These methods fit distributional regressors or apply conformal procedures to absolute residuals.

• Ensemble Effects: Varying the ensemble size (E) reveals metric-dependent gains. While accuracy and proper scores generally improve with larger ensembles, the effect on calibration diagnostics is less uniform.

• Coverage-Width Trade-off: The analysis explicitly links coverage error to interval width. Figure 44 demonstrates that as ensemble size increases, there are corresponding changes in marginal and worst-slab coverage errors relative to Vanilla intervals.

Synthesis and Central Conclusion

The combined research paints a picture where predictive accuracy, marginal calibration, subgroup reliability, and interval efficiency are distinct properties answering different questions.

The overarching conclusion is that reliable tabular prediction cannot be achieved by optimizing a single metric. Instead, the paper asserts that reliable TFM selection requires a multidimensional evaluation approach. For any given target dataset and decision point, one must jointly evaluate these properties:

1. Point Prediction Accuracy (What is the best guess?)

2. Marginal Calibration (How reliable are the scores across all data points?)

3. Subgroup Reliability (Is the model equally reliable for different segments of the data?)

4. Interval Efficiency/Sharpness (How tight and trustworthy are the uncertainty bounds?)

The research strongly cautions against relying on aggregate rankings or single-metric improvements, emphasizing that post-processing methods must be chosen based on which specific reliability dimension—marginal error reduction, worst-slab coverage improvement, or interval sharpness—is most critical for the application at hand.

1. textbfConditional Coverage Assessment for Regression TFMs: Develop a reliability profile that jointly evaluates marginal coverage, subgroup coverage, interval width, and degenerate outputs. This allows for a more nuanced understanding of PPD reliability than relying on aggregate rankings alone, as the paper states: Reliable interval evaluation therefore requires examining marginal coverage, subgroup coverage, interval width, and degenerate outputs jointly.

2. textbfContext-Aware Model Selection: Implement a meta-learning framework using dataset meta-features to predict which TFM will perform better rather than simply predicting which TFM will perform best on a new dataset, as the analysis shows that meta-features predict whether TFMs will generally perform well more readily than they predict which TFM will perform best.

3. textbfDataset-Specific Calibration Diagnostics: Use dataset meta-features to identify interpretable warning signs for classification and regression reliability, as the paper finds that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration.

Tabular foundation models (TFMs) are increasingly popular for their strong predictive performance. Many also provide posterior predictive distributions (PPDs), whose reliability remains less studied. We assess reliability through point prediction, distributional quality, marginal and subgroup calibration, and interval efficiency. We then introduce TabReliBench, a large classification and regression benchmark for these comparisons and dataset-level analyses. We find that TFMs' rankings change between TabArena and our benchmark and across reliability metrics. Regression TFMs usually achieve near-nominal marginal coverage and retain narrow intervals, but this favorable marginal behavior does not imply valid conditional coverage. Dataset-level analyses show that classification difficulty often accompanies poor calibration, whereas regression predictive performance is not a reliable indicator of calibration. Ensembling improves predictive performance but does not consistently improve calibration of PPDs; TFMs also benefit little from conformal post-processing, which does not reliably resolve subgroup gaps. Reliable TFM selection therefore requires multidimensional evaluation beyond aggregate rankings. Code is available at https://github.com/Tianqi-Zhao/TabReliBench.