Test-Time Compute for Tabular Foundation Models: Mechanisms, Gains, and Limits
The research tested three ways to improve pre-trained tabular foundation models during inference: parameter adaptation (DiagScale), prediction aggregation (Greedy Selection), and context construction. Findings show that combining parameter-efficient fine-tuning with greedy selection yields the best results, though it requires significant computation. Context construction benefits are task-dependent, and current predictors cannot reliably predict success on new data.