Tabular Numeric Stretch Transformation

This paper introduces the stretch transformation framework, a principled approach to numeric feature preprocessing for tabular deep learning that formulates numeric feature preprocessing as an optimization problem to make the target function smoother and thus more learnable. The framework has two variants: unsupervised stretch and supervised stretch.

The key insight is that "feature transformation should not merely normalize distributions but make the target function smoother in the transformed space. This reduces the design of numeric feature transformations to a width-allocation problem: how much transformed-space width should be assigned to different regions of a numeric feature?"

The paper addresses the challenge that numeric tabular features can combine heterogeneous quantities, binary flags, counts, percentages, monetary values, and heavy-tailed continuous variables, each with distinct scales and statistical properties. This heterogeneity creates a challenging optimization landscape for neural networks, which typically assume well-conditioned inputs.

The authors note that although target information (e.g., labels) is typically available during training, most existing numeric feature transformations are unsupervised. The only existing supervised variant, PLE-T, uses targets only to set bin boundaries, not to optimize the transformation itself from the perspective of target-function learnability.

For a scalar feature x with range [xmin, xmax], the domain is partitioned into T intervals with quantile-based binning. The transformation is defined as:

s(x) = c t-1 + (x - b t-1)/(b t - b t-1) · w t, for x ∈ [b t-1, b t)

where w t > 0 is the width allocated to interval t, and c t is the cumulative width. The slope within each interval is w t/(b t - b t-1), allowing control over local stretch or squeeze of each region.

The framework uses Dirichlet energy minimization. Through a kernel-smoothing perspective with Gaussian RBF kernels, the authors show that maximizing kernel smoothness is equivalent to minimizing the Dirichlet energy:

E[g] = ∫01 ∥g′(y)∥22 dy

The discrete version becomes:

E disc = Σi ∥Δfi∥22/Δyi

where Δfi = f(x i+1) - f(x i) and Δyi = y i+1 - y i.

Without target information, the authors adopt a minimax principle and minimize the worst-case Dirichlet energy over all target functions with bounded local variation. This yields uniform spacing allocation:

w t* = 1/T for all t ∈ 1,..., T

Connection to empirical CDF: As T → n, our method converges to the exact empirical CDF, which maps each sample to its normalized rank. This provides a theoretical justification for the empirical finding in Beyazit et al. [3] that CDF transformation reduces frequency content and improves learnability.

Connection to PLE: Unsupervised stretch precisely parameterizes the PLE manifold by normalized arc length. Both achieve identical quantile-based density redistribution, differing only by coordinate representation. However, unsupervised stretch offers O(1) versus O(T) memory per feature and no dimensional expansion.

When target information is available, the authors optimize the discrete Dirichlet energy directly. Using a robust lower bound based on bin-wise total variation:

For bin t with width w t and total variation S t = Σ i∈I t ∥Δfi∥2, the optimal width is:

w t* = S t / Σu S u

"Intuitively, in the transformed space, we allocate a larger space to the regions where the target function varies rapidly, effectively equalizing the slope magnitudes across bins and creating a smoother target function in the transformed space."

Connection to target encoding: When T = n (one bin per unique value), the optimal width for the interval [x i, x i+1] becomes f i+1 - f i / Σ k f k+1 - f k. This is remarkably similar to applying min-max scaling to target encoding.

Out-of-fold estimation: To prevent information leakage, the authors use K-fold cross-validation with adaptive Nadaraya-Watson kernel regression to obtain f̂(x i) for each sample without using its own target value.

The paper evaluates on 38 datasets from the TALENT benchmark (26 classification, 12 regression), using five neural architectures: FT-Transformer, MLP, MLP-PLR, RealMLP, and ResNet, yielding 190 dataset-model combinations. Seven baselines are compared: standardization, Yeo-Johnson, quantile transformation, min-max scaling, RobustScale+SmoothClip, PLE, and PLE-T.

1. Supervised stretch achieves the strongest aggregate performance, especially in regression. "Across all evaluations, supervised stretch stands out as the strongest method, with its advantage most pronounced in regression, where it maintains a substantial margin over all competitors across significance thresholds. The win-rate analysis shows decisive regression win-loss records of 18-7 against PLE, 28-9 against PLE-T, and 14-4 against standardization."

2. Unsupervised stretch is a strong target-agnostic method. It "consistently ranks as a strong runner-up: it outperforms all methods except its supervised counterpart under the aggregate pairwise comparison and is especially competitive in classification, with classification win-loss records of 38–25 vs. PLE, 38–18 vs. PLE-T, and 26–13 vs. Standardization."

3. Transformation preferences vary by architecture and task type. Regression benefits more from advanced transformations than classification, with supervised stretch yielding the largest gains in continuous prediction tasks.

4. Non-uniform marginal signals help explain why smoothing is beneficial. "Real-world tabular features exhibit non-uniform marginal distributions with localized regions of sharp variation (high relative marginal slopes); this structural heterogeneity creates high-frequency components that are difficult for neural networks to learn due to spectral bias."

The paper tests robustness to label noise: "Supervised Stretch does not collapse under noise. The mean rank of Supervised Stretch stays in a competitive range over the full noise sweep and does not show the gradual degradation seen for some baselines. At the highest noise level we test (η = 0.5), it actually has the best mean rank among the five methods."

The paper provides theoretical explanations for existing empirical observations:

• Unsupervised stretch explains why CDF transformation can reduce frequency content and improve learnability, despite being label-agnostic.

• "Supervised stretch reveals deep connections to target encoding, providing a theoretical justification for why target-based transformations can be effective while offering principled regularization through the number of bins T."

The authors conclude that explicitly optimizing for target function smoothness is a powerful and underexplored strategy for tabular deep learning, and their results challenge the prevailing paradigm of unsupervised feature transformation in tabular deep learning and open new avenues for supervised preprocessing methods.

The paper notes: "Our framework's design could be enhanced by exploring smoother, spline-based alternatives, and adaptive binning strategies. The current transformation is marginal and therefore does not optimize a joint or conditional smoothness objective... Theoretically, the analysis could be generalized beyond the small-σ RBF kernel approximation. Applying Stretch as an option in a TabPFN inference-time preprocessing ensemble is also an interesting direction."

Improvements to AI Systems:

1. Adaptive Preprocessing Module for Tabular Deep Learning Models

• Integrate supervised stretch as a trainable preprocessing layer that dynamically allocates feature-space width based on target-function variation, replacing static normalization (e.g., standard scaling, quantile transforms).

• The improved system can automatically detect regions of rapid target change (e.g., income brackets, dosage-response curves) and stretch those regions to reduce spectral bias, leading to faster convergence and higher accuracy on heterogeneous tabular data (e.g., medical records, financial transactions).

2. Noise-Resilient Target Encoding for Regression and Classification

• Replace standard target encoding with supervised stretch using out-of-fold kernel-smoothed estimates and bin-width optimization.

• The improved system maintains predictive performance under label noise (up to 50% corruption) where conventional target encoding degrades, enabling robust learning in real-world noisy datasets (e.g., sensor logs, crowdsourced labels).

3. Unsupervised Feature Warping for Model-Agnostic Pipelines

• Deploy unsupervised stretch as a drop-in replacement for quantile/PLE transformations in any tabular pipeline (XGBoost, neural nets, etc.), with O(1) memory per feature.

• The improved system achieves CDF-like smoothing without dimensional expansion, reducing overfitting in small-sample regimes and improving generalization on heavy-tailed features (e.g., insurance claim amounts, web traffic counts).

4. Spectral-Bias-Aware Curriculum Learning

• Use the Dirichlet energy metric to order training samples or features during curriculum learning, prioritizing high-variation regions first.

• The improved system learns high-frequency target patterns earlier in training, reducing the number of epochs needed to fit sharp transitions (e.g., threshold effects in credit scoring, phase changes in material science).

5. Architecture-Specific Transformation Selection

• Implement a meta-learner that predicts whether supervised or unsupervised stretch will benefit a given architecture (e.g., MLP vs. Transformer) based on dataset properties (marginal slope distribution, target smoothness).

• The improved system automatically selects the optimal preprocessing strategy per model, yielding consistent gains across 190+ dataset-model combinations without manual tuning.

6. Joint Smoothness Optimization for Multi-Feature Interactions

• Extend the marginal stretch framework to a joint objective that minimizes Dirichlet energy over feature pairs or groups, using conditional binning.

• The improved system captures interaction effects (e.g., age × dosage) by allocating width in multi-dimensional space, improving performance on datasets with strong feature interactions (e.g., personalized medicine, recommendation systems).

7. Real-Time Adaptive Preprocessing for Streaming Data

• Implement an online variant of supervised stretch that updates bin widths incrementally as new labeled data arrives, using exponential moving averages of bin-wise total variation.

• The improved system adapts to concept drift in real-time (e.g., fraud detection, demand forecasting), maintaining smooth target functions without full retraining.

8. Explainable Preprocessing for Model Debugging

• Provide a visualization tool that plots the learned stretch function per feature, highlighting regions of high width allocation as high-information zones.

• The improved system enables data scientists to identify which feature ranges drive predictions, aiding feature engineering and model auditing (e.g., identifying critical thresholds in loan approval models).