SoftMCC: An MCC-Brier Calibration Bridge for Threshold-Free Model Selection under Class Imbalance

SoftMCC is a post-training MCC validation framework on established probability-valued

confusion counts, coupling an MCC-specific calibrated identity with a tie-aware, sharedpool selection protocol. Its core score is a covariance-normalized probability-label association, reduces exactly to MCC for hard predictions, and is Pearson-bounded. Under

perfect population calibration it equals the Brier skill score with identical candidate ordering;

outside that regime the gap does not identify calibration error. Across 18 settings with

12 duplicate-safe grouped repeats, SoftMCC attains the best stability mean rank (2.31)

and highest mean tie-corrected Kendall’s W (0.659), with a significant Friedman test

(p = 0.007); Nemenyi analysis separates it from AUPRC and MCC@0.5, while 14-sourcefamily sensitivity retains only the latter. Selected-model utility shows no advantage.

Three of six prespecified comparisons have negative mean test-MCC differences, only

F1 @best survives Holm correction (p = 0.014), and the dataset-level test is not significant

(p = 0.117). Label permutation lowers mean W to 0.092; temperature scaling shifts

SoftMCC rankings (mean Spearman 0.851) whereas rank-based and threshold-optimized

metrics remain invariant. SoftMCC is a calibration-sensitive MCC-family selector with

bounded stability and utility evidence.

Improvements to AI Systems:

1. Calibration-Aware Model Selection: Integrate SoftMCC as a post-training validation metric for classification systems, replacing or supplementing AUPRC/MCC when selecting between candidate models. This ensures the chosen model has both discriminative power and probability calibration, reducing silent failures in high-stakes deployment (e.g., medical diagnosis, fraud detection).

2. Tie-Aware and Duplicate-Safe Evaluation: Use SoftMCC’s shared-pool selection protocol to handle tied predictions and repeated grouped data (e.g., multi-site clinical trials, user sessions) without inflating stability metrics. This improves the reliability of model comparison in real-world, non-IID datasets.

3. Bounded Stability Monitoring: Deploy SoftMCC’s Pearson-bounded score as a continuous monitoring signal during model updates. If SoftMCC drops significantly while hard-MCC stays stable, it flags calibration drift—enabling early intervention before downstream decision quality degrades.

4. Calibration-Sensitive Early Stopping: During training, use SoftMCC as a validation criterion for early stopping. Unlike accuracy or MCC, it penalizes overconfident wrong predictions, guiding the model toward better probability estimates without sacrificing ranking performance.

5. Robust Benchmarking for AutoML: In automated machine learning pipelines, replace single-point MCC with SoftMCC’s mean rank across multiple repeats and label permutations. This yields more stable leaderboards and reduces the chance of selecting a model that wins by chance on a fixed test set.

What the Improved AI System Can Do:

• Select models that generalize better under distribution shift, because SoftMCC rewards calibrated probabilities, not just hard-label accuracy.

• Detect silent calibration failures in production—e.g., when a model’s confidence inflates after fine-tuning, SoftMCC drops even if accuracy stays flat.

• Compare models fairly on grouped, imbalanced, or duplicated data (e.g., patient records with multiple visits) without bias from tie-breaking or repeated samples.

• Provide statistically sound model rankings with known significance (Friedman + Nemenyi), enabling defensible model selection in regulated environments.

• Maintain ranking stability under temperature scaling or threshold optimization, meaning the chosen model remains optimal even if downstream decision thresholds are tuned separately.