What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
Kunwoong Kim, Dongha Kim
cs.LG, cs.AI, stat.ML
Submitted: 2026-06-29
Code: https://github.com/Minqi824/ADBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no
Terminology
Abstract
Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder, we show that, under mild assumptions, the model can successfully memorize inliers while failing to memorize outliers during certain stages of early training. In particular, we characterize not only the emergence of the IM effect, but also its strength and persistence, and analyze how these properties depend on the data distribution and parameter initialization. In addition, building on these insights, we derive simple yet practical guidelines for enhancing the IM effect, including data preprocessing and parameter initialization schemes, achieving state-of-the-art performance on the ADBench datasets. Our findings provide a theoretical foundation for the IM effect and offer actionable directions for improving IM-based outlier detection methods.
Sources
- Deep Learning for Anomaly Detection: A Survey
- GradStop: Exploring Training Dynamics in Unsupervised Outlier Detection through Gradient
- Deep Anomaly Detection Using Geometric Transformations
- On Diffusion Modeling for Anomaly Detection
- Auto-Encoding Variational Bayes
- Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
- Adam: A Method for Stochastic Optimization
- A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities
- Representation Learning with Contrastive Predictive Coding
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks