HAND: A Biologically-Inspired Activation Function that Improves Generalisation and Sample Efficiency in Image Classification
cs.CV, cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Code: https://github.com/pytorch/vision
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: DNNs exhibit robustness and generalisation issues not seen in humans.
Terminology
Abstract
DNNs exhibit robustness and generalisation issues not seen in humans. They are also far less data-efficient learners, requiring considerably more training samples to accurately classify novel exemplars. Inductive bias could help with these issues by providing in-built mechanisms to improve generalisation, and hence, reduce reliance on learning from data. We incorporate a biologically-inspired inductive bias into a new activation function, HAND (Homeostasis, Accelerating Nonlinearity, and Divisive-nomalisation), and show its effectiveness with CNNs trained on image classification. Using HAND a ConvNeXt-tiny required 25 training epochs to reach the same accuracy on ImageNet1k as the unmodified model achieved after 200 epochs. Consistent with the effects of an inductive bias, the performance gap reduced with training time and increased data augmentation. When the volume of training data was reduced and unevenly distributed between classes (Long-tailed ImageNet) the improvements in accuracy were even larger and did not reduce with increased training time. Generalisation performance with the common-corruptions data, and the ability to reject samples from unknown classes, were unaffected or improved by HAND. Results generalised across CNN architectures and training data-sets. HAND can, therefore, reduce the required training time and/or the required volume and variety of training data, helping to improve sample efficiency.
Sources
- Synthesizing Robust Adversarial Examples
- Layer Normalization
- Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- On Evaluating Adversarial Robustness
- WDiscOOD: Out-of-Distribution Detection via Whitened Linear Discriminant Analysis
- Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection
- Analyzing Redundancy in Pretrained Transformer Models
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- Generalisation in humans and deep neural networks
- Partial success in closing the gap between human and machine vision
- Deep Residual Learning for Image Recognition
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Deep Anomaly Detection with Outlier Exposure
- Natural Adversarial Examples
- Scaling Out-of-Distribution Detection for Real-World Settings
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Universal Approximation with Deep Narrow Networks
- Gradient-Based Adversarial and Out-of-Distribution Detection
- A ConvNet for the 2020s
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models