A Composite Activation Function for Learning Stable Binary Representations
The paper introduces Heavy-Tailed Activation Function (HTAF), a smooth approximation of the Heaviside function that allows neural networks to be trained stably using standard gradient descent, overcoming issues with non-differentiable binary activations. HTAF preserves a large gradient near zero while allowing controlled decay in the tail regions. This enables stable training for Spiking Neural Networks, Binary Neural Networks, and Deep Heaviside networks.