On the expressivity of deep Heaviside networks
The episode discusses a paper on deep Heaviside networks (DHNs) and their expressivity limitations. Hosts discuss how adding structural augmentations like skip connections or linear neurons significantly boosts what these networks can represent, improving approximation rates and complexity bounds.