Gradient descent dynamics for deep equilibrium models
This work studies how gradient descent trains deep equilibrium models (DEQs), which are used for infinitely deep neural networks. For linear DEQs, a conservation law keeps parameters on spheres, ensuring well-conditioned gradients and exponential convergence to a global minimum. Nonlinear single-index models also show exponential convergence under certain conditions.