What can linear attention learn from nonlinear teachers in-context?

Name: What can linear attention learn from nonlinear teachers in-context?
Uploaded: 2026-10-14T17:27:46.110416+00:00
Duration: 259 s
Description: Linear attention extracts only the linear Hermite component of a nonlinear function during in-context learning. The remaining nonlinear structure acts as effective noise, limiting performance to a new error floor ($ ho_{ ext{nonlinear}}$) even with more diverse training data. This shows the model learns the linear part but cannot capture complex nonlinear signals.

Linear attention extracts only the linear Hermite component of a nonlinear function during in-context learning. The remaining nonlinear structure acts as effective noise, limiting performance to a new error floor ($ ho_{ ext{nonlinear}}$) even with more diverse training data. This shows the model learns the linear part but cannot capture complex nonlinear signals.

Open the full episode