Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
The paper argues that extending in-context learning to lifelong settings demands parametric attention because non-parametric methods cause memory caches to grow indefinitely, which is impossible under fixed hardware limits. The solution is using online, parametrically defined representations of context instead of storing every past example. This involves redefining attention as an online learning algorithm that continuously updates its internal model.