Fast weight programming and linear transformers: from machine learning to neurobiology
Fast Weight Programmers (FWPs) use 2D matrices as hidden states to model dynamic synaptic weights, bridging machine learning and neuroscience. The research shows how different local loss functions translate into specific weight update rules, demonstrating that FWPs can instantiate various sequence models like Transformers and offer a biological perspective on plasticity timescales.