RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
RAM-Net reconciles full attention's capacity with linear models' efficiency by mapping inputs to high-dimensional sparse vectors acting as explicit memory addresses. It uses a differentiable address decoder to create K-hot vectors, allowing it to scale state size exponentially without extra parameters, improving retrieval accuracy while maintaining fast computation.