Interdomain Attention: Beyond Token-Level Key-Value Memory
Interdomain Attention integrates a State Space Model (SSM) into an attention module using kernel methods to create query-conditioned attention over a fixed-size state. This method improves performance over existing token mixers, maintains length-flat behavior in long contexts, and allows for efficient per-token generation independent of sequence length.