TANGO: Treating Tokens as Operators
TANGO replaces standard self-attention and feed-forward networks with a single cross-token gated residual update. It computes weights based on source token similarities to modulate destination features, allowing a gate at one position to influence another. This method shows superior performance across language modeling benchmarks.