TraceRelay: Attention-Aligned Recurrence over Rolling Traces
TraceRelay is an attention-aligned recurrent model that organizes recurrence around attention using rolling sequences of low-dimensional traces. It introduces a delayed write mechanism where right writers create causally delayed increments, which are then transported by a phase accumulation process. The left reader accesses these augmented representations, showing superior performance in specific tasks based on the trace width.