FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking

FeedbackTrack: Visual-Cortex-Inspired Cross-Frame Feedback for Transformer Tracking

Abstract Summary:

Visual object tracking requires continuous integration of target information across frames, yet most Transformer trackers rely on predominantly feed-forward visual feature extraction. Existing temporal mechanisms commonly update templates, prompts, queries, or prediction states, while intermediate representations from previous frames rarely modulate corresponding stages of current-frame processing. The paper proposes FeedbackTrack, a visual-cortex-inspired framework that introduces sparse, group-level layer-aligned cross-frame feedback into pretrained Transformer trackers. Previous-frame outputs from selected Transformer groups are detached, cached, and returned to the corresponding groups in the current frame. FeedbackTrack contains two complementary pathways: Query Feedback transforms historical search-token states into low-rank query biases and applies per-sample, per-head, and per-token RMS alignment to control their magnitude; Gate Feedback uses pooled historical context to provide bounded channel-group modulation of the projected attention output. The mechanism retains the original tracking pipeline and requires only a fixed-size one-frame cache. Across SPMTrack and ARTrackV2, FeedbackTrack consistently improves five backbone configurations on LaSOT and GOT-10k. It improves GOT-10k AO by 2.3–4.1 points and LaSOT AUC by 1.1–1.8 points, reaching 83.4 AO and 79.1 AUC with SPMTrack-G, while adding less than 1% parameters in all configurations. Controlled comparisons show that cross-frame feedback outperforms same-frame modulation by 1.8–3.2 AO points, confirming that the gains primarily arise from recurrent historical information rather than additional modulation capacity. Analysis of the learned feedback scales reveals a non-uniform depth-dependent organization, demonstrating that recurrent cross-frame feedback provides an effective and scalable mechanism for Transformer tracking.

Introduction Summary:

Visual object tracking aims to continuously localize an arbitrary target throughout a video given only its initial state. A reliable tracker must preserve target identity while adapting to appearance variation, occlusion, fast motion, background clutter, and similar distractors. Recent advances have shifted tracking architectures from convolutional Siamese matching toward Transformer-based target–search interaction. In particular, one-stream and sequence-based trackers jointly model template and search information within a shared backbone, enabling strong feature interaction and accurate target localization. Despite their strong performance, existing Transformer trackers still rely predominantly on feed-forward visual feature extraction within each frame. Temporal information has been incorporated through updated templates, historical prompts, autoregressive queries, candidate associations, and additional temporal tokens. These mechanisms effectively preserve target history, but temporal states are commonly introduced at the input, prompt, query, or prediction level and then processed by another largely feed-forward visual encoder. Intermediate representations formed at a particular backbone stage in the previous frame are rarely returned to the corresponding processing stage in the current frame. Consequently, the visual feature hierarchy itself lacks an explicit recurrent pathway through which previously formed representations can directly modulate ongoing feature extraction. Biological vision suggests a different computational organization: the mammalian visual system combines ascending feed-forward pathways with extensive recurrent and feedback connections, allowing previously formed representations to influence subsequent sensory processing. Recent studies further indicate that visual feedback is distributed across processing stages, pathway-specific, and modulatory rather than a simple reversal of feed-forward activity. These findings motivate investigating whether recurrent reuse of intermediate visual states can improve Transformer tracking. The paper does not seek to reproduce specific cortical structures or neural dynamics, but instead abstracts the general principle that previously formed representations can provide stage-corresponding modulation of current visual processing.

The paper proposes FeedbackTrack, which introduces sparse, group-level layer-aligned cross-frame feedback into pretrained Transformer trackers. The visual encoder is partitioned into groups of consecutive Transformer blocks. For each selected group, the previous-frame group output is detached, cached, and returned to the first block of the corresponding group in the current frame. This design converts the originally stateless feature hierarchy into a lightweight recurrent system while retaining the original feed-forward pathway, tracking interface, and prediction modules. Only one-frame states are retained, so the cache size remains constant with respect to video length. FeedbackTrack implements recurrent modulation through two complementary pathways: Query Feedback transforms previous search-token states through a low-rank projection to generate token-wise biases for current search queries, with per-sample, per-head, and per-token RMS alignment controlling the bias magnitude relative to the pretrained queries; Gate Feedback pools the complete previous group state and generates bounded, context-dependent channel-group modulation of the projected attention output. Query Feedback influences where current attention retrieves evidence, whereas Gate Feedback regulates the resulting attention response.

The paper instantiates FeedbackTrack on two representative tracking frameworks, SPMTrack and ARTrackV2. On SPMTrack, FeedbackTrack improves GOT-10k AO from 76.5 to 80.6 for ViT-B, from 80.0 to 82.6 for ViT-L, and from 81.0 to 83.4 for ViT-G. The corresponding LaSOT AUC scores increase from 74.9 to 76.4, from 76.8 to 77.9, and from 77.4 to 79.1. FeedbackTrack also improves ARTrackV2-B and ARTrackV2-L to 81.2 and 81.8 AO on GOT-10k, with corresponding LaSOT AUC scores of 74.8 and 75.4. Across all five configurations, the additional parameter cost remains below 1%. Moreover, under the same modulation architecture and parameter count, cross-frame feedback outperforms its same-frame counterpart by 3.2, 2.3, and 1.8 AO points for ViT-B, ViT-L, and ViT-G, respectively, demonstrating that the improvements primarily arise from recurrent historical information rather than additional modulation capacity alone. Although all Query Feedback scales are initialized identically, the trained models develop a non-uniform depth-dependent organization, with relatively weak feedback in shallow groups and stronger modulation in intermediate and deep groups. This emergent pattern is qualitatively consistent with the hierarchical and pathway-dependent organization of biological visual feedback, while representing a computational correspondence rather than an anatomical equivalence.

Main Contributions:

• The paper introduces FeedbackTrack, a general framework that equips pretrained Transformer trackers with sparse, group-level layer-aligned cross-frame feedback, enabling intermediate previous-frame states to recurrently modulate corresponding stages of current-frame visual processing.

• The paper develops two complementary lightweight pathways: Query Feedback performs low-rank, RMS-aligned modulation of current search queries, while Gate Feedback applies bounded, context-dependent channel-group rescaling to the projected attention output.

• The paper demonstrates consistent improvements across two tracking frameworks and five backbone configurations on LaSOT and GOT-10k, while introducing less than 1% additional parameters. Controlled same-frame comparisons further verify the importance of recurrent historical information.

• The paper analyzes the learned feedback strengths and identifies an emergent depth-dependent organization, showing that cross-frame feedback is allocated non-uniformly across visual representation stages.

Related Work Summary:

Visual object tracking has progressively shifted from convolutional Siamese matching to Transformer-based architectures with explicit target–search interaction. Representative methods include TransT, STARK, OSTrack, SeqTrack, ARTrack, and ARTrackV2, while more recent approaches further explore adaptive temporal queries and parameter-efficient tracking. Although these trackers may use historical templates, prompts, or autoregressive predictions, intermediate visual representations are generally not recurrently returned to corresponding backbone stages in the following frame. FeedbackTrack complements these architectures by introducing sparse, group-level layer-aligned cross-frame feedback inside the visual encoder. Existing trackers exploit temporal context through historical templates, candidate associations, memory representations, prompts, autoregressive queries, and temporal tokens. These approaches typically introduce temporal information at the input, prompt, memory, or prediction level, after which the visual encoder remains predominantly feed-forward. In contrast, FeedbackTrack treats intermediate Transformer group outputs as recurrent visual states and returns each previous-frame group output to the corresponding current-frame group. This design is complementary to existing temporal modeling strategies, including the prompt-based autoregressive pipeline of ARTrackV2. Biological vision combines feed-forward and recurrent processing, with distributed and pathway-specific feedback modulating ongoing sensory representations across the visual hierarchy. FeedbackTrack does not reproduce specific cortical structures or neural pathways, but abstracts several general computational principles, including recurrent reuse of previous representations, correspondence between feedback states and processing stages, and residual modulation of current visual computation. These principles are implemented through group-level layer-aligned cross-frame connections together with Query Feedback and Gate Feedback.

Method Summary:

FeedbackTrack is formulated as a generic feedback mechanism for Transformer-based trackers and is instantiated on two representative frameworks, SPMTrack and ARTrackV2. For SPMTrack, the feedback modules are inserted into its DINOv2-based Transformer backbone, while the patch embedding, tracking embeddings, TMoE adapters, and prediction head remain unchanged. For ARTrackV2, only the visual encoder is modified, while the original trajectory prompts, appearance prompts, autoregressive prediction modules, and training objective are retained. FeedbackTrack augments the original visual feature extractor with sparse cross-frame connections while preserving the task-specific tracking interface and prediction pipeline.

The Transformer backbone is partitioned into groups of G consecutive blocks. Let Xt(i) and Ut(i) denote the input and output of group i at frame t, respectively. The output of the corresponding group in the previous frame is detached and stored as a one-frame cache: Ct−1(i) = StopGrad(Ut−1(i)). At frame t, the cached representation is provided to the first Transformer block of the same group: Ut(i) = Gfb(i)(Xt(i), Ct−1(i)), where Gfb(i) denotes group i equipped with the proposed feedback module. Because the output of group i at frame t−1 is returned to the first block of the same group i at frame t, the temporal connection preserves the hierarchical stage at which the representation was generated. This design is referred to as group-level layer-aligned cross-frame feedback. Only the first block of each group contains the feedback module, whereas the remaining blocks retain their original computation. A group size of G = 4 is used. For example, the output produced after blocks 0–3 at frame t−1 is returned to block 0 at frame t, while the output produced after blocks 4–7 is returned to block 4. Accordingly, feedback modules are inserted at blocks 0, 4, 8 for ViT-B, 0, 4, 8, 12, 16, 20 for ViT-L, and 0, 4, 8, 12, 16, 20, 24, 28, 32, 36 for ViT-G. The same grouping strategy is applied to the visual encoders of ARTrackV2-B and ARTrackV2-L.

The feedback module contains two complementary pathways. Query Feedback preserves token-level spatial information and recurrently modulates the current search-token queries. Gate Feedback summarizes the complete previous group state and applies context-dependent channel-group modulation to the projected attention output. The two pathways therefore inject historical information at different locations in the attention computation.

For Norm-Controlled Query Feedback, let Mt−1(q,i) ∈ R(B×Ns×D) denote the search-token subset selected from the cached representation Ct−1(i), where B is the batch size, Ns is the number of search tokens, and D is the feature dimension. The historical search-token states are first transformed using LayerNorm and a low-rank projection: Bt(q,i) = Up(i)(Down(i)(LN(Mt−1(q,i)))), where the down- and up-projections map the feature dimension as D → r → D, with a bottleneck dimension of r = 16. The projected historical bias is then reshaped into the multi-head form Bt(q,i) ∈ R(B×H×Ns×dh), where H is the number of attention heads and D = Hdh. Directly adding the projected historical bias to the pretrained queries may result in a substantial magnitude mismatch. Therefore, their root-mean-square magnitudes are aligned independently for each sample, attention head, and search token: B̂t,bhs:(q,i) = Bt,bhs:(q,i) · ρ / RMS(Bt,bhs:(q,i)), where b, h, and s index the sample, attention head, and search token, respectively. The RMS operator is defined as RMS(x) = sqrt((1/dh) Σ x2d + ϵ). The paper sets ρ = 0.05 and ϵ = 10−4. The aligned historical bias is added to the current search-token queries using a learnable head-wise scale vector sq(i) ∈ R H: Q̃t,bhs:(i) = Qt,bhs:(i) + sq,h(i) B̂t,bhs:(q,i). All q-scales are initialized to 0.01, allowing the modified tracker to start close to the pretrained model. Only the current search-token queries are modified, while all non-search queries, keys, and values remain unchanged.

For Gate Feedback, the complete previous group state is summarized by averaging over all cached tokens: pt−1(i) = (1/N) Σ Ct−1,n(i), where pt−1(i) ∈ R(B×D) and N denotes the total number of cached tokens. A lightweight MLP maps the pooled context from D to a hidden dimension of 384 and then to H channel groups. A tanh-bounded contextual signal is combined with a learnable channel-group scale vector sg(i) ∈ R H: gt(i) = tanh(MLP(i)(LN(pt−1(i)))) ⊙ sg(i), where gt(i) ∈ R(B×H) contains one modulation coefficient for each channel group. The learnable gate scales are initialized to 0.01. Let At(i) denote the attention output after head concatenation, output projection, projection dropout, and the original LayerScale. The feature dimension is reshaped into H channel groups, each containing dh channels, and the following is applied: Ãt,bnh:(i) = (1 + gt,bh(i)) At,bnh:(i). The same gate is broadcast across all tokens and applied before the original residual connection. Because the output projection has already mixed information from the original attention heads, this operation should be interpreted as channel-group rescaling of the projected attention output rather than direct modulation of individual attention heads.

During training, consecutive search frames are processed sequentially, and detached previous-frame group outputs serve as feedback caches for the current frame, avoiding cross-frame backpropagation through time. The original feed-forward and residual pathways are retained, while the small initial query and gate scales keep the model close to the pretrained base tracker. During inference, each selected Transformer group maintains a one-frame cache that is updated after every frame. When no valid cache is available, the original base-tracker computation is used. Because only the immediately preceding group states are stored, the cache memory remains constant with respect to video length.

Experiments Summary:

The paper evaluates FeedbackTrack on two widely used visual tracking benchmarks, LaSO…

Improvements to AI Systems:

1. Add recurrent cross-frame feedback to Transformer-based video models – Instead of purely feed-forward processing, cache intermediate group outputs from the previous frame and inject them into corresponding stages of the current frame. This converts stateless visual encoders into lightweight recurrent systems with constant memory (one-frame cache), improving temporal consistency without retraining the base model.

2. Implement dual-pathway modulation for temporal context – Use Query Feedback (low-rank, RMS-aligned token-wise bias on queries) to guide where attention retrieves evidence, and Gate Feedback (pooled context with tanh-bounded channel-group scaling) to regulate attention output magnitude. This separates spatial-precise and global-contextual temporal influence, improving robustness to appearance change and occlusion.

3. Enable stage-aligned temporal reuse – Partition the backbone into groups of blocks and return each previous group’s output to the first block of the same group in the current frame. This preserves hierarchical feature correspondence, allowing shallow features to reuse fine details and deep features to reuse semantic context, improving long-term tracking under distractors.

4. Add magnitude-controlled feedback injection – Use per-sample, per-head, per-token RMS alignment to scale historical bias relative to current queries, and initialize learnable scales to small values (0.01). This prevents destabilizing pretrained representations, enabling stable fine-tuning and fast convergence with minimal parameter overhead (<1%).

5. Provide a plug-and-play temporal module for existing trackers – The feedback mechanism can be inserted into any Transformer-based tracker (e.g., SPMTrack, ARTrackV2) without altering the original pipeline, prediction heads, or training objectives. This allows upgrading existing models with recurrent temporal reasoning while retaining their strengths.

6. Enable efficient online adaptation – Since only one-frame caches are stored and cross-frame backpropagation is avoided (detached states), the system supports streaming inference with constant memory and minimal throughput loss (5–7% for large models, <1% for autoregressive trackers), making it suitable for real-time video applications.

---

What the Improved AI System Can Do:

• Track arbitrary objects across long videos with higher accuracy – Achieves +2.3–4.1 AO on GOT-10k and +1.1–1.8 AUC on LaSOT across five backbone configurations, improving robustness to occlusion, fast motion, and similar distractors.

• Maintain target identity under appearance changes – Recurrent query modulation and gate scaling adapt feature extraction based on historical visual states, reducing drift and improving success rate at strict overlap thresholds (e.g., +5.8 SR0.75 on ARTrackV2-B).

• Generalize to other video tasks – The same group-level cross-frame feedback can be applied to video object segmentation, action recognition, video understanding, and video generation, providing a general mechanism for injecting temporal recurrence into pretrained video Transformers.

• Operate in real-time with minimal overhead – Adds <1% parameters and preserves 93–99% inference throughput, enabling deployment on edge devices or in interactive applications where latency matters.

• Learn task-specific temporal hierarchies – The depth-dependent feedback allocation (weak in shallow, strong in intermediate/deep layers) emerges automatically, allowing the system to allocate recurrent computation where it matters most for the task, without manual tuning.