DeepSTEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "DeepSTEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans".
Jane: Here's the gist of the paper. STEP introduces a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans We are inspired by the fact that the human brain exploits spatiotemporal continuity and performs concurrent localization and pose estimation despite the specialization of brain areas for form and motion processing Traditional discriminative models typically require predefined target states for determining model weights, a challenge we address through Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) Modules Our method starts with a known target state in the initial frame of a given video sequence and then seamlessly tracks the target and estimates keypoints of anatomical importance as output for subsequent frames Unlike prevalent top-down pose estimation methods, our approach doesn’t rely on per-frame target detections due to its tracking capability This facilitates a significant advancement in inference efficiency and potential applications Pruning redundant reasoning tokens The paper addresses the limitations of traditional top-down pose estimation techniques which require precise bounding box data, sourced either from automated detectors or manually annotated ground truths, to appropriately crop, center, and scale the images of the targets To handle non-standard aspect ratios without relying on per-frame object detections used in existing top-down methods, the proposed method concurrently estimates weights for three tasks: localization, tracking, and pose estimation This integration of tracking eliminates the need for per-frame object detections used in existing top-down methods How the training signal is built The network adopts an end-to-end learning approach to predict the weights of the target model purely from data, adhering to the discriminative model prediction approach introduced by Mayer et al.
Tom: (2022) and expanding it to learn target model weights specifically for the keypoint regression task The training features for the Transformer module are created by combining extracted deep features X with target state information, as denoted in Equation 3 Specifically, the training features are defined as fi = X+Ψb(Bom)+ϕloc·Bgm+ϕkp·Kt gm The GMSP Module is trained exclusively on training frames to predict a soft map of all possible key points within these frames, ensuring it is not exposed to test-frame data during the encoding determination process Where the gains come from The core innovation lies in the Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) modules which work together to accurately localize key points corresponding to the target state, ensuring both efficiency and precision in pose keypoint estimation The GMSP Module learns a soft map relevant to anatomically crucial keypoints of the target by training exclusively on training frames, using Πkp as supervision to predict a soft map of all possible key points within these frames The Offset Map Regression Adapter Module collaborates with the GMSP Module for the test frame through a two-tower-based architecture, allowing it to reason the final keypoints specific to the object of interest Inference and Robustness The inference procedure requires initialization with a bounding box around the target object for the initial frame s0 to establish target state information The network's memory is updated during inference when at least 50% of the total keypoints and the target localization both are predicted with a confidence value greater than τm = 0.6 This iterative process continues until the end of the sequence, enhancing performance speed because training frames and their corresponding target encodings remain unaltered The proposed approach achieves state-of-the-art performance in both tracking and pose estimation tasks while simultaneously performing the tracking task also The framework demonstrates satisfactory performance, effectively quantifying behavioral changes in a setting with high variability in animal occupancy, scale, and pose Limitations Our proposed approach requires initialization through a bounding box for the first-most frame in a given video to infer target state encodings Errors originating from one task can negatively impact the accuracy of the other task, which is evident from the metrics evaluated between ”Ours” and ”Ours” in Table 1 The network tends to estimate the pose keypoints within the predicted bounding box or, conversely, predicts a bounding box that encompasses the predicted keypoints The framework requires initialization through a bounding box for the first-most frame in a given video to infer target state encodings This work proposes a novel framework for Simultaneous Tracking and Estimation of Pose across diverse animal species and humans Our method estimates the pose of the object of interest while simultaneously tracking it, utilizing a Transformer-based discriminative target model prediction approach The key contributions of our framework include the Gaussian Soft Map Prediction module and the Offset Map Regression Adapter module, which effectively removes the dependency on keypoint initialization to estimate target state encodings We show that the network equipped with the proposed modules performs better than the version relying on keypoint initialization The proposed STEP framework achieves state-of-the-art performance in both tracking and pose estimation tasks Lastly, we showcase the robustness of our approach in challenging video scenarios with high animal occupancy, underscoring its effectiveness in both tracking and pose estimation Acknowledgments We thank Prof.
Jane: Kazunori Yamada from Osaka University and Hisami-san from the Awaji Island Monkey Center for granting us access to the center’s video feeds for our analysis We also thank Dr.
Tom: Mami Matsuda for facilitating communication with the Awaji Monkey Center Additionally, we acknowledge the support of the Jibaben Patel Chair in AI for this work Data Availability All experiments are performed on publically available datasets: APT36K Yang et al.
Paper summary: Tom: We're diving into the DeepSTEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans paper now. We've covered the main idea of using a Transformer-based discriminative model prediction to do both tracking and pose estimation at the same time across different animals and humans.
Jane: That was a good start, Tom. So, what's the thesis again? What exactly is the core claim this paper is making about how this works?
Lu: The thesis is that they are tackling traditional top-down pose estimation techniques that need precise bounding box data for cropping and scaling by proposing a transformer-based method to concurrently estimate weights for localization, tracking, and pose estimation.
Meng: So, the key claim there is that integrating tracking eliminates the need for those per-frame object detections used in existing top-down methods.
Tom: That's right. They introduce GMSP and OMRA modules specifically to handle non-standard aspect ratios without needing those per-frame detections.
Jane: But how do they make the weights learn all three tasks—localization, tracking, and pose estimation—purely from data instead of predefining them?
Lu: They use an end-to-end learning approach following the discriminative model prediction method introduced by Mayer et al. that they expanded to learn target model weights for the keypoint regression task.
Meng: What are those training features they are feeding into the Transformer module? How do they get that data ready?
Tom: They create training features by combining extracted deep features X with target state information, which is defined by equation three as f i equals X plus psi b of Bom plus phi loc times Bg m plus phi kp times Kt gm.
Jane: So the network gets this feature set, and then GMSP is trained exclusively on training frames to predict a soft map of all possible key points within those frames.
Lu: That's because it’s trained only on training frames, and it isn't exposed to test-frame data during the encoding determination process.
Meng: And what about the Offset Map Regression Adapter module? What does its role look like in this overall system?
Tom: The Offset Map Regression Adapter module works with GMSP for the test frame using a two-tower-based architecture, which lets it reason about the final keypoints specific to the object of interest.
Jane: So, if I had to sum up what we just covered, they are proposing a system that predicts weights for tracking and pose estimation while simultaneously handling localization and tracking tasks.
Lu: That’s right. And their main contribution is the GMSP and OMRA modules which work together to accurately localize the keypoints corresponding to the target state for efficiency and precision in pose keypoint estimation.
Meng: So, what's the practical application of this framework beyond just animal tracking? Where does it get tested?
Tom: They showcase their effectiveness through a case study on Awaji Monkey Center video streams, demonstrating satisfactory performance in environments with high variability in animal occupancy and scale.
Conclusion: Jane: So we've gone through the DeepSTEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans paper. What’s the final word on this framework? What does it all boil down to for us?
Tom: The title is DeepSTEP, which stands for Simultaneous Tracking and Estimation of Pose for Animals and Humans, focusing on getting both tasks done concurrently.
Lu: It essentially proposes a method that estimates the pose of the object while simultaneously tracking it by utilizing a Transformer-based discriminative model prediction approach.
Meng: The main contribution is the Gaussian Soft Map Prediction module and the Offset Map Regression Adapter module which effectively remove the dependency on keypoint initialization to estimate target state encodings.
Jane: So, in simple terms, this means they’ve shown that their network performs better than versions that required keypoint initialization.
Tom: They claim this framework achieves state-of-the-art performance for both tracking and pose estimation tasks at the same time.
Lu: The robustness of their approach is also highlighted in challenging video scenarios with high animal occupancy, which shows it works well in those situations too.
Meng: From a deployment view, the limitation they state is that you still need to initialize it with a bounding box for the first frame to get those target state encodings.
Jane: So while it’s powerful for its simultaneous tracking and pose estimation capabilities, that initial bounding box setup is something engineers have to account for.
Tom: Right. It's a system that performs well, but it requires that initial setup to get started on the right track with the target state encoding.
Department of Electrical Engineering, Indian Institute of Technology Gandhinagar · National Institute of Mental Health, National Institutes of Health
cs.CV
Submitted: 2025-03-17
Updated: 2026-11-04
Code: https://github.com/pandorgan/APT-36K
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: STEP introduces a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans We are inspired…
Key concepts
- Gaussian Map Soft Prediction (GMSP)
- This module is trained only on training frames to predict a 'soft map' of all possible key points. It learns which anatomical regions are important for the target's pose, helping the system accurately localize key points without needing prior knowledge of every possible point.
- Offset Map Regression Adapter (OMRA)
- Working with GMSP, this module uses a two-tower architecture to refine the final keypoints for the test frame. It collaborates with GMSP to ensure precise localization of the target's specific pose keypoints during inference.
- Discriminative Model Prediction
- The network learns the weights of a target model directly from data, inspired by how human brains handle spatiotemporal continuity. This allows the system to predict necessary model parameters for tracking and pose estimation concurrently without relying on predefined target states.
Terminology
Summary
STEP introduces a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans We are inspired by the fact that the human brain exploits spatiotemporal continuity and performs concurrent localization and pose estimation despite the specialization of brain areas for form and motion processing Traditional discriminative models typically require predefined target states for determining model weights, a challenge we address through Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) Modules Our method starts with a known target state in the initial frame of a given video sequence and then seamlessly tracks the target and estimates keypoints of anatomical importance as output for subsequent frames Unlike prevalent top-down pose estimation methods, our approach doesn’t rely on per-frame target detections due to its tracking capability This facilitates a significant advancement in inference efficiency and potential applications
Pruning redundant reasoning tokens
The paper addresses the limitations of traditional top-down pose estimation techniques which require precise bounding box data, sourced either from automated detectors or manually annotated ground truths, to appropriately crop, center, and scale the images of the targets To handle non-standard aspect ratios without relying on per-frame object detections used in existing top-down methods, the proposed method concurrently estimates weights for three tasks: localization, tracking, and pose estimation This integration of tracking eliminates the need for per-frame object detections used in existing top-down methods
How the training signal is built
The network adopts an end-to-end learning approach to predict the weights of the target model purely from data, adhering to the discriminative model prediction approach introduced by Mayer et al. (2022) and expanding it to learn target model weights specifically for the keypoint regression task The training features for the Transformer module are created by combining extracted deep features X with target state information, as denoted in Equation 3 Specifically, the training features are defined as fi = X+Ψb(Bom)+ϕloc·Bgm+ϕkp·Kt gm The GMSP Module is trained exclusively on training frames to predict a soft map of all possible key points within these frames, ensuring it is not exposed to test-frame data during the encoding determination process
Where the gains come from
The core innovation lies in the Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) modules which work together to accurately localize key points corresponding to the target state, ensuring both efficiency and precision in pose keypoint estimation The GMSP Module learns a soft map relevant to anatomically crucial keypoints of the target by training exclusively on training frames, using Πkp as supervision to predict a soft map of all possible key points within these frames The Offset Map Regression Adapter Module collaborates with the GMSP Module for the test frame through a two-tower-based architecture, allowing it to reason the final keypoints specific to the object of interest
Inference and Robustness
The inference procedure requires initialization with a bounding box around the target object for the initial frame s0 to establish target state information The network's memory is updated during inference when at least 50% of the total keypoints and the target localization both are predicted with a confidence value greater than τm = 0.6 This iterative process continues until the end of the sequence, enhancing performance speed because training frames and their corresponding target encodings remain unaltered The proposed approach achieves state-of-the-art performance in both tracking and pose estimation tasks while simultaneously performing the tracking task also The framework demonstrates satisfactory performance, effectively quantifying behavioral changes in a setting with high variability in animal occupancy, scale, and pose
Limitations
Our proposed approach requires initialization through a bounding box for the first-most frame in a given video to infer target state encodings Errors originating from one task can negatively impact the accuracy of the other task, which is evident from the metrics evaluated between ”Ours” and ”Ours” in Table 1 The network tends to estimate the pose keypoints within the predicted bounding box or, conversely, predicts a bounding box that encompasses the predicted keypoints The framework requires initialization through a bounding box for the first-most frame in a given video to infer target state encodings
Conclusion
This work proposes a novel framework for Simultaneous Tracking and Estimation of Pose across diverse animal species and humans Our method estimates the pose of the object of interest while simultaneously tracking it, utilizing a Transformer-based discriminative target model prediction approach The key contributions of our framework include the Gaussian Soft Map Prediction module and the Offset Map Regression Adapter module, which effectively removes the dependency on keypoint initialization to estimate target state encodings We show that the network equipped with the proposed modules performs better than the version relying on keypoint initialization The proposed STEP framework achieves state-of-the-art performance in both tracking and pose estimation tasks Lastly, we showcase the robustness of our approach in challenging video scenarios with high animal occupancy, underscoring its effectiveness in both tracking and pose estimation
Acknowledgments
We thank Prof. Kazunori Yamada from Osaka University and Hisami-san from the Awaji Island Monkey Center for granting us access to the center’s video feeds for our analysis We also thank Dr. p. 2 This
update integrates previous frame outputs to
supervise and estimate subsequent outputs effectively. We showcase the effectiveness of our
approach through a case study on Awaji Monkey
Center (AMC) video streams.Where the paper says it
Data Availability
All experiments are performed on publically available datasets: APT36K Yang et al. (2022) (https://github. p. 1 The top-down approaches
Xu et al. (2022) Sun et al. We evaluate the models on the APT36K
Yang et al. (2022), APT10K Yu et al. (2021a) Li et al. (2021b)
Li et al. (2022), APT10K Yu et al. (2021b), CrowdPose Li et al. (2022) Sun et al. (2019) first detect
the object of interest employing object detectors
and subsequently conduct pose estimation for each
detection. (2019), and DeepLabCut Benchmark datasets Lauer
et al. (2022), which include the TriMouse, Fish, and Marmoset datasets. (2019), and DeepLabCut Benchmark datasets Lauer
et al. (2022), which include the TriMouse, Fish, and Marmoset datasets. Notably, markerless pose estimation is gaining popularity in neuroscience and the study of animal
behaviors Coria-Avila et al. Expanding on
the widely used CNN-based architecture of the
DeepLabCut model Mathis et al. This facilitates a
significant advancement in inference efficiency and potential applications. We train and validate our
approach on datasets encompassing diverse species. 2 Network Architecture
Our proposed network adopts an end-to-end learning approach to predict the weights of the target
model purely from data. The overview of our
methodology is depicted in Figure 1.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Simultaneous Tracking and Pose Estimation: The improved system can simultaneously track objects across video sequences while estimating their pose, overcoming limitations of existing top-down methods that require per-frame object detections.
-
Robustness to Target State Dependency: By introducing the Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) modules, the system removes the necessity of predefined target states for model weights, streamlining the process and improving efficiency.
-
Enhanced Pose Estimation Precision: The framework can achieve state-of-the-art performance in pose estimation compared to existing methods, as shown by superior results on datasets like APT36K across multiple species.
-
Tracking Robustness through Memory Update: Inference speed and tracking accuracy are enhanced by a confidence based memory update of the tracker, which integrates previous frame outputs to supervise and estimate subsequent outputs effectively.
-
Initialization Independence: The system can be initialized with only an initial bounding box for the first frame, as it relies on
initialization with a bounding box around the target object for the initial frame s0 to establish target state information,
minimizing dependency on per-frame ground truth detections during tracking.
Abstract
Maintaining a consistent identity across video frames while estimating pose is crucial for downstream tasks such as behavior analysis and action recognition, yet existing pipelines largely decouple tracking and pose estimation. Top-down methods rely on per-frame detections followed by identity association, leading to increased computational cost and error propagation, whereas bottom-up approaches often struggle to preserve identity reliably in multi-instance settings. To address these limitations, we introduce DeepSTEP (STEP), a Transformer-based framework for joint tracking and pose estimation across diverse animal species and humans. STEP unifies localization, identity maintenance, and keypoint estimation into a single model, eliminating the need for per-frame detections or post hoc matching. Our approach leverages discriminative model weight prediction to propagate target identity across frames while predicting anatomically meaningful keypoints. Discriminative model prediction typically requires predefined target states. In contrast, STEP assumes only an initial bounding box in the first frame, removing the need for explicit keypoint initialization. This is enabled by two components: Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA), which allow direct keypoint prediction and simplify inference. Starting from the initial bounding box, STEP tracks the subject and estimates its pose in subsequent frames. We train and evaluate STEP on datasets spanning multiple species, demonstrating strong performance and greater efficiency than existing methods. By jointly modeling tracking and pose, STEP provides an effective and scalable solution.
Sources
- Pytorch-Wildlife: A Collaborative Deep Learning Framework for Conservation
- RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs