Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras".
Jane: Here's the gist of the paper. Markerless multiview motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras and reconstruct 3D keypoints and dense scene point maps at metric scale Pipeline Overview The pipeline is designed for modularity and efficiency, enabling scalable processing of long video sequences Given videos from unsynchronized and uncalibrated cameras, the pipeline first estimates temporal offsets between video streams via audio-based synchronization Next, automatic camera calibration is performed using 2D keypoints detected by a replaceable off-the-shelf detector A graphbased formulation is used to recover an optimal global camera configuration through minimum spanning tree selection Using the recovered cameras, 3D keypoints and scene point maps are reconstructed in arbitrary units To obtain metric-scale reconstructions, Kineo proposes two strategies: leveraging a human body prior using the SMPL model or employing a monocular metric depth estimator for subject-agnostic scaling Calibration and Synchronization Mechanisms Temporal synchronization is achieved by adopting an audio-based approach that computes the temporal lag gammak for each camera by aligning its audio track with that of a reference camera using cross-correlation This method uses compact audio descriptors like Mel-Frequency Cepstral Coefficients (MFCCs) to improve efficiency during cross-correlation Automatic camera calibration is performed by modeling the multi-view system as a graph, where nodes are cameras and edges are weighted by a pairwise calibration quality metric, such as the average Sampson distance The keypoint sampling strategy selects high-quality correspondences based on their 2D detector confidences to reduce noise and outliers while ensuring computational efficiency 3D Reconstruction and Confidence Scoring The pipeline reconstructs 3D keypoints by triangulating the 2D keypoints using a weighted Direct Linear Transform (DLT) method, where the triangulation is weighted by the 2D confidence score of each individual keypoint A novel confidence score is introduced as the pairwise reprojection confidence score, which quantifies the reliability of 3D estimations for downstream tasks This score is defined as the weighted mean reprojection agreement across all views, where each view contribution is modulated by its 2D keypoint confidence Metric Scale Recovery Strategies To obtain metric-scale reconstructions, two strategies are proposed: first, leveraging a human body prior using the SMPL model to estimate the global scale factor by optimizing the scale factor while enforcing bone length consistency Second, employing a monocular metric depth estimator to recover scale even in scenes without humans by computing the ratio between metric units and arbitrary units Performance and System Efficiency Kineo demonstrates substantial improvements over prior calibration-free methods, reducing camera translation error by approximately 83–85% and world mean-per-joint error (WMPJPE) by 83–91% on the EgoHumans and Human3.6M datasets The method is efficient in real-world scenarios, processing multi-view sequences faster than their duration in specific configuration, such as processing six views of five-minute videos requiring only 11 minutes on an NVIDIA RTX 3080 Ti The pipeline is modular and supports offline and online modes for rapid deployment in both unconstrained real-world capture and real-time applications Future Research Directions Promising research directions include integrating the proposed 3D confidence scores into temporal filtering to enhance robustness and temporal coherence Furthermore, camera pose estimation could be refined by incorporating a photometric loss, following Choi et al.
6: , who demonstrated that dynamic NeRF optimization can be used to jointly refine camera poses and reconstruct volumetric representations of the scenes The modular design allows for seamless integration of improved 2D detectors without altering the architecture, instantly enhancing calibration and reconstruction accuracy Acknowledgments This work was supported by the Auvergne-Rhône-Alpes region as part of the PROMESS project This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD010614830 made by GENCI We also express our gratitude to the Guédelon Castle for kindly welcoming us and permitting the captures that were essential to this study References
1: Ijaz Akhter, Tomas Simon, Sohaib Khan, Iain Matthews, and Yaser Sheikh.
Tom: Bilinear Spatiotemporal Basis Models
2: Federica Arrigoni, Andrea Fusiello, and Beatrice Rossi.
Paper summary: Tom: So we've touched on what Kineo claims it can do—it’s calibration-free and metric scale reconstruction using standard 2D keypoints <ref:2510.24464#pg1>.
Jane: The next thing I want to know is how they manage the tricky parts, like getting those cameras lined up when they weren't calibrated beforehand.
Lu: They start by estimating temporal offsets between the video streams using audio-based synchronization, which finds the time lags between cameras by aligning their audio tracks using cross-correlation on compact audio descriptors like Mel-Frequency Cepstral Coefficients.
Meng: That sounds like a practical way to handle unsynchronized footage without needing dedicated hardware for every single capture.
Tom: And then, they use a graph-based formulation to recover the optimal global camera configuration by selecting edges based on a pairwise calibration quality metric, specifically using the average Sampson distance as that weight.
Jane: So they build this whole system as a graph where nodes are cameras and edges represent how well those cameras see each other.
Lu: They also have this clever keypoint sampling strategy that selects correspondences based on the confidence of the 2D detector to cut down on noise and outliers while keeping the calibration cost fixed, no matter how long the video sequence is <ref:2510.24464#pg1>.
Tom: That's interesting because fixing that computational cost regardless of sequence duration is a big win for real-world deployment.
Conclusion: Jane: So putting it all together, Kineo is this fully automatic pipeline that takes unsynchronized, uncalibrated consumer videos and recovers three dee keypoints and scene point maps at metric scale by using standard 2D keypoints for both camera calibration and reconstruction <ref:2510.24464#pg1>.
Tom: The authors are trying to move away from the high computational cost of older calibration-free methods while still achieving reasonable accuracy.
Lu: What really stands out is their introduction of a novel pairwise reprojection confidence score, which quantifies the reliability of those three dee estimations for any downstream task you might want to do with that data <ref:2510.24464#pg1>.
Meng: The implication for production is that if you can get a confidence score, you know exactly when your three dee reconstruction is probably trustworthy enough to use in an application <ref:2510.24464#pg1>.
Jane: It means they're not just giving you a point cloud; they're giving you a measure of how good that point cloud actually is for what your application needs.
Tom: And the authors explicitly mention that this confidence score lets people leverage the precision of 2D keypoints without needing complex neural network architectures for the core process, sticking to traditional Structure-from-Motion principles <ref:2510.24464#pg1>.
École Centrale de Lyon · CNRS · INSA Lyon · Université Claude Bernard Lyon 1 · Université Lumière Lyon 2 · LIRIS · UMR5205 · ENISE
cs.CV
Submitted: 2025-10-28
Updated: 2026-11-05
Code: https://github.com/cjaverliat/hsfm
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Importance score: 92/100
The gist: Markerless multiview motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures Existing calibration-free…
Key concepts
- Temporal Synchronization
- Since videos are often uncalibrated and unsynchronized, this method aligns them using audio tracks. It calculates a temporal lag ($\gamma_k$) for each camera by cross-correlating their audio signals, using features like MFCCs to find the best match between streams.
- Automatic Camera Calibration
- The system models the cameras as a graph where nodes are cameras and edges represent calibration quality. It uses a minimum spanning tree selection based on pairwise metrics (like Sampson distance) to recover an optimal global camera configuration from 2D keypoint correspondences.
- Metric Scale Recovery Strategies
- To get real-world measurements, Kineo employs two methods: first, optimizing the SMPL model to find a consistent scale factor based on bone lengths. Second, it uses a monocular metric depth estimator to recover scale even in scenes without humans by calculating the ratio between arbitrary and metric units.
Terminology
Summary
Markerless multiview motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras and reconstruct 3D keypoints and dense scene point maps at metric scale
Pipeline Overview
The pipeline is designed for modularity and efficiency, enabling scalable processing of long video sequences Given videos from unsynchronized and uncalibrated cameras, the pipeline first estimates temporal offsets between video streams via audio-based synchronization Next, automatic camera calibration is performed using 2D keypoints detected by a replaceable off-the-shelf detector A graphbased formulation is used to recover an optimal global camera configuration through minimum spanning tree selection Using the recovered cameras, 3D keypoints and scene point maps are reconstructed in arbitrary units To obtain metric-scale reconstructions, Kineo proposes two strategies: leveraging a human body prior using the SMPL model or employing a monocular metric depth estimator for subject-agnostic scaling
Calibration and Synchronization Mechanisms
Temporal synchronization is achieved by adopting an audio-based approach that computes the temporal lag gammak for each camera by aligning its audio track with that of a reference camera using cross-correlation This method uses compact audio descriptors like Mel-Frequency Cepstral Coefficients (MFCCs) to improve efficiency during cross-correlation Automatic camera calibration is performed by modeling the multi-view system as a graph, where nodes are cameras and edges are weighted by a pairwise calibration quality metric, such as the average Sampson distance The keypoint sampling strategy selects high-quality correspondences based on their 2D detector confidences to reduce noise and outliers while ensuring computational efficiency
3D Reconstruction and Confidence Scoring
The pipeline reconstructs 3D keypoints by triangulating the 2D keypoints using a weighted Direct Linear Transform (DLT) method, where the triangulation is weighted by the 2D confidence score of each individual keypoint A novel confidence score is introduced as the pairwise reprojection confidence score, which quantifies the reliability of 3D estimations for downstream tasks This score is defined as the weighted mean reprojection agreement across all views, where each view contribution is modulated by its 2D keypoint confidence
Metric Scale Recovery Strategies
To obtain metric-scale reconstructions, two strategies are proposed: first, leveraging a human body prior using the SMPL model to estimate the global scale factor by optimizing the scale factor while enforcing bone length consistency Second, employing a monocular metric depth estimator to recover scale even in scenes without humans by computing the ratio between metric units and arbitrary units
Performance and System Efficiency
Kineo demonstrates substantial improvements over prior calibration-free methods, reducing camera translation error by approximately 83–85% and world mean-per-joint error (WMPJPE) by 83–91% on the EgoHumans and Human3.6M datasets The method is efficient in real-world scenarios, processing multi-view sequences faster than their duration in specific configuration, such as processing six views of five-minute videos requiring only 11 minutes on an NVIDIA RTX 3080 Ti The pipeline is modular and supports offline and online modes for rapid deployment in both unconstrained real-world capture and real-time applications
Future Research Directions
Promising research directions include integrating the proposed 3D confidence scores into temporal filtering to enhance robustness and temporal coherence Furthermore, camera pose estimation could be refined by incorporating a photometric loss, following Choi et al. [6], who demonstrated that dynamic NeRF optimization can be used to jointly refine camera poses and reconstruct volumetric representations of the scenes The modular design allows for seamless integration of improved 2D detectors without altering the architecture, instantly enhancing calibration and reconstruction accuracy
Acknowledgments
This work was supported by the Auvergne-Rhône-Alpes region as part of the PROMESS project This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD010614830 made by GENCI We also express our gratitude to the Guédelon Castle for kindly welcoming us and permitting the captures that were essential to this study
References
[1] Ijaz Akhter, Tomas Simon, Sohaib Khan, Iain Matthews, and Yaser Sheikh. 2012. Bilinear Spatiotemporal Basis Models
[2] Federica Arrigoni, Andrea Fusiello, and Beatrice Rossi. 2015. On Computing the Translations Norm in the Epipolar Graph
[3] Fabien Baradel, Matthieu Armando, Salma Galaaoui, Romain Brégier, Philippe Weinzaepfel, Grégory Rogez, and Thomas Lucas. 2025. Multi-HMR: Multi-person Whole-Body Human Mesh Recovery in a Single Shot
[4] Oleksandr Bogdan, Viktor Eckstein, Francois Rameau, and Jean-Charles Bazin. 2018. DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras
[5] Laura Bragagnolo, Matteo Terreran, Davide Allegro, and Stefano Ghidoni. 2025. Multi-view Pose Fusion for Occlusion-Aware 3D Human Pose Estimation
[6] Changwoon Choi, Jeongjun Kim, Geonho Cha, Minkwan Kim, Dongyoon Wee, and Young Min Kim. 2025. Humans as a Calibration Pattern: Dynamic 3D Scene Reconstruction from Unsynchronized and Uncalibrated Videos
[7] Sai Kumar Dwivedi, Cordelia Schmid, Hongwei Yi, Michael J. Black, and Dimitrios Tzionas. 2024. POCO: 3D Pose and Shape Estimation with Confidence
[8] Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, Yao Feng, and Michael J. Black. 2024. TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation
[9] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. 2023. Humans in 4D: Reconstructing and Tracking Humans with Transformers
[10] Richard I. Hartley. 1997. Kruppa’s Equations Derived from the Fundamental Matrix
[11] Nils Hasler, Bodo Rosenhahn, Thorsten Thormahlen, Michael Wand, Juergen Gall, and Hans-Peter Seidel. 2009. Markerless Motion Capture with Unsynchronized Moving Cameras
[12] Charlie Hewitt, Tadas Baltrušaitis, Erroll Wood, Lohit Petikam, Louis Florentin, and Hanz Cuevas Velasquez. 2023. Procedural Humans for Computer Vision
[13] Charlie Hewitt, Fatemeh Saleh, Sadegh Aliakbarian, Lohit Petikam, Shideh Rezaeifar, Louis Florentin, Zafiirah Hosenie, Thomas J. Cashman, Julien Valentin, Darren Cosker, and Tadas Baltrušaitis. 2024. Look Ma, no markers: holistic performance capture without the hassle
[14] Charlie Hewitt, Fatemeh Saleh, Sadegh Aliakbarian, Lohit Petikam, Shideh Rezaeifar, Louis Florentin, Zafiirah Hosenie, Thomas J. Cashman, Julien Valentin, Darren Cosker, and Tadas Baltrusaitis. 2024. Look Ma No Markers: Holistic Performance Capture without the Hassle
[15] Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Javier Romero, Ijaz Akhter, and Michael J. p. 2 While computationally efficient, this approach is sensitive to noise. To improve robustness, Akhter et al.Where the paper says it
[16] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. 2014. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
[17] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. 2019. Learnable Triangulation of Human Pose
[18] Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. 2023. RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
[19] Rawal Khirodkar, Aayush Bansal, Lingni Ma, Richard Newcombe, Minh Vo, and Kris Kitani. 2023. EgoHumans: An Egocentric 3D Multi-Human Benchmark
[20] Muhammed Kocabas, Chun-Hao P Huang, Joachim Tesch, Lea Müller, Otmar Hilliges, and Michael J. Black. 2021. SPEC: Seeing People in the Wild with an Estimated Camera
[21] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. 2021. Probabilistic Modeling for Human Mesh Recovery
[23] Erwin Kruppa. 1913. To determine a 3D object from two perspective views with known inner orientation
[24] Richard Latimer, Jason Holloway, Ashok Veeraraghavan, and Ashutosh Sabharwal. 2015. SocialSync: Sub-Frame Synchronization in a Smartphone Camera Network
[25] Vincent Leroy, Yohann Cabon, and Jerome Revaud. 2024. Grounding Image Matching in 3D with MASt3R
[26] Amy Lin, Jason Y. Zhang, Deva Ramanan, and Shubham Tulsiani. 2024. RelPose++: Recovering 6D Poses from Sparse-view Observations
[27] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: a skinned multi-person linear model
[28] Fengjun Lv, Tao Zhao, and R. Nevatia. 2006. Camera Calibration from Video of a Walking Human
[30] NaturalPoint. 2025. Optitrack Motion Capture Systems
[31] D. Nister. 2004. An efficient solution to the five-point relative pose problem
[32] Priyanka Patel and Michael J. Black. 2025. CameraHMR: Aligning People with Perspective
[33] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman, Dimitrios Tzionas, and Michael J. Black. 2019. Expressive Body Capture: 3D Hands Face and Body From a Single Image
[34] Jens Puwein, Luca Ballan, Remo Ziegler, and Marc Pollefeys. 2015. Joint Camera Pose Estimation and 3D Human Pose Estimation in a Multi-camera Setup
[35] Haibo Qiu, Chunyu Wang, Jingdong Wang, Naiyan Wang, and Wenjun Zeng. 2019. Cross View Fusion for 3D Human Pose Estimation
[56] Yan Xu and Kris Kitani. 2022. Multi-View Multi-Person 3D Pose Estimation with Uncalibrated Camera Networks
[57] Yan Xu, Yu-Jhe Li, Xinshuo Weng, and Kris Kitani. 2021. Wide-Baseline Multi-Camera Calibration using Person Re-Identification
[58] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2022. ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
[59] Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. 2023. Effective wholebody pose estimation with two-stages distillation
[60] Hang Ye, Wentao Zhu, Chunyu Wang, Rujie Wu, and Yizhou Wang. 2022. Faster VoxelPose: Real-time 3D Human Pose Estimation by Orthographic Projection
[61] Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. 2023. Decoupling Human and Camera Motion from Videos in the Wild
[62] Zhe Zhang, Chunyu Wang, Weichao Qiu, Wenhu Qin, and Wenjun Zeng. 2021. AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild
Table 7
Table 7: Comparison of the W-MPJPE metric (lower is better) across Human3.6M sequences for Kineo (blue), HSfM (red), and ground truth (black)
Table 5
Table 5: Ablation study on the impact of different global scale estimation strategies (SMPL-based and metric depth-based using MoGe [54]) on reconstruction accuracy (W-MPJPE) and camera translation error (TE)
Table 3
Table 3: Ablation study for different combinations of ground truth (GT), estimated (est) parameters and omitted parameters, evaluated on Human3.6M and EgoHumans datasets
Table 4
Table 4: Impact of 2D detector on the PA-MPJPE (lower is better)
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improved automatic camera calibration by leveraging 2D keypoints to simultaneously calibrate cameras including Brown-Conrady distortion coefficients, ensuring robust reconstruction even with consumer-grade RGB cameras.
-
Enhanced 3D reconstruction fidelity by introducing a pairwise reprojection confidence score that
quantifies 3D reconstruction reliability for downstream tasks.
-
Increased computational efficiency and scalability by employing a
confidence-driven spatio-temporal keypoint sampling strategy
combined with graph-based global optimization, which ensures afixed computational cost independent of sequence length.
-
Achieved robust camera pose recovery through a
Graph-Based Global Calibration
model where camera networks are weighted by pairwise calibration quality and optimized via a minimum spanning tree (MST). -
Enabled accurate metric scale recovery through two strategies: one leveraging the SMPL model for human scenes and another employing a monocular metric depth estimator like MoGe for subject-agnostic scaling.
-
Improved temporal synchronization in unsynchronized, uncalibrated streams by adopting an audio-based approach using compact audio descriptors like Mel-Frequency Cepstral Coefficients (MFCCs) for efficient cross-correlation.
-
Enhanced 3D point confidence estimation by defining a
pairwise reprojection confidence score
that captures how well a 3D point aligns with the 2D detections across all cameras, allowing for view-consistent fusion. -
Optimized reconstruction accuracy during bundle adjustment by integrating estimated distortion coefficients into the optimization process following the Brown-Conrady model, which is
omitted in prior works.
-
Produced metric-scale 3D keypoint outputs that are reliable for downstream tasks by lifting 2D keypoints to 3D space using a weighted Direct Linear Transform (DLT) method, where
triangulation is weighted by the 2D confidence score of each individual keypoint.
Abstract
Markerless multi-view motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures. Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy. We present Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras. Kineo leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras, including Brown-Conrady distortion coefficients, and reconstruct 3D keypoints and dense scene pointmaps at metric scale. A confidence-aware spatio-temporal keypoint subsampling strategy, combined with graph-based global optimization, enables robust calibration at a fixed computational cost independent of sequence length. We further introduce a pairwise reprojection consistency score, a heuristic measure of the agreement between views, which we use to weight bone lengths in metric scale recovery. Evaluations on EgoHumans and Human3.6M demonstrate substantial improvements over prior calibration-free methods. Compared to previous state-of-the-art approaches, Kineo reduces camera translation error by approximately 83-85%, camera angular error by 86-92%, and world mean-per-joint position error (W-MPJPE) by 83-91%. Kineo is also efficient in real-world scenarios, processing multi-view sequences faster than their duration in a specific configuration (e.g., 36 min to process 1 h 23 min of footage). The full pipeline and evaluation code are openly released to promote reproducibility and practical adoption at: https://liris-xr.github.io/kineo/
Sources
- Multi-view Pose Fusion for Occlusion-Aware 3D Human Pose Estimation
- Procedural Humans for Computer Vision
- Towards Accurate Markerless Human Shape and Pose Estimation over Time
- RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
- SPEC: Seeing People in the Wild with an Estimated Camera
- Reconstructing People, Places, and Cameras
- Wide-Baseline Multi-Camera Calibration using Person Re-Identification
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs