Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras
summary
The gist
Markerless multiview motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures Existing calibration-free…
In short
Kineo is a calibration-free pipeline for markerless motion capture from consumer RGB videos. It uses 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras and reconstruct 3D keypoints. The system achieves metric scale by using human body priors (SMPL) or monocular depth estimators, significantly reducing errors compared to prior methods.
Key concepts
- Temporal Synchronization
- Since videos are often uncalibrated and unsynchronized, this method aligns them using audio tracks. It calculates a temporal lag ($\gamma_k$) for each camera by cross-correlating their audio signals, using features like MFCCs to find the best match between streams.
- Automatic Camera Calibration
- The system models the cameras as a graph where nodes are cameras and edges represent calibration quality. It uses a minimum spanning tree selection based on pairwise metrics (like Sampson distance) to recover an optimal global camera configuration from 2D keypoint correspondences.
- Metric Scale Recovery Strategies
- To get real-world measurements, Kineo employs two methods: first, optimizing the SMPL model to find a consistent scale factor based on bone lengths. Second, it uses a monocular metric depth estimator to recover scale even in scenes without humans by calculating the ratio between arbitrary and metric units.
Terminology used across episodes
This episode discusses
- Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras · Paper Radio
- Multi-view Pose Fusion for Occlusion-Aware 3D Human Pose Estimation
- Procedural Humans for Computer Vision
- Towards Accurate Markerless Human Shape and Pose Estimation over Time
- RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose
- SPEC: Seeing People in the Wild with an Estimated Camera
- Reconstructing People, Places, and Cameras
- Wide-Baseline Multi-Camera Calibration using Person Re-Identification
The paper
Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras · Read on arXiv
École Centrale de Lyon · CNRS · INSA Lyon · Université Claude Bernard Lyon 1 · Université Lumière Lyon 2 · LIRIS · UMR5205 · ENISE
Markerless multi-view motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures. Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy. We present Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras. Kineo leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras, including Brown-Conrady distortion coefficients, and reconstruct 3D keypoints and dense scene pointmaps at metric scale. A confidence-aware spatio-temporal keypoint subsampling strategy, combined with graph-based global optimization, enables robust calibration at a fixed computational cost independent of sequence length. We further introduce a pairwise reprojection consistency score, a heuristic measure of the agreement between views, which we use to weight bone lengths in metric scale recovery. Evaluations on EgoHumans and Human3.6M demonstrate substantial improvements over prior calibration-free methods. Compared to previous state-of-the-art approaches, Kineo reduces camera translation error by approximately 83-85%, camera angular error by 86-92%, and world mean-per-joint position error (W-MPJPE) by 83-91%. Kineo is also efficient in real-world scenarios, processing multi-view sequences faster than their duration in a specific configuration (e.g., 36 min to process 1 h 23 min of footage). The full pipeline and evaluation code are openly released to promote reproducibility and practical adoption at: https://liris-xr.github.io/kineo/
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras".
Jane: Here's the gist of the paper. Markerless multiview motion capture is often constrained by the need for precise camera calibration, limiting accessibility for non-experts and in-the-wild captures Existing calibration-free approaches mitigate this requirement but suffer from high computational cost and reduced reconstruction accuracy Kineo, a fully automatic, calibration-free pipeline for markerless motion capture from videos captured by unsynchronized, uncalibrated, consumer-grade RGB cameras leverages 2D keypoints from off-the-shelf detectors to simultaneously calibrate cameras and reconstruct 3D keypoints and dense scene point maps at metric scale Pipeline Overview The pipeline is designed for modularity and efficiency, enabling scalable processing of long video sequences Given videos from unsynchronized and uncalibrated cameras, the pipeline first estimates temporal offsets between video streams via audio-based synchronization Next, automatic camera calibration is performed using 2D keypoints detected by a replaceable off-the-shelf detector A graphbased formulation is used to recover an optimal global camera configuration through minimum spanning tree selection Using the recovered cameras, 3D keypoints and scene point maps are reconstructed in arbitrary units To obtain metric-scale reconstructions, Kineo proposes two strategies: leveraging a human body prior using the SMPL model or employing a monocular metric depth estimator for subject-agnostic scaling Calibration and Synchronization Mechanisms Temporal synchronization is achieved by adopting an audio-based approach that computes the temporal lag gammak for each camera by aligning its audio track with that of a reference camera using cross-correlation This method uses compact audio descriptors like Mel-Frequency Cepstral Coefficients (MFCCs) to improve efficiency during cross-correlation Automatic camera calibration is performed by modeling the multi-view system as a graph, where nodes are cameras and edges are weighted by a pairwise calibration quality metric, such as the average Sampson distance The keypoint sampling strategy selects high-quality correspondences based on their 2D detector confidences to reduce noise and outliers while ensuring computational efficiency 3D Reconstruction and Confidence Scoring The pipeline reconstructs 3D keypoints by triangulating the 2D keypoints using a weighted Direct Linear Transform (DLT) method, where the triangulation is weighted by the 2D confidence score of each individual keypoint A novel confidence score is introduced as the pairwise reprojection confidence score, which quantifies the reliability of 3D estimations for downstream tasks This score is defined as the weighted mean reprojection agreement across all views, where each view contribution is modulated by its 2D keypoint confidence Metric Scale Recovery Strategies To obtain metric-scale reconstructions, two strategies are proposed: first, leveraging a human body prior using the SMPL model to estimate the global scale factor by optimizing the scale factor while enforcing bone length consistency Second, employing a monocular metric depth estimator to recover scale even in scenes without humans by computing the ratio between metric units and arbitrary units Performance and System Efficiency Kineo demonstrates substantial improvements over prior calibration-free methods, reducing camera translation error by approximately 83–85% and world mean-per-joint error (WMPJPE) by 83–91% on the EgoHumans and Human3.6M datasets The method is efficient in real-world scenarios, processing multi-view sequences faster than their duration in specific configuration, such as processing six views of five-minute videos requiring only 11 minutes on an NVIDIA RTX 3080 Ti The pipeline is modular and supports offline and online modes for rapid deployment in both unconstrained real-world capture and real-time applications Future Research Directions Promising research directions include integrating the proposed 3D confidence scores into temporal filtering to enhance robustness and temporal coherence Furthermore, camera pose estimation could be refined by incorporating a photometric loss, following Choi et al.
6: , who demonstrated that dynamic NeRF optimization can be used to jointly refine camera poses and reconstruct volumetric representations of the scenes The modular design allows for seamless integration of improved 2D detectors without altering the architecture, instantly enhancing calibration and reconstruction accuracy Acknowledgments This work was supported by the Auvergne-Rhône-Alpes region as part of the PROMESS project This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD010614830 made by GENCI We also express our gratitude to the Guédelon Castle for kindly welcoming us and permitting the captures that were essential to this study References
1: Ijaz Akhter, Tomas Simon, Sohaib Khan, Iain Matthews, and Yaser Sheikh.
Tom: Bilinear Spatiotemporal Basis Models
2: Federica Arrigoni, Andrea Fusiello, and Beatrice Rossi.
Paper summary: Tom: So we've touched on what Kineo claims it can do—it’s calibration-free and metric scale reconstruction using standard 2D keypoints <ref:2510.24464#pg1>.
Jane: The next thing I want to know is how they manage the tricky parts, like getting those cameras lined up when they weren't calibrated beforehand.
Lu: They start by estimating temporal offsets between the video streams using audio-based synchronization, which finds the time lags between cameras by aligning their audio tracks using cross-correlation on compact audio descriptors like Mel-Frequency Cepstral Coefficients.
Meng: That sounds like a practical way to handle unsynchronized footage without needing dedicated hardware for every single capture.
Tom: And then, they use a graph-based formulation to recover the optimal global camera configuration by selecting edges based on a pairwise calibration quality metric, specifically using the average Sampson distance as that weight.
Jane: So they build this whole system as a graph where nodes are cameras and edges represent how well those cameras see each other.
Lu: They also have this clever keypoint sampling strategy that selects correspondences based on the confidence of the 2D detector to cut down on noise and outliers while keeping the calibration cost fixed, no matter how long the video sequence is <ref:2510.24464#pg1>.
Tom: That's interesting because fixing that computational cost regardless of sequence duration is a big win for real-world deployment.
Conclusion: Jane: So putting it all together, Kineo is this fully automatic pipeline that takes unsynchronized, uncalibrated consumer videos and recovers three dee keypoints and scene point maps at metric scale by using standard 2D keypoints for both camera calibration and reconstruction <ref:2510.24464#pg1>.
Tom: The authors are trying to move away from the high computational cost of older calibration-free methods while still achieving reasonable accuracy.
Lu: What really stands out is their introduction of a novel pairwise reprojection confidence score, which quantifies the reliability of those three dee estimations for any downstream task you might want to do with that data <ref:2510.24464#pg1>.
Meng: The implication for production is that if you can get a confidence score, you know exactly when your three dee reconstruction is probably trustworthy enough to use in an application <ref:2510.24464#pg1>.
Jane: It means they're not just giving you a point cloud; they're giving you a measure of how good that point cloud actually is for what your application needs.
Tom: And the authors explicitly mention that this confidence score lets people leverage the precision of 2D keypoints without needing complex neural network architectures for the core process, sticking to traditional Structure-from-Motion principles <ref:2510.24464#pg1>.
More episodes
- 2603.28113-The multiplicative Lipschitz bound as an inductive bias
- 2603.28595-Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound