COMPASS: Fusion-Matched Supervision for Missing-Modality Human Sensing
cs.CV
Submitted: 2026-04-02
Updated: 2026-09-24
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal human activity recognition (HAR) and human pose estimation (HPE) must cope with modalities missing at inference.
Terminology
Abstract
Multimodal human activity recognition (HAR) and human pose estimation (HPE) must cope with modalities missing at inference. Completions generated for a fixed fusion model must preserve the fusion readout: what fusion uses from each modality. We propose COMPASS, a completion-and-fusion framework in which each modality occupies a fixed slot filled by an observed representation or a completion inferred from available inputs. Its core principle, fusion-matched supervision, supervises completions with same-sample real targets at readout granularity: token means for HAR, where fusion averages tokens, and per-joint representations for HPE, where fusion preserves joint structure. Trained with complete multimodal samples, COMPASS improves subset-averaged performance over the strongest baselines on XRF55 and MM-Fi for HAR and on MM-Fi for HPE. At fixed architecture, fusion-matched supervision outperforms full-token and centered-token matching in HAR, and joint-mean matching and no matching in HPE. The code is available at: https://github.com/haowangcoder/COMPASS.
Sources
- Distilling the Knowledge in a Neural Network
- Learning Factorized Multimodal Representations
- Deep Multimodal Learning with Missing Modality: A Survey
- Diverse via bounded Agreement: Geometric Regularization for Multimodal Fusion
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models