Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations

arXiv:2602.08755 · cs.LG · Submitted 2026-02-09 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Align and Adapt".

Jane: The gist The proposed model, AliAd, integrates multimodal multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in human activity recognition.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Looking at this paper, "Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations," the core contribution is AliAd, which manages arbitrary view availability during both training and inference without needing to reconstruct missing views.

Jane: The authors show that using an adjusted center contrastive loss allows the model to align views in a hyperspherical latent space while factoring in view quality, which helps mitigate the impact of missing data on the fusion process >

Lu: They also use a mixture-of-experts module with a specialized load balancing strategy to address those remaining discrepancies between different view combinations that standard contrastive learning misses >

Meng: The paper proves that this approach works well across inertial and human pose datasets, handling view counts from three to nine, which suggests it has good practical performance and robustness when you run into missing views during real-world scenarios >

Tom: So, for the listener, this means we can build recognition systems that are much more flexible about what data they receive—they don't have to be perfect combinations of views to work reliably >

Jane: It moves the focus away from trying to perfectly fill in gaps and instead focuses on aligning what you *do* have, using view weights computed by an attention module for dynamic fusion >

Lu: The geometric relationship they highlight among the components in their model shows how these different parts combine effectively within that latent space when things are missing >

Meng: It’s a robust way to deploy multimodal AI because it doesn't require perfect, complete data sets to function reliably in production environments >

Conclusion: Tom: So we’ve seen how AliAd handles missing views and those weird view combinations, but let's talk about the actual title now, "Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations."

Jane: It sounds a little technical for a driver or someone cooking dinner. What does that actually mean in plain English?

Lu: It means the model is designed to handle any situation where you don't have all the data you want at once. If you only have three views instead of nine, it still works well because it adapts.

Meng: From a deployment standpoint, I worry about how much extra complexity this adds when we are running this on our servers. Does "Adapt" mean we need a massive GPU cluster just to handle the shifting view weights?

Lalam: The paper says the adjustment is done through an adjusted center contrastive loss that puts the center closer to higher quality views, which means it’s smart about which data it trusts.

Jane: So, instead of trying to perfectly rebuild what's missing data we can use a weighted sum based on view quality. That sounds like a practical way to make sense of incomplete information.

Tom: Exactly. The authors aren't trying to reconstruct the missing views, which saves us from needing extra complex reconstruction modules we’d have to train separately for every possible gap.

Lu: And they also use this mixture of experts part specifically to fill in the gaps where the contrastive learning isn't picking up any signal from a specific view combination.

Meng: I'm still thinking about the cost and complexity tradeoff here. If we switch from O(V squared) to O(V) for that loss, it’s good for training speed but does it keep inference fast enough when we have nine views?

Lalam: The computational complexity drops from O of V squared to O of V, meaning it scales linearly with the number of views. That should help keep things manageable on the edge devices.

Jane: It sounds like they’ve managed to make a very flexible system that doesn't break just because some sensors fail or a view is missing. What does this mean for actually building real-world activity recognition apps?

Tom: It means we can deploy these models in environments where sensor data is never perfect, which is most of the time. This flexibility opens up applications like monitoring people in messy homes or on uneven terrain.

Lu: The implication is that multimodal learning doesn't have to be a rigid pipeline anymore; it can be more resilient to real-world noise and incompleteness.

Meng: So for production, this suggests we don't need perfect, synchronized data streams to get high accuracy in activity recognition systems. That’s a big win for reliability.

Lalam: I'm confident the method described here offers a solid way to handle these variable inputs reliably across different modalities.

University College Dublin

cs.LG

Submitted: 2026-02-09

Updated: 2026-02-18

DOI: 10.1109/JSEN.2026.3718255

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 85/100

The gist: The gist The proposed model, AliAd, integrates multimodal multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in

Key concepts

Adjusted Center Contrastive Loss (LAC): This loss function aligns views in a hyperspherical latent space while incorporating view quality. It pulls all views toward higher-quality ones, effectively mitigating the impact of missing views on the fusion process by weighting them based on their perceived quality.
Mixture of Experts (MoE) Module: A sparse MoE architecture is used for classification to handle discrepancies between different view combinations. It uses a load balancing loss to ensure the gating mechanism distributes input tokens evenly among experts, adapting to arbitrary view sets.
Attention-based View Fusion: The model uses an attention module to dynamically capture data quality and task relevance across views. Fusion is achieved by taking a weighted sum of views, where weights are computed based on the latent representation's relationship with high-quality views on the hypersphere.

Terminology

Contrastive Learning — 2× in this paper · explained in 7 episodes
Alignment — 1× in this paper
Latent Space — 1× in this paper · explained in 7 episodes
Representation Learning — 1× in this paper · explained in 3 episodes

Summary

The gist The proposed model, AliAd, integrates multimodal multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in human activity recognition.

Proposed Model and Core Concepts

AliAd is a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference in human activity recognition. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on the fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from O(V 2) to O(V), where V is the number of views.

Adjusted Center Contrastive Loss

The adjusted center contrastive loss mitigates the impact of view missing on the fusion by aligning views in the hyperspherical latent space. This loss function takes view quality into account. The adjusted center contrastive loss is defined as LAC = 1/(V-1) sum a (1 - w(a)) stop grad Lpair z(a), X V v!=a w(v) z(v) stop grad. The weight term w(v) puts the center closer to higher-quality views, causing all views to be pulled more toward those of higher quality.

Attention-based View Fusion

The model employs an attention module to capture differences in data quality and task relevance among views. Fusion is performed via a weighted sum followed by vector magnitude normalization. The fused representation lies in the center of the individual views, while being closer to high-quality views on the hypersphere. View weights are computed as w(v) = softmax(MLP(z stop grad))v (7). This fused representation is used for both classification and contrastive loss.

Mixture of Expert Classifiers

A sparse mixture-of-experts architecture is used for the classification head to address the remaining discrepancies among different view combinations not captured by the contrastive loss. The crossentropy loss function H is used: Lcls = 1/(V+1) H(y, yˆ(wf)) + sum v V v H(y, yˆ(v))! A load balancing loss encourages the gate to distribute input tokens more evenly among the experts.

Experimental Validation and Results

AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility. Strong empirical results demonstrate the effectiveness of the proposed method and its robustness to missing views compared to baseline methods. AliAd achieves the highest scores overall when simulating missing views randomly.

Time Complexity Analysis

The adjusted center contrastive loss reduces time complexity from O(V 2) to O(V), where V is the number of views. Empirically, for optimization time, all methods behave similarly except for the full graph loss, which becomes increasingly costly as the number of views grows. The empirical results are consistent with theoretical expectations.

Conclusion

AliAd can handle arbitrarily missing views during both training and inference, while avoiding unnecessary data reconstruction. The model is trained with a load balancing loss for MoE, an adjusted center contrastive loss, and a classification loss. It leverages both labeled and unlabeled data since contrastive loss does not require labels. The attention module assigns view weights, dynamically adjusting view fusion for both contrastive learning and classification. The MoE head addresses residual discrepancies among view combinations that arise from view-specific features and are not captured by contrastive learning. This approach samples negative pairs for contrastive loss within each batch, which is effective when the data are diverse. The model processes views separately to accommodate missing data. While it retains useful view-specific features, it does not model cross-view interactions, except at the fusion step.

Improvements for AI systems

Bold header: Enhanced Robustness to View Omission

The system can robustly tackle view missing problems by relying on view alignment, without missing view reconstruction or filling, as stated in the abstract, by using an adjusted center contrastive loss which mitigates the impact of missing views on fusion.

Bold header: Efficient and Flexible View Fusion

AliAd supports arbitrary view combinations during both training and inference by utilizing an attention-based weighting mechanism to perform a weighted sum for fusion, ensuring that the fused vector is oriented more toward the vector with a higher weight.

Bold header: Adaptive Classification via Mixture of Experts

The system can address residual discrepancies not captured by contrastive learning by employing a mixture-of-experts module with a specialized load balancing strategy, which allows it to adapt to arbitrary view combinations and generalize to unseen view combinations.

Bold header: Improved Feature Space Consistency

To ensure consistency across components, the model projects all representations onto a hyperspherical space using magnitude normalization, which places feature vectors across all views onto a hypersphere.

Bold header: Optimized Time Complexity

The system achieves a reduced time complexity of O(V) for contrastive loss compared to O(V 2), where V is the number of views, by contrasting each view with the center of the other views, which is reduced time complexity from O(V 2) to O(V).

Abstract

Multimodal multiview learning seeks to integrate information from diverse sources to enhance task performance. Existing approaches often struggle with flexible view configurations, including arbitrary view combinations, numbers of views, and heterogeneous modalities. Focusing on the context of human activity recognition, we propose AliAd, a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on multiview fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from O(V 2) to O(V), where V is the number of views. To address residual discrepancies not captured by contrastive learning, we employ a mixture-of-experts module with a specialized load balancing strategy, tasked with adapting to arbitrary view combinations. We highlight the geometric relationship among components in our model and how they combine well in the latent space. AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.