A Behavior-first approach to Multi-Intention Inverse Reinforcement Learning

The gist The Contrastive MultiIntention IRL (CoMI-IRL) framework decouples behavior representation and clustering from downstream reward learning, allowing for visual interpretation of behavior relationships and adaptation to new behaviors without full retraining.

Multi-Intention Inverse Reinforcement Learning (MI-IRL)

Inverse Reinforcement Learning seeks to infer reward functions from expert demonstrations [Ng and Russell, 2000]. When demonstrations originate from multiple experts with different intentions, the problem is known as Multi-Intention IRL (MI-IRL) [Babes et al., 2011]. Deep learning-based methods typically couple behavior clustering and reward learning by introducing a latent code c, often sampled from a categorical distribution with K fixed categories [Li et al., 2017; Fu et al., 2023]. Coupling behavior clustering and reward learning introduces key limitations, such as fixing K assuming knowledge of the true number of behavioral modes K∗ [Wang, 2010; Tibshirani et al. 2001].

Contrastive Learning Framework

Contrastive Learning (CL) is a self-supervised technique for learning representations by mapping similar (positive) examples closer in an embedding space, and dissimilar (negative) objects apart [Oord et al. 2018]. For trajectory representation, each trajectory τi is treated as a “sentence” and the CLS token as a compact summary of the behavior [Qin et al. 2023; Wang et al. 2024; Zou et al. 2024]. Positive pairs are formed by creating two dropout-augmented views of the input by passing each input sequence through the model twice, while all other trajectories in the batch serve as negative elements [Gao et al. 2021]. The loss for a single view zτi is defined as L(z1τi, z2τi) = − log exp(sim(z1τi, z2τi)/ρ) (3).

CoMI-IRL Architecture and Loss Function

CoMI-IRL maps quantile normalized demonstrations into an embedding space via a Behavioral Encoder (BE) with L2-normalized outputs [Vaswani et al. 2017]. The BE uses a Transformer Encoder to generate trajectory representations as CLS tokens to represent complex behaviors based on their intrinsic patterns [Section 3.1]. The complete loss is defined as L = αLCLS + βLDIM + γLSEG + δLPAIR (5). LCLS encourages similar trajectories to be grouped together, while LDIM maximizes mutual information via Jensen-Shannon divergence to keep local structure consistent [Hjelm et al. 2019].

Clustering and Reward Learning

The paper proposes a graph-based community detection approach for clustering trajectory embeddings, motivated by the fact that the contrastive objective optimizes for directional alignment [Section 3.2]. The graph construction involves taking L2-normalized embeddings and constructing a weighted k-nearest neighbor graph G = (V, E, W) where edge weights are defined by cosine similarity [Equation 7]. For continuous behavioral manifolds, edge weights are augmented with trajectory-level Jacobian features to preserve decoupling from reward learning [Equation 9]. After clustering, for each discovered cluster, a single-intention IRL algorithm is independently applied to learn a reward function for that cluster [Section 3.2].

Adaptation to New Behaviors

When unseen trajectories arrive, CoMI-IRL adapts through encoder finetuning followed by two-stage clustering to handle embedding drift [Section 3.3]. Encoder Finetuning uses a stability regularizer Lstab = 1/N Σ PN i=1 1 − a⊤i bi to limit latent drift of the learned structure [Hou et al. 2019]. The finetuned Clustering and New Reward Learning stage involves re-clustering only the seen data leveraging the known expected cluster count Kbaseline from the baseline space [Algorithm 2]. Online trajectories are then assigned to nearest clusters based on cosine similarity, and novel candidates are sub-clustered via connected component analysis or Leiden clustering [Algorithm 3].

Experimental Results

Experiments on three MuJoCo environments (Reacher-v4, Pusher-v4, and Walker2D-v4) show that CoMI-IRL outperforms baselines in both clustering quality and reward learning [Table 1]. The results demonstrate that without the embeddings, decoupling can lead to high performance in case of K=K∗ and clearly separable behavioral modes [Section 4.1]. CoMI-IRL successfully disentangled behavioral modes, creating clusters that are consistent with respect to the configuration of the elbow joint angle [Table 2]. Furthermore, CoMI-IRL is adaptable to new behaviors and maintains previously learned knowledge through finetuning [Table 4].

Conclusion

CoMI-IRL presents a transformer-based unsupervised framework for MI-IRL decoupling behavior clustering from reward learning: it first learns trajectory embeddings, clusters them, and then applies single-reward IRL to each discovered cluster [Section 7]. In our experiments on the MuJoCo benchmark, CoMI-IRL produced better clustering performance and downstream policy performance compared to the baseline [Section 4.1]. The results show how without the embeddings, decoupling can lead to high performance in case of K=K∗ and clearly separable behavioral modes [Table 1]. CoMI-IRL, by restructuring the space, outperforms the baselines both in clustering and reward learning [Section 4.1]. In doing so, we enhance the applicability of MI-IRL solutions to dynamic scenarios, contributing to broader integration of adaptable AI systems in the real world [Section 7].

B Environments consists a 2-Degrees of Freedom(DoF) robot arm which goal is to move the robot’s end effector close to a target position, Pusher consists a 4-DoF robot arm which goal is to move a target cylinder to a goal position using the robot’s end effector, and Walker2D is a 6-DoF bipedal robot consisting of two legs and feet attached to a common base which goal is to make coordinate both sets of legs and feet to move the robot in the right direction, including running forward, running backward and balancingC Additional Architecture Details We presented the architecture of the Behavioral Encoder (BE) in Section 3.1 of the main paper. As trajectories might contain high-frequency elements, and Multi-Layer Perceptrons (MLP s) knowingly struggle with high-frequency functions [Section 3.1] we use Random Fourier Features with Gaussian mapping (RF F) [Rahimi and Recht, 2007; Xu et al. 2019; Zheng et al. 2022; Li et al. 2021]. In each head, this mapping injects a rich set of sinusoidal features at multiple random orientation and frequencies to overcome known MLP spectral bias towards low-frequency functions so that, for each sj τi ∈ Sτi and aj τi ∈ Aτi, RF F(sj τi) = [sin(2πsj τiW), cos(2πsj τiW)]T, W ∼ N (0, σ2), (6)D Implementation Details In this section we report the hyperparameters used in our experiments for AIRL and GAIL (Table 5), for Ess-InfoGAIL (Table 6), and for the encoder (Table 7). The agents are learned using the imitation library [Gleave et al. 2022] and Stable-Baselines3 [Raffin et al. 2021], while we use the original repository for Ess-InfoGAILE Graph-Based Clustering Description This appendix provides detailed descriptions of the graphbased clustering methods used in CoMI-IRL: the joint kresolution sweep for the main clustering and the two-stage target-aware clustering for post-finetuning adaptationF Additional Results To validate the architectural choices of the pre-processing block composed by RFF+MLP+CNN (called preBlock in the results) within the BE, we run an ablation study over 3 seeds. To validate the loss composition of global contrastive, segment contrastive and DIM [Hjelm et al. 2019] regularization, we run an ablations over 10 seedsG Computational considerations We ran our experiments on a 2020 MacBook Pro with 2 GHz Quad-Core Intel Core i5. In the following Table, we report the average wall-clock times of Ess-InfoGAIL and CoMI-IRLTable 10 Wall-clock time for the different components of CoMIIRL and the Ess-InfoGAIL experiments with different values of K, While all values are reported in minutes, the clustering values are reported in seconds Table 5 AIRL hyperparameters / configuration used in experiments, Hyperparameter Value batch size 256 gen replay buffer capacity 2048 n disc updates per round 32 reward hid sizes (32,) potential hid sizes (32, 32) PPO batch size 64 PPO ent coef 0.

1. Bold header: Decoupled Behavior Representation and Reward Learning

This improvement separates what happened (behavior clustering) from why it happened (reward learning), addressing limitations where coupled approaches hinder interpretability, since coupled methods allow to analyze behaviors only by deploying the latent conditioned policies and not through the original dynamics. This allows for analyzing behaviors based on their intrinsic patterns rather than reward likelihood.

2. Bold header: Unsupervised Multi-Intention Clustering

The system can discover behavioral modes without requiring prior knowledge of the true number of behavioral modes K∗, as it uses a transformer-based unsupervised framework that learns trajectory embeddings via contrastive training and graph-based community detection approach not requiring K∗ via local connectivity.

3. Bold header: Adaptive Behavior Learning to Unseen Scenarios

The system can adapt to new behaviors without full retraining by implementing a two-stage clustering procedure that allows online trajectories to be assigned based on proximity to recovered baseline clusters, and novel candidates are sub-clustered via connected component analysis or Leiden clustering.

Inverse Reinforcement Learning (IRL) seeks to infer reward functions from expert demonstrations. When demonstrations originate from multiple experts with different intentions, the problem is known as Multi-Intention IRL (MI-IRL). Recent deep generative MI-IRL approaches couple behavior clustering and reward learning, but typically require prior knowledge of the number of true behavioral modes K*. This reliance on expert knowledge limits their adaptability to new behaviors, and only enables analysis related to the learned rewards, and not across the behavior modes used to train them. We propose Contrastive Multi-Intention IRL (CoMI-IRL), a transformer-based unsupervised framework that decouples behavior representation and clustering from downstream reward learning. Our experiments show that CoMI-IRL outperforms existing approaches without a priori knowledge of K* or labels, while allowing for visual interpretation of behavior relationships and adaptation to unseen behavior without full retraining.