D-CLOT: Double Closed Loop Optimal Transport for Unsupervised Action Segmentation

Published in: IEEE Transactions on Pattern Analysis and Machine Intelligence (arXiv:2608.05877v1 [cs.CV], 6 Aug 2026)

The paper addresses unsupervised action segmentation in untrimmed videos using optimal transport (OT). The authors state: "Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. They identify this representation–prototype inconsistency as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions."

The proposed solution builds on CLOT (Closed Loop Optimal Transport), which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, D-CLOT "introduces a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry."

Two instantiations are studied: "D-CLOT updates the prototypes using k-means, whereas D-CLOTB updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to +12.7 F1 and +10.2 mIoU (YTI) and activity-level gains of up to +8.9 F1 (FS-Eval). The paper also establishes the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work."

Temporal action segmentation assigns an action label to each frame of an untrimmed video and underlies downstream tasks such as activity recognition and procedural video understanding. Unsupervised action segmentation aims to discover recurring actions and their temporal boundaries without manual labels.

The paper traces the OT-based line of work: "ASOT [13] removed the need for a known action order by enforcing temporal consistency directly in the transport problem. HVQ [15] improved short-action detection via hierarchical vector quantization. VASOT [14] added cross-video correspondences as regularisation. CLOT [16] unifies these threads through a multi-level cyclic mechanism: frame embeddings are aligned to action prototypes, a parallel decoder infers segment embeddings from the aligned sequence, and the resulting segment structure is propagated back to the frame domain via cross-attention before a final OT alignment."

The core identified problem: "This cyclic formulation leaves an asymmetry unaddressed: while frame and segment representations are iteratively refined, the action embeddings A that define the transport costs at every stage are not re-estimated from this refined geometry. They are initialised once via k-means and thereafter evolve only through the classification gradient, so as OT refinement re-organises the frame representation space, the prototypes meant to organise it can fall behind, a representation–prototype inconsistency. This is most damaging around ambiguous temporal transitions, where refined embeddings of adjacent frames from different actions overlap, and for short or infrequent actions, whose limited transport mass is easily absorbed by dominant modes. Additionally, since the refinement stages optimise purely for the OT-induced classification loss, nothing prevents them from introducing associations between frames that were not neighbours in the original encoder geometry."

D-CLOT "leaves the CLOT backbone and its three-stage OT objective intact, while introducing two successive operations. First, it stabilizes the refined frame and segment representations through a graph-constrained regularizer that encourages their local neighbourhood structure to remain consistent with that of the encoder output. Second, it periodically re-anchors the action-embedding to this stabilized representation geometry, thereby mitigating the drift that can arise when they are updated solely through gradient descent."

The paper lists five contributions:

1. Identification of the representation-prototype inconsistency: the latent action embeddings that define the OT costs are not re-estimated as the refined frame and segment representations evolve.

2. "An action-embedding refinement step that periodically re-estimates the latent prototypes from the refined frame embeddings FR and blends the resulting estimates with the current prototypes. Together with the graph constraint... this closes the representation–prototype loop without modifying the CLOT backbone or its training objective."

3. A graph-constrained regularization module that stabilizes the OT-refined frame and segment representations by preserving the local neighborhood structure of the encoder's feature space.

4. Two instantiations — "D-CLOT, which uses a k-means-based prototype update, and D-CLOTB, which updates prototypes as OT barycenters weighted by the refined transport plan. Both variants achieve state-of-the-art performance on YTII, FS, DA, and BF."

5. The first unsupervised action-segmentation baseline on Assembly101, a large-scale, procedural, fine-grained testbed.

Given a collection of B untrimmed activity videos, each with precomputed frame descriptors Xb = [xb1;...; xbNb] ∈ RNb×D, and a predefined number of latent action classes K, the objective is to infer frame-level action assignments without using frame-level labels, transcripts, or action-order constraints during training. The latent action space is parameterised by a shared action-embedding matrix A = [a1,..., aK]⊤ ∈ RK×d, with prototypes constrained to the unit sphere (∥ak∥2 = 1), shared across all videos of an activity.

The backbone comprises three components:

Encoder with feature dispatching: Maps X into frame embeddings F = norm(MLP(X)) ∈ RN×d. A feature-dispatching mechanism injects prototype information: for each frame fi and prototype ak, the affinity is φ(ak, fi) = σ(bφ + sφ · a⊤k fi / (∥ak∥2 ∥fi∥2)), and the embedding is updated through a prototype-conditioned residual fi′ = fi + (1/K) Σk φ(ak, fi) ak.

Parallel segment decoder: A transformer decoder maps F into segment embeddings S ∈ RK′×d using a set of learnable queries Q, with K′ ≤ K, interacting through self-attention and cross-attention with the encoded frame sequence.

Cross-attention refinement: FR = F + softmax(FS⊤/(τ·√d))S, where each frame aggregates its most relevant segment embeddings, combining local evidence with higher-level temporal context.

Given a representation Z and validity mask m, the normalized pairwise affinity is:

Gij = wij / Σ(p,q)∈E wpq, with wij = mi mj exp(−(1 − cos(zi, zj))/h)

where h > 0 is a bandwidth. The graph-constrained loss minimizes the cross-entropy between the affinity distributions of the fixed encoder anchor Fenc and the auxiliary representation Z ∈ F, FR:

LG(Fenc, Z) = −Σij G0ij log(Gij + ε)

Minimising (5) discourages the OT refinement stages from forming inter-action connections absent in the encoder geometry. By default, the fidelity term is imposed as a soft penalty added to the CLOT objective.

Both variants apply a conservative, normalised blend rather than a hard replacement: A ← norm((1 − β)A + βÂ), with blend factor β ∈ (0, 1].

D-CLOT (k-means refinement): Â ← KMeans(F, K), applied to the refined frame features pooled over all videos of the activity. This update is assignment-agnostic: it uses no transport plan and no assumption beyond K. Its stability therefore rests entirely on the graph penalty.

D-CLOTB (barycentric refinement): Enforces the graph constraint through an explicit auxiliary variable, solving the inner problem F̃R = arg minV λG LG(Fenc, V) + λa ∥V − FR∥2F, with the inner steps... differentiably unrolled during training. Each prototype is then updated as the OT barycenter of the stabilised features, weighted by their soft assignment mass:

Âk = norm(Σb Σi (T̃R)i,k (f̃R)i / πk) if πk > ε, else ak

where πk is the total soft mass of action k. "Unlike the k-means update, the barycentric update is assignment-aware: each frame contributes to a prototype in proportion to its OT confidence... This is what protects the long tail. A rare action whose frames are assigned with low confidence exerts a correspondingly small pull on its prototype."

Three OT problems (T, TS, TR) underlie the pipeline, using the unbalanced OT formulation of ASOT: the Kantorovich problem with the Gromov-Wasserstein term (encoding structural priors such as temporal consistency) and an unbalanced relaxation with a soft KL penalty on the action marginal, allowing the long-tailed action distributions typical of untrimmed video. The visual cost is Ckij = (x⊤i aj)/(∥xi∥ ∥aj∥) − ρZij, where Z is a temporal prior regularising the coupling towards a banded-diagonal structure, and a Sliced-Wasserstein cost complements the cosine cost.

For D-CLOT: Ltrain D-CLOT = LCLOT + λFG LG(Fenc, F) + λRG LG(Fenc, FR). For D-CLOTB, the fidelity term is consumed inside the inner problem (8), and the outer objective reduces to LCLOT evaluated on the graph-stabilised representations. At inference, the final segmentation solves the refined OT problem between FR (or F̃R) and A, assigning each frame ŷi = arg maxk (TR)i,k.

The backbone is unchanged from CLOT. Optimization uses Adam with weight decay 10−4 and a learning rate around 10−3. Action embeddings A are initialised by k-means, and the number of action clusters K is set to the ground-truth action count of each dataset. Videos are sampled to 256 frames per training iteration, except on Assembly101, where the longer average duration motivates a budget of 512 frames. Neither module introduces learnable parameters, so both variants have the same parameter count as CLOT.

• Breakfast (BF): 1,700 videos, 10 activities, 48 fine-grained action classes, IDT Fisher-vector features.

• YouTube Instructions (YTI): 150 videos, 5 activities, background frames dominate the footage (approximately 75%); HOF + VGG16 conv5 features.

• 50 Salads (FS): 50 videos, Mid (19 actions) and Eval (12 coarser classes) granularities.

• Desktop Assembly (DA): 76 videos, 22 temporally ordered actions.

• Assembly101 (A101): 200 view-specific videos, 15 toy categories, The number of observed coarse actions per toy category ranges from 11 to 42, V-JEPA2 features, 42.1% background.

Per-video matching: "D-CLOT attains the best F1 and mIoU on YTI and FS-Mid, improving over CLOT by +12.7 F1 and +10.2 mIoU on YTI, and by +6.2 F1 and +4.2 mIoU on FS-Mid. D-CLOTB provides the strongest gains on FS-Eval and DA, increasing F1 by +5.9 and +6.3 points, respectively, and mIoU by +2.2 and +4.3 points."

Activity-level matching: D-CLOT leads on YTI (+1.3 MoF, +1.2 F1, +3.9 mIoU over CLOT) and FS-Mid (+0.4 MoF, +1.2 F1, +1.4 mIoU), while D-CLOTB is best on FS-Eval (+5.5 MoF, +8.9 F1, +1.6 mIoU) and attains the top DA mIoU. On Breakfast, both variants trade a small MoF decrease for higher segment-level scores: D-CLOTB improves F1 by +1.2 and mIoU by +0.6.

The choice between variants is dataset-dependent: "D-CLOT tends to benefit datasets with sharper cluster structure (YTI, FS-Mid), while the assignment-aware update of D-CLOTB is more effective on datasets with more procedurally consistent, imbalanced action durations (BF, FS-Eval, DA)."

"Both D-CLOT and D-CLOTB improve substantially over CLOT in F1 and mIoU at the video level, at the cost of a small MoF drop... D-CLOTB obtains the best overall average, indicating that graph-constrained action refinement generalises beyond the cooking and instructional domains to large-scale, fine-grained assembly video. Our method provides the strongest first baseline, but the scores indicate substantial room for future work."

The ablation isolates graph regularization on the frame embeddings (F̃), graph regularization on the refined frame embeddings (F̃R), and action-embedding refinement (A).

Key findings: "action refinement is most effective when applied to a graph-stabilized representation. Updating A alone is unstable for D-CLOT, since prototypes are re-anchored to an unconstrained refined space. Conversely, graph regularization alone yields dataset-dependent gains, but does not consistently match the full configuration... D-CLOTB shows a more stable pattern: its full configuration dominates all partial variants across datasets and metrics. This supports the assignment-aware barycentric update as a more robust refinement mechanism."

The qualitative ablation (Fig. 3) explains the complementary roles: "CLOT disrupts the action order, while GC recovers more coherent temporal boundaries. However, GC still assigns frames to a spurious action prototype... Updating A corrects this prototype mismatch: in the full D-CLOT model, the blue segment disappears... Thus, GC mainly regularizes temporal structure, whereas action refinement corrects the prototype-to-frame assignment. On this video, D-CLOT improves MoF from 0.663 to 0.939, F1 from 0.333 to 1.000, and mIoU from 0.348 to 0.883."

The authors analyze the graph fidelity weights (λFG, λRG), graph bandwidth h, blend factor β, and stabilization budget. On DA, F1 peaks sharply at λFG = 0.05 and λRG = 0.1 for both variants: a moderate amount of graph regularisation stabilises the refined frame geometry, while excessive regularisation over-constrains it. On YTI, F1 varies little: with 62% background frames, there is limited geometry to anchor to, so the modules act as a stabiliser rather than a strong amplifier, the intended fallback behaviour, not a shortcoming. D-CLOTB's inner-step count S matters: The three datasets peak sharply at S = 2, our default. Overall, the sensitivity curves demonstrate that graph-constrained action refinement is robust to moderate hyperparameter variations. No parameter causes a substantial performance degradation.

Both variants have the same parameter count as CLOT (92.5K parameters). "D-CLOT adds 2.06 s per training epoch due to the periodic K-means prototype update... D-CLOTB adds 0.47 s per epoch. Its barycentric re-estimation costs only 26 ms every ten epochs. At inference, the inner refinement increases latency by 8.2 ms per video. Peak memory remains below 82 MB for all variants."

The authors summarize: "We introduced D-CLOT, a graph-constrained extension of CLOT for unsupervised temporal action segmentation. The proposed framework addresses the mismatch between progressively refined frame representations and latent action prototypes by jointly regularizing the representation geometry and periodically updating the prototypes. The ablation shows graph-based regularization and prototype refinement are complementary: graph regularization promotes temporally coherent representations, whereas prototype refinement maintains alignment between latent actions and the evolving feature space. Among the proposed variants, the assignment-aware barycentric refinement consistently provides the most robust performance across datasets. They conclude: These results suggest that continuously adapting…

Here are the concrete improvements to AI systems suggested by the D-CLOT paper, followed by what the improved system can now do.

1. Continuously re-anchor latent prototypes to the refined feature space.

Instead of letting action prototypes drift while frame/segment embeddings are optimized, the system periodically re-estimates prototypes from the refined frame embeddings and blends them with the current prototypes (A ← norm((1−β)A + βÂ)). This closes the “representation–prototype loop” and prevents the prototypes from lagging behind the evolving geometry.

2. Add a graph-constrained regularization module.

The system explicitly penalizes changes in local neighborhood structure between the encoder output and the OT-refined frame/segment embeddings. This prevents the optimal-transport refinement from creating spurious inter-action links that were absent in the original encoder geometry.

3. Use assignment-aware prototype updates via OT barycenters (D-CLOTB).

Instead of a generic k-means update, the system computes each prototype as the OT barycenter of stabilized frame features, weighted by the soft transport-plan mass. This protects short and infrequent actions: a rare action with low-confidence assignments exerts a correspondingly small, controlled pull on its prototype, avoiding absorption by dominant modes.

4. Make the graph constraint differentiable through unrolled inner steps.

D-CLOTB enforces the fidelity term through an explicit auxiliary variable solved by unrolled inner iterations (F̃R = arg min V λG LG(Fenc,V) + λa V − FR2F). The system can therefore train end-to-end while still stabilizing representations before each prototype update.

5. Keep the backbone and training objective intact.

Both variants leave the CLOT backbone and its three-stage OT objective unchanged, add no learnable parameters, and add minimal compute (≈2.06 s/epoch for D-CLOT, ≈0.47 s/epoch for D-CLOTB, 8.2 ms extra inference latency, <82 MB peak memory). This makes the improvements directly applicable to existing OT-based segmentation pipelines without retraining from scratch or increasing model size.

6. Adapt the refinement to dataset characteristics.

The system offers two instantiations: k-means prototype re-anchoring (D-CLOT) for datasets with sharper cluster structure, and barycentric assignment-aware re-anchoring (D-CLOTB) for procedurally consistent, imbalanced datasets. A user can select the variant based on the data distribution.

• Segment untrimmed videos into actions without any frame-level labels, transcripts, or action-order constraints, by jointly refining frame representations, segment representations, and latent action prototypes.

• Produce more accurate temporal boundaries around ambiguous transitions, because graph regularization keeps refined embeddings locally consistent with the encoder geometry.

• Detect short and infrequent actions more reliably, thanks to assignment-aware barycentric prototype updates that prevent long-tail actions from being absorbed by dominant classes.

• Achieve state-of-the-art unsupervised segmentation scores on YTI, 50 Salads, Breakfast, and Desktop Assembly, with per-video gains of up to +12.7 F1 and +10.2 mIoU on YTI, +6.2 F1 / +4.2 mIoU on FS-Mid, +5.9 F1 / +2.2 mIoU on FS-Eval, and +6.3 F1 / +4.3 mIoU on DA, plus activity-level gains up to +8.9 F1 on FS-Eval.

• Generalize to large-scale, fine-grained procedural video: it provides the first unsupervised action-segmentation baseline on Assembly101, handling long videos, 11–42 actions per category, and 42% background frames.

• Remain robust in background-heavy settings (e.g., YTI with 75% background frames), where the modules act as stabilizers rather than amplifiers — an intended fallback behavior.

• Maintain stable temporal action order where CLOT fails: in the qualitative example, the full system improves MoF from 0.663 to 0.939, F1 from 0.333 to 1.000, and mIoU from 0.348 to 0.883 by removing a spurious prototype assignment.

• Run efficiently in production-like settings, since the added modules introduce no new parameters and only marginal training/inference overhead.

Optimal transport (OT) has emerged as an effective framework for unsupervised action segmentation. Yet, in existing OT-based methods, the latent action prototypes that define the OT costs are not re-estimated from the refined frame geometry. Instead, they evolve solely through gradients from the pseudo-label loss. We identify this representation--prototype inconsistency as a central bottleneck, particularly around ambiguous transitions and for short or infrequent actions. To address this issue, we build on the recently introduced CLOT, which refines frame embeddings based on estimated segment embeddings, and further re-estimates the action prototypes from the refined frame embeddings. Specifically, we introduce a graph-constrained module that regularizes the OT-refined frame and segment representations by preserving the local neighborhood geometry of the encoder output. An action-embedding refinement step then periodically re-anchors the prototypes to this stabilized representation geometry. We study two instantiations that share the same backbone, graph module, and objective: D-CLOT updates the prototypes using k-means, whereas D-CLOT B updates them as OT barycenters weighted by the refined transport plan, yielding an assignment-aware prototype update consistent with the current transport geometry. Across five established benchmarks, both variants improve segment-level quality over CLOT, with per-video gains of up to +12.7 F1 and +10.2 mIoU (YTI) and activity-level gains of up to +8.9 F1 (FS-Eval). We further establish the first unsupervised action-segmentation baseline on Assembly101, a procedural and substantially more fine-grained benchmark than those commonly used in prior work. Extensive ablations and sensitivity analyses demonstrate that the two refinement mechanisms are complementary and robust.