CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning

arXiv:2608.24374 · cs.SD · Submitted 2026-08-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning".

Jane: Here's the gist of the paper. CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function system.

Tom: The Core Problem Addressed Conventional audio language models (ALMs) struggle to effectively process multi-event audio sequences in daily spatial scenarios because they primarily rely on coarse-grained contrastive learning with global auditory and textual features.

Jane: Current approaches lack the resolution to distinguish multiple sequential events.

Tom: Relying solely on global representations imposes a severe information bottleneck by compressing dynamic, high-dimensional content into a single holistic vector, which often induces “context drift” and fails to resolve the distinct semantic identity of individual events.

Paper summary: Tom: So looking at the CoSTALA paper, it seems their main contribution is proposing this specific training paradigm which moves from global alignment to fine-grained spatio-temporal reasoning using that multi-grain loss system. They’re addressing the issue of context drift caused by compressing dynamic audio into a single vector.

Jane: Exactly, and the authors are showing through their results that decoupling and modeling those temporal dependencies explicitly makes a huge difference, especially when you look at how much better it performs on tasks involving multiple sequential events in space.

Lu: The paper establishes this framework as a robust way to build systems capable of compositional reasoning in audio-language tasks by using those different levels of alignment—semantic, local, and temporal—to guide the training process.

Meng: From a deployment standpoint, it’s interesting that they found that the feature consistency loss actually helps maintain local semantic purity even when dealing with global contexts during training. That suggests a stability benefit we might see in production models if we can manage the complexity of those anchors.

Tom: So, to put it simply, CoSTALA is about transitioning from just matching general audio to language to actually understanding the sequence and spatial arrangement of events within that audio. The authors are providing a new training blueprint for achieving that level of detail in complex audio scenes.

Conclusion: Tom: So we've seen how CoSTALA uses that complex system of losses to break down audio understanding into smaller pieces, and now we're wrapping up this paper titled CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning.

Jane: It really is about taking those different levels of alignment—the semantic pairs and the temporal sequences—and stitching them together in a new way to get a finer picture of what's happening in a long audio scene.

Lu: The core idea they push is moving away from just matching the whole audio clip to text, and instead forcing the model to learn how individual sounds relate spatially and temporally on their own.

Meng: From my side, it’s interesting that they show that adding those local alignment losses really makes a difference in performance when you look at real-world scenarios where events are close together.

Tom: Yeah, so what does this mean for us? If we can train models this way, we might finally get audio systems that actually understand the sequence of things happening, not just the overall vibe.

Jane: Exactly. It means future audio language models could handle complex stories or scenes where multiple distinct sounds happen one after another in a specific place.

Lu: The implication is building representations that are much richer, because instead of one big blob of information, you've got structured knowledge about where and when things occur.

Meng: But the catch is how they set up those losses; if you don't get the anchoring right for each individual sound, the whole system might just collapse into a global average again.

Tom: Right, so it’s not just about making one model better; it’s about building a training structure that forces the model to learn composition instead of just memorizing patterns.

Jane: And that's what makes this paper significant because it gives us a concrete blueprint for how to build those more granular models going forward.

Peiwei Ren, Jinbo Hu, Fang Kang, Shan Liang, Yin Cao

Xi’an Jiaotong Liverpool University · MiLM Plus, Xiaomi Inc. · Ã Center for Machine Vision and Signal Analysis, University of Oulu, Finland · 旡 Institute of Acoustics, Chinese Academy of Sciences

cs.SD

Submitted: 2026-08-25

Updated: 2026-08-25

DOI: 10.21437/Interspeech.2026-1110

Code: https://github.com/Cell778/CoSTALA26

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function…

Key concepts

Hierarchical Loss Functions
This is a system of multiple training rules used during model development. Instead of one simple rule, CoSTALA uses several losses—like contrastive loss and spatio-temporal loss—that work at different levels. This allows the model to learn both broad relationships and very fine details about how sounds relate to each other in space and time.
Spatio-Temporal Reasoning
This refers to the ability of an AI to understand not just what a sound is, but exactly where it is coming from (spatial) and when it happened (temporal). CoSTALA aims for this by decoupling the audio processing so the model can focus separately on location and sequence, rather than treating everything as one single block of information.
3-Way Spatio-Temporal Loss (Lst)
This specific loss function forces the model to learn a crucial distinction: whether an error is due to incorrect timing or incorrect spatial placement. By using the correct text embedding as a reference, this loss helps the model suppress confusing audio candidates that might look similar but are temporally or spatially wrong.
Local Alignment Loss (Llocal)
This component focuses on anchoring individual sound events to their specific descriptions. It aligns short, specific chunks of raw audio directly with their corresponding spatial captions. This ensures that the model learns the exact correspondence between a single sound and its precise location description.

Terminology

Summary

CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function system. p. 0

Where the paper says it

To overcome these limitations, we propose CoSTALA—a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity.

The Core Problem Addressed

Conventional audio language models (ALMs) struggle to effectively process multi-event audio sequences in daily spatial scenarios because they primarily rely on coarse-grained contrastive learning with global auditory and textual features. Current approaches lack the resolution to distinguish multiple sequential events. p. 0

Where the paper says it

However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events.

Relying solely on global representations imposes a severe information bottleneck by compressing dynamic, high-dimensional content into a single holistic vector, which often induces “context drift” and fails to resolve the distinct semantic identity of individual events. p. 1
Where the paper says it

This coarse-grained approach imposes a severe information bottleneck by compressing dynamic, high-dimensional content into a single holistic vector. Such compression often induces “context drift”, where the model averages out precise local semantics and consequently fails to resolve the distinct semantic identity of individual events.

The CoSTALA Architecture

To achieve fine-grained spatio-temporal reasoning, CoSTALA explicitly decouples and models the temporal dependencies of spatial events. The architecture integrates a RoBERTa-based text encoder with a hierarchical audio backbone driven by HTSAT. p. 1

Where the paper says it

To implement the proposed paradigm, the CoSTALA architecture explicitly maps these decoupled attributes to a shared latent space. We integrate a RoBERTa-based text encoder [18] with a hierarchical audio backbone driven by HTSAT [19], as shown in Figure 1.

This design strictly decouples pure acoustic semantic extraction from spatial localization. p. 1
Where the paper says it

We integrate a RoBERTa-based text encoder [18] with a hierarchical audio backbone driven by HTSAT [19], as shown in Figure 1. Inspired by the EINV2 dual-branch design [20], we strictly decouple pure acoustic semantic extraction from spatial localization.

Furthermore, a dedicated Transformer-based temporal encoder equipped with Rotated Position Embedding (RoPE) is incorporated to precisely capture the temporal sequence of acoustic events. p. 1
Where the paper says it

Inspired by the EINV2 dual-branch design [20], we strictly decouple pure acoustic semantic extraction from spatial localization. Crucially, to precisely capture the temporal sequence of acoustic events, we incorporate a dedicated Transformer-based [21] temporal encoder equipped with Rotated Position Embedding (RoPE) [22].

Hierarchical Loss Functions

The training utilizes a multi-grain hierarchical loss function system to enforce different levels of alignment. The system includes several components:

  1. Contrastive learning loss (Lcl) establishes multi-grain spatial audio-text mappings through a purely semantic pair [Esem, Tsem] and a joint macroscopic pair [Eglobal, Tst]. p. 2

    Where the paper says it

    We define two distinct alignment pairs: a purely semantic pair [Esem, Tsem] and a joint macroscopic pair [Eglobal, Tst]. Denoting the contrastive losses for these two pairs as L sem cl and L st cl respectively, the total contrastive loss Lcl is formulated as the sum of these two components.

  2. 3-way spatio-temporal loss (Lst) explicitly models chronological dependencies by forcing the model to differentiate between chronological and spatial errors using the positive text embedding as an anchor. p. 2

    Where the paper says it

    The objective of L text→audio st is to force the model to explicitly discriminate between chronological errors and spatial localization errors, using the positive text embedding, T (i) st , as the anchor. Analogously, we formulate the symmetric audio-to-text loss L audio→text st by using the anchor audio embedding, E (i) st , with negative text embeddings, T˜ (i) st and T˙ (i) st .

  3. Local alignment loss (Llocal) anchors individual acoustic events by aligning hard-encoded audio chunks with their corresponding spatial captions. p. 2

    Where the paper says it

    To mitigate the erosion of fine-grained details inherent in global alignment, we incorporate a local alignment objective. By explicitly aligning the independent hardencoded audio chunks, E ci hard, with their corresponding spatial captions, T ci spa, we establish robust semantic anchors for individual acoustic events.

  4. Feature consistency loss (Lconsist) enforces a Mean Squared Error constraint on soft embeddings using hard anchors to preserve local semantic purity within global contexts. p. 2

    Where the paper says it

    To mitigate these effects, we introduce a feature consistency loss Lconsist. By leveraging the independently encoded hard chunks, E ci hard, as anchors, we enforce a Mean Squared Error (MSE) constraint on the corresponding soft embeddings E ci soft.

Experimental Results and Findings

The ablation study demonstrates that the full framework, CoSTALA (Lcl + Lst + Llocal + Lconsist), achieves the peak performance. Specifically, applying Lconsist without Llocal leads to performance degradation. p. 3

Where the paper says it

Crucially, our ablation study demonstrates a potent synergy between local alignment loss, Llocal, and feature consistency loss, Lconsist. Notably, applying Lconsist without Llocal leads to performance degradation.

The introduction of the 3-way spatio-temporal loss (Lst) yields a significant increase in R@1 by successfully suppressing partially similar candidates. p. 3
Where the paper says it

Building upon this global pool discrimination, the introduction of the 3-way spatio-temporal loss, Lst, yields a significant increase in R@1. This suggests that its rigorous decision boundary successfully suppresses partially similar candidates (e.

Empirical validation shows that relying solely on the semantic audio representation leads to a catastrophic performance decline compared to the joint representation. p. 3
Where the paper says it

As shown in the right section of Table 2, relying solely on semantic audio representation leads to a catastrophic performance decline compared to the joint representation. This empirical finding rigorously demonstrates that while semantic modeling can identify acoustic entities, it inherently lacks the spatial granularity required to disentangle complex multi-event scenarios.

Conclusion

In conclusion, CoSTALA introduces a novel training paradigm for comprehensive understanding of long-duration, multievent spatial audio sequences by integrating explicit temporal modeling with multi-grain hierarchical losses. This work establishes a robust and fine-grained foundational framework for future spatio-temporal audio-language research. p. 3

Where the paper says it

By integrating explicit temporal modeling with a suite of multi-grain hierarchical losses, our approach effectively overcomes the fundamental limitations of existing Audio-Language Models in processing extended spatial contexts. Ultimately, this work establishes a robust and fine-grained foundational framework, paving the way for future spatio-temporal audio-language research.

Generative AI Use Disclosure

The generative AI is used for reviewing grammar, checking format, and polishing manuscript.

Acknowledgements

This work was supported by the Research Development Fund (RDF) of Xi’an Jiaotong-Liverpool University under Grant No. RDF-22-01-084. p. 4

Where the paper says it

Acknowledgements This work was supported by the Research Development Fund (RDF) of Xi’an Jiaotong-Liverpool University under Grant No. RDF-22-01-084.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Transition from coarse-grained contrastive learning to multi-grain hierarchical loss functions allows for explicit modeling of temporal dependencies and anchoring individual acoustic events to preserve their semantic purity. This enables the system to move beyond single-event mapping to fine-grained spatio-temporal reasoning across multiple sequential events.

  2. Decoupling pure acoustic semantic extraction from spatial localization, as inspired by EINV2 dual-branch design, allows the model to leverage specialized encoders for each task. This separation ensures that spatial information and semantic meaning are not conflated during the learning process.

  3. Introducing a 3-way spatio-temporal loss Lst forces the model to explicitly discriminate between chronological and spatial errors by compelling it to differentiate between these two types of mistakes. This capability allows the system to robustly handle combinatorial nuances in complex auditory environments.

  4. Implementing the local alignment loss Llocal establishes robust semantic anchors for individual acoustic events by aligning independent hardencoded audio chunks with their corresponding spatial captions. This dense supervision prevents local features from being submerged by the global sequence context, maintaining high semantic purity.

  5. Incorporating the feature consistency loss Lconsist enforces a Mean Squared Error constraint on soft embeddings using hard chunk anchors, ensuring that soft representations preserve high semantic fidelity within complex global contexts. This regularization mitigates representational collapse during long-sequence learning.

  6. The resulting CoSTALA system can perform comprehensive spatio-temporal audio understanding by jointly optimizing multi-grain representations and temporal reasoning capabilities across different granularities. It can achieve superior spatial retrieval performance compared to models like SALM and T-CLAP by leveraging a unified negative pool encompassing both spatially and temporally inconsistent samples.

  7. The system demonstrates the necessity of joint representations, as relying solely on semantic audio representation leads to catastrophic performance decline when paired with complex spatio-temporal text representations. This proves that spatial granularity is essential for disentangling complex multi-event scenarios.

Abstract

Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.

Related papers