CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning".
Jane: Here's the gist of the paper. CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function system.
Tom: The Core Problem Addressed Conventional audio language models (ALMs) struggle to effectively process multi-event audio sequences in daily spatial scenarios because they primarily rely on coarse-grained contrastive learning with global auditory and textual features.
Jane: Current approaches lack the resolution to distinguish multiple sequential events.
Tom: Relying solely on global representations imposes a severe information bottleneck by compressing dynamic, high-dimensional content into a single holistic vector, which often induces “context drift” and fails to resolve the distinct semantic identity of individual events.
Paper summary: Tom: So looking at the CoSTALA paper, it seems their main contribution is proposing this specific training paradigm which moves from global alignment to fine-grained spatio-temporal reasoning using that multi-grain loss system. They’re addressing the issue of context drift caused by compressing dynamic audio into a single vector.
Jane: Exactly, and the authors are showing through their results that decoupling and modeling those temporal dependencies explicitly makes a huge difference, especially when you look at how much better it performs on tasks involving multiple sequential events in space.
Lu: The paper establishes this framework as a robust way to build systems capable of compositional reasoning in audio-language tasks by using those different levels of alignment—semantic, local, and temporal—to guide the training process.
Meng: From a deployment standpoint, it’s interesting that they found that the feature consistency loss actually helps maintain local semantic purity even when dealing with global contexts during training. That suggests a stability benefit we might see in production models if we can manage the complexity of those anchors.
Tom: So, to put it simply, CoSTALA is about transitioning from just matching general audio to language to actually understanding the sequence and spatial arrangement of events within that audio. The authors are providing a new training blueprint for achieving that level of detail in complex audio scenes.
Conclusion: Tom: So we've seen how CoSTALA uses that complex system of losses to break down audio understanding into smaller pieces, and now we're wrapping up this paper titled CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning.
Jane: It really is about taking those different levels of alignment—the semantic pairs and the temporal sequences—and stitching them together in a new way to get a finer picture of what's happening in a long audio scene.
Lu: The core idea they push is moving away from just matching the whole audio clip to text, and instead forcing the model to learn how individual sounds relate spatially and temporally on their own.
Meng: From my side, it’s interesting that they show that adding those local alignment losses really makes a difference in performance when you look at real-world scenarios where events are close together.
Tom: Yeah, so what does this mean for us? If we can train models this way, we might finally get audio systems that actually understand the sequence of things happening, not just the overall vibe.
Jane: Exactly. It means future audio language models could handle complex stories or scenes where multiple distinct sounds happen one after another in a specific place.
Lu: The implication is building representations that are much richer, because instead of one big blob of information, you've got structured knowledge about where and when things occur.
Meng: But the catch is how they set up those losses; if you don't get the anchoring right for each individual sound, the whole system might just collapse into a global average again.
Tom: Right, so it’s not just about making one model better; it’s about building a training structure that forces the model to learn composition instead of just memorizing patterns.
Jane: And that's what makes this paper significant because it gives us a concrete blueprint for how to build those more granular models going forward.
Peiwei Ren, Jinbo Hu, Fang Kang, Shan Liang, Yin Cao
Xi’an Jiaotong Liverpool University · MiLM Plus, Xiaomi Inc. · Ã Center for Machine Vision and Signal Analysis, University of Oulu, Finland · 旡 Institute of Acoustics, Chinese Academy of Sciences
cs.SD
Submitted: 2026-08-25
Updated: 2026-08-25
DOI: 10.21437/Interspeech.2026-1110
Code: https://github.com/Cell778/CoSTALA26
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 83/100
The gist: CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function…
Key concepts
- Hierarchical Loss Functions
- This is a system of multiple training rules used during model development. Instead of one simple rule, CoSTALA uses several losses—like contrastive loss and spatio-temporal loss—that work at different levels. This allows the model to learn both broad relationships and very fine details about how sounds relate to each other in space and time.
- Spatio-Temporal Reasoning
- This refers to the ability of an AI to understand not just what a sound is, but exactly where it is coming from (spatial) and when it happened (temporal). CoSTALA aims for this by decoupling the audio processing so the model can focus separately on location and sequence, rather than treating everything as one single block of information.
- 3-Way Spatio-Temporal Loss (Lst)
- This specific loss function forces the model to learn a crucial distinction: whether an error is due to incorrect timing or incorrect spatial placement. By using the correct text embedding as a reference, this loss helps the model suppress confusing audio candidates that might look similar but are temporally or spatially wrong.
- Local Alignment Loss (Llocal)
- This component focuses on anchoring individual sound events to their specific descriptions. It aligns short, specific chunks of raw audio directly with their corresponding spatial captions. This ensures that the model learns the exact correspondence between a single sound and its precise location description.
Terminology
Summary
CoSTALA proposes a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning by constructing a multi-granularity hierarchical loss function system. p. 0 To overcome these
limitations, we propose CoSTALA—a novel training paradigm
that transitions from purely global alignment to fine-grained
spatio-temporal reasoning. By constructing a multi-granularity
hierarchical loss function system, we achieve explicit modeling
of temporal dependencies, and successfully anchors individual
acoustic events to preserve their semantic purity.Where the paper says it
The Core Problem Addressed
Conventional audio language models (ALMs) struggle to effectively process multi-event audio sequences in daily spatial scenarios because they primarily rely on coarse-grained contrastive learning with global auditory and textual features. Current approaches lack the resolution to distinguish multiple sequential events. p. 0 However, in daily spatial scenarios, they still cannot
effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning
with global auditory and textual features, lacking the resolution
to distinguish multiple sequential events. This coarse-grained approach imposes a severe information bottleneck by compressing dynamic,
high-dimensional content into a single holistic vector. Such compression often induces “context drift”, where the model averages
out precise local semantics and consequently fails to resolve the
distinct semantic identity of individual events.Where the paper says it
Where the paper says it
The CoSTALA Architecture
To achieve fine-grained spatio-temporal reasoning, CoSTALA explicitly decouples and models the temporal dependencies of spatial events. The architecture integrates a RoBERTa-based text encoder with a hierarchical audio backbone driven by HTSAT. p. 1 To implement the proposed paradigm, the CoSTALA architecture explicitly maps these decoupled attributes to a shared
latent space. We integrate a RoBERTa-based text encoder [18]
with a hierarchical audio backbone driven by HTSAT [19], as
shown in Figure 1. We integrate a RoBERTa-based text encoder [18]
with a hierarchical audio backbone driven by HTSAT [19], as
shown in Figure 1. Inspired by the EINV2 dual-branch design [20], we strictly decouple pure acoustic semantic extraction
from spatial localization. Inspired by the EINV2 dual-branch design [20], we strictly decouple pure acoustic semantic extraction
from spatial localization. Crucially, to precisely capture the temporal sequence of acoustic events, we incorporate a dedicated
Transformer-based [21] temporal encoder equipped with Rotated
Position Embedding (RoPE) [22].Where the paper says it
Where the paper says it
Where the paper says it
Hierarchical Loss Functions
The training utilizes a multi-grain hierarchical loss function system to enforce different levels of alignment. The system includes several components:
-
Contrastive learning loss (Lcl) establishes multi-grain spatial audio-text mappings through a purely semantic pair [Esem, Tsem] and a joint macroscopic pair [Eglobal, Tst]. p. 2
Where the paper says it
We define two distinct alignment pairs: a purely semantic pair [Esem, Tsem] and a joint macroscopic pair [Eglobal, Tst]. Denoting the contrastive losses for these two pairs as L sem cl and L st cl respectively, the total contrastive loss Lcl is formulated as the sum of these two components.
-
3-way spatio-temporal loss (Lst) explicitly models chronological dependencies by forcing the model to differentiate between chronological and spatial errors using the positive text embedding as an anchor. p. 2
Where the paper says it
The objective of L text→audio st is to force the model to explicitly discriminate between chronological errors and spatial localization errors, using the positive text embedding, T (i) st , as the anchor. Analogously, we formulate the symmetric audio-to-text loss L audio→text st by using the anchor audio embedding, E (i) st , with negative text embeddings, T˜ (i) st and T˙ (i) st .
-
Local alignment loss (Llocal) anchors individual acoustic events by aligning hard-encoded audio chunks with their corresponding spatial captions. p. 2
Where the paper says it
To mitigate the erosion of fine-grained details inherent in global alignment, we incorporate a local alignment objective. By explicitly aligning the independent hardencoded audio chunks, E ci hard, with their corresponding spatial captions, T ci spa, we establish robust semantic anchors for individual acoustic events.
-
Feature consistency loss (Lconsist) enforces a Mean Squared Error constraint on soft embeddings using hard anchors to preserve local semantic purity within global contexts. p. 2
Where the paper says it
To mitigate these effects, we introduce a feature consistency loss Lconsist. By leveraging the independently encoded hard chunks, E ci hard, as anchors, we enforce a Mean Squared Error (MSE) constraint on the corresponding soft embeddings E ci soft.
Experimental Results and Findings
The ablation study demonstrates that the full framework, CoSTALA (Lcl + Lst + Llocal + Lconsist), achieves the peak performance. Specifically, applying Lconsist without Llocal leads to performance degradation. p. 3 Crucially, our ablation study demonstrates a
potent synergy between local alignment loss, Llocal, and feature
consistency loss, Lconsist. Notably, applying Lconsist without
Llocal leads to performance degradation. Building upon this global pool discrimination, the introduction of the 3-way spatio-temporal loss, Lst, yields a significant increase in R@1. This suggests that its rigorous decision
boundary successfully suppresses partially similar candidates
(e. As shown in the right section
of Table 2, relying solely on semantic audio representation leads
to a catastrophic performance decline compared to the joint
representation. This empirical finding rigorously demonstrates
that while semantic modeling can identify acoustic entities, it
inherently lacks the spatial granularity required to disentangle
complex multi-event scenarios.Where the paper says it
Where the paper says it
Where the paper says it
Conclusion
In conclusion, CoSTALA introduces a novel training paradigm for comprehensive understanding of long-duration, multievent spatial audio sequences by integrating explicit temporal modeling with multi-grain hierarchical losses. This work establishes a robust and fine-grained foundational framework for future spatio-temporal audio-language research. p. 3 By integrating explicit temporal
modeling with a suite of multi-grain hierarchical losses, our
approach effectively overcomes the fundamental limitations of
existing Audio-Language Models in processing extended spatial contexts. Ultimately, this work establishes a robust and
fine-grained foundational framework, paving the way for future
spatio-temporal audio-language research.Where the paper says it
Generative AI Use Disclosure
The generative AI is used for reviewing grammar, checking format, and polishing manuscript.
Acknowledgements
This work was supported by the Research Development Fund (RDF) of Xi’an Jiaotong-Liverpool University under Grant No. RDF-22-01-084. p. 4 Acknowledgements
This work was supported by the Research Development Fund
(RDF) of Xi’an Jiaotong-Liverpool University under Grant No. RDF-22-01-084.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Transition from coarse-grained contrastive learning to multi-grain hierarchical loss functions allows for explicit modeling of temporal dependencies and anchoring individual acoustic events to preserve their semantic purity. This enables the system to move beyond single-event mapping to fine-grained spatio-temporal reasoning across multiple sequential events.
-
Decoupling pure acoustic semantic extraction from spatial localization, as inspired by EINV2 dual-branch design, allows the model to leverage specialized encoders for each task. This separation ensures that spatial information and semantic meaning are not conflated during the learning process.
-
Introducing a 3-way spatio-temporal loss Lst forces the model to
explicitly discriminate between chronological and spatial errors
by compelling it to differentiate between these two types of mistakes. This capability allows the system to robustly handle combinatorial nuances in complex auditory environments. -
Implementing the local alignment loss Llocal establishes
robust semantic anchors for individual acoustic events
by aligning independent hardencoded audio chunks with their corresponding spatial captions. This dense supervision prevents local features from being submerged by the global sequence context, maintaining high semantic purity. -
Incorporating the feature consistency loss Lconsist enforces a Mean Squared Error constraint on soft embeddings using hard chunk anchors, ensuring that
soft representations preserve high semantic fidelity within complex global contexts.
This regularization mitigates representational collapse during long-sequence learning. -
The resulting CoSTALA system can perform comprehensive spatio-temporal audio understanding by jointly optimizing multi-grain representations and temporal reasoning capabilities across different granularities. It can achieve superior spatial retrieval performance compared to models like SALM and T-CLAP by leveraging a unified negative pool encompassing both spatially and temporally inconsistent samples.
-
The system demonstrates the necessity of joint representations, as relying solely on semantic audio representation leads to
catastrophic performance decline
when paired with complex spatio-temporal text representations. This proves that spatial granularity is essential for disentangling complex multi-event scenarios.
Abstract
Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.
Related papers
- Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play
- MuLaCover: Flexible Cover Song Generation via Symbolic Tonal Control
- Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems