Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

arXiv:2610.11846 · cs.CV, cs.LG, cs.MM, cs.SD, eess.IV · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion".

Jane: The gist The dominant pipeline uses a frozen multimodal foundation model (e.g.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today titled Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion by Praveen and his team. Basically, they're tackling how to label video segments with events, even classes they never saw during training.

Jane: That’s right, and the main idea is that instead of using a standard way to combine visual information and audio information to pick an event class, they use complex numbers for that fusion.

Lu: It's interesting because they are replacing the existing scalar fusion rule—the one that just takes a geometric mean or a weighted average—with something much richer called a complex-valued neural network, or CVNN.

Meng: So instead of getting one number from the visual similarity and another from the audio similarity, they are creating two complex similarities for every segment against every class: visual–text and audio–text.

Tom: Exactly, and then they compute these two cosine similarities for each segment against each candidate class name. That feeds into their core idea of using this complex-valued neural network to learn how to fuse them.

Jane: What makes this approach different is that they treat the standard representation of each modality as the real part of a complex value, and they use a paired companion stream to supply the imaginary part.

Lu: For the visual side, the real part comes from the RGB content itself, but they add an imaginary part using an iHSV transform of that RGB frame before feeding it into a frozen vision encoder.

Meng: And for audio, it's symmetrical; the real part is taken from the log-mel magnitude content from a frozen audio encoder, and they use a CycleGAN-translated phase spectrogram to supply the imaginary part, which also feeds into that same frozen audio encoder.

Tom: They actually have two main ways to set this up: either a two-stream fusion where they package both modalities into one complex channel—real for visual similarity and imaginary for audio similarity—or a four-stream extension.

Jane: That four-stream extension is where they enrich each modality with those paired companion streams, and they stack those as two complex channels into the same ComplexCNN structure.

Paper summary: Lu: The ComplexCNN itself uses five layers, where the input to any layer is treated as a complex feature map x equals xR plus iX I, and the learnable kernel W equals WR plus iWI >

Meng: And they define a complex convolution operation where Re(W star x) equals WR star xR minus WI star xI, and Im(W star x) equals WR * xI plus WI * xr, which is a real two-D convolution over the time and class grid <ref:2610.11846#pg1>.

Tom: After those five layers of ComplexCNN, they get this final output magnitude, which they use to get per-segment class scores P hat t in R T times C plus one.

Jane: These scores are then min–max rescaled along the class axis to give you the final per-segment class scores. That's how the model spits out its decision for each segment and each possible event category.

Lu: Now, looking at what they trained, they kept things frozen, which means modules like the ImageBind encoders and the CycleGAN generator are trained offline.

Meng: The training only focuses on the trainable components, which include four temporal-attention blocks—Φv, ΦI v, Φa, and Φφ—and that ComplexCNN itself.

Tom: The objective for training is a per-segment cross-entropy loss L equals minus sigma log P hat t y t >

Jane: So the paper shows that the four-stream complex model achieves state-of-the-art results on both open set audio–visual event localization benchmarks, reaching sixty-six point five/fifty-nine point one/fifty-four point one percent accuracy for segments, F1 score for segments, and Event F1 score on the open split of OV-AVEBench.

Lu: And when they look at the isolated contributions by doing separate ablations, they show that complex parameterization is the main driver of improvement on both datasets.

Meng: Specifically, real to complex at two streams yields a gain of seven point nine Event F1 on OVAVEBench > <ref:2610.11846#pg1>

Tom: Adding those auxiliary streams—the iHSV for vision and the CycleGAN-phase for audio—adds a smaller, complementary gain, lifting unseen Event F1 by zero point eight on OV-AVEBench >

Paper summary: Jane: The paper also notes that the CycleGAN translation step for audio phase actually outperforms using raw IF spectrograms, improving every metric on OV-AVEBench by one point eight/two point four/two point two for accuracy, segment F1, and event F1 >

Lu: They also compared this against AVEOV and found the CycleGAN translation for audio phase improved every metric by one point nine on OV-AVEBench and tied on Segment F1 on AVEOV > <ref:2610.11846#pg1>

Meng: The authors conclude that the four-stream approach outperformed the two-stream approach on both datasets, although they admit its complexity might lead to lesser cross-dataset transfer capabilities > <ref:2610.11846#pg2>

Tom: What this means for us is that complex fusion is clearly the dominant element here, but those auxiliary streams add a smaller boost.

Jane: So if you think about what this does for someone who only listens to the show, it means we’re moving towards systems that can identify events in video and audio they weren't specifically trained on, which is key for open-vocabulary applications.

Lu: The paper proposes two layered architectural contributions: complex-valued fusion and a four-stream extension >

Meng: It also points out that this method is competitive with the dedicated closed set method AVSGN even when restricted to the closed set regime >

Tom: This work establishes a new state of the art on both OV-AVEL benchmarks, showing that complex parameterization is what gives them the lead in these open-vocabulary settings.

Jane: It confirms that complex fusion and auxiliary streams are complementary ingredients rather than substitutes when you compare them against real fusion at matched parameters >

Lu: The paper provides code available at https://github.com/AnirudhPraveen/Complex-Valued-OV-AVEL >

Meng: So, if we look forward, the authors suggest that the four-stream approach might be too complex for some scenarios regarding cross-dataset transfer > <ref:2610.11846#pg2>

Tom: It’s a trade off, really; you get these high accuracy numbers, but you might sacrifice some versatility when moving the system to a totally different set of data.

Jane: Overall, Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion shows that complex modeling provides richer representations than traditional real-valued networks in this specific multimodal fusion task >

Conclusion: Tom: So we're wrapping up this look at "Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion." Basically, they took standard video and audio data and fused them using complex numbers to identify events that weren't in their original training set.

Jane: That's right, the core idea is using a complex network—a CVNN—to combine visual and audio information instead of just using simple addition or averaging of real numbers.

Lu: They use this approach to create two different types of similarity scores: one for the visual part and one for the audio part, both operating in this complex space.

Meng: And they enrich each input with extra streams—like an iHSV transform for the picture and a CycleGAN phase spectrogram for the sound—to give those complex numbers more detail to work with.

Lalam: From a model perspective, it shows that when you feed richer, paired information into a complex structure, the resulting representations become much more nuanced for open-vocabulary tasks.

Tom: The authors found this four-stream setup gets them state-of-the-art results on these audio visual event localization benchmarks.

Jane: They showed that the four stream approach outperformed the two stream version across both datasets, meaning adding those extra streams actually helps the system perform better overall.

Lu: But they also noted that this complex model, while powerful for finding new events, might be a bit too heavy or complex for some kinds of cross-dataset transfers.

Meng: So the practical reality is that you get high accuracy on unseen events, but you have to consider the complexity when you try to move that system to a completely different type of data.

Tom: It really highlights how much more information we can pack into these fusion models by using complex math instead of just standard real-valued calculations.

Jane: It suggests that for things like event localization, moving from simple real-number fusion to complex representations opens up a whole new way to interpret what we're seeing and hearing in the world.

Lu: And this opens the door for thinking about how other multimodal tasks could benefit from this type of complex feature learning.

PRAVEEN ET AL., Anirudh Praveen, Koteswar Rao Jerripothula, Pratik Joshi, Aveen Dayal, Neela Sawant

IIT Kanpur · Dolby Laboratories

cs.CV, cs.LG, cs.MM, cs.SD, eess.IV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted to British Machine Vision Conference (BMVC) 2026

Code: https://github.com/AnirudhPraveen/Complex-Valued-OV-AVEL

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: The gist The dominant pipeline uses a frozen multimodal foundation model (e.g.

Key concepts

Complex Fusion: This technique treats data from different modalities (like visual and audio) as complex numbers, where the real part represents one type of similarity (e.g., visual–text) and the imaginary part represents another (e.g., audio–text). This richer mathematical representation allows for more nuanced interaction between the inputs than simple real-valued fusion.
CVNN: The Complex-Valued Neural Network is a core component that performs convolution operations on complex feature maps. Instead of standard real convolution, it uses a complex kernel and defines the operation to compute both the real and imaginary components simultaneously, leading to more expressive feature extraction.
Auxiliary Streams: These are extra data streams added to the main modalities. For vision, this involves an iHSV transform; for audio, it involves a CycleGAN-translated phase spectrogram. These companion streams enrich the primary representations by providing complementary information that further boosts localization accuracy.
ComplexCNN: This is a five-layer convolutional network designed to process complex feature maps. The input and the learnable kernels are both complex, allowing the network to learn intricate relationships between visual and audio features within a single, unified complex processing pipeline.

Terminology

Ablation — 5× in this paper · explained in 1 episode
Benchmark — 5× in this paper · explained in 2 episodes
Consistency — 5× in this paper · explained in 3 episodes
Adam — 4× in this paper · explained in 2 episodes
Computer Vision and Pattern Recognition — 4× in this paper
Foundation Model — 4× in this paper · explained in 3 episodes
Cosine Similarity — 3× in this paper · explained in 3 episodes
Embeddings — 3× in this paper · explained in 2 episodes
In European Conference on Computer Vision — 3× in this paper
International Conference on Computer Vision — 3× in this paper
Transformer — 3× in this paper · explained in 2 episodes
Accuracy — 2× in this paper · explained in 1 episode
Contrastive Learning — 2× in this paper · explained in 5 episodes
In International Conference on Learning Representations — 2× in this paper
Ablation Studies — 1× in this paper · explained in 1 episode
Adversarial Training — 1× in this paper · explained in 1 episode
Alignment — 1× in this paper · explained in 1 episode
Core — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Loss Function — 1× in this paper · explained in 2 episodes
Multimodal Fusion — 1× in this paper · explained in 3 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Scaffolding — 1× in this paper · explained in 3 episodes
Semantic Alignment — 1× in this paper · explained in 3 episodes
Winter Conference on Applications of Computer Vision — 1× in this paper

Summary

The gist The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual–text and audio–text > <ref:2610.11846#pg6>

How it works

The paper addresses Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) by replacing the existing scalar fusion rule with a complex-valued neural network (CVNN) to compute similarities, which yields richer representations than real-valued networks > <ref:2610.11846#pg4>

The core idea involves treating each modality's standard representation as the real part of a complex value and using a paired companion stream to supply the imaginary part > <ref:2610.11846#pg6>

For the visual modality, the real part is derived from RGB content, while the imaginary part is supplied by an iHSV transform of an RGB frame, which is then fed into a frozen vision encoder > <ref:2610.11846#pg6>

Symmetrically, for the audio modality, the real part is the log-mel magnitude content from a frozen audio encoder, and the imaginary part is supplied by a CycleGAN-translated phase spectrogram that is fed into the same frozen audio encoder > <ref:2610.11846#pg6>

The architecture can be implemented in two main forms:

Two-stream fusion, which packages both modalities into a single complex channel, where the real part carries visual–text similarity and the imaginary part carries audio–text similarity > <ref:2610.11846#pg6>
Four-stream extension, where each modality is enriched with a paired companion stream (iHSV for vision and CycleGAN-phase for audio), both of which are stacked as two complex channels into the same ComplexCNN > <ref:2610.11846#pg6>

Complex Convolution and Fusion

The fusion network utilizes a five-layer ComplexCNN where the input to a layer is treated as a complex feature map x = xR +ixI, and the learnable kernel is itself complex, W = WR + iWI > <ref:2610.11846#pg6>

The complex convolution operation is defined as Re(W∗ x) = WR ∗ xR − WI ∗ xI and Im(W∗ x) = WR ∗ xI + WI ∗ xr, where the operation denotes real 2-D convolution over the (time,class) grid > <ref:2610.11846#pg6>

The final output magnitude of this five-layer ComplexCNN is used to obtain per-segment class scores Pˆ ∈ R T×(C+1), which are then min–max rescaled along the class axis to yield the final per-segment class scores > <ref:2610.11846#pg6>

Training and Trainable Components

The frozen modules include the ImageBind encoders (Ea, Ev, Et) and the CycleGAN generator (GIF→mel), which are trained offline > <ref:2610.11846#pg6>

The trainable modules consist of four temporal-attention blocks (Φv, ΦI v, Φa, Φφ) and the ComplexCNN itself > <ref:2610.11846#pg6>

The training objective is a per-segment cross-entropy loss L = −∑t logPˆt,yt > <ref:2610.11846#pg6>

Experimental Results and Contributions

The 4-stream complex model achieves state-of-the-art results on both OV-AVEL benchmarks, reaching 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 on the open (unseen-class) split of OV-AVEBench > <ref:2610.11846#pg6>

The two contributions are isolated by separate ablations, showing that complex parameterisation is the dominant lever, with real→complex at 2 streams yielding +7.9 Event-F1 on OVAVEBench > <ref:2610.11846#pg6>

The auxiliary streams add a smaller, complementary gain, as adding iHSV + CycleGAN-phase to the 2-stream complex model lifts unseen Event-F1 by +0.8 on OV-AVEBench > <ref:2610.11846#pg6>

Furthermore, the CycleGAN translation step for audio phase outperforms raw IF spectrograms, improving every metric on (a) OV-AVEBench (+1.8/+2.4/+2.2 Acc/Seg/Eve) and Event-F1 on (b) AVEOV (+1.9; Seg-F1 tied) > <ref:2610.11846#pg6>

The 4-stream approach outperformed the 2-stream approach on both datasets, although it performed worse in one of the cross-dataset transfer experiments > <ref:2610.11846#pg6>

The final architecture demonstrates that complex fusion is the dominant lever, and auxiliary streams add a smaller additive boost > <ref:2610.11846#pg6>

The paper concludes that the two contributions are isolated by separate ablations, showing that complex fusion is the dominant lever (real→complex at 2 streams: +7.9 Event-F1 OV-AVEBench, +15.9 AVE-OV) and the auxiliary streams add a smaller additive boost (adding iHSV + CycleGAN-phase to the 2-stream complex model: +0.8 / +2.3 Event-F1) > <ref:2610.11846#pg6>

The architecture successfully reaches 87.8/82.7/82.8 Acc/Seg/Eve on OV-AVEBench and 42.0/34.4/34.3 Acc/Seg Eve Avg on AVE-OV > <ref:2610.11846#pg6>

The paper proposes two layered architectural contributions for OV-AVEL, focusing on complex-valued fusion and a four-stream extension > <ref:2610.11846#pg6>

The proposed method is competitive with the dedicated closedset method AVSGN even when restricted to the closed-set regime > <ref:2610.11846#pg6>

This work establishes a new state of the art on both OV-AVEL benchmarks > <ref:2610.11846#pg6>

The paper provides code available at https://github.com/AnirudhPraveen/Complex-Valued-OV-AVEL > <ref:2610.11846#pg6>

The authors state that the 4-stream approach outperformed the 2-stream approach on both datasets, but its complexity might lead to lesser cross-dataset transfer capabilities > <ref:2610.11846#pg6>

The paper confirms that the two ingredients are complementary rather than substitutes when comparing complex fusion against real fusion at matched parameters > <ref:2610.11846#pg6>

The authors note that the CycleGAN inherently preserves the audio phase information in the generated mel-spectrogram, which is why it can be reconstructed via cycle-consistency loss > <ref:2610.

Improvements for AI systems

Complex-Valued Fusion for OV-AVEL: Replace scalar fusion rules with a five-layer ComplexCNN [24] that treats visual–text and audio–text similarities as the real and imaginary parts of a complex input, which accounts for the bulk of the gain over the baseline. This allows the model to leverage directional orthogonality between modalities, treating audio information as directionally orthogonal to visual information, analogous to depth.
Four-Stream Modality Enrichment: Implement a four-stream architecture where each modality is enriched with a paired companion—iHSV [29] transform on the visual side and a CycleGAN-translated phase spectrogram on the audio side—to give each modality its own complex (content + structure) pair. This adds a smaller, complementary gain on top of (1).
Phase Information Utilization: Utilize an unpaired CycleGAN to translate the instantaneous frequency spectrogram into the mel manifold, allowing frozen encoders to ingest phase-derived input without retraining. This is crucial because raw phase information is too noisy for magnitude-trained encoders, and this translation allows the encoder to ingest phase-derived input without retraining.
Parameter Efficiency in Fusion: Employ a two-stream packaging where both modalities into one complex channel—real for visual, imaginary for audio, which results in a simpler, more efficient, and redundancy-reduced representation. This structure is proposed because it leverages the intuition that audio information is directionally orthogonal to visual information.
Cross-Dataset Transfer Capabilities: The 2-stream complex fusion maintains a positive margin over the fine-tuned baseline in cross-dataset transfer experiments (+3.6/+8.5 Avg), while the 4-stream variant shows uneven transfer, suggesting that its complicated nature, as it might have over-fitted on the larger dataset it was trained on and ended up exhibiting lesser cross-dataset transfer capabilities.