Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

The gist The proposed Queryguided Spatial–Temporal–Frequency (QSTar) interaction method enhances audio–visual understanding for music audio-visual question answering by incorporating query-guided clues and exploiting spatial, temporal, and frequency domain characteristics.

How it works

The QSTar method is designed to jointly consider query guidance for both audio and visual feature learning and enhance cross-modal interactions across multiple dimensions. The core of the method involves a query-guided multimodal correlation module that refines audio and visual features conditioned on the question from the early stage. This module consists of three components: modality-specific self-enhancing, cross-modal guidance capturing, and information propagation.

The proposed spatial–temporal–frequency interaction module further enriches the query-guided features by emphasizing semantic relevance across spatial, temporal, and frequency dimensions. This module includes two submodules:

** Spatial–Temporal Interaction (STI): This submodule selectively refines the previously obtained query-guided features by enhancing the patch-level visual features Fp to better align fine-grained spatial details with the query-guided audio context. It also computes a temporal interaction between the query-guided audio and visual features by computing the dot product between F′aq and F′vq, followed by a softmax function. The outputs are then integrated by concatenating the corresponding features after reshape, followed by a Feed-Forward Network. This fusion enables the model to yield refined spatial–temporal visual features conditioned on the query. **

** Temporal–Frequency Interaction (TFI): This submodule refines the audio representation by utilizing a pretrained Audio Spectrogram Transformer (AST) Gong et al. (2021) to extract rich frequency-aware features from the audio waveform, denoted as Fast. It introduces a frequency-wise attention over the AST features to explicitly emphasize frequency bands that are most relevant for query-guided instrument recognition. The final audio features F′ast and F′aq are then aligned and integrated by concatenating them and applying a convolutional fusion block composed of two convolutional layers with batch normalization and ReLU activation. This fusion allows the model to effectively associate discriminative spectral patterns with the corresponding instruments, producing more informative and query-guided audio features. **

Query Guidance and Reasoning

The Query Context Reasoning (QCR) block is employed to refine and fuse features before prediction, inspired by prompting techniques. This block uses prompting to incorporate linguistic context and task-specific constraints for more precise reasoning. Specifically, it first encodes context-related keywords using the same CLIP text encoder, yielding prompt embeddings denoted as Fprompt. These are concatenated and passed through a multi-head self-attention unit to produce the query context feature Fqc. Next, cross-attention is applied using Fqc as the query to guide the refinement of visual and audio features Fvi and Fai, respectively. This process enables the model to extract the most informative audio and visual cues under the linguistic query context. For final answer prediction, a simple multimodal fusion and classification is performed, where the fused representation is obtained by Fav = FC(Concat(Ffv, Ffa)). The final feature e is then used to predict the final answer from a predefined vocabulary set.

Experimental Validation and Contributions

Extensive experiments conducted on several AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The main contributions are summarized as follows:

** We propose QSTar, a novel framework that integrates query guidance throughout the entire pipeline to refine modality-specific features. By embedding linguistic context early, QSTar enhances both audio and visual representations in a question-aware manner, enabling more precise cross-modal reasoning. **

** We introduce a fine-grained interaction module that emphasizes semantic cues across spatial, temporal, and frequency dimensions. This design enhances discriminative understanding, especially in polyphonic scenarios where subtle audio or visual cues are critical. **

** We design a reasoning block that injects task-aware linguistic context through prompting to guide final predictions, improving semantic alignment between the question and multimodal features. **

Quantitative results on the MUSIC-AVQA benchmark show that QSTar achieved state-of-the-art results across most question types, outperforming TSPM and QA-TIGER by 2.19% and 1.36% in overall accuracy, respectively. Notably, QSTar surpassed existing AVQA approaches that primarily rely on visual processing techniques such as frame selection or object-level perception. The ablation studies confirm the effectiveness of each module, showing that removing STI significantly impacted performance, and eliminating the TFI module caused a sharp decrease in Audio QA and Audio-Visual QA performance.

Conclusion

In this work, we propose a Query-Guided Spatial–Temporal–Frequency Interaction (QSTar) method for the music AVQA task. We design a query-guided multimodal correlation module that incorporates query guidance into audio–visual feature learning to enhance relevant representations. We also propose a spatial, temporal, and frequency interaction module to effectively align audio, visual, and textual modalities across these dimensions. Furthermore, a prompting-based query reasoning block is employed to incorporate linguistic context before final prediction. Extensive experiments on the standard AVQA datasets demonstrate that QSTar consistently outperforms state-of-the-art methods, while ablation studies confirm the effectiveness of its components. The model demonstrates strong reasoning and robust performance even in challenging cases involving rare instruments. Future work will explore integrating dynamic prompting or large language model-based reasoning for more flexible context modeling. The approach is also planned to be extended to handle open-ended questions beyond the current predefined setup. The model will also incorporate object-level and relational reasoning for fine-grained visual tasks. Expanding the approach to other multimodal QA benchmarks will further validate its generalizability. The proposed method achieves state-of-the-art results on the MUSIC-AVQA benchmark. The model's ability to interpret diverse audio–visual cues and generate accurate answers is showcased in the qualitative results. The model's ability to localize query-relevant instruments across space, time, and frequency enables accurate reasoning in complex musical scenes. The model demonstrates strong reasoning and robust performance even in challenging cases involving rare instruments. The results confirm the effectiveness of its components through ablation studies. The proposed method achieves state-of-the-art results across most question types, outperforming TSPM and QA-TIGER by 2.19% and 1.

1. Spatial–Temporal–Frequency Interaction Module (STI) enhancement: The system can better distinguish instruments by refining patch-level visual features using query-guided audio context, as shown in Equation (5), which encourages the model to focus on sounding regions that correspond to the question.

2. Temporal–Frequency Interaction (TFI) module refinement: The system can capture subtle timbral differences in polyphonic scenarios by utilizing a pre-trained Audio Spectrogram Transformer (AST) and frequency-wise attention, as detailed in Equations (8), (9), and (10), which allow the model to explicitly emphasize frequency bands that are most relevant for query-guided instrument recognition.

3. Query Context Reasoning Block (QCR) integration: The improved system can perform more precise reasoning by incorporating linguistic context via prompting, as the QCR block uses prompt-based conditioning Khattak et al. (2023) to guide refinement, leading to better semantic alignment between the query and multimodal features.

4. Early-stage query guidance optimization: By integrating query guidance at three stages—beginning (QGMC), middle (TFI via question embeddings), and final (QCR via prompting)—the system can achieve improved performance, as ablation studies show that early query-guided refinement helps identify representative audio and visual features (Table 3).

5. Handling longer video sequences: Future extensions should focus on integrating dynamic prompting or large language model-based reasoning to enable more flexible context modeling, allowing the model to handle longer videos that require temporal abstraction or memory mechanisms beyond fixed-length segments.

Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.