RAG-Audio: Retrieval-Augmented Generation for Faithful Brain-to-Audio Reconstruction

RAG-Audio introduces a retrieval-augmented generation approach to address prior domination in brain-to-audio reconstruction, where a pretrained generator conditioned on a weak neural signal produces realistic but stimulus-inaccurate audio. The paper states: when a powerful pretrained generator receives a weak or noisy brain-derived condition, its learned distribution over plausible audio can outweigh the evidence carried by that condition.

The method decodes fMRI into a semantic CLAP audio embedding, retrieves a matching real-audio exemplar from a memory bank, and initializes the frozen generator's sampling trajectory from that exemplar at an intermediate time while retaining the decoded embedding as conditioning. For diffusion-based generators like AudioLDM, this follows the SDEdit principle of partially noising the exemplar latent before reverse diffusion; for flow-based generators like TangoFlux, it uses deterministic interpolation along the rectified-flow trajectory. The anchoring strength s controls the trade-off between faithfulness and novelty.

On the Brain2Music dataset, RAG-Audio improves 10-way stimulus identification from 0.14–0.18 for direct generation (near the 0.10 chance level) to 0.40–0.43, comparable to retrieval. It also reduces Fréchet Audio Distance by roughly an order of magnitude, from 13.49 to 1.25 for AudioLDM. The paper notes: RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio.

An autoregressive negative control (MusicGen), which lacks an initializable latent trajectory, shows no comparable gain (0.18 to 0.20), attributing the improvement to trajectory initialization rather than retrieval alone. The paper concludes: These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.

Key contributions include: identifying and quantifying prior domination as a failure mode of the decode-then-generate recipe; introducing exemplar anchoring that recovers retrieval-level identification while keeping generation genuinely generative and reducing FAD by 10.8× for AudioLDM and 3.3× for TangoFlux; using MusicGen as a negative control to attribute the recovery to latent-trajectory initialization; and providing an anatomical validity check localizing the decoder signal to auditory cortex (78% of top voxels in bilateral superior temporal gyrus).

Improvements to AI Systems:

1. Add a retrieval-anchored initialization module to any latent-trajectory generative model (diffusion or flow-based). The system will condition generation on a weak or noisy input (e.g., brain signals, low-resolution text, partial sketches) by first mapping the input to a semantic embedding, retrieving the nearest real exemplar from a memory bank, and then partially noising or interpolating that exemplar’s latent to serve as the starting point for reverse diffusion or rectified flow. This directly counters “prior domination” where the generator ignores weak conditioning.

2. Implement an adjustable anchoring strength parameter (s) that dynamically balances faithfulness vs. novelty. The improved system will allow users to set s per task or per sample. At high s, the output stays close to the retrieved exemplar (high fidelity to the condition); at low s, it explores more novel variations. This enables a tunable trade-off, useful for creative generation (e.g., music, images) where both accuracy and diversity are desired.

3. Add a negative-control diagnostic to detect prior domination. The system will include a test mode that runs the same conditioning through an autoregressive model (e.g., MusicGen) that lacks an initializable latent trajectory. If the autoregressive model shows no improvement from retrieval, the system flags that the gain is due to trajectory initialization, not mere retrieval—this helps validate whether the generative architecture is suitable for weak-condition tasks.

4. Integrate a memory-bank retrieval layer that is decoupled from the generator’s weights. The improved system will keep the generator frozen and only add a retrieval module (e.g., a nearest-neighbor search over CLAP embeddings). This allows the system to upgrade its memory bank without retraining the generator, enabling rapid adaptation to new domains (e.g., new brain datasets, new audio styles) by simply swapping the exemplar database.

5. Add an anatomical validity check for brain-decoding systems. For any neural-signal-to-output system, the improved AI will include a post-hoc localization module that verifies whether the decoded features originate from task-relevant brain regions (e.g., auditory cortex for audio). This provides a confidence score and helps reject spurious decodes, improving reliability in neuroimaging applications.

What the improved AI system can do:

• Reconstruct audio from fMRI with 10-way stimulus identification accuracy of 0.40–0.43 (vs. 0.14–0.18 without anchoring), while still generating novel audio rather than replaying recordings.

• Generate music or speech from weak, noisy conditions (e.g., single-trial EEG, low-SNR text) with a 3–10× reduction in Fréchet Audio Distance, producing outputs that are both perceptually realistic and semantically aligned to the input.

• Provide a user-controllable knob to switch between “faithful replay” (high anchoring) and “creative reinterpretation” (low anchoring), useful for assistive music composition or brain-computer interfaces.

• Diagnose whether a given generative architecture is suitable for weak-condition tasks by running an autoregressive negative control, saving development time.

• Validate the neural origin of decoded signals, ensuring that the system is actually reading task-relevant brain activity rather than noise, which is critical for clinical or neuroscientific use.

Brain-to-audio reconstruction is limited by prior domination: when a pretrained generator is conditioned on a weak neural signal, it produces realistic but stimulus-inaccurate audio. We introduce RAG-Audio, which decodes fMRI into a semantic audio embedding, retrieves a matching real-audio exemplar, and initializes the frozen generator's sampling trajectory from that exemplar while retaining the decoded embedding as conditioning. On Brain2Music, RAG-Audio improves 10-way stimulus identification from 0.14 -- 0.18 for direct generation, near the 0.10 chance level, to 0.40 -- 0.43, comparable to retrieval. It also reduces Fr'echet Audio Distance by roughly an order of magnitude, from 13.49 to 1.25 for AudioLDM. RAG-Audio approaches nearest-neighbor retrieval in identification while remaining generative; its higher FAD is expected because retrieval directly replays real audio. An autoregressive negative control, which lacks an initializable latent trajectory, shows no comparable gain, attributing the improvement to trajectory initialization. These results suggest that retrieval-guided initialization can mitigate prior domination in brain-to-audio generation.