Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception

summary

The gist

Extended reality (XR) enables controlled yet naturalistic multisensory experiments, but its validity depends on whether virtualized cues are perceptually plausible and lead to behavior comparable to…

In short

This study tested how virtualizing sound in 6DoF XR affects perception and distance judgment. They found that while auditory virtualization caused a systematic underestimation of sound source distance, visual-only virtualization mainly increased uncertainty. The findings suggest that high plausibility alone is insufficient; task-specific behavioral validation is required to confirm real-world equivalence.

Key concepts

6DoF XR Systems
These are immersive virtual reality systems that track a user's position and orientation in three dimensions plus six degrees of freedom (movement). The study used a custom system combining live room acoustics, binaural audio rendering, and visual rendering to create an interactive environment where sound is placed realistically around the user.
Plausibility Assessment
This experiment checks if virtualized sound cues feel realistic compared to real sounds. The researchers found that while the system produced plausible renderings in both VR and AAR, the auditory virtualization condition resulted in significantly lower sensitivity, meaning participants were less able to tell real from virtual sound.
Dynamic Auditory Distance Perception (ADP)
This task measures how accurately users judge the distance to sound sources while moving. The study showed that auditory virtualization caused a systematic bias toward perceiving sound sources as closer than they actually were, which changed depending on how far away the speakers were placed.

Terminology used across episodes

This episode discusses

The paper

Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception · Read on arXiv

Johannes M. Arend, Christian Scheer, Melissa Ramírez, Stefan Weinzierl, Fabian Brinkmann

Aalto University · Technische Universität Berlin · Institute of Computer and Communication Technology TH K¨oln - University of Applied Sciences

Extended reality (XR) enables controlled yet naturalistic multisensory experiments, but its validity depends on whether virtualized cues are perceptually plausible and lead to behavior comparable to the real world. To the best of our knowledge, this is the first study to perceptually evaluate a six-degree-of-freedom (6DoF) XR system that combines real-time room acoustic simulation, binaural rendering, and an audio-visually matched virtual replica of a large laboratory room in terms of both plausibility and behavior. In two experiments with 45 participants, we investigated how auditory and visual virtualization affect plausibility and dynamic auditory distance perception (ADP). A plausibility experiment compared real and virtual sound sources in audio-augmented reality (AAR) and virtual reality (VR), while a locomotion-based ADP experiment assessed distance-judgment accuracy, signed error, and task effort across four virtualization conditions. The system produced plausible binaural rendering in both AAR and VR, with higher plausibility in VR, indicating that perceptual requirements for XR audio depend on the visual reference frame. In the ADP task, any form of virtualization reduced accuracy and increased effort, although performance remained high overall. Virtual audio produced a bias toward nearer perceived source distances, whereas visual-only virtualization reduced accuracy without a significant directional bias. These effects did not add up in full VR. Overall, the findings demonstrate the strong potential of current 6DoF XR systems for interactive perceptual research, while revealing that task-specific behavioral validation may be necessary.

Transcript

Introduction to the show: ident: Electrical Engineering. The day in electrical engineering: signals, systems and the hardware that carries them.

Rosa: We're looking at the paper: "Auditory and Visual Virtualization in 6DoF XR".

Dev: Here's the gist of the paper. Extended reality (XR) enables controlled yet naturalistic multisensory experiments, but its validity depends on whether virtualized cues are perceptually plausible and lead to behavior comparable to the real world The single most informative sentence about the paper's subject and finding is that "Overall, the findings demonstrate the strong potential of current 6DoF XR systems for interactive perceptual research, while revealing that task-specific behavioral validation may be necessary".

Rosa: System Implementation Details The study utilized a custom 6DoF XR system combining liveRAZR real-time room acoustic simulation and binaural rendering with Unreal Engine-based visual rendering The system employed Unreal Engine 5.1.1 for visual rendering and OpenXR and Open Sound Control (OSC) for data distribution For audio, liveRAZR 0.6.0 was used for real-time room acoustic simulation and 6DoF binaural rendering The XR hardware included an HTC Vive Pro Eye HMD without headphones or a HTC Vive Tracker 3.0 attached to the employed headphones combined with a Samsung Galaxy Tab S3 for AAR contexts Binaural signals were presented via AKG K1000 extra-aural headphones connected to a Sennheiser EW IEM G4 wireless audio transmitter The entire system ran on a Windows 10 Pro PC with 64 GB of memory, a 16-core AMD Ryzen 9 5950X CPU, and an NVIDIA GeForce RTX 3070 GPU.

Dev: Plausibility Assessment Findings The plausibility experiment compared real and virtual sound sources in audio-augmented reality (AAR) and virtual reality (VR) The system produced plausible binaural rendering in both AAR and VR, with higher plausibility in VR.

Rosa: The results showed that the mean d′ of the VR condition was 0.16, compared to 0.41 for the AAR condition.

Title and authors: Rosa: So we're looking at this paper called "Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception," and it basically tackles whether these virtual sound cues actually feel real <ref:2611.07612#pg1,Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic>.

Dev: Right, the title itself is pretty direct about testing how much auditory and visual virtualization affects how believable things are in a six-degree-of-freedom extended reality system.

Rosa: Yeah, what interests me is that they aren't just looking at whether it sounds good or not, but specifically how this virtualization impacts something like distance perception when you're moving around.

Taro: I’m curious if they actually test this in a way that shows how the system handles things when the world gets weird, like when someone is walking and the sound comes from a strange direction.

Rosa: They do, and what they found is that while it’s hard to tell if it sounds real—the plausibility assessment—the actual distance judgments got messed up in a specific way depending on whether you used audio or just visuals to fake things.

Dev: That’s the core finding, right? They showed that any form of virtualization generally made performance worse and took more effort when trying to judge distances dynamically.

Rosa: Exactly, and the interesting part is that auditory virtualization caused a systematic bias where people tended to think sound sources were closer than they really were.

Taro: That means if you’re relying on virtual audio cues while moving, you’re systematically underestimating how far away those things actually are.

Dev: And visual-only virtualization was different; it didn't cause that kind of directional bias toward nearer sources, but it still reduced overall accuracy in the distance estimation task.

Rosa: So what they’re saying is that auditory virtualization and visual-only virtualization had different effects on distance perception, and they didn't just add up to create a bigger problem in full virtual reality.

Taro: It seems like the way the audio was virtualized was more impactful for distance estimation than just having visual stuff layered on top.

Dev: That points toward needing a more nuanced approach when designing these systems because you can’t just assume that if one modality works, it helps the other modality perform better in a complex task.

Rosa: So what they suggest next is that you need to move beyond just making things sound plausible and start testing how those virtual cues actually affect the specific perceptual processes you care about.

Taro: Which means if your goal is to study something fundamental, like how the brain perceives distance, you need to validate the system for that specific process rather than just checking if it looks right.

Dev: It sounds like they’re saying plausibility and actual behavioral accuracy aren't exactly the same thing, which is a big distinction for anyone building these kinds of interactive systems.

Rosa: That’s what they’re highlighting, and that means researchers need to use task-specific validation when they want to generalize what they learn about perception.

Taro: So the implication for us is that just making the sound look convincing isn't enough; we have to prove it helps people perform the specific task we're studying correctly.

Dev: Exactly, and that means we can’t just rely on a high sensitivity parameter like d prime alone to say everything about how real the experience feels.

Rosa: So, to wrap up this paper, the authors are suggesting that while these 6DoF XR systems have huge potential for interactive research, they need to be validated against the actual perceptual and behavioral processes they are trying to study <ref:2611.07612#pg1>.

Taro: I just want to emphasize that if you're building something for real-world use or serious cognitive study, testing it in a controlled lab room isn't enough; you have to see how it handles the messiness of a real environment.

Dev: And from an engineering standpoint, this means our safety cases need to account for these kinds of systematic biases that aren't just about latency or tracking errors but about how the sensory input is being virtualized.

Rosa: So, if you’re working on XR audio design right now, this paper shows that you need to think beyond just achieving a high plausibility score and focus on the specific behavioral outcome of your system.

Taro: We should probably start thinking about how we can build validation layers into our systems that specifically check for those types of distance biases before we even deploy them widely.

Dev: That makes sense; it shifts the engineering focus from just making the rendering loop run smoothly to ensuring the perceptual consequences of that rendering are sound.

Rosa: So, the paper "Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception" ultimately tells us that for immersive systems to be useful in research, they need more than just looking convincing; they need to perform correctly on the specific perceptual tasks we set up <ref:2611.07612#pg1,Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic>.

The paper's summary: Rosa: The authors found that while these six-degree-of-freedom systems can be really useful for interactive research, whether it *feels* real depends on whether the virtual cues actually make sense to us.

Taro: So it's not just about if the sound is loud or not, but if our brains are tricked into believing something is closer or farther away than it really is.

Dev: Exactly. They tested plausibility—how convincing the audio and visuals are—and then they tested dynamic distance perception while we were moving along a line.

Rosa: And what they showed was that auditory virtualization caused a specific bias, making us think sound sources were systematically nearer than they actually were, and visual-only changes didn't cause that specific directional bias.

Taro: That’s interesting because it means the way you fake distance with audio is different from how you fake it with visuals.

Dev: Right. The study showed that any kind of virtualization generally made performance worse and took more effort when trying to judge distances dynamically, but auditory virtualization was the biggest problem there.

Rosa: And the authors actually ended up saying that all these effects—the sound stuff and the visual stuff—don't just add up to make a bigger mess in full virtual reality.

Taro: So, if you’re building an immersive system, you can’t just assume that making it look plausible is enough to guarantee the distance judgments will be correct.

Dev: That’s what they are suggesting. Plausibility tells you if the environment looks consistent, but task-specific validation is still necessary when you want to study something fundamental about how we perceive distance in real life.

Rosa: It means researchers can't just check a box saying their system is plausible; they have to prove it works for the actual perceptual process they are studying, whether that’s distance estimation or something else.

Taro: So, the implication is that we need to validate these immersive systems not just for overall plausibility but for the specific things we intend to study or train with them.

Dev: It shifts the focus from just making a cool simulation to ensuring that when you use that system in a lab, it actually produces accurate behavioral results for whatever you're testing.

Rosa: And that leads us to asking how we can build systems that check for those specific biases before we even deploy them widely.

The paper's improvements: Rosa: So, looking at what they found, the authors are suggesting that just checking if something sounds plausible isn't enough to prove its real-world worth in research.

Dev: They're saying that plausibility can confirm the environment is free of obvious artifacts, but task-specific validation is still necessary when you want to study fundamental perceptual processes or generalize behavioral findings.

Taro: That means researchers have to move beyond just making things look convincing and start testing how those virtual cues actually affect the specific perceptual process they are trying to study.

Rosa: Exactly. They’re pointing out that plausibility and actual behavioral accuracy aren't exactly the same thing, which is a big distinction for anyone building these kinds of interactive systems.

Dev: So, if you’re working on an immersive setup, you can't just rely on a high sensitivity parameter to say everything about how real the experience feels across different tasks.

Taro: That means we need validation layers into our systems that specifically check for those types of distance biases before we even deploy them widely.

Rosa: And that implies that when we look at these 6DoF XR systems, the goal shouldn't just be achieving a high plausibility score, but ensuring they perform correctly on the actual tasks we set up <ref:2611.07612#pg1>.

Dev: It means shifting the focus from just making the rendering loop run smoothly to ensuring that the perceptual consequences of that rendering are sound for whatever you’re testing.

Taro: So, if you’re building something for serious cognitive study, you need to prove it works on those specific tasks rather than just checking if it looks right in a controlled room.

Rosa: That's the point. The future work they suggest is to examine whether similar behavioral results emerge with other room types or more detailed acoustic models, but that’s still lab work for now.

Dev: And they also flag a limitation: the study was done in one big lab room with specific geometry, which might mean those findings don't hold up everywhere.

Taro: So, if you're building something like this today, you need to think about how it handles the world misbehaving outside that perfect lab setting.

Rosa: That’s what we need to focus on next—how do we build validation for these systems when they aren't perfectly controlled environments?

Conclusion: Rosa: So we're wrapping up on "Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception <ref:2611.07612#pg1,Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic>." It boils down to this—the paper shows that even when a system sounds convincing, if it’s not validated for the specific task you’re doing, its performance might still be off.

Dev: Right. The main implication is that we can't just chase high plausibility numbers in these six-degree-of-freedom systems; we have to check if they actually work for the perceptual process we care about.

Taro: I agree with that, it means the system needs to be validated against what it’s supposed to do when things get messy in the real world, not just checking a score in a clean lab setting.

Rosa: That’s right. The paper emphasizes that plausibility and behavioral validity aren't interchangeable; researchers need task-specific validation when they want to generalize what they learn about perception.

Dev: It’s about making sure the controls engineer knows what the safety case is, not just that the rendering loop runs smoothly with low latency.

Taro: I just want to push on how this changes things for autonomy research; it means we need validation for failure modes, not just for when everything works perfectly in a straight line.

Rosa: So, if you’re building an immersive system, the lesson is that you need to go beyond checking if it looks right and prove it performs correctly on the specific distance estimation task.

Dev: That means we have to integrate that validation layer into our design from the start, so we're not just focusing on rendering fidelity but also perceptual accuracy under dynamic conditions.

Taro: It makes sense for autonomy; if a system misjudges distance because of how it virtualizes sound, that’s a failure mode you need to address gracefully.

Rosa: Exactly. We can't just assume the sensory input is fine just because the visualization looks good on screen or in the simulation.

Dev: So, for anyone working with XR audio right now, don't just aim for a high sensitivity number; focus on proving that auditory virtualization doesn't introduce systematic biases in distance judgment.

Taro: And that leads us to thinking about those future directions they mentioned, like testing this on people with hearing difficulties or in different room types.

Rosa: We’ll be talking about those more next time, but for now, remember that "Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic Auditory Distance Perception" is a great starting point for figuring out what makes a system truly usable in research <ref:2611.07612#pg1,Auditory and Visual Virtualization in 6DoF XR: Effects on Plausibility and Dynamic>.

More episodes

← Home