EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR".

Jane: Here's the gist of the paper. The gist The EyeNavGS dataset and its accompanying software tools provide valuable resources for advancing research in 6-DoF viewport prediction, adaptive streaming, 3D saliency, and foveated rendering for 3DGS scenes> EyeNavGS Dataset Overview The paper introduces EyeNavGS, the first publicly available 6-DoF navigation dataset featuring traces from 46 participants exploring twelve diverse, real-world 3DGS scenes> The dataset was collected at two sites, using the Meta Quest Pro headsets, recording the head pose and eye gaze data for each rendered frame during free world standing 6-DoF navigation> For each of the twelve scenes, we performed careful scene initialization to correct for scene tilt and scale, ensuring a perceptually-comfortable VR experience> The dataset includes per-frame head pose and eye gaze data, captured with Meta Quest Pro headsets during free world standing exploration> Record-n-Replay Software The authors release an open-source software, a fork of the SIBR viewer

8: for 3DGS, enhanced with record-and-replay functionality> This fork includes capabilities for recording user traces and replaying these traces frame-by-frame for visualization, video generation, and detailed offline analysis> In the Record Mode, our modified OpenXR module captures fine-grained, per-frame data during a user’s VR session> For each rendered frame and for each eye (left and right), we record a comprehensive set of parameters: the field-of-view (FOV), eye position, head orientation (as a quaternion), as well as the eye gaze position and gaze orientation quaternion> In the Replay Mode, the recorded trace is parsed and the logged data is injected line-by-line into the rendering pipeline, overriding the HMD pose information> To generate video output, rendered frames are captured directly from GPU memory using OpenGL’s glGetTexImage API> The replay mode generates two separate videos, one for each eye, that precisely replicate the original stereoscopic experience> Scene Initialization and Calibration To prepare each trained 3DGS scene for immersive exploration, we correct each raw scene to align with human assumptions about the physical world> These corrections include i) correcting the initial quaternion to fix scene tilt, ii) selecting a per-scene scale factor to ensure objects match their real-world proportions, and iii) establishing an example starting viewing position> Scene Tilt and Orientation Correction involves applying a corrective transformation when each scene is loaded, setting the initial quaternion via the poseInReferenceSpace member of the XrReferenceSpaceCreateInfo structure> To find the amount of scene tilt, we implemented a robust procedure using Blender

7: and the KIRI Engine add-on

24: to insert a reference plane perpendicular to the Y axis (which aligns with gravity) and manually adjust the scene’s orientation> Scene Scale Calibration involves applying a per-scene scale factor at runtime that maps movements in real-world metric measurements to the scene’s virtual units> We introduced dimensionally accurate reference objects, e.g., a 1-meter cube, into each scene by comparing known dimensions from the real scene to their 3DGS representations> Data Output and Utility Tools The recorded traces are stored in a hierarchical folder structure where each top-level folder corresponds to the specific 3DGS scenes (e.g., “train”) and within each scene, each user trace is stored as a csv file named as: user scene.csv> Table 2 outlines the structure of the recorded csv traces, which includes columns such as FOV1 (rad), FOV2 (rad), Pos X, Pos Y, Pos Z Eye position (i.e., head position offset by half of IPD) in the world space, and GazePos X,Y,Z Eye gaze position in the world space> We provide a utility to convert these traces to “physical stage coordinates”, which represent the user’s movements in the physical world on a 1:1 metric scale> This conversion undoes the scene initialization transforms including the scene tilt correction, scene scaling, and initial view positioning> Compatibility with Other Frameworks is ensured by including two utility tools: csv2json.cpp for converting recorded traces to.json format in the correct coordinate system and json2csv.cpp for convert.json files exported from other frameworks to.csv format, also in the correct coordinate system> Dataset Analysis and Use Cases The EyeNavGS dataset offers opportunities for research in several areas> 6-DoF Viewport Prediction and Streaming Optimization can be informed by the fine-grained head pose and eye gaze of reconstructed real-world scenes to develop 6-DoF viewport prediction algorithms that inform the design of adaptive media streaming algorithms> 3D Saliency and Saliency in VR can be studied by aggregating per-frame eye gaze information across participants to create groundtruth 3D saliency maps of the 3DGS scenes> Foveated Rendering Optimization is facilitated by the per-frame eye gaze traces collected during free world standing 6-DoF navigation, allowing for evaluation and optimization of real-world performance of these foveated rendering techniques> The collection involved a total of 46 participants across two sites: 22 at RU and 24 at NTHU> Participants explored twelve scenes for one minute each, in the free world standing (fws) mode, with a one-minute break between each scene exploration to allow for participant rest and for the system to load the next scene> The dataset captures stereoscopic pose, field-of-view, and eye gaze traces> The EyeNavGS dataset focuses on capturing detailed and realistic behavior by collecting user traces in scenes of real-world environments, carefully calibrated (tilt, scale, and starting viewpoint) to maintain realism> Our dataset captures stereoscopic pose, field-of-view, and eye gaze information> Multi-site data collection (46 participants across two institutions) ensures a diverse population of participants> The realism, detail, and diversity of EyeNavGS fills a critical gap for immersive media researchers, allowing user-centric evaluation in core areas such as streaming, rendering, and compression> Page 1 --- EyeNavGS: A 6-DoF Navigation Dataset and Record-n-Replay Software for Real-World 3DGS Scenes in VR<ref:2506.02380#pg2> 3D Gaussian Splatting (3DGS) is an emerging media representation that reconstructs real-world 3D scenes in high fidelity, enabling 6-degrees-of-freedom (6-DoF) navigation in virtual reality (VR)<ref:2506.02380#pg2> However, developing and evaluating 3DGS-enabled applications and optimizing their rendering performance, require realistic user navigation data<ref:2506.02380#pg2> Such data is currently unavailable for photorealistic 3DGS reconstructions of real-world scenes<ref:2506.02380#pg2> This paper introduces EyeNavGS, the first publicly available 6-DoF navigation dataset featuring traces from 46 participants exploring twelve diverse, real-world 3DGS scenes<ref:2506.02380#pg2> The dataset was collected at two sites, using the Meta Quest Pro headsets, recording the head pose and eye gaze data for each rendered frame during free world standing 6-DoF navigation<ref:2506.02380#pg2> For each of the twelve scenes, we performed careful scene initialization to correct for scene tilt and scale, ensuring a perceptually-comfortable VR experience<ref:2506.02380#pg2> We also release our open-source SIBR viewer software fork with record-and-replay functionalities and a suite of utility tools for data processing, conversion, and visualization<ref:2506.02380#pg2> The EyeNavGS dataset and its accompanying software tools provide valuable resources for advancing research in 6-DoF viewport prediction, adaptive streaming, 3D saliency, and foveated rendering for 3DGS scenes<ref:2506.02380#pg2> The EyeNavGS dataset is available at: https://symmru.github.io/EyeNavGS.<ref:2506.02380#pg2> 1 Introduction Since its introduction in 2023, 3D Gaussian Splatting (3DGS)

21: has quickly emerged as a popular immersive media format for 3D scene representation, enabling high-fidelity, 6-degrees-of-freedom (6-DoF) exploration of complex real-world environments<ref:2506.02380#pg2> Due to its fast training time and real-time rendering speed, it has received significant attention from both academia and industry

6, 14, 20, 22, 26, 30, 46: <ref:2506.02380#pg2> 3DGS has unlocked new possibilities, including rendering on mobile devices with WebGL support

2: and extending traditional video streaming paradigms to full 6-DoF volumetric content<ref:2506.02380#pg2> For example, recent works such as SGSS

47: and L3GS

44: have proposed streaming approaches for static 3DGS scenes<ref:2506.

Paper summary: Tom: Moving into that overview, we need to focus on what EyeNavGS actually captures. It features traces from forty-six participants exploring twelve diverse, real-world three deeGS scenes <ref:2506.02380#pg1,participants exploring twelve diverse, real-world 3DGS scenes>.

Jane: So the dataset was collected at two sites using Meta Quest Pro headsets and recorded the head pose and eye gaze data for every single rendered frame during free world standing six-DoF navigation <ref:2506.02380#pg1,the dataset was collected at two sites using>.

Lu: When we look closer at the coordinate systems, there's a known mismatch because Gaussians in a three deeGS scene are trained from an initial point cloud obtained via COLMAP, which uses a righthanded system where the plus Y axis points downwards and the plus Z axis points forward <ref:2506.02380#pg1>.

Meng: That conflicts with OpenXR’s coordinate systems where plus Y points upwards, so they had to rotate the camera one hundred eighty degrees around the X-axis in that SIBR viewer source code to ensure it renders upright to the user <ref:2506.02380#pg3,the camera 180 degrees around the X-axis>.

Lalam: This is handled by rotating the camera one hundred eighty degrees around the X-axis in the SIBR viewer source code to ensure a scene is rendered upright to the user <ref:2506.02380#pg3,the camera 180 degrees around the X-axis>.

Tom: It sounds like they had this coordinate system mismatch as a technical hurdle they had to solve before collecting that data.

Jane: They also did careful scene initialization for each of those twelve scenes, correcting for scene tilt and scale to make sure it was perceptually comfortable for VR users.

Lu: They corrected the initial quaternion to fix scene tilt, selected a per-scene scale factor to ensure objects match their real-world proportions, and established an example starting viewing position.

Meng: And they used dimensionally accurate reference objects like a one-meter cube in each scene by comparing known dimensions from the real scene to their three deeGS representations <ref:2506.02380#pg1>.

Tom: That level of detail in calibration is what makes this dataset different from just having raw tracking data.

Jane: It gives researchers the necessary ground truth for things like six-DoF viewport prediction and view-adaptive streaming algorithms <ref:2506.02380#pg1>.

Conclusion: Tom: Now let's talk about that record-and-replay software they released. It's a fork of the SIBR viewer eight enhanced with record-and-replay functionality.

Jane: This fork allows users to record their traces and then replay those traces frame by frame for visualization, video generation, and detailed offline analysis.

Lu: In Record Mode, their modified OpenXR module captures fine-grained, per-frame data during a user’s VR session. For each rendered frame and for each eye, they record the field-of-view FOV2 in radians.

Meng: And they also record the eye position in world space and the head orientation as a quaternion, plus the eye gaze position in world space and gaze orientation quaternion.

Lalam: They record a comprehensive set of parameters for each frame, including FOV1, FOV2, eye position, head orientation as a quaternion, and eye gaze position and gaze orientation quaternion.

Tom: In Replay Mode, the recorded trace is parsed and the logged data is injected line-by-line into the rendering pipeline overriding the HMD pose information.

Jane: To generate video output from this replay mode, they capture rendered frames directly from GPU memory using OpenGL’s glGetTexImage API.

Lu: This means they generate two separate videos, one for each eye, that precisely replicate the original stereoscopic experience when replaying the trace.