Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art

summary

The gist

Immersive exhibitions increasingly combine large-scale projection displays and mixed reality (MR) head-mounted displays (HMDs)…

In short

The study investigated how users perceive transitions between large-scale projections and mixed reality (MR) head-mounted displays (HMDs) in immersive art. Manipulating inconsistencies in spatial alignment, visual appearance, and latency resulted in lower presence and higher workload. Findings suggest designers must prioritize consistency for interaction-critical assets like static meshes to ensure a coherent cross-reality experience.

Key concepts

Cross-Reality (CR) Transitions
This refers to the process of moving between different visualization environments, specifically from a large projection display into an MR HMD. The research focuses on how users experience these shifts when the visual and spatial contexts are different, aiming for a seamless flow in immersive art.
Spatial Alignment
This concept measures how consistently objects and scenes are positioned across the projection surface and the MR HMD view. Misalignment causes users to feel a lack of predictability during interactions, disrupting their sense of control over the virtual space.
Asset-Type Sensitivity
This examines how different types of digital art elements—like static meshes (rigid objects), skeletal meshes (animated objects), and particle systems—affect the perception of transition inconsistencies. The study found that rigid static meshes are most sensitive to spatial and visual mismatches.
Presence and Workload
Presence measures the user's feeling of being truly immersed in the experience, while workload reflects how mentally demanding a task is. Inconsistent transitions led to decreased presence and increased mental effort for users, indicating that technical flaws directly impact immersion quality.

Terminology used across episodes

This episode discusses

The paper

Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art · Read on arXiv

Trinity College Dublin

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art".

Jane: Here's the gist of the paper. Immersive exhibitions increasingly combine large-scale projection displays and mixed reality (MR) head-mounted displays (HMDs), yet their integration into a coherent experience remains underexplored, particularly in terms of how users perceive transitions across heterogeneous visualization environments Abstract—Immersive exhibitions increasingly combine large-scale projection displays and mixed reality (MR) head-mounted displays (HMDs), yet their integration into a coherent experience remains underexplored, particularly in terms of how users perceive transitions across heterogeneous visualization environments This paper investigates cross-reality (CR) object- and scene-level transitions between projection and an MR HMD through a hybrid immersive art installation spanning projection, augmented reality (AR), and virtual reality (VR) Quantitative results confirmed that the manipulation produced a clear experiential contrast, with lower presence and higher workload under inconsistency These findings provide empirical insight into how users perceive projection–MR transitions and inform asset-aware design strategies for coherent hybrid immersive art experiences Pruning redundant reasoning tokens The study was structured around examining transition cues (RQ1-RQ2) and asset-related dimensions (RQ3) The researchers focused on three cue dimensions relevant to continuity: spatial alignment, visual appearance, and temporal responsiveness These cues were operationalized by introducing controlled discrepancies into a bundled inconsistency condition The study also examined three common immersive-art asset types: static meshes, skeletal meshes, and particle systems How the training signal is built The experimental setup involved a hybrid projection–MR system that enabled spatially coherent rendering and interaction across large-scale projection mapping and an MR HMD The system consisted of a projection component, an MR component, and a calibration–synchronization layer to align coordinate frames Spatial calibration was achieved by establishing a shared hybrid coordinate space through aligning each physical projection surface with an identical planar proxy in the MR view This allowed for the consistent transformation of object and camera poses between the projection and MR applications during runtime Where the gains come from The core investigation utilized a within-subject study with N=24 participants comparing a calibrated condition against a bundled inconsistency condition The manipulation introduced spatial, appearance, and latency discrepancies to serve as a controlled diagnostic probe for design rather than establishing hardware targets Qualitative analysis revealed that spatial misalignment disrupted action–outcome predictability during precision interaction Appearance mismatch affected aesthetic coherence, while latency reduced perceived responsiveness Asset-type sensitivity The study examined how different asset structures mediated the perceptual impact of transition inconsistencies Static meshes were most frequently cited as revealing inconsistencies because their rigid geometry and stable surface appearance provided clear spatial and visual anchors Skeletal meshes elicited mixed but generally weaker sensitivity compared to static assets, as participants often evaluated them in terms of overall motion impression Particle effects were perceived as most tolerant, often being evaluated in terms of atmosphere and overall impression rather than small geometric mismatches Design implications for CR design The findings suggest that coherent hybrid projection–MR experiences require both overall technical consistency and an understanding of where specific inconsistencies become most perceptually consequential Designers should allocate stricter consistency budgets to diagnostic assets and interaction-critical moments, such as static meshes used for precision targeting Maintaining appearance consistency is crucial for supporting aesthetic coherence, especially for assets with rich surface detail Latency should be minimized during moments where users directly manipulate objects or expect immediate feedback to maintain perceived control Stage scene transitions according to whether the goal is efficiency or experiential continuity, utilizing portals and expansion effects when narrative continuity is prioritized Together, these findings suggest that coherent hybrid projection–MR experiences require both overall technical consistency and an understanding of where specific inconsistencies become most perceptually and experientially consequential They motivate asset-aware transition strategies that prioritize spatial and temporal consistency during interaction-critical moments, maintain appearance coherence for visually diagnostic assets, and support continuity across projection, AR, and VR The authors release the core system as an open-source Unreal Engine plugin to support future projection–MR CR research and prototyping References

1: S.

Tom: Butscher, S.

Jane: Hubenschmid, J.

Tom: Müller, J.

Paper summary: Tom: So we're looking at this paper called Exploring Cross-Reality Transitions between Projections and Head-Mounted Displays for Immersive Digital Art. Basically, the big question is how people feel when they jump between a huge projection display and a head-mounted display like mixed reality.

Jane: Right, it's about figuring out how users perceive those transitions when they are moving between different visualization environments, especially since we don't really know much about that integration yet.

Tom: The paper investigates this by setting up a hybrid immersive art installation that uses projection, augmented reality and virtual reality to study object and scene level transitions across these displays.

Lu: What's interesting is how they structure the study around three main things: transition cues, which are things like spatial alignment and visual appearance, and asset related dimensions, which look at different types of digital art objects.

Meng: So what kind of objects are they looking at? Are we talking just simple shapes or more complex things?

Tom: They looked at static meshes, which are rigid geometry dominated objects with a stable structure, skeletal meshes that are articulated and change through animation, and particle systems which are distributed effects.

Jane: That’s helpful for us to see where the study focuses its attention when we think about what makes a transition feel good or bad.

Tom: The core of their experimental setup was comparing a calibrated condition against a bundled inconsistency condition that introduced noticeable mismatches in spatial alignment, visual appearance, and cross-device latency.

Lu: They used this degraded condition not to find some hardware target or error bound, but as a controlled diagnostic probe for design.

Meng: So they were intentionally making things inconsistent to see what cues participants would actually notice when the experience went wrong at scale? That's a good way to test real-world deployment issues.

Tom: Exactly, and the quantitative results confirmed that this manipulation produced a clear experiential contrast, showing lower presence and higher workload when there was inconsistency.

Jane: So if we look at those numbers, it seems like even small inconsistencies really matter for how present the user feels in the experience.

Tom: Interviews further revealed that spatial misalignment disrupted action outcome predictability during precision interaction, appearance mismatch affected aesthetic coherence, and latency reduced perceived responsiveness.

Lu: It’s telling that they broke down the cues by showing how each one shaped a different part of the user's experience, like control versus aesthetics.

Meng: And when you look at the asset types they tested—static meshes seemed to reveal inconsistencies most frequently because their rigid geometry and stable surface appearance provided clear spatial and visual anchors.

Jane: That makes sense, because if you have a precise object that should stay put across both displays, any shift in its alignment is going to be immediately noticeable.

Paper summary: Tom: Skeletal meshes showed mixed but generally weaker sensitivity compared to static assets, while particle systems were perceived as most tolerant because they were evaluated more in terms of atmosphere and overall impression rather than small geometric mismatches.

Lu: So the study suggests that for certain digital art assets, like those with lots of surface detail, maintaining appearance consistency is really crucial for supporting aesthetic coherence.

Meng: From a production standpoint, if we're deploying something where users are manipulating things precisely, you need to be extremely strict about spatial and temporal consistency during those interaction critical moments.

Jane: What about the scene transitions themselves? The paper mentions that scene level transitions were designed to match the embodied and spatial nature of the immersive art experience.

Tom: They used a localized expansion effect that gradually replaces the current scene while preserving spatial continuity, which they found was commonly described as a gradual transformation of the surrounding space.

Lu: That’s how they handled scene-level transitions when prioritizing narrative flow and embodied feeling over just pure efficiency.

Meng: So for us deploying something, if we want that continuity, we have to design those expansion effects very carefully so they don't feel jarring or laggy for the user.

Jane: What did the study say about how users actually perceived these inconsistencies in their overall sense of experience?

Tom: Six participants described the overall experience as coherent, attributing connectedness to narrative flow and persistent entities traversing modalities, while five participants reported a more fragmented impression in the less consistent session.

Lu: That difference suggests that while some people can hold it together through narrative structure, others are much more sensitive to the underlying technical inconsistencies.

Meng: It means we have to design for both—we need the technical consistency so everyone has a good experience, but we also need to think about how narrative flow helps people cope when things aren't perfectly aligned.

Jane: So what does this all mean for future designs in hybrid projection and MR experiences?

Tom: The conclusion is that coherent hybrid experiences require both overall technical consistency and an understanding of where specific inconsistencies become most perceptually consequential.

Lu: This motivates asset aware transition strategies that prioritize spatial and temporal consistency during interaction critical moments, maintain appearance coherence for visually diagnostic assets, and support continuity across projection, AR, and VR.

Meng: For production teams, that means setting stricter budgets for consistency specifically on those diagnostic assets like static meshes or skeletal meshes during user interaction.

Jane: And they also suggest scene transitions should be designed based on whether the goal is efficiency or experiential continuity by using things like portals and expansion effects when narrative continuity is prioritized.

Tom: This work also releases their core system as an open-source Unreal Engine plugin called HUICRSync to support future research and prototyping in this area.

Lu: That plugin gives everyone a tool to start building these cross reality experiences without having to reinvent the entire synchronization layer from scratch.

Conclusion: Tom: So we've been looking at how people feel when they jump between massive projection displays and head-mounted mixed reality headsets, and this paper is about that cross-reality transition stuff.

Jane: It’s about how users perceive those shifts across different visualization environments, especially since that whole integration hasn't really been explored much before.

Tom: The authors used a hybrid installation—projection, augmented reality, virtual reality—to look at object and scene level transitions between the projection and the headset.

Lu: They framed it around three cue dimensions for continuity: spatial alignment, visual appearance, and temporal responsiveness.

Meng: And they tested how different types of digital art assets affected these cues; static meshes seemed to show inconsistencies most clearly because they're rigid geometry.

Jane: So what did all that testing actually tell us about the experience itself? What were the main findings?

Tom: The results showed a clear experiential contrast. When there was inconsistency, presence went down and the workload went up significantly.

Lu: The paper found that spatial misalignment messed with action outcome predictability during precise interaction, while appearance mismatch hurt the aesthetic coherence of things.

Meng: And latency, which is how slow it feels when you move things around, really reduced how responsive the whole experience felt to the user.

Jane: So for someone just listening, what does this mean? What’s actually changing for them?

Tom: It means that technical consistency isn't just a nice feature; it directly impacts whether a user feels present or if they feel like they’re struggling to control things.

Lu: The authors suggest designers need to be specific about where they apply strict consistency budgets, especially for those diagnostic assets like static meshes used for targeting.

Meng: I see that as needing a tiered approach; you don't treat every single asset the same way when you’re shipping something at scale.

Jane: And when it comes to scene transitions, how does that change? How should we design those shifts?

Tom: They suggest scene transitions should match the embodied and spatial nature of the art, using things like expansion effects to gradually reveal the next scene while keeping spatial continuity.

Lu: The authors conclude that coherent experiences need both technical consistency and an understanding of where those specific inconsistencies cause the most problems experientially.

Jane: It sounds like this work is pushing designers to think more deeply about how technical glitches translate into actual human feelings within a mixed reality space.

Tom: Exactly. This research gives us some real data on what’s actually causing users to lose their sense of presence in these complex setups.

More episodes

← Home