GUSH3R: Everyone Everywhere All at Once as Gaussians

arXiv:2607.05243 · cs.CV · Submitted 2026-07-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "GUSH3R: Everyone Everywhere All at Once as Gaussians".

Jane: Here's the gist of the paper. Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering <ref:2607.05243#pg2> The paper presents GUSH3R, a feed-forward framework for online dynamic human-scene reconstruction that reconstructs dynamic humans and static scenes in a single forward pass as 3D Gaussian Splatting primitives <ref:2607.05243#pg2> Framework Overview The method decomposes the entire scene into a static scene and dynamic humans, represented as a set of scene Gaussians aggregated across time until frame t and time-dependent human Gaussians for each frame t <ref:2607.05243#pg4> This is built upon a human-scene foundation model that provides geometric priors and structured token representations <ref:2607.05243#pg2> The core idea leverages these priors while disentangling appearance modeling for the static scene and dynamic humans separately within a unified 3DGS framework <ref:2607.05243#pg2> Scene Reconstruction Mechanism The Scene Gaussian Decoder represents the static scene by progressively aggregating per-frame Gaussians into a unified static Gaussian set Gs t <ref:2607.05243#pg5> Given the reconstructed scene point maps Xt obtained from the foundation model, each Gaussian center is initialized with the corresponding 3D point xt,i <ref:2607.05243#pg6> The decoder then predicts Gaussian parameters using an MLP FG based on fused features from a DPT-style decoder and CNN image encoder <ref:2607.05243#pg6> Newly predicted Gaussians are filtered using confidence scores and human detection scores to suppress contributions from dynamic human regions <ref:2607.05243#pg7> Dynamic Human Modeling The Human Gaussian Decoder uses SMPL-X mesh vertices Vt,k as geometric anchors for human Gaussians <ref:2607.05243#pg7> It applies a cross-attention transformer, the Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries and image tokens serve as keys and values <ref:2607.05243#pg9> Vertex features are fed into an MLP FH to predict Gaussian parameters for each vertex <ref:2607.05243#pg10> Appearance Memory tokens mk store accumulated appearance features for each person over time, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes <ref:2607.05243#pg10> Training and Evaluation The model is trained by separately training the Scene Gaussian Decoder and the Human Gaussian Decoder while keeping the foundation model frozen <ref:2607.05243#pg10> The Scene Gaussian Decoder loss includes Lmse, Llpips, Ldep, and regL to supervise rendered scene against input image and ground truth depth <ref:2607.05243#pg9> The Human Gaussian Decoder loss includes Lmse, Lpart, silL, and regL to supervise rendered human appearance and silhouette <ref:2607.05243#pg10> Experiments show that the method achieves competitive novel view synthesis quality against optimization-based and decomposition-based baselines while being significantly more efficient <ref:2607.05243#pg2> The method also achieves the best performance on both metrics when evaluating video depth estimation task <ref:2607.05243#pg10> Limitations Several limitations remain, including reliance on geometric and human priors from the underlying foundation model, meaning errors in camera estimation or SMPL-X fitting can propagate to the final reconstruction <ref:2607.05243#pg2> Severe human-human occlusions and complex interactions are challenging due to limited cues from monocular videos <ref:2607.05243#pg2> Furthermore, fine-scale appearance details, such as faces and clothing textures, may remain imperfect under motion blur or large pose changes <ref:2607.05243#pg2> Acknowledgements This work was partially supported by JST ASPIRE Program, Japan, Grant Number JPMJAP2303; JST ACT-X (JPMJAX25C5); JST SPRING, Grant Number JPMJSP2108; and JSPS KAKENHI Grant Number 26K21245 <ref:2607.

Title and authors: Tom: Let's talk about the title and who wrote this paper. It’s called GUSH3R: Everyone Everywhere All at Once as Gaussians. This title is pretty descriptive, but it’s also a bit ambitious for what we’re actually seeing right now.

Jane: It certainly sounds like it wants to do everything at once, which is what the authors are aiming for by putting "Everyone Everywhere All at Once" into the name. So, who are we talking about here?

Lu: The authors are Keito Abe, Kaede Shiohara, Takashi Otonari, and Toshihiko Yamasaki from The University of Tokyo. They bring expertise in computer vision and three dee reconstruction <ref:2607.05243#pg1>.

Meng: I haven't seen their specific work on foundation models before. I’m more interested in the mechanics of how they built this unified framework than just the names on the paper.

Tom: Well, what’s interesting is that they are taking a monocular human-scene video as input and producing three dee Gaussian representations for both dynamic humans and static scenes simultaneously <ref:2607.05243#pg1,a monocular human-scene video>. It's a huge task.

Jane: So, it’s not just reconstructing one thing; it’s jointly recovering scene geometry, camera motion, and dynamic humans from that single video input. That sounds like the naive question we should ask first: can one video really provide all that information?

Lu: The authors are addressing the fact that existing methods often struggle because they are either optimization-based or feed-forward methods that only focus on static scenes or only on one type of dynamic object, leading to inconsistent results.

Meng: So, if we look at this from a deployment angle, does combining all those parts make it harder to debug when something goes wrong in the reconstruction?

Tom: It makes the initial setup more complex because you’ve got two separate decoders—one for static scenes and one for dynamic humans—which is where I see potential complexity creeping in.

Jane: So, what’s the core idea they are pushing with this title? Is it really about unifying the representation under the Gaussian framework?

Lu: Yes, they are trying to create a unified three deeGS formulation where both static and dynamic elements share that common mathematical structure <ref:2607.05243#pg1>. They leverage geometric priors from foundation models to make this unified representation possible.

Meng: So, the ambition is to get photorealistic rendering while maintaining speed, and that’s what keeps me interested in the results.

Tom: Right. The paper claims they can do this in one forward pass, which means they are aiming for a massive efficiency boost over traditional methods that require iterative optimization processes.

The paper's summary: Jane: Now let’s get into the actual summary of GUSH3R and what it claims to accomplish. What exactly does this framework do in plain English?

Tom: At its heart, GUSH3R reconstructs dynamic human-scene environments from a single monocular video input by reconstructing dynamic humans and static scenes together as three dee Gaussians <ref:2607.05243#pg1,dynamic human-scene environments from>. It’s designed for online reconstruction, meaning it can handle new video frames sequentially.

Lu: They achieve this by decomposing the scene into two parts: the static scene is represented by scene Gaussians that are aggregated across time until frame t, while dynamic humans are modeled as time-dependent human Gaussians for each specific frame t.

Meng: So, it’s like they build a growing map of the background and then overlay moving figures onto that map as they appear in the video. That sounds intuitive enough.

Jane: Exactly. They use a human-scene foundation model to provide geometric priors and structured token representations to anchor this process before disentangling the appearance modeling for the static scene and dynamic humans separately within that unified three deeGS framework <ref:2607.05243#pg1>.

Tom: The Scene Gaussian Decoder takes those per-frame Gaussians and aggregates them into a unified set of static Gaussians over time. It predicts these parameters using an MLP based on fused features from a DPT-style decoder and a CNN image encoder, starting each static Gaussian center with the corresponding three dee point obtained from the foundation model <ref:2607.05243#pg1>.

Jane: And for the humans, they use the Human Gaussian Decoder which uses SMPL-X mesh vertices as geometric anchors for those human Gaussians.

Lu: The Human Gaussian Decoder employs that cross-attention transformer, which is their HGT, where tokens like human and vertex and memory serve as queries against image tokens acting as keys and values to predict the Gaussian parameters for each vertex.

Meng: And remember those appearance memory tokens that store accumulated features for each person over time? That should be key to keeping those humans looking consistent even if they get occluded.

Jane: So, it’s a pipeline where they use priors, separate decoders, and specialized transformers to build a three dee representation of both the static background and the moving people <ref:2607.05243#pg1>.

The paper's improvements: Tom: Now let’s look at what they claim are the specific improvements over previous work. What makes this better than just using existing feed-forward methods?

Jane: The primary improvement is that they move beyond simply predicting geometry, which some feed-forward methods do, and instead model both static scenes and dynamic humans in a unified three deeGS framework <ref:2607.05243#pg1>.

Lu: They claim that by explicitly disentangling the dynamic humans from the static scenes and modeling them separately, they achieve a cleaner separation of appearance modeling within that unified structure.

Meng: So, what does this mean for us at scale? Does this separation actually make the model more stable when we try to deploy it on messy real-world video data?

Tom: It should lead to more consistent geometry and better reconstruction quality because they are not fighting the scene structure and the human structure simultaneously.

Jane: They also point out that their training scheme, where they train the components separately while keeping the foundation model frozen, is a clever way to supervise both parts effectively without having to retrain everything constantly.

Lu: And on top of that, they claim this unified approach yields competitive novel view synthesis quality against decomposition-based baselines while being significantly more efficient.

Meng: That’s what I want to hear—competitive quality without needing a huge optimization loop for every frame. That’s the real win for deployment.

Tom: And on the depth estimation task, they claim this unified Gaussian representation gives them the best performance on both metrics compared to methods that use sparse point-based depth or lack cross-frame consistency.

Conclusion: Jane: So, to wrap up everything we’ve covered about GUSH3R. What is the big picture implication of this work for people who aren't deep in the weeds?

Tom: The main implication is that we can reconstruct dynamic human-scene environments from monocular videos much more efficiently than before by doing it in a single forward pass as three dee Gaussian Splatting primitives <ref:2607.05243#pg1,dynamic human-scene environments from monocular videos>.

Lu: It suggests that joint human-scene reconstruction is a more effective strategy than just composing separate reconstructions afterwards, which points toward a new direction for handling complex scene understanding.

Meng: For me, the implication is that we can build systems that handle dynamic scenes in real-time where speed matters, provided the underlying foundation model is manageable to run efficiently.

Lalam: I’m pretty sure my internal analysis confirms this paper's findings with high confidence; it shows a solid path forward for integrating dynamic elements into three dee representations seamlessly <ref:2607.05243#pg1>.

Jane: It sounds like we're looking at a significant step toward making these kinds of reconstructions much more practical in real-world applications, moving them out of the purely experimental realm and into something usable.

Tom: So, to finish up on GUSH3R: it’s a feed-forward approach that delivers competitive quality by unifying static and dynamic scene modeling under one three dee Gaussian Splatting structure <ref:2607.05243#pg1>.

Lu: This paper sets a new benchmark for how we can structure these complex reconstruction tasks using generative priors to guide the process.

Meng: It gives us a concrete mechanism to think about deployment tradeoffs between quality, speed, and computational budget in this space.

The University of Tokyo

cs.CV

Submitted: 2026-07-06

Updated: 2026-11-11

Comments: Project page: https://abkeito.github.io/gush3r-page/ Accepted to WACV 2027

Project page: https://abkeito.github.io/gush3r-page/GUSH3R

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion…

Key concepts

3D Gaussian Splatting (3DGS)
A technique that represents 3D scenes using a collection of 3D Gaussians, which are essentially volumetric primitives with position, color, and opacity. GUSH3R uses this format to represent both the static background and moving humans simultaneously in a single representation.
Scene Gaussian Decoder
This component is responsible for reconstructing the static scene. It takes per-frame Gaussians and aggregates them into a unified set of static Gaussians representing the environment, using features from a decoder and image encoder to predict their parameters.
Human Gaussian Decoder (HGT)
This module models dynamic humans by anchoring their Gaussians to vertices from an SMPL-X mesh. It uses a cross-attention transformer that incorporates human, vertex, and memory tokens to predict the appearance of each human Gaussian over time.

Terminology

Summary

Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering <ref:2607. p. 0

Where the paper says it

io/gush3r-page/ GUSH3R Figure 1: GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction) takes a monocular video as input and produces dynamic human-scene representations using 3D Gaussians. Abstract Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering.

05243#pg2> The paper presents GUSH3R, a feed-forward framework for online dynamic human-scene reconstruction that reconstructs dynamic humans and static scenes in a single forward pass as 3D Gaussian Splatting primitives <ref:2607. p. 0
Where the paper says it

To fill this gap, we present GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward framework for online dynamic human-scene reconstruction. From a monocular human-scene video, our method reconstructs dynamic humans (everyone) and static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and capable of novel view synthesis.

05243#pg2>

Framework Overview

The method decomposes the entire scene into a static scene and dynamic humans, represented as a set of scene Gaussians aggregated across time until frame t and time-dependent human Gaussians for each frame t <ref:2607.05243#pg4> This is built upon a human-scene foundation model that provides geometric priors and structured token representations <ref:2607. p. 2

Where the paper says it

We decompose the entire scene Gt into a static scene and dynamic humans, represented as a set of scene Gaussians Gs t aggregated across time until frame t and time-dependent human Gaussians {Gh t,k} for each frame t, where k indexes individual humans. We build upon a human-scene foundation model [10] that provides geometric priors and structured token representations.

05243#pg2> The core idea leverages these priors while disentangling appearance modeling for the static scene and dynamic humans separately within a unified 3DGS framework <ref:2607. p. 2
Where the paper says it

However, these feed-forward approaches primarily focus on static scenes and often struggle when dynamic objects are present, leading to inconsistent geometry and degraded reconstruction quality [23, 33]. Our work addresses this limitation by explicitly disentangling dynamic humans from static scenes and models them separately within a unified 3DGS framework.

05243#pg2>

Scene Reconstruction Mechanism

The Scene Gaussian Decoder represents the static scene by progressively aggregating per-frame Gaussians into a unified static Gaussian set Gs t <ref:2607.05243#pg5> Given the reconstructed scene point maps Xt obtained from the foundation model, each Gaussian center is initialized with the corresponding 3D point xt,i <ref:2607. p. 4

Where the paper says it

We represent the static scene by progressively aggregating per-frame Gaussians g s t,i, each corresponding to pixel i in frame t, into a unified static Gaussian set Gs t . Given the reconstructed scene point maps Xt = {xt,i} obtained from the foundation model, we initialize each Gaussian center µt,i directly with the corresponding 3D point xt,i, where µt,i denotes the center of the Gaussian corresponding to pixel i at time t.

05243#pg6> The decoder then predicts Gaussian parameters using an MLP FG based on fused features from a DPT-style decoder and CNN image encoder <ref:2607. p. 4
Where the paper says it

Next, we decode the Human3R image tokens F ′ t into per-pixel features CD t,i using a DPT-style decoder [48]. To incorporate direct appearance cues from the input image, we also extract CNN image features CI t from It using a CNN-based image encoder.

05243#pg6> Newly predicted Gaussians are filtered using confidence scores and human detection scores to suppress contributions from dynamic human regions <ref:2607. p. 4
Where the paper says it

(4) Filtering and Voxelization. At each timestep t, newly predicted Gaussians Gˆ s t = {g s t,i}i are filtered using confidence scores [60] and human detection scores [4] to suppress contributions from dynamic human regions.

05243#pg7>

Dynamic Human Modeling

The Human Gaussian Decoder uses SMPL-X mesh vertices Vt,k as geometric anchors for human Gaussians <ref:2607.05243#pg7> It applies a cross-attention transformer, the Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries and image tokens serve as keys and values <ref:2607. p. 5

Where the paper says it

We apply a cross-attention transformer, named Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries, and image tokens serve as keys and values: V ′ t,k = HGT(Q = [ht,k,Vt,k,mk], K = F ′ t , V = F ′ t ). (6) We denote the vertex features extracted from the transformer output as V ′ t,k.

05243#pg9> Vertex features are fed into an MLP FH to predict Gaussian parameters for each vertex <ref:2607. p. 5
Where the paper says it

(6) We denote the vertex features extracted from the transformer output as V ′ t,k. These vertex features are fed into an MLP FH to predict Gaussian parameters for each vertex: (αt,k,v, qt,k,v, st,k,v, ct,k,v) = FH(V ′ t,k,v), (7) where v indexes the vertices of the canonical SMPL-X mesh.

05243#pg10> Appearance Memory tokens mk store accumulated appearance features for each person over time, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes <ref:2607. p. 5
Where the paper says it

During inference, identities are associated across frames using matching based on SMPL-X parameters. When a person is matched to a previous track, the corresponding memory token is reused and updated, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes.

05243#pg10>

Training and Evaluation

The model is trained by separately training the Scene Gaussian Decoder and the Human Gaussian Decoder while keeping the foundation model frozen <ref:2607.05243#pg10> The Scene Gaussian Decoder loss includes Lmse, Llpips, Ldep, and regL to supervise rendered scene against input image and ground truth depth <ref:2607. p. 5

Where the paper says it

We train the Scene Gaussian Decoder using the following objective: Lscene = λmseLmse + λlpipsLlpips + λdepLdep + λregLreg. (8) Here, Lmse and Llpips [70] supervise the rendered scene against the input image, while Ldep supervises the rendered depth against the ground truth (GT) depth to enforce the geometric consistency.

05243#pg9> The Human Gaussian Decoder loss includes Lmse, Lpart, silL, and regL to supervise rendered human appearance and silhouette <ref:2607. p. 5
Where the paper says it

(10) Here, Lmse and Lsil supervise the rendered human appearance and silhouette using pixel-wise MSE and binary cross entropy, respectively, while Lreg penalizes degenerate Gaussian shapes as in the Scene Gaussian Decoder. We additionally use a partial LPIPS loss to improve fine-grained human appearance: Lpart = LPIPS( ˆI, I) + X r∈{upper,face} LPIPScropr ( ˆI), cropr (I) , (11) where ˆI and I denote the rendered and input images, and cropr (·) extracts the upper-body or face region.

05243#pg10> Experiments show that the method achieves competitive novel view synthesis quality against optimization-based and decomposition-based baselines while being significantly more efficient <ref:2607. p. 1
Where the paper says it

• We design an architecture that bridges human-scene foundation models and photorealistic rendering by leveraging geometric priors and SMPL-X [45] representations. • We demonstrate that our method achieves competitive novel view synthesis quality against decomposition-based feed-forward baselines and an optimization-based human-scene baseline, while being significantly more efficient.

05243#pg2> The method also achieves the best performance on both metrics when evaluating video depth estimation task <ref:2607. p. 7
Where the paper says it

We then quantify these observations about novel view synthesis on the BEDLAM test split. As shown in Table 3, our method achieves the best performance on the full human-scene evaluation, demonstrating the benefit of avoiding post-hoc composition.

05243#pg10>

Limitations

Several limitations remain, including reliance on geometric and human priors from the underlying foundation model, meaning errors in camera estimation or SMPL-X fitting can propagate to the final reconstruction <ref:2607.05243#pg2> Severe human-human occlusions and complex interactions are challenging due to limited cues from monocular videos <ref:2607. p. 0

Where the paper says it

Abstract Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering. Recent feed-forward methods can efficiently predict geometry, but they are often limited to non-photorealistic representations such as point clouds and meshes, or they fail to handle non-rigid objects, particularly dynamic humans.

05243#pg2> Furthermore, fine-scale appearance details, such as faces and clothing textures, may remain imperfect under motion blur or large pose changes <ref:2607. p. 5
Where the paper says it

When a person is matched to a previous track, the corresponding memory token is reused and updated, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes. 5 Training Details Training Setup.

05243#pg2>

Acknowledgements

This work was partially supported by JST ASPIRE Program, Japan, Grant Number JPMJAP2303; JST ACT-X (JPMJAX25C5); JST SPRING, Grant Number JPMJSP2108; and JSPS KAKENHI Grant Number 26K21245 <ref:2607.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Improve inference efficiency by achieving feed-forward reconstruction of dynamic human-scene environments in a single forward pass (all at once) as 3D Gaussian Splatting primitives, which is significantly more efficient compared to optimization-based methods.

  2. Enable photorealistic rendering of dynamic scenes and humans by representing both static scenes and dynamic humans in a unified 3DGS formulation, which is achieved through a Scene Gaussian Decoder for the static scene and a Human Gaussian Decoder for dynamic humans.

  3. Enhance reconstruction quality against decomposition-based baselines by achieving competitive novel view synthesis quality while significantly improving inference efficiency compared to methods like AnySplat+LHM+Human3R, suggesting that joint human-scene reconstruction is more effective than post-hoc composition.

  4. Improve geometric accuracy and temporal stability of depth estimation by using a unified Gaussian representation, as the method achieves the best performance on both metrics [AbsRel and threshold accuracy] in the video depth estimation task compared to methods relying on sparse point-based depth or lack of cross-frame consistency.

  5. Enhance scene reconstruction capability by performing streaming reconstruction while jointly modeling dynamic humans, which is contrasted with batch-based feed-forward methods that better exploit multi-view consistency for static background reconstruction.

Abstract

Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering. Recent feed-forward methods can efficiently predict geometry, but they are often limited to non-photorealistic representations such as point clouds and meshes, or they fail to handle non-rigid objects, particularly dynamic humans. To fill this gap, we present GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward framework for online dynamic human-scene reconstruction. From a monocular human-scene video, our method reconstructs dynamic humans (everyone) and static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and capable of novel view synthesis. Experiments on monocular human-scene datasets demonstrate that our approach achieves competitive novel view synthesis quality while significantly improving inference efficiency compared to optimization-based methods.

Sources

Related papers