GUSH3R: Everyone Everywhere All at Once as Gaussians
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "GUSH3R: Everyone Everywhere All at Once as Gaussians".
Jane: Here's the gist of the paper. Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering <ref:2607.05243#pg2> The paper presents GUSH3R, a feed-forward framework for online dynamic human-scene reconstruction that reconstructs dynamic humans and static scenes in a single forward pass as 3D Gaussian Splatting primitives <ref:2607.05243#pg2> Framework Overview The method decomposes the entire scene into a static scene and dynamic humans, represented as a set of scene Gaussians aggregated across time until frame t and time-dependent human Gaussians for each frame t <ref:2607.05243#pg4> This is built upon a human-scene foundation model that provides geometric priors and structured token representations <ref:2607.05243#pg2> The core idea leverages these priors while disentangling appearance modeling for the static scene and dynamic humans separately within a unified 3DGS framework <ref:2607.05243#pg2> Scene Reconstruction Mechanism The Scene Gaussian Decoder represents the static scene by progressively aggregating per-frame Gaussians into a unified static Gaussian set Gs t <ref:2607.05243#pg5> Given the reconstructed scene point maps Xt obtained from the foundation model, each Gaussian center is initialized with the corresponding 3D point xt,i <ref:2607.05243#pg6> The decoder then predicts Gaussian parameters using an MLP FG based on fused features from a DPT-style decoder and CNN image encoder <ref:2607.05243#pg6> Newly predicted Gaussians are filtered using confidence scores and human detection scores to suppress contributions from dynamic human regions <ref:2607.05243#pg7> Dynamic Human Modeling The Human Gaussian Decoder uses SMPL-X mesh vertices Vt,k as geometric anchors for human Gaussians <ref:2607.05243#pg7> It applies a cross-attention transformer, the Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries and image tokens serve as keys and values <ref:2607.05243#pg9> Vertex features are fed into an MLP FH to predict Gaussian parameters for each vertex <ref:2607.05243#pg10> Appearance Memory tokens mk store accumulated appearance features for each person over time, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes <ref:2607.05243#pg10> Training and Evaluation The model is trained by separately training the Scene Gaussian Decoder and the Human Gaussian Decoder while keeping the foundation model frozen <ref:2607.05243#pg10> The Scene Gaussian Decoder loss includes Lmse, Llpips, Ldep, and regL to supervise rendered scene against input image and ground truth depth <ref:2607.05243#pg9> The Human Gaussian Decoder loss includes Lmse, Lpart, silL, and regL to supervise rendered human appearance and silhouette <ref:2607.05243#pg10> Experiments show that the method achieves competitive novel view synthesis quality against optimization-based and decomposition-based baselines while being significantly more efficient <ref:2607.05243#pg2> The method also achieves the best performance on both metrics when evaluating video depth estimation task <ref:2607.05243#pg10> Limitations Several limitations remain, including reliance on geometric and human priors from the underlying foundation model, meaning errors in camera estimation or SMPL-X fitting can propagate to the final reconstruction <ref:2607.05243#pg2> Severe human-human occlusions and complex interactions are challenging due to limited cues from monocular videos <ref:2607.05243#pg2> Furthermore, fine-scale appearance details, such as faces and clothing textures, may remain imperfect under motion blur or large pose changes <ref:2607.05243#pg2> Acknowledgements This work was partially supported by JST ASPIRE Program, Japan, Grant Number JPMJAP2303; JST ACT-X (JPMJAX25C5); JST SPRING, Grant Number JPMJSP2108; and JSPS KAKENHI Grant Number 26K21245 <ref:2607.
Title and authors: Tom: Let's talk about the title and who wrote this paper. It’s called GUSH3R: Everyone Everywhere All at Once as Gaussians. This title is pretty descriptive, but it’s also a bit ambitious for what we’re actually seeing right now.
Jane: It certainly sounds like it wants to do everything at once, which is what the authors are aiming for by putting "Everyone Everywhere All at Once" into the name. So, who are we talking about here?
Lu: The authors are Keito Abe, Kaede Shiohara, Takashi Otonari, and Toshihiko Yamasaki from The University of Tokyo. They bring expertise in computer vision and three dee reconstruction <ref:2607.05243#pg1>.
Meng: I haven't seen their specific work on foundation models before. I’m more interested in the mechanics of how they built this unified framework than just the names on the paper.
Tom: Well, what’s interesting is that they are taking a monocular human-scene video as input and producing three dee Gaussian representations for both dynamic humans and static scenes simultaneously <ref:2607.05243#pg1,a monocular human-scene video>. It's a huge task.
Jane: So, it’s not just reconstructing one thing; it’s jointly recovering scene geometry, camera motion, and dynamic humans from that single video input. That sounds like the naive question we should ask first: can one video really provide all that information?
Lu: The authors are addressing the fact that existing methods often struggle because they are either optimization-based or feed-forward methods that only focus on static scenes or only on one type of dynamic object, leading to inconsistent results.
Meng: So, if we look at this from a deployment angle, does combining all those parts make it harder to debug when something goes wrong in the reconstruction?
Tom: It makes the initial setup more complex because you’ve got two separate decoders—one for static scenes and one for dynamic humans—which is where I see potential complexity creeping in.
Jane: So, what’s the core idea they are pushing with this title? Is it really about unifying the representation under the Gaussian framework?
Lu: Yes, they are trying to create a unified three deeGS formulation where both static and dynamic elements share that common mathematical structure <ref:2607.05243#pg1>. They leverage geometric priors from foundation models to make this unified representation possible.
Meng: So, the ambition is to get photorealistic rendering while maintaining speed, and that’s what keeps me interested in the results.
Tom: Right. The paper claims they can do this in one forward pass, which means they are aiming for a massive efficiency boost over traditional methods that require iterative optimization processes.
The paper's summary: Jane: Now let’s get into the actual summary of GUSH3R and what it claims to accomplish. What exactly does this framework do in plain English?
Tom: At its heart, GUSH3R reconstructs dynamic human-scene environments from a single monocular video input by reconstructing dynamic humans and static scenes together as three dee Gaussians <ref:2607.05243#pg1,dynamic human-scene environments from>. It’s designed for online reconstruction, meaning it can handle new video frames sequentially.
Lu: They achieve this by decomposing the scene into two parts: the static scene is represented by scene Gaussians that are aggregated across time until frame t, while dynamic humans are modeled as time-dependent human Gaussians for each specific frame t.
Meng: So, it’s like they build a growing map of the background and then overlay moving figures onto that map as they appear in the video. That sounds intuitive enough.
Jane: Exactly. They use a human-scene foundation model to provide geometric priors and structured token representations to anchor this process before disentangling the appearance modeling for the static scene and dynamic humans separately within that unified three deeGS framework <ref:2607.05243#pg1>.
Tom: The Scene Gaussian Decoder takes those per-frame Gaussians and aggregates them into a unified set of static Gaussians over time. It predicts these parameters using an MLP based on fused features from a DPT-style decoder and a CNN image encoder, starting each static Gaussian center with the corresponding three dee point obtained from the foundation model <ref:2607.05243#pg1>.
Jane: And for the humans, they use the Human Gaussian Decoder which uses SMPL-X mesh vertices as geometric anchors for those human Gaussians.
Lu: The Human Gaussian Decoder employs that cross-attention transformer, which is their HGT, where tokens like human and vertex and memory serve as queries against image tokens acting as keys and values to predict the Gaussian parameters for each vertex.
Meng: And remember those appearance memory tokens that store accumulated features for each person over time? That should be key to keeping those humans looking consistent even if they get occluded.
Jane: So, it’s a pipeline where they use priors, separate decoders, and specialized transformers to build a three dee representation of both the static background and the moving people <ref:2607.05243#pg1>.
The paper's improvements: Tom: Now let’s look at what they claim are the specific improvements over previous work. What makes this better than just using existing feed-forward methods?
Jane: The primary improvement is that they move beyond simply predicting geometry, which some feed-forward methods do, and instead model both static scenes and dynamic humans in a unified three deeGS framework <ref:2607.05243#pg1>.
Lu: They claim that by explicitly disentangling the dynamic humans from the static scenes and modeling them separately, they achieve a cleaner separation of appearance modeling within that unified structure.
Meng: So, what does this mean for us at scale? Does this separation actually make the model more stable when we try to deploy it on messy real-world video data?
Tom: It should lead to more consistent geometry and better reconstruction quality because they are not fighting the scene structure and the human structure simultaneously.
Jane: They also point out that their training scheme, where they train the components separately while keeping the foundation model frozen, is a clever way to supervise both parts effectively without having to retrain everything constantly.
Lu: And on top of that, they claim this unified approach yields competitive novel view synthesis quality against decomposition-based baselines while being significantly more efficient.
Meng: That’s what I want to hear—competitive quality without needing a huge optimization loop for every frame. That’s the real win for deployment.
Tom: And on the depth estimation task, they claim this unified Gaussian representation gives them the best performance on both metrics compared to methods that use sparse point-based depth or lack cross-frame consistency.
Conclusion: Jane: So, to wrap up everything we’ve covered about GUSH3R. What is the big picture implication of this work for people who aren't deep in the weeds?
Tom: The main implication is that we can reconstruct dynamic human-scene environments from monocular videos much more efficiently than before by doing it in a single forward pass as three dee Gaussian Splatting primitives <ref:2607.05243#pg1,dynamic human-scene environments from monocular videos>.
Lu: It suggests that joint human-scene reconstruction is a more effective strategy than just composing separate reconstructions afterwards, which points toward a new direction for handling complex scene understanding.
Meng: For me, the implication is that we can build systems that handle dynamic scenes in real-time where speed matters, provided the underlying foundation model is manageable to run efficiently.
Lalam: I’m pretty sure my internal analysis confirms this paper's findings with high confidence; it shows a solid path forward for integrating dynamic elements into three dee representations seamlessly <ref:2607.05243#pg1>.
Jane: It sounds like we're looking at a significant step toward making these kinds of reconstructions much more practical in real-world applications, moving them out of the purely experimental realm and into something usable.
Tom: So, to finish up on GUSH3R: it’s a feed-forward approach that delivers competitive quality by unifying static and dynamic scene modeling under one three dee Gaussian Splatting structure <ref:2607.05243#pg1>.
Lu: This paper sets a new benchmark for how we can structure these complex reconstruction tasks using generative priors to guide the process.
Meng: It gives us a concrete mechanism to think about deployment tradeoffs between quality, speed, and computational budget in this space.
The University of Tokyo
cs.CV
Submitted: 2026-07-06
Updated: 2026-11-11
Comments: Project page: https://abkeito.github.io/gush3r-page/ Accepted to WACV 2027
Project page: https://abkeito.github.io/gush3r-page/GUSH3R
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 91/100
The gist: Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion…
Key concepts
- 3D Gaussian Splatting (3DGS)
- A technique that represents 3D scenes using a collection of 3D Gaussians, which are essentially volumetric primitives with position, color, and opacity. GUSH3R uses this format to represent both the static background and moving humans simultaneously in a single representation.
- Scene Gaussian Decoder
- This component is responsible for reconstructing the static scene. It takes per-frame Gaussians and aggregates them into a unified set of static Gaussians representing the environment, using features from a decoder and image encoder to predict their parameters.
- Human Gaussian Decoder (HGT)
- This module models dynamic humans by anchoring their Gaussians to vertices from an SMPL-X mesh. It uses a cross-attention transformer that incorporates human, vertex, and memory tokens to predict the appearance of each human Gaussian over time.
Terminology
Summary
Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering <ref:2607. p. 0 io/gush3r-page/
GUSH3R
Figure 1: GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction) takes a monocular video
as input and produces dynamic human-scene representations using 3D Gaussians. Abstract
Reconstructing dynamic human-scene environments from monocular videos is a
challenging problem that requires jointly modeling scene geometry, camera motion,
and non-rigid human dynamics while enabling photorealistic rendering. To fill this gap, we present
GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward
framework for online dynamic human-scene reconstruction. From a monocular
human-scene video, our method reconstructs dynamic humans (everyone) and
static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian
Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and
capable of novel view synthesis.Where the paper says it
Where the paper says it
Framework Overview
The method decomposes the entire scene into a static scene and dynamic humans, represented as a set of scene Gaussians aggregated across time until frame t and time-dependent human Gaussians for each frame t <ref:2607.05243#pg4> This is built upon a human-scene foundation model that provides geometric priors and structured token representations <ref:2607. p. 2 We
decompose the entire scene Gt into a static scene and dynamic humans, represented as a set of scene
Gaussians Gs
t
aggregated across time until frame t and time-dependent human Gaussians {Gh
t,k} for
each frame t, where k indexes individual humans. We build upon a human-scene foundation model [10] that provides geometric priors and structured
token representations. However, these feed-forward approaches primarily focus on static scenes and often struggle when
dynamic objects are present, leading to inconsistent geometry and degraded reconstruction quality [23,
33]. Our work addresses this limitation by explicitly disentangling dynamic humans from static
scenes and models them separately within a unified 3DGS framework.Where the paper says it
Where the paper says it
Scene Reconstruction Mechanism
The Scene Gaussian Decoder represents the static scene by progressively aggregating per-frame Gaussians into a unified static Gaussian set Gs t <ref:2607.05243#pg5> Given the reconstructed scene point maps Xt obtained from the foundation model, each Gaussian center is initialized with the corresponding 3D point xt,i <ref:2607. p. 4 We represent the static scene by progressively aggregating per-frame
Gaussians g
s
t,i, each corresponding to pixel i in frame t, into a unified static Gaussian set Gs
t
. Given
the reconstructed scene point maps Xt = {xt,i} obtained from the foundation model, we initialize
each Gaussian center µt,i directly with the corresponding 3D point xt,i, where µt,i denotes the
center of the Gaussian corresponding to pixel i at time t. Next, we decode the Human3R image tokens F
′
t
into per-pixel features CD
t,i using a DPT-style
decoder [48]. To incorporate direct appearance cues from the input image, we also extract CNN
image features CI
t
from It using a CNN-based image encoder. (4)
Filtering and Voxelization. At each timestep t, newly predicted Gaussians Gˆ s
t = {g
s
t,i}i are filtered
using confidence scores [60] and human detection scores [4] to suppress contributions from dynamic
human regions.Where the paper says it
Where the paper says it
Where the paper says it
Dynamic Human Modeling
The Human Gaussian Decoder uses SMPL-X mesh vertices Vt,k as geometric anchors for human Gaussians <ref:2607.05243#pg7> It applies a cross-attention transformer, the Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries and image tokens serve as keys and values <ref:2607. p. 5 We apply a cross-attention transformer, named Human Gaussian Transformer (HGT), where human,
vertex, and memory tokens serve as queries, and image tokens serve as keys and values:
V
′
t,k = HGT(Q = [ht,k,Vt,k,mk], K = F
′
t
, V = F
′
t
). (6)
We denote the vertex features extracted from the transformer output as V
′
t,k. (6)
We denote the vertex features extracted from the transformer output as V
′
t,k. These vertex features
are fed into an MLP FH to predict Gaussian parameters for each vertex:
(αt,k,v, qt,k,v, st,k,v, ct,k,v) = FH(V
′
t,k,v), (7)
where v indexes the vertices of the canonical SMPL-X mesh. During
inference, identities are associated across frames using matching based on SMPL-X parameters. When a person is matched to a previous track, the corresponding memory token is reused and updated,
allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes.Where the paper says it
Where the paper says it
Where the paper says it
Training and Evaluation
The model is trained by separately training the Scene Gaussian Decoder and the Human Gaussian Decoder while keeping the foundation model frozen <ref:2607.05243#pg10> The Scene Gaussian Decoder loss includes Lmse, Llpips, Ldep, and regL to supervise rendered scene against input image and ground truth depth <ref:2607. p. 5 We train the Scene Gaussian Decoder using the following objective:
Lscene = λmseLmse + λlpipsLlpips + λdepLdep + λregLreg. (8)
Here, Lmse and Llpips [70] supervise the rendered scene against the input image, while Ldep supervises the rendered depth against the ground truth (GT) depth to enforce the geometric consistency. (10)
Here, Lmse and Lsil supervise the rendered human appearance and silhouette using pixel-wise MSE
and binary cross entropy, respectively, while Lreg penalizes degenerate Gaussian shapes as in the
Scene Gaussian Decoder. We additionally use a partial LPIPS loss to improve fine-grained human
appearance:
Lpart = LPIPS( ˆI, I) + X
r∈{upper,face}
LPIPScropr
(
ˆI), cropr
(I)
, (11)
where ˆI and I denote the rendered and input images, and cropr
(·) extracts the upper-body or face
region. • We design an architecture that bridges human-scene foundation models and photorealistic
rendering by leveraging geometric priors and SMPL-X [45] representations. • We demonstrate that our method achieves competitive novel view synthesis quality against
decomposition-based feed-forward baselines and an optimization-based human-scene baseline, while being significantly more efficient. We then quantify these observations about novel view synthesis on the BEDLAM test split. As
shown in Table 3, our method achieves the best performance on the full human-scene evaluation,
demonstrating the benefit of avoiding post-hoc composition.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations
Several limitations remain, including reliance on geometric and human priors from the underlying foundation model, meaning errors in camera estimation or SMPL-X fitting can propagate to the final reconstruction <ref:2607.05243#pg2> Severe human-human occlusions and complex interactions are challenging due to limited cues from monocular videos <ref:2607. p. 0 Abstract
Reconstructing dynamic human-scene environments from monocular videos is a
challenging problem that requires jointly modeling scene geometry, camera motion,
and non-rigid human dynamics while enabling photorealistic rendering. Recent
feed-forward methods can efficiently predict geometry, but they are often limited to
non-photorealistic representations such as point clouds and meshes, or they fail to
handle non-rigid objects, particularly dynamic humans. When a person is matched to a previous track, the corresponding memory token is reused and updated,
allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes. 5 Training Details
Training Setup.Where the paper says it
Where the paper says it
Acknowledgements
This work was partially supported by JST ASPIRE Program, Japan, Grant Number JPMJAP2303; JST ACT-X (JPMJAX25C5); JST SPRING, Grant Number JPMJSP2108; and JSPS KAKENHI Grant Number 26K21245 <ref:2607.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Improve inference efficiency by achieving feed-forward reconstruction of dynamic human-scene environments
in a single forward pass (all at once)
as 3D Gaussian Splatting primitives, which is significantly more efficient compared to optimization-based methods. -
Enable photorealistic rendering of dynamic scenes and humans by representing both static scenes and dynamic humans in a
unified 3DGS formulation,
which is achieved through a Scene Gaussian Decoder for the static scene and a Human Gaussian Decoder for dynamic humans. -
Enhance reconstruction quality against decomposition-based baselines by achieving
competitive novel view synthesis quality while significantly improving inference efficiency
compared to methods like AnySplat+LHM+Human3R, suggesting thatjoint human-scene reconstruction is more effective than post-hoc composition.
-
Improve geometric accuracy and temporal stability of depth estimation by using a unified Gaussian representation, as the method achieves
the best performance on both metrics [AbsRel and threshold accuracy]
in the video depth estimation task compared to methods relying on sparse point-based depth or lack of cross-frame consistency. -
Enhance scene reconstruction capability by performing streaming reconstruction while jointly modeling dynamic humans, which is contrasted with batch-based feed-forward methods that
better exploit multi-view consistency for static background reconstruction.
Abstract
Reconstructing dynamic human-scene environments from monocular videos is a challenging problem that requires jointly modeling scene geometry, camera motion, and non-rigid human dynamics while enabling photorealistic rendering. Recent feed-forward methods can efficiently predict geometry, but they are often limited to non-photorealistic representations such as point clouds and meshes, or they fail to handle non-rigid objects, particularly dynamic humans. To fill this gap, we present GUSH3R (Gaussian-Unified Scene Human 3D Reconstruction), a feed-forward framework for online dynamic human-scene reconstruction. From a monocular human-scene video, our method reconstructs dynamic humans (everyone) and static scenes (everywhere) in a single forward pass (all at once) as 3D Gaussian Splatting (3DGS) primitives (as gaussians), which are geometrically consistent and capable of novel view synthesis. Experiments on monocular human-scene datasets demonstrate that our approach achieves competitive novel view synthesis quality while significantly improving inference efficiency compared to optimization-based methods.
Sources
- Virtual KITTI 2
- DGGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images
- The Replica Dataset: A Digital Replica of Indoor Spaces
- RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos
- Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs