GenIA: Generative Reconstruction with Test-Time Input Alignment

arXiv:2610.12388 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "GenIA: Generative Reconstruction with Test-Time Input Alignment".

Tom: The gist: GenIA introduces an inference-time framework that aligns SAM3D’s generative prior to observations using geometric and photometric constraints, improving pose and appearance consistency across monocular, multi-view,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about who wrote this. The paper is titled "GenIA: Generative Reconstruction with Test-Time Input Alignment," and the authors are Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger and Jonathon Luiten.

Jane: They're a solid team from the Tübingen AI Center at the University of Tübingen. It shows they're working on cutting-edge research in three dee foundation models and generative reconstruction <ref:2610.12388#pg1>.

Lu: What this title really tells us is that they are focusing on alignment during inference, which means making sure the output you get when you run the model matches your input data right when it happens, rather than fixing it during the initial training phase.

Meng: So, if you're driving a car and need a quick three dee understanding of something in front of you—maybe a pedestrian or an obstacle—this work aims to make that process more accurate on the fly <ref:2610.12388#pg1>.

Lalam: It implies they are trying to bridge the gap between having a model that can generate complete three dee shapes and actually being able to use that shape accurately in real-world, input-dependent scenarios <ref:2610.12388#pg1>.

The paper's summary: Tom: So, what's the actual gist of GenIA? They are showing how to align generation to observations at three different levels. First, they recover translation and scale using depth cues while keeping the generative rotation prior.

Jane: Then they condition appearance on geometry and visibility, directing local features toward the image regions that observe them, and fusing predictions across views or frames with visibility weighting.

Lu: And finally, they use differentiable rendering to provide both in-trajectory guidance for dynamic inputs and an optional post-denoising refinement step. That’s a multi-pronged approach to getting the geometry, pose, and appearance right all at once.

Meng: It sounds like they are moving beyond just one image—monocular input—to handle multi-view and dynamic inputs with this framework.

Lalam: The paper explains that their grounding mechanisms are agnostic to the specific geometry predictor being used, meaning it can even support externally supplied shapes if you have them available.

The paper's improvements: Tom: The main improvement is how they handle pose prediction. They use SAMthree dee’s coarse geometry model to predict the world-space pose, but they estimate translation and scale from geometric cues instead of relying solely on image-space errors.

Jane: They measure three things from the camera-space pointmap: the object’s depth, its center, and its angular size. Then they run five fixed-point iterations to update scale, depth, and lateral position.

Lu: For dynamic objects, since the injected shapes already capture non-rigid motion and local rotation, they only need to recover a single global root orientation rather than one pose per frame.

Meng: And for appearance prediction, they use visibility attention bias to direct the model toward patches that actually see each voxel in the three dee representation <ref:2610.12388#pg1>. They also use rendering guidance during denoising and a loss function combining photometric and depth supervision.

Lalam: A big part of this is how they handle optional shape injection. If you have external geometry, they replace SAMthree dee’s coarse prediction with an occupancy grid inversion, which then conditions the pose denoising step.

Conclusion: Tom: So, to wrap up the GenIA paper, they show that this framework improves pose and appearance reconstruction across monocular, multi-view and dynamic inputs without retraining the foundation model. It’s about aligning it at three levels: geometry initialization, visibility reasoning, and differentiable rendering.

Jane: The results on ActionMesh show that GenIA ranks second to HiMoR on DAVIS training-view PSNR and δavg, while achieving the lowest reprojection error and highest CLIP-I across dynamic settings.

Lu: It suggests that by grounding SAMthree dee’s prior in observations this way, we get better color and lighting alignment than we see with models like SAMthree dee alone in a single view setting.

Meng: For practical applications, especially with dynamic scenes, this consistency across frames is important because per-frame SAMthree dee was temporally inconsistent. This work offers a more stable tracking correspondence for dynamic object reconstruction.

Lalam: Overall, GenIA is about taking the power of a large generative prior and making it reliable by forcing it to respect the physical constraints of what's actually in front of us at inference time.

Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger

Tübingen AI Center, University of Tübingen

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Project page: https://facebookresearch.github.io/GenIA

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 79/100

The gist: The gist: GenIA introduces an inference-time framework that aligns SAM3D’s generative prior to observations using geometric and photometric constraints, improving pose and appearance consistency

Key concepts

Test-Time Input Alignment: This is the core idea: taking a pre-trained generative model (like SAM3D) and modifying its inference process so that its output matches specific input data (images or video frames). GenIA achieves this by grounding the model's predictions in observed geometry and photometric data at test time, ensuring the generated 3D asset is consistent with what is actually seen.
Geometry-Derived Pose Estimation: Instead of relying solely on image-based pose estimation, GenIA uses depth cues derived from the input to accurately determine an object's translation (where it is in space) and scale (how big it is). This geometric information helps refine the object's world-space placement while preserving the rotation learned by the foundation model.
Visibility-Biased Attention: This technique directs the model's attention mechanism during appearance prediction. It biases the attention towards image patches that actually observe specific voxels, effectively ignoring occluded or unobserved areas. This ensures that only visible parts of the object influence its predicted appearance and shape details.
Differentiable Rendering Guidance: This involves using a differentiable rendering process during the denoising steps of the generative model. It provides an explicit supervision signal by comparing the rendered output from a predicted 3D model with the actual input image. This guidance helps correct geometric and photometric errors in real-time.

Terminology

Computer Vision and Pattern Recognition — 18× in this paper
Ablation — 14× in this paper · explained in 1 episode
Alignment — 9× in this paper
Gaussian Splatting — 9× in this paper · explained in 4 episodes
Consistency — 8× in this paper · explained in 3 episodes
Foundation Model — 8× in this paper · explained in 3 episodes
Benchmark — 7× in this paper · explained in 2 episodes
Fidelity — 4× in this paper · explained in 2 episodes
Foundation Models — 4× in this paper · explained in 7 episodes
3D Gaussian Splatting — 3× in this paper · explained in 2 episodes
Accuracy — 3× in this paper · explained in 1 episode
Adaptation — 3× in this paper · explained in 1 episode
Grounding — 3× in this paper · explained in 3 episodes
Pattern Analysis and Machine Intelligence — 3× in this paper
Temporal consistency — 3× in this paper · explained in 3 episodes
Transformer — 3× in this paper · explained in 2 episodes
Adam — 2× in this paper · explained in 2 episodes
Gradient Flow — 2× in this paper · explained in 2 episodes
Latent Representation — 2× in this paper · explained in 2 episodes
Latent Space — 2× in this paper · explained in 7 episodes
Rectified Flow — 2× in this paper · explained in 3 episodes
Aggregation — 1× in this paper
Calibration — 1× in this paper · explained in 7 episodes
Core — 1× in this paper · explained in 2 episodes
Diffusion Model — 1× in this paper · explained in 10 episodes
Diffusion Models — 1× in this paper · explained in 9 episodes
Distillation — 1× in this paper · explained in 2 episodes
Embodied AI — 1× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Flow Matching — 1× in this paper · explained in 15 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 3 episodes
Point Clouds — 1× in this paper · explained in 2 episodes
Quantization — 1× in this paper · explained in 13 episodes
Reconstruction Error — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes

Summary

The gist: GenIA introduces an inference-time framework that aligns SAM3D’s generative prior to observations using geometric and photometric constraints, improving pose and appearance consistency across monocular, multi-view, and dynamic inputs without retraining the foundation model.

How it works

GenIA improves object pose by deriving translation and scale from geometry while retaining the learned rotation priorThe core challenge is to combine generative priors that complete unseen regions with reconstruction constraints that faithfully reproduce the observed evidence<ref:2610.12388#pg3>GenIA aligns generation to the observations at three levels. First, we recover translation and scale using depth-derived cues while retaining the generative rotation prior<ref:2610.12388#pg3>Second, we condition appearance on geometry and visibility, directing local features toward the image regions that observe them and fusing predictions across views or frames with visibility weighting<ref:2610.12388#pg3>Third, differentiable rendering provides both in-trajectory guidance and optional post-denoising refinement<ref:2610.12388#pg3>Our grounding mechanisms are geometry-predictor agnostic and therefore also support externally supplied shapes<ref:2610.12388#pg3>.

Alignment Levels

GenIA aligns generation to the observations at three levels<ref:2610.12388#pg3>First, we recover translation and scale using depth-derived cues while retaining the generative rotation prior<ref:2610.12388#pg3>. Second, we condition appearance on geometry and visibility, directing local features toward the image regions that observe them and fusing predictions across views or frames with visibility weighting<ref:2610.12388#pg3>. Third, differentiable rendering provides both in-trajectory guidance and optional post-denoising refinement<ref:2610.12388#pg3>.

Optional Shape Injection

When external geometry is available, GenIA replaces SAM3D’s coarse shape prediction by voxelizing S into an occupancy grid O and inverting it into the clean latent x O 1 expected by SAM3D’s frozen occupancy decoder<ref:2610.12388#pg5>Pose denoising (Sec. 4.2) is then conditioned on the injected latent; without external geometry, injection is skipped and SAM3D denoises coarse shape and pose jointly<ref:2610.12388#pg5>. For dynamic inputs, this external geometry additionally specifies an object-local deformation field δ, relating the canonical shape to each frame (details in Sec. A.3)<ref:2610.12388#pg5>.

Pose Prediction

GenIA uses SAM3D’s coarse geometry model to predict the reconstructed object’s world-space pose ρ k:= (R k, t k, s k) at frame k<ref:2610.12388#pg6>With an injected shape latent x O k1, the fixed shape conditions the pose integration (1) for 10 steps<ref:2610.12388#pg6>. For dynamic objects, the injected shapes already capture non-rigid motion and local rotation, so we only need to recover a single global root orientation R rather than one per frame<ref:2610.12388#pg6>. Depth-grounded translation and scale dominate SAM3D’s image-space pose errors (ablated in Sec. A.1)<ref:2610.12388#pg6>We therefore retain the denoised rotation R k and instead estimate translation and scale from geometric cues<ref:2610.12388#pg6>Given the camera-space pointmap P k, object mask M k, and intrinsics K k = (fx, fy, cx, cy), we measure three quantities: the object’s depth d k (the median z of P k over the mask), its center (¯u k, v¯u), and its angular size a k = max(∆u k/fx, ∆v k/fy)<ref:2610.12388#pg6>We then perform five fixed-point iterations that update, in order, scale, depth, and lateral position<ref:2610.12388#pg6>.

Appearance Prediction

With the coarse shape O fixed and the object pose ρ given, the appearance stage predicts a structured latent representation Z that encodes fine geometry and appearance<ref:2610.12388#pg6>Conditioned on one observation, the SAM3D backbone predicts the velocity for the latent trajectory<ref:2610.12388#pg6>At every denoising step, we first bias cross-attention toward image patches that observe each voxel and then apply rendering guidance to the resulting velocity prediction<ref:2610.12388#pg6>. The per-observation guided velocity predictions are then fused at every denoising step using visibility-based weights, yielding a shared latent trajectory<ref:2610.12388#pg6>Per-voxel visibility. A voxel occluded in a view receives no reliable appearance information from it<ref:2610.12388#pg6>We therefore compute a per-observation voxel visibility mask ν i(b) ∈ (0, 1)<ref:2610.12388#pg6>. Visibility attention bias. The appearance predictor crossattends to conditioning patches c<ref:2610.12388#pg6>We bias this attention toward the patches that directly observe the voxels<ref:2610.12388#pg6>For each voxel i and observation b, we add +α (α=5) to the logits of Π i(b) in the full- and cropped-RGB conditioning groups; occluded voxels receive no bias<ref:2610.12388#pg6>. Rendering guidance. The visibility bias acts through the frozen conditioning pathway<ref:2610.12388#pg6>Rendering guidance, instead, directly supervises the decoded reconstruction during the flow trajectory<ref:2610.12388#pg6>At each ODE step τ, we estimate the clean latent by extrapolating along the flow using the flow-matching analogue of Tweedie’s formula: Zˆ b(τ) = Z b(τ) + (1 − τ)v(b)θ<ref:2610.12388#pg6>. This is decoded to Gaussians and rendered under the fixed pose ρ b to obtain (ˆI b, D̂ b)<ref:2610.12388#pg6>The per-observation rendering loss combines photometric and depth supervision: L(b)rend = Lrgb(ˆI b, I b) + 0.2Lssim(ˆI b, I b) + 0.5Llpips(ˆI b, I b) + 0.1Ldepth(D̂ b, D b)<ref:2610.12388#pg6>.

Optional Test-time Refinement

We optionally refine a low-rank Gaussian-decoder adapter ∆Dg, together with the canonical latent Z and the world-space poses ρ for 100 iterations per object<ref:2610.12388#pg6>Using the rendering objective of (5), we optimize these test-time variables by rendering each observation as R(Dg(Z; ∆Dg), ρ k) and minimizing Lttr = Xnf k=1 Xnv j=1 L(k,j)rend + µ Z − Z 0 squared<ref:2610.12388#pg6>The latent anchor in (7) is an MSE to the initial latent Z 0 with µ = 0.01<ref:2610.12388#pg6>.

Datasets and Evaluation

We demonstrate this with ActionMesh [60], which provides canonical dynamic geometry but neither world-space placement nor appearance; GenIA complements it with input-aligned canonical appearance and temporally stable rigid placement<ref:2610.12388#pg3>Static datasets include GSO-30 [24], a synthetic multi-view benchmark with a single object on a white background, and selected CO3D [58] scenes, a real multi-view dataset with one object; both are also evaluated with a single input view<ref:2610.12388#pg8>Dynamic datasets include ActionBench [60], a synthetic monocular benchmark with a single object on a white background, which we re-render with one held-out view per timestamp; and DAVIS [54], comprising monocular in-the-wild videos with one object per scene (16 frames at stride 2 over the first 32 frames)<ref:2610.12388#pg8>.

Results Summary

Across datasets, GenIA aligns the frozen SAM3D model to the inputs, improving pose and appearance reconstruction (Tab. 1) while remaining competitive on runtime (Fig. 6)<ref:2610.12388#pg9>In the single-view setting, SAM3D, RecGen, and TRELLISv2 can realistically complete unobserved geometry but often misalign appearance, especially for symmetric objects<ref:2610.12388#pg9>Relative to SAM3D, we better resolve these ambiguities and improve color and lighting alignment<ref:2610.12388#pg9>On dynamic scenes, per-frame SAM3D is temporally inconsistent in geometry and appearance and provides no tracking correspondences<ref:2610.12388#pg9>HiMoR and Any4D reconstruct only observed regions<ref:2610.12388#pg9>With TTR, GenIA ranks second to HiMoR on DAVIS training-view PSNR and δavg, while achieving the lowest reprojection error and highest CLIP-I<ref:2610.12388#pg9>. On ActionBench, HiMoR leads training-view PSNR and co-PSNR<ref:2610.12388#pg9>, while GenIA is on par with per-frame SAM3D and achieves the best held-out LPIPS<ref:2610.12388#pg9>.

Ablations

We cumulatively ablate the three components of our appearance stage on top of SAM3D’s appearance prediction (Base): visibility attention bias (+Attn), rendering guidance (+RG), and test-time refinement (+TTR)<ref:2610.12388#pg14>The cumulative ladder shows that the combination of all three components yields the best results across static and dynamic settings<ref:2610.12388#pg15>. The performance gains from depth-grounded translation and scale provide the largest gains, reducing reprojection error by 1.4–2.8× and relative jitter by 2–4× across the dynamic settings while also improving static-view PSNR<ref:2610.

Improvements for AI systems

Bold Header: Inference-Time Input-Aligned Generation Framework

This framework grounds SAM3D’s generative prior in geometric and photometric observations without retraining the foundation model, improving object pose by deriving translation and scale from geometry while retaining the learned rotation prior.

Bold Header: Multi-View Reconstruction Capability

The system can perform reconstruction from monocular, multi-view, and dynamic inputs by aligning generation across all settings using a mechanism that spans monocular, multi-view, and dynamic inputs.

Bold Header: Dynamic Object Tracking and Consistency

When given temporal shapes of dynamic objects, the framework enables recovers a shared, input-aligned canonical appearance and stable world-space placement, leading to temporally consistent dynamic video reconstructions.

Bold Header: Appearance Alignment via Visibility Constraints

The system aligns appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising, ensuring that the predicted geometry is accompanied by a faithful appearance.

Bold Header: Optional Test-Time Refinement for Fidelity

A post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations, allowing for object-specific capacity beyond the frozen decoder’s range during inference.

Bold Header: Geometry Agnostic Grounding

The framework supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement, demonstrating that grounding mechanisms are geometry-predictor agnostic.

Abstract

Reconstructing complete 3D object assets from monocular or sparse multi-view observations remains challenging. Generative 3D foundation models can complete object geometry beyond the observed views, but their predictions may not faithfully reproduce the observed geometry, appearance, or pose. We introduce GenIA, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model. We improve object pose by deriving translation and scale from geometry while retaining the learned rotation prior, and align appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising. An optional post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations. Our framework also supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement. Across synthetic and real benchmarks, GenIA improves pose prediction and object reconstruction from monocular, multi-view, and dynamic inputs, outperforming recent optimization-based, per-frame image-to-3D, and video-to-4D methods. Our project page is available at https://facebookresearch.github.io/GenIA.