Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement

arXiv:2610.11857 · cs.CV · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support Token Refinement".

Jane: The gist: FreeInpaint presents a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions, eliminating reliance on precomputed camera poses.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper titled "Pose-Free Feed-Forward three dee Inpainting via Learnable Mask Attention and Support Token Refinement <ref:2610.11857#pg1,Pose-Free Feed-Forward 3D Inpainting via Learnable Mask Attention and Support>." What the authors are trying to do is basically create a way to fill in missing or hidden parts of a three dee scene, but without needing those precomputed camera poses that everyone else usually requires <ref:2610.11857#pg1>.

Jane: That’s right, they present this new feed-forward framework called FreeInpaint. The main idea is that you can generate complete and three dee-consistent scenes directly from a set of unposed multi-view images where some parts are masked out <ref:2610.11857#pg1,complete and 3D-consistent scenes directly from>.

Lu: What makes it novel is that instead of relying on accurately calibrated camera poses first, they extend a three dee foundation model to propagate those masked regions from one reference view to all the other unposed views <ref:2610.11857#pg1,a 3D foundation model to propagate>. It bridges the gap between three dee reconstruction and scene inpainting while keeping the model’s ability to recover geometry and camera parameters intact <ref:2610.11857#pg1,3D reconstruction and scene inpainting while>.

Meng: So, if I understand this right, it means we don't need a heavy pre-processing step like running Structure-from-Motion or COLMAP before we can start inpainting?

Tom: Exactly. The paper points out that existing methods often require that SfM step first, which is time-consuming and brittle when you have sparse views or masked observations. They are trying to make this process faster and more interactive for casual use cases like in-the-wild captures, by eliminating that initial pose estimation bottleneck.

Jane: And they tackle two specific challenges when adapting these feed-forward three dee foundation models to inputs with masks <ref:2610.11857#pg1,feed-forward 3D foundation models to>. First, masked regions can mess up the cross-view correspondence reasoning, which hurts both pose estimation and geometry recovery.

Lu: To handle that, they introduce a Learnable Mask Attention mechanism. This mechanism is designed to keep the spatial anchoring of reliable observations while letting the masked regions gradually soak up useful context in deeper layers. It calculates cross-view attention using this formula: Attention Q K top root d plus B(l) V, where B(l) is a token-level mask bias that starts from an explicit input occlusion mask and gets refined through learnable residuals across layers.

Paper summary: Meng: That sounds like a clever way to stop the masked areas from corrupting the overall scene understanding while still letting them participate in the process. But what about when things are really dark, like severe occlusions, where a single forward pass just doesn't have enough appearance evidence for high-fidelity completion?

Tom: Ah, that’s where their second key challenge comes in. To address that under severe occlusions, they propose a Support Token Refinement strategy. This involves three steps: first, Blind Spot Detection. They compute a visibility score Vi equals P alpha i render times M i over P M i plus epsilon; low opacity means there's not enough support there.

Jane: Then, for the area that needs refinement, they generate a Support View using a diffusion prior conditioned on the current scene context, using Stochastic Differential Editing and IP-Adapter to create a complementary view.

Lu: And finally, they inject those generated support view tokens into the main model in a confidence-weighted manner to help refine those under-observed regions while keeping the original spatial anchor. It's like adding expert guidance when the model feels uncertain.

Meng: From an engineering standpoint, I’m interested in how they handle training this whole system together without needing perfect ground truth poses upfront. The training objectives look pretty detailed on page three and beyond, involving mask generation and depth-guided self-distillation using a frozen copy of DA3 to generate pseudo ground truth depth maps Dprior <ref:2610.11857#pg1>.

Tom: They use a hybrid masking strategy mixing random masks and geometric masks at a ratio of zero point four to zero point six, where the geometric masks are projected from the reference view to the source views using metric depth estimated by the original pre-trained DA3.

Jane: The overall loss function combines a photometric loss, Lrgb, which uses MSE alongside an LPIPS perceptual penalty, with a multi-view depth consistency loss called Ldepth equals norm Dpred times M minus Dprior times M.

Lu: The results they show are quite strong across different datasets like SPIn-NeRF, three hundred sixty-USID, and LLFF. On the SPIn-NeRF dataset alone, FreeInpaint achieves a PSNR of zero point two thousand eight hundred nineteen and an LPIPS of zero point two thousand five hundred twenty-six.

Meng: And the inference speed is impressive for what it does, taking only about two hundredths of a second on a single A6000 GPU for one forward pass. That's fast enough to be useful in many applications.

Paper summary: Tom: Fast is good, but they also highlight some limitations with this paper. They admit that the Support Token Refinement relies on 2D diffusion models to fill those blind spots, so the final three dee quality is limited by how good that 2D prior is <ref:2610.11857#pg1>.

Jane: And they also mention that in extreme cases, like when you only have one or two restricted views of a complex scene, error can accumulate during the support token refinement process. Future work could look at integrating native three dee or multi-view consistent video diffusion priors to make the geometric consistency more stable across arbitrary refinement steps <ref:2610.11857#pg1>.

Lu: They also suggest a persistent spatial memory mechanism for future iterations to maintain long-term geometric consistency across multiple refinement steps, which would solve those accumulation issues. This paper establishes a framework for pose-free three dee scene inpainting, and it shows how you can do this without the traditional heavy preprocessing overhead <ref:2610.11857#pg1>.

Tom: So, what does this mean practically? It means we can get high-quality three dee scenes from just a few images taken casually, bypassing the need for that complex initial calibration step that used to slow everything down <ref:2610.11857#pg1>.

Jane: Basically, it’s about making three dee scene reconstruction and inpainting accessible in scenarios where you don't have perfect camera knowledge from the start <ref:2610.11857#pg1>. This framework, FreeInpaint, shows how to keep a feed-forward model powerful while making it adaptable to real-world data inputs.

Lu: It opens up possibilities for creating interactive applications where users can manipulate three dee scenes directly using unposed inputs without needing a separate SfM pipeline running on the backend <ref:2610.11857#pg1>.

Meng: For practical implementation, the focus will be on making that Support Token Refinement robust enough to handle those truly sparse or heavily occluded scenes we encounter in real-world capture.

Tom: And for the listener, it means future tools for creating three dee environments or editing existing ones won't always need you to start by meticulously setting up a camera rig; they can just give them the images <ref:2610.11857#pg1>.

Jane: That’s what this paper is about: taking a complex task and showing how to solve it using a more direct, feed-forward approach that handles the input variability much better than previous reference-based methods.

Conclusion: Tom: So we've looked at how FreeInpaint tackles three dee scene inpainting without needing those tricky precomputed camera poses that usually slow everything down.

Jane: It really focuses on making this process direct, using unposed images with masked spots to generate a whole, consistent three dee world.

Lu: The authors are doing this by extending an existing three dee foundation model to let it propagate information across those missing parts from one view to all the others.

Meng: So they’re aiming for something fast and adaptable without needing that initial calibration step first.

Tom: Exactly, they’re trying to bypass that heavy pre-processing bottleneck entirely so you can just give them the images and get a three dee scene back quickly.

Jane: And it does this by using two main tricks, learnable mask attention to handle the corrupted views and a support token refinement strategy for when things are really dark.

Lu: The mechanism they use to stop masked areas from messing up pose estimation is called learnable mask attention, which uses a per-token bias that gets refined through layers.

Meng: I see how that would be useful in practice because it keeps the reliable parts anchored while letting the masked regions slowly absorb context.

Tom: And when those areas are totally occluded, they inject support tokens generated by a diffusion model to give the model confidence where it’s blind.

Jane: The overall training involves mixing random and geometric masks and using depth-guided self-distillation to keep the geometry consistent during learning.

Lu: They train it by forcing the predicted depth map to match a pseudo ground truth from a frozen copy of Depth Anything three which anchors the three dee structure.

Meng: On the results side, they show good numbers on datasets like SPIn-NeRF, achieving PSNRs around zero point two eight and inference times under one second on an A6000 GPU.

Tom: Those are solid numbers for pose-free inpainting, but we have to keep an eye on their limitations because they admit the support token refinement still relies on 2D diffusion models for those blind spots <ref:2610.11857#pg1>.

Jane: And they flag that extreme cases, like very few views of a complex scene, can still lead to error accumulation during that refinement step.

Lu: The implication here is that while this is a powerful feed-forward approach, the final three dee quality in the most difficult scenes depends heavily on how good those initial 2D priors are <ref:2610.11857#pg1>.

Meng: It means for real-world use, you might still need a robust way to handle those highly constrained environments if you want perfect geometry.

Tom: So basically, FreeInpaint gives us a path toward quick, pose-free three dee scene generation from just photos, but we’re still working on making it perfectly stable in the worst capture scenarios.

Jane: Next time we talk about this paper, we’ll be looking at how they plan to integrate native three dee diffusion priors to fix those geometric stability issues.

Jingyi Pan, Dan Xu, Qiong Luo

Hong Kong University of Science and Technology

cs.CV

Submitted: 2026-10-08

Updated: 2026-10-08

Comments: Accepted to NeurIPS 2026 (poster). Project page: https://rorisis.github.io/FreeInpaint/

Project page: https://rorisis.github.io/FreeInpaint

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The gist: FreeInpaint presents a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions, eliminating reliance on

Key concepts

Learnable Mask Attention
This mechanism prevents masked areas from ruining pose and geometry estimation. Instead of blocking masked tokens entirely, it uses a learnable bias that is progressively refined across layers. This allows the model to preserve reliable spatial anchors while enabling masked regions to gradually absorb useful context from deeper parts of the network.
Support Token Refinement
This strategy fixes poor completion in severely occluded spots during inference. It detects blind spots, generates a complementary support view using 2D diffusion models, and injects this evidence into the model as confidence-weighted tokens. This helps refine under-observed regions while respecting the original spatial anchors.
Depth-Guided Self-Distillation
During training, the method ensures geometric consistency by treating depth reconstruction like a self-distillation process. It uses a frozen depth estimation model to create pseudo ground truth depths, which the main model then tries to match. This loss forces the trainable model to learn geometrically consistent scene structures.
Mask Generation
The training uses a hybrid masking approach combining random masks and geometric masks. Geometric masks are projected from one view onto others using depth estimates derived from a pre-trained model. This ensures the training process learns how to handle both random missing data and structurally related missing areas.

Terminology

Summary

The gist: FreeInpaint presents a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions, eliminating reliance on precomputed camera poses.

How it works

FreeInpaint extends a 3D foundation model to propagate masked regions from a reference view to other unposed views, bridging 3D reconstruction and scene inpainting while preserving the model’s native ability to recover camera poses and scene geometry The method addresses two key challenges in adapting feed-forward 3D foundation models to masked inputs First, masked regions can corrupt crossview correspondence reasoning, degrading pose estimation and geometry recovery. To address this, FreeInpaint introduces a Learnable Mask Attention mechanism that preserves the spatial anchoring of reliable observations while allowing masked regions to progressively absorb useful context in deeper layers. Second, under severe occlusions, a single forward pass often lacks sufficient appearance evidence for high-fidelity completion. Therefore, we propose a Support Token Refinement strategy, which injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions while preserving the original spatial anchor.

Learnable Mask Attention

The Learnable Mask Attention mechanism is designed to prevent masked regions from corrupting pose and geometry estimation under masked inputs A straightforward strategy is to apply a hard binary attention mask that permanently blocks masked regions, but this formulation is overly restrictive for inpainting. The mechanism introduces a per-token, key-side attention bias initialized from explicit input occlusion masks and progressively updated across layers through learnable residuals. For each cross-view attention layer l, we derive a token-level mask bias B(l) from the corresponding source tokens and binary masks. This bias is initialized from the binary mask and then progressively refined through a lightweight layer-wise network conditioned on the current source tokens. The cross-view attention is computed as Attention(Q, K, V) = Softmax QK⊤√d + B(l)V.

Support Token Refinement

The Support Token Refinement strategy is proposed to improve blind-spot completion at inference time when a single feed-forward pass remains under-confident in severely occluded regions. This involves three steps:

  1. Blind Spot Detection: We compute a visibility score over the masked region, Vi = Pαi render ⊙ Mi / PMi + ϵ, where low opacity indicates insufficient support.

  2. Support View Generation: For the selected anchor view, we introduce a diffusion prior to generate a complementary support view conditioned on the current scene context using Stochastic Differential Editing (SDEdit) and IP-Adapter.

  3. Support Token Injection: The generated support view is encoded into auxiliary tokens, which are then injected into the model in a confidence-weighted manner to refine under-observed regions.

Training Objectives and Optimization

The training strategy involves several components to ensure geometric consistency during inpainting

** Mask Generation:**

**: We employ a hybrid masking strategy that mixes random masks and geometric masks at a ratio of 0.4:0.6, where geometric masks are projected from the reference view to source views using the metric depth estimated by the original pre-trained DA3. **

** Depth-Guided Self-Distillation:**

**: We formulate the geometric reconstruction as a self-distillation process, leveraging the strong spatial priors embedded in the original foundation model by passing unmasked source and novel views through a frozen copy of DA3 to generate pseudo ground-truth depth maps Dprior. The trainable model predicts Dpred, and we enforce geometric consistency via a masked distillation loss Ldepth = ∥Dpred ⊙ M − Dprior ⊙ M∥1. **

** Overall Loss:**

**: The overall training objective is formulated as a combination of a photometric loss and a multi-view depth consistency loss: Ltotal = Lrgb + λdepthLdepth. The photometric loss Lrgb is computed using an MSE loss alongside an LPIPS perceptual penalty. **

Results and Evaluation

Extensive experiments across diverse datasets demonstrate that FreeInpaint achieves superior inpainting quality, eliminating the reliance on precomputed camera poses while keeping a fast inference speed. The method is evaluated on SPIn-NeRF [27], 360-USID [41], and LLFF [24] datasets. Quantitative comparisons show that FreeInpaint consistently outperforms prior methods across most metrics while maintaining rapid inference speeds. For example, on the SPIn-NeRF dataset, FreeInpaint achieves a PSNR of 0.2819 and an LPIPS of 0.2526. The inference time is measured on a single A6000 GPU and requires only ∼0.4s for a single forward pass. Furthermore, the learnable mask attention significantly sharpens structural details by suppressing invalid features.

Limitations and Future Work

While FreeInpaint establishes a robust framework for pose-free 3D scene inpainting, it has some limitations. Primarily, our support token refinement relies on 2D diffusion models to inpaint blind spots, making the final 3D quality inherently bounded by the performance of the 2D prior. Future research could integrate native 3D or multi-view consistent video diffusion priors to ensure more stable geometric consistency. Additionally, extreme cases involving only 1 or 2 restricted views of a complex 360-degree scene remain challenging due to error accumulation during support token refinement. To address this, future iterations could incorporate a persistent spatial memory mechanism to maintain long-term geometric consistency across arbitrary refinement steps.

Acknowledgments

The work of Qiong Luo and Jingyi Pan is supported by a startup grant from the Hong Kong University of Science and Technology (Guangzhou). The work of Dan Xu is supported by the Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321, ITF PRP/046/24FX, Science & Technology Cooperation Program of Shandong under grant No. SDST26EG01, SAIL Research Project, and HKUST-Zeekr Collaborative Research Fund.

References

The references cited in the paper are listed in the References section. The references include works such as Depth Anything 3 [39] and 3D Gaussian Splatting [40]. The paper also cites methods like Spin-nerf [27] and Aurafusion360 [41]. The references cover foundational models like DINOv2 [28] and Swin Transformer [20]. The paper also cites methods like Ip-adapter [44] and Sdedit [23]. The references include works such as Mapanything [33] and Structure-from-Motion revisited [33]. The references also include works such as Mvinpainter [5] and InFusion [21]. The references cover foundational models like Nerf [25] and 3D Gaussian Splatting [3DGS] [40].

Improvements for AI systems

  1. No reliance on pre-computed camera poses for 3D scene inpainting: FreeInpaint, a novel feed-forward framework that generates complete and 3D-consistent scenes directly from unposed multi-view images with masked regions. This enables application in casual, in-the-wild scenarios by eliminating the need for time-consuming Structure-from-Motion (SfM) preprocessing.

  2. Robust handling of corrupted crossview correspondence: The Learnable Mask Attention mechanism prevents masked regions from corrupting pose and geometry estimation by introducing a bias term to attention logits, allowing the network to adaptively regulate how masked regions participate in cross-view reasoning at different depths. This ensures spatial anchoring of reliable observations is preserved.

  3. High-fidelity completion in severe occlusions: The Support Token Refinement strategy injects diffusion-generated support evidence as confidence-weighted auxiliary tokens to refine under-observed regions, specifically addressing scenarios where a single forward pass lacks sufficient appearance evidence for high-fidelity completion.

  4. Geometric consistency through self-distillation: During training, the model enforces geometric constraints by using a frozen DA3 as a teacher network to generate pseudo ground truth depth maps (Dprior) and minimizing the loss Ldepth =∥Dpred ⊙ M − Dprior ⊙ M∥1, which mitigates hallucinated textures onto incorrect depth planes.

  5. Adaptive mask-aware reasoning: The framework utilizes learnable mask attention to determine how strongly masked regions should participate in cross-view reasoning, as evidenced by the improvement of camera rotation and translation accuracy using Absolute Trajectory Error (ATE) and Relative Pose Error (RPE).

  6. Optimized inference efficiency: The method achieves a fast inference speed of just 0.4s via a single forward pass, which is orders of magnitude faster than optimization baselines, offering an efficient solution for real-time or interactive user-facing applications.

Sources

Related papers