TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
summary
The gist
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry…
In short
This research introduces Test-Time Baking (TTB), a method to improve decoder-only view synthesis models by repurposing existing Test-Time Training layers. TTB learns cross-view attention mappings explicitly from input/output pairs, achieving state-of-the-art quality while maintaining the fast rendering speed of standard TTT. It resolves modality misalignment and uses local duplication to scale quality with more context views.
Key concepts
- Test-Time Baking (TTB)
- A strategy that learns global scene information from cross-view attention into a lightweight MLP during inference. Instead of just using pre-trained weights, TTB trains this small MLP on the fly using a specific loss function derived from ground truth novel views, allowing it to quickly decode high-quality novel views.
- Cross-View Attention Mapping
- The core idea is to use the fast weight MLP's learning process to explicitly learn how different context tokens relate across various input views. This mapping is learned by setting the inner-loop training target to the output of this attention mechanism, effectively making the model understand how different views influence each other for novel view generation.
- Modality Disalignment Resolution
- The paper addresses a mismatch where context tokens used for prefilling contain image content, while novel-view tokens are camera-only. This is fixed by separating context information into two distinct sets: 'context ground-truth tokens' (for mapping targets) and 'context rendering tokens' (for TTB input), disentangling the inputs for better learning.
- Local-Duplicate Inference
- To handle the fixed capacity of the fast weight MLP, this technique clusters context views into spatially-local groups. Each novel view only queries a corresponding copy of the baked MLP, ensuring O(1) per-token rendering cost while allowing quality to scale as more context views are introduced.
Terminology used across episodes
This episode discusses
- TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models · Paper Radio
- Long-LRM++: Preserving Fine Details in Feed-Forward Wide-Coverage Reconstruction
- Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention
- Test-Time Training with KV Binding Is Secretly Linear Attention
- GLU Variants Improve Transformer
- DINOv3
- DVSM: Decoder-only View Synthesis Model Done Right
- tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction
- MeshLRM: Large Reconstruction Model for High-Quality Meshes
- Test-Time Training Done Right
- E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
The paper
TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models · Read on arXiv
Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
NVIDIA Department of Computer Science National Yang Ming Chiao Tung University
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models".
Jane: Here's the gist of the paper. Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality The second key addresses a modality misalignment between context prefilling and novel-view rendering by disentangling context tokens into cameraonly context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target Our method achieves up to 23x rendering speedup with quality on par with the state-of-the-art across several benchmarks.
Tom: Pruning redundant reasoning tokens The authors propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding This strategy is motivated by the idea of trading context-view processing time for faster and higher-fidelity novel-view rendering, achieving the Pareto frontier of quality and speed shown in Figure 1.
Jane: Specifically, the inner-loop learning target becomes the output tokens from cross-view attention, instead of the raw value tokens used in TTT How the training signal is built The authors formalize their task as decomposing novel-view synthesis into two stages: Recon(I, P, K) and Rend(P′, K′, S).
Tom: The model is trained end-to-end against ground-truth novel views I′ by the objective J = MSE(¯I, I′) + λ · Percep(¯I, I′).
Title and authors: Tom: So we're looking at this paper called "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models," and the authors are trying to fix a big problem with how these large view synthesis models work, which is about making them run faster without losing too much quality.
Jane: That sounds like a challenge. So what's the core idea they’re proposing? Is it about making them faster, or is it about improving the quality of those novel views they generate?
Tom: It’s both, really. They started by looking at Test-Time Training layers, which are fast because they don't need to rebuild a lot of things during inference. But TTT sacrifices quality for that speed. This paper says you can repurpose those fast layers instead of just replacing the whole attention mechanism with something else entirely, achieving similar speed while getting much better results.
Lu: The main trick they use is to learn the cross-view attention mapping directly from its input and output pairs during training, instead of just using the raw value tokens as they did in earlier Test-Time Training methods. This lets them trade some context processing time for a faster and higher fidelity novel view decoding.
Meng: But that sounds complicated. How do you actually make sure this new method is better than just sticking with TTT, especially when you're shipping models to production? What are the practical trade-offs here?
Jane: Well, they tackle two main issues: first, they address a mismatch between the context prefilling and the actual novel view rendering. They disentangle those context tokens into two types: ground-truth tokens that carry image content, and camera only rendering tokens that mimic what is needed for novel view rendering.
Tom: That addresses a modality misalignment issue. It means the fast weight MLPs aren't struggling to understand different kinds of information when they move from prefilling to rendering. How does that actually help the model learn better?
Lu: By pairing those tokens, they optimize the fast weight MLP to map those camera only rendering tokens to their corresponding context ground-truth tokens, which carry the image and cross-view attended information. That way, the baked MLP is shared between both for retrieving scene info, keeping everything consistent across layers.
Meng: So it’s about making sure the model knows what it’s looking at during rendering, not just relying on a general prefilling step. What about the speedup they claim?
Jane: They say this approach can give up to a twenty-three times rendering speedup compared to their cross-attention baseline while keeping the quality very close to what you see in the state of the art models. It sits right on that trade-off line between quality and speed.
Tom: Twenty-three times faster is huge for deployment scenarios, especially if you're dealing with high frame rates. They also point out that this technique can preserve those fine details, like thin structures or small text, that other methods tended to blur out at higher speeds.
Title and authors: Lu: And they use a local-duplicate inference mechanism to handle the fixed capacity of their fast weight MLP. They cluster context views into spatially local groups using k means on per view camera poses, so each copy only needs to absorb about B views, and the stored state grows linearly with K, which is much better than quadratic scaling with the number of context views V.
Meng: That sounds like a practical way to manage memory and computation on hardware. If the memory footprint stays fixed regardless of how many context views you feed in, that’s something we can actually deploy. What about the training process itself?
Jane: They formalize it as two stages: one for recognition and one for rendering, with an objective function that includes both a mean squared error loss on ground-truth novel views and a perceptual loss term to keep the quality high. The TTB layer specifically uses a fast weight update rule where the inner loop target becomes the output tokens from cross-view attention instead of just raw value tokens.
Tom: That specific change in the inner loop target is key because it shifts Test-Time Baking beyond just interpreting linear attention and makes it collaborate with the attention mechanism itself. It's a targeted way to bake global scene information into that lightweight MLP.
Lu: They also mention that they removed the asymmetric query key projection and used a shared linear projection instead, and they use a SwiGLU MLP architecture for this fast weight part, plus an extra linear layer to predict per token learning rates. This helps control how much each part of the model learns during that baking process.
Meng: I’m curious about the inner loop objective again. If you're optimizing for something specific like cross-view attention output, does that mean you get a more accurate representation of the scene structure being baked?
Jane: Yes, because they show that swapping the inner loop target from just value tokens to the cross-view attention output broadens the effective view field of that baked fast weight MLP because it’s already aggregating across all context views. They stack all these components together—the context pairing, switching to MSE with three inner steps for TTB, and replacing the linear projection target with cross-view attention output—and they close the quality gap to models like DVSM after only five to seven views at every view count V.
Tom: So it’s a combination of smarter token handling and a more targeted training signal that lets them match DVSM’s quality at a constant twenty-three frames per second. That's a solid number when you compare it to the baseline.
Lu: And the scene representation size for TTB is actually fixed, just three hundred twenty-four megabytes across all L blocks in floating point thirty-two, which means it’s independent of V, and that compactness is what enables their constant time per token rendering.
Title and authors: Meng: That fixed memory footprint is what makes it practical for deployment. But they did admit a limitation: the capacity of that fast weight MLP is fixed, which can cause quality to plateau as the number of context views V gets really large. They use local-duplicate routing as a heuristic to manage that capacity ceiling.
Jane: And they also noted that the inner loop cross-view attention itself makes reconstruction quadratic in V, which is a drawback for scenarios where you want a single shot result without any context views at all. But when you stack all three things—the context pairing, the convergence-oriented inner loop training with three steps, and using cross-view attention as the target—they manage to match DVSM quality while keeping that speed from TTT.
Tom: So the implication is that baking scene information from attention outputs provides a new way to integrate complex attention mechanisms into a state of fixed size, which could be useful in any domain where you need to amortize an expensive but powerful attention module.
Lu: They suggest this baking perspective isn't just for novel view synthesis. It’s applicable to any domain where you have an expensive attention module that needs to be baked into a fixed size state, like in long context language models or physical perception systems that handle streaming sensor inputs.
Meng: From my side, the real question is whether this fixed size state is robust enough for real-world traffic. They’re constrained by that fixed capacity ceiling, and the local-duplicate routing is a bit of a workaround for when V gets massive.
Jane: It sounds like they’ve found a sweet spot where you get high quality and high speed without having to rebuild the entire attention mechanism every single time you want to render a new view. The whole concept behind TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models is about creating this compact, V independent state that performs well across the board.
Tom: Yeah, it’s a really neat piece of work if they can maintain those quality metrics across different benchmarks while maintaining that kind of speedup. It shows how repurposing existing components can yield big gains when you do it smartly.
Lu: And the results on few-view lower resolution settings show that TTB reproduces the cross-view attention results faithfully, which is a strong validation point, as reflected by comparable results with their pretrained DVSM models.
Meng: I just hope that in production environments, the fixed memory footprint doesn't become a bottleneck when you need to handle extremely high context view counts consistently.
Jane: That’s the practical reality of deployment, isn't it? But for now, TTB offers a very promising approach to getting fast and high-quality results from these decoder-only view synthesis models.
The paper's summary: Tom: So, we’ve been talking about Test-Time Baking for Efficient Rendering of Decoder-only View Synthesis Models, or TTB, and now we're going to look at what the whole paper is saying in summary and why it matters for us.
Jane: Okay, so to recap what we’ve got is that they took a fast way to generate novel views—Test-Time Training layers—and they repurposed them. They aren't just swapping out the attention mechanism; they are using those fast layers to learn how to bake the scene information directly from cross-view attention outputs during training.
Lu: Exactly. The core idea is trading off computation time for quality by learning that mapping explicitly from input and output pairs, which lets them hit a much better quality level on the Pareto frontier of speed and fidelity compared to just using those fast layers as they were before.
Tom: What’s the kicker there? They address a serious problem called modality misalignment between when you are prefilling context information and when you are actually rendering those novel views. They fix this by splitting the context tokens into ground-truth tokens that have the image content, and camera only rendering tokens that just have the camera information.
Jane: That separation means they make sure the fast weight MLPs don't get confused about what kind of information they’re supposed to be using during those different stages of processing. It lets them interpolate features much better across those layers that used to be mismatched.
Lu: And then they introduce a trick called local-duplicate inference. They cluster the context views into spatially local groups, so each novel view only has to query a copy of the baked fast weight MLP for about B views, instead of needing something that scales quadratically with every single context view V.
Meng: That fixed memory footprint is what I care about most. If the scene representation stays a fixed size and doesn't grow as you feed in more context, that makes deploying this model on hardware way more feasible for us right now.
Tom: They show that stacking all these pieces—the token pairing, switching the inner loop target to cross-view attention output, and using three steps in the MSE loss—gets them matching the quality of a model called DVSM after only five or seven views at every context view count V.
Jane: So what this means for you listening right now is that they’ve found a way to get performance that rivals state-of-the-art models, but with a rendering speedup up to twenty three times, and it keeps those fine details like thin structures and small text from getting blurred out.
Lu: The implication is they have created this compact, V independent scene representation—just three hundred twenty-four megabytes across all layers—which is incredibly efficient for deployment scenarios with limited GPU memory.
Meng: I see the limitation right away though: that fast weight MLP still has a fixed capacity, which means if you feed it way more context views than they planned for, the quality will start to plateau because of that capacity ceiling.
Tom: Right, so the local-duplicate routing is essentially a workaround for when V gets really big and you need to keep things running fast. They also admit that baking this way makes reconstruction quadratic in V, which is bad if you want a single shot result without any context views at all.
Jane: But they showed that by combining everything—the targeted training signal, the attention output target, and the local clustering—they manage to close the quality gap to DVSM while keeping that speed from Test-Time Training.
Lu: The authors conclude that this baking perspective is a new way to integrate complex attention mechanisms into a state of fixed size, which could be useful in any domain where you need to amortize an expensive but powerful attention module.
Tom: It’s basically showing how repurposing existing components can yield big gains when you do it smartly, moving beyond just replacing the whole thing.
Jane: And they even suggest this baking method isn't just for novel view synthesis. It’s applicable to any domain where you have an expensive attention module that needs to be baked into a fixed size state, like in long context language models or physical perception systems that handle streaming sensor inputs.
Meng: That sounds ambitious for real-world deployment, but the memory savings are definitely compelling enough to keep us watching this space.
The paper's improvements: Tom: So we’re moving on to what they actually did to make TTB better than just a basic version of it, and Jane, what are the specific improvements they suggest?
Jane: They focused on making the training signal more targeted during that inner loop optimization. Instead of just using an unnormalized dot-product loss, they switched to a convergence-oriented inner loop where they train the fast weight MLP via a mean squared error loss over three steps.
Lu: That change is significant because it means they are learning the residual to the mapping target, which makes it much easier to bake that information into the fast weights than if they were just trying to learn the final target directly.
Tom: So, what does that do for us on a practical level? Does it mean faster convergence or better accuracy?
Lu: It means their rendering loss reflects the exact dependence of those final fast weights on the outer loop parameters, which should lead to much more accurate baking of the scene structure itself.
Jane: That feeds right back into what we talked about earlier—it helps them close that quality gap to models like DVSM by ensuring the baked representation is really faithful to what they are trying to render.
Tom: Okay, so we’ve got better training signal, now let’s talk about how they handle those context tokens again. What else did they change in their setup?
Lu: They also made a change to how the queries and keys are created inside that fast weight layer. Instead of using an asymmetric query key projection, they used a shared linear projection to produce both the queries and the keys.
Jane: That's about making things more symmetric, which should help with stability during the baking process itself. It simplifies how those tokens interact within the MLP structure.
Meng: From an engineering standpoint, simplifying those projections is good for deployment because it reduces complexity in the inference path, even if it doesn't change the final speed much.
Tom: And they also added a layer to predict per token learning rates within that fast weight part. It’s like giving each part of the model a little nudge on how fast to learn during baking, which lets them control that process more finely.
Lu: That fine-grained control is what lets them manage the fixed capacity issue better, as you can tune how much influence each component has on the overall baked representation.
Jane: So to put it simply, they are tightening the screws on the training process—making it smarter about what information it’s baking and how fast each part learns—to get a higher fidelity result without needing massive amounts of context views at every step.
Tom: It sounds like they’re really pushing that boundary where baking and attention mapping start to work together more tightly than before.
Lu: Exactly. They're showing that the way you structure the inner loop objective and how you project those keys and queries matters a lot in making this baking strategy effective.
Jane: And this whole approach suggests that if you can bake complex attention mechanisms into a fixed size state using these targeted training signals, it opens up new possibilities for systems where an expensive attention module needs to be compressed.
Meng: That compression is key; we don't want to pay the computational price every single time we want a new view.
Conclusion: Tom: So we're wrapping up our look at TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models, and Jane, what's the final word on what this actually means?
Jane: To sum it all up, this paper shows how you can take a fast rendering technique and bake scene information directly from attention outputs during training, which gives you a quality boost while keeping rendering speeds very high.
Lu: It’s about creating a fixed-size scene representation that doesn't depend on the number of context views V, which is something we really need for practical deployment.
Meng: For me, it means if we can get this kind of memory footprint down to a few hundred megabytes, it becomes viable for running complex view synthesis models on standard hardware instead of needing massive GPU clusters just to handle the context window.
Lalam: I'm confident in my assessment; the paper is very solid. It proves that you can decouple the expensive attention mechanism from every single inference step by baking its useful knowledge into a small, baked structure.
Tom: It’s impressive how they managed to keep it at constant time per token rendering while still improving quality over models like DVSM, especially on those in-the-wild datasets where fine details matter.
Jane: And the fact that it recovers things like thin structures and small text that other methods blur out is a big win for visual fidelity.
Lu: The core idea is powerful because it lets you integrate complex attention into a fixed size state, which is exactly what we need when dealing with long context inputs or streaming sensor data.
Meng: Yeah, the capacity ceiling they mentioned still worries me though; if you push V too far, that fixed state might start limiting quality again.
Tom: And they use local-duplicate routing as a heuristic to manage that capacity problem, so it’s not just a theoretical concept but something they’ve actually implemented.
Jane: It really shows how much attention and rendering can be integrated in new ways beyond just simple view synthesis.
Lu: This baking perspective is definitely applicable beyond novel view synthesis; it could be useful in any domain where you need to compress an expensive attention module into a fixed state.
Meng: I just hope that when we move this into real-world traffic, the memory management doesn't become the bottleneck for our inference pipeline.
Lalam: I'm very sure about that; TTB achieves what it sets out to do with solid evidence, and it provides a much more efficient route to high-quality novel view generation.
Tom: So we’ve seen how repurposing existing components can yield big gains when you do it smartly in the paper titled "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models."
Jane: It’s a really neat piece of work that shows how to get fast and high-quality results from these decoder-only view synthesis models.
Lu: We're looking forward to seeing how this fixed representation idea plays out in other areas where we need to amortize expensive attention modules.
More episodes
- 2603.29765-Training-Free Dynamic Upcycling of Expert Language Models
- 2605.00778-Observable Performance Does Not Fully Reflect Adaptive System Organization: A Multi-Level Analysis of Gait Dynamics Under Occlusal Constraint
- 2605.06906-TraXion: Rethinking Pre-training Frameworks for Mobility and Beyond
- 2605.09165-Sparse Layers are Critical to Scaling Looped Language Models
- 2605.09439-Inverse Design for Conditional Distribution Matching
- 2605.10313-Signature Approach for Contextual Bandits with Nonlinear and Path-dependent Rewards: Regret Bound and Statistical Inference
- 2605.10878-Minimum Neural Weight Norm = Kolmogorov Complexity
- 2605.11617-MIST: Reliable Streaming Decision Trees for Online Class-Incremental Learning via McDiarmid Bound
- 2605.15085-From Data to Action: Accelerating Refinery Optimization with AI
- 2605.23500-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation