TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models".
Jane: Here's the gist of the paper. Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality The second key addresses a modality misalignment between context prefilling and novel-view rendering by disentangling context tokens into cameraonly context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target Our method achieves up to 23x rendering speedup with quality on par with the state-of-the-art across several benchmarks.
Tom: Pruning redundant reasoning tokens The authors propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding This strategy is motivated by the idea of trading context-view processing time for faster and higher-fidelity novel-view rendering, achieving the Pareto frontier of quality and speed shown in Figure 1.
Jane: Specifically, the inner-loop learning target becomes the output tokens from cross-view attention, instead of the raw value tokens used in TTT How the training signal is built The authors formalize their task as decomposing novel-view synthesis into two stages: Recon(I, P, K) and Rend(P′, K′, S).
Tom: The model is trained end-to-end against ground-truth novel views I′ by the objective J = MSE(¯I, I′) + λ · Percep(¯I, I′).
Title and authors: Tom: So we're looking at this paper called "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models," and the authors are trying to fix a big problem with how these large view synthesis models work, which is about making them run faster without losing too much quality.
Jane: That sounds like a challenge. So what's the core idea they’re proposing? Is it about making them faster, or is it about improving the quality of those novel views they generate?
Tom: It’s both, really. They started by looking at Test-Time Training layers, which are fast because they don't need to rebuild a lot of things during inference. But TTT sacrifices quality for that speed. This paper says you can repurpose those fast layers instead of just replacing the whole attention mechanism with something else entirely, achieving similar speed while getting much better results.
Lu: The main trick they use is to learn the cross-view attention mapping directly from its input and output pairs during training, instead of just using the raw value tokens as they did in earlier Test-Time Training methods. This lets them trade some context processing time for a faster and higher fidelity novel view decoding.
Meng: But that sounds complicated. How do you actually make sure this new method is better than just sticking with TTT, especially when you're shipping models to production? What are the practical trade-offs here?
Jane: Well, they tackle two main issues: first, they address a mismatch between the context prefilling and the actual novel view rendering. They disentangle those context tokens into two types: ground-truth tokens that carry image content, and camera only rendering tokens that mimic what is needed for novel view rendering.
Tom: That addresses a modality misalignment issue. It means the fast weight MLPs aren't struggling to understand different kinds of information when they move from prefilling to rendering. How does that actually help the model learn better?
Lu: By pairing those tokens, they optimize the fast weight MLP to map those camera only rendering tokens to their corresponding context ground-truth tokens, which carry the image and cross-view attended information. That way, the baked MLP is shared between both for retrieving scene info, keeping everything consistent across layers.
Meng: So it’s about making sure the model knows what it’s looking at during rendering, not just relying on a general prefilling step. What about the speedup they claim?
Jane: They say this approach can give up to a twenty-three times rendering speedup compared to their cross-attention baseline while keeping the quality very close to what you see in the state of the art models. It sits right on that trade-off line between quality and speed.
Tom: Twenty-three times faster is huge for deployment scenarios, especially if you're dealing with high frame rates. They also point out that this technique can preserve those fine details, like thin structures or small text, that other methods tended to blur out at higher speeds.
Title and authors: Lu: And they use a local-duplicate inference mechanism to handle the fixed capacity of their fast weight MLP. They cluster context views into spatially local groups using k means on per view camera poses, so each copy only needs to absorb about B views, and the stored state grows linearly with K, which is much better than quadratic scaling with the number of context views V.
Meng: That sounds like a practical way to manage memory and computation on hardware. If the memory footprint stays fixed regardless of how many context views you feed in, that’s something we can actually deploy. What about the training process itself?
Jane: They formalize it as two stages: one for recognition and one for rendering, with an objective function that includes both a mean squared error loss on ground-truth novel views and a perceptual loss term to keep the quality high. The TTB layer specifically uses a fast weight update rule where the inner loop target becomes the output tokens from cross-view attention instead of just raw value tokens.
Tom: That specific change in the inner loop target is key because it shifts Test-Time Baking beyond just interpreting linear attention and makes it collaborate with the attention mechanism itself. It's a targeted way to bake global scene information into that lightweight MLP.
Lu: They also mention that they removed the asymmetric query key projection and used a shared linear projection instead, and they use a SwiGLU MLP architecture for this fast weight part, plus an extra linear layer to predict per token learning rates. This helps control how much each part of the model learns during that baking process.
Meng: I’m curious about the inner loop objective again. If you're optimizing for something specific like cross-view attention output, does that mean you get a more accurate representation of the scene structure being baked?
Jane: Yes, because they show that swapping the inner loop target from just value tokens to the cross-view attention output broadens the effective view field of that baked fast weight MLP because it’s already aggregating across all context views. They stack all these components together—the context pairing, switching to MSE with three inner steps for TTB, and replacing the linear projection target with cross-view attention output—and they close the quality gap to models like DVSM after only five to seven views at every view count V.
Tom: So it’s a combination of smarter token handling and a more targeted training signal that lets them match DVSM’s quality at a constant twenty-three frames per second. That's a solid number when you compare it to the baseline.
Lu: And the scene representation size for TTB is actually fixed, just three hundred twenty-four megabytes across all L blocks in floating point thirty-two, which means it’s independent of V, and that compactness is what enables their constant time per token rendering.
Title and authors: Meng: That fixed memory footprint is what makes it practical for deployment. But they did admit a limitation: the capacity of that fast weight MLP is fixed, which can cause quality to plateau as the number of context views V gets really large. They use local-duplicate routing as a heuristic to manage that capacity ceiling.
Jane: And they also noted that the inner loop cross-view attention itself makes reconstruction quadratic in V, which is a drawback for scenarios where you want a single shot result without any context views at all. But when you stack all three things—the context pairing, the convergence-oriented inner loop training with three steps, and using cross-view attention as the target—they manage to match DVSM quality while keeping that speed from TTT.
Tom: So the implication is that baking scene information from attention outputs provides a new way to integrate complex attention mechanisms into a state of fixed size, which could be useful in any domain where you need to amortize an expensive but powerful attention module.
Lu: They suggest this baking perspective isn't just for novel view synthesis. It’s applicable to any domain where you have an expensive attention module that needs to be baked into a fixed size state, like in long context language models or physical perception systems that handle streaming sensor inputs.
Meng: From my side, the real question is whether this fixed size state is robust enough for real-world traffic. They’re constrained by that fixed capacity ceiling, and the local-duplicate routing is a bit of a workaround for when V gets massive.
Jane: It sounds like they’ve found a sweet spot where you get high quality and high speed without having to rebuild the entire attention mechanism every single time you want to render a new view. The whole concept behind TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models is about creating this compact, V independent state that performs well across the board.
Tom: Yeah, it’s a really neat piece of work if they can maintain those quality metrics across different benchmarks while maintaining that kind of speedup. It shows how repurposing existing components can yield big gains when you do it smartly.
Lu: And the results on few-view lower resolution settings show that TTB reproduces the cross-view attention results faithfully, which is a strong validation point, as reflected by comparable results with their pretrained DVSM models.
Meng: I just hope that in production environments, the fixed memory footprint doesn't become a bottleneck when you need to handle extremely high context view counts consistently.
Jane: That’s the practical reality of deployment, isn't it? But for now, TTB offers a very promising approach to getting fast and high-quality results from these decoder-only view synthesis models.
The paper's summary: Tom: So, we’ve been talking about Test-Time Baking for Efficient Rendering of Decoder-only View Synthesis Models, or TTB, and now we're going to look at what the whole paper is saying in summary and why it matters for us.
Jane: Okay, so to recap what we’ve got is that they took a fast way to generate novel views—Test-Time Training layers—and they repurposed them. They aren't just swapping out the attention mechanism; they are using those fast layers to learn how to bake the scene information directly from cross-view attention outputs during training.
Lu: Exactly. The core idea is trading off computation time for quality by learning that mapping explicitly from input and output pairs, which lets them hit a much better quality level on the Pareto frontier of speed and fidelity compared to just using those fast layers as they were before.
Tom: What’s the kicker there? They address a serious problem called modality misalignment between when you are prefilling context information and when you are actually rendering those novel views. They fix this by splitting the context tokens into ground-truth tokens that have the image content, and camera only rendering tokens that just have the camera information.
Jane: That separation means they make sure the fast weight MLPs don't get confused about what kind of information they’re supposed to be using during those different stages of processing. It lets them interpolate features much better across those layers that used to be mismatched.
Lu: And then they introduce a trick called local-duplicate inference. They cluster the context views into spatially local groups, so each novel view only has to query a copy of the baked fast weight MLP for about B views, instead of needing something that scales quadratically with every single context view V.
Meng: That fixed memory footprint is what I care about most. If the scene representation stays a fixed size and doesn't grow as you feed in more context, that makes deploying this model on hardware way more feasible for us right now.
Tom: They show that stacking all these pieces—the token pairing, switching the inner loop target to cross-view attention output, and using three steps in the MSE loss—gets them matching the quality of a model called DVSM after only five or seven views at every context view count V.
Jane: So what this means for you listening right now is that they’ve found a way to get performance that rivals state-of-the-art models, but with a rendering speedup up to twenty three times, and it keeps those fine details like thin structures and small text from getting blurred out.
Lu: The implication is they have created this compact, V independent scene representation—just three hundred twenty-four megabytes across all layers—which is incredibly efficient for deployment scenarios with limited GPU memory.
Meng: I see the limitation right away though: that fast weight MLP still has a fixed capacity, which means if you feed it way more context views than they planned for, the quality will start to plateau because of that capacity ceiling.
Tom: Right, so the local-duplicate routing is essentially a workaround for when V gets really big and you need to keep things running fast. They also admit that baking this way makes reconstruction quadratic in V, which is bad if you want a single shot result without any context views at all.
Jane: But they showed that by combining everything—the targeted training signal, the attention output target, and the local clustering—they manage to close the quality gap to DVSM while keeping that speed from Test-Time Training.
Lu: The authors conclude that this baking perspective is a new way to integrate complex attention mechanisms into a state of fixed size, which could be useful in any domain where you need to amortize an expensive but powerful attention module.
Tom: It’s basically showing how repurposing existing components can yield big gains when you do it smartly, moving beyond just replacing the whole thing.
Jane: And they even suggest this baking method isn't just for novel view synthesis. It’s applicable to any domain where you have an expensive attention module that needs to be baked into a fixed size state, like in long context language models or physical perception systems that handle streaming sensor inputs.
Meng: That sounds ambitious for real-world deployment, but the memory savings are definitely compelling enough to keep us watching this space.
The paper's improvements: Tom: So we’re moving on to what they actually did to make TTB better than just a basic version of it, and Jane, what are the specific improvements they suggest?
Jane: They focused on making the training signal more targeted during that inner loop optimization. Instead of just using an unnormalized dot-product loss, they switched to a convergence-oriented inner loop where they train the fast weight MLP via a mean squared error loss over three steps.
Lu: That change is significant because it means they are learning the residual to the mapping target, which makes it much easier to bake that information into the fast weights than if they were just trying to learn the final target directly.
Tom: So, what does that do for us on a practical level? Does it mean faster convergence or better accuracy?
Lu: It means their rendering loss reflects the exact dependence of those final fast weights on the outer loop parameters, which should lead to much more accurate baking of the scene structure itself.
Jane: That feeds right back into what we talked about earlier—it helps them close that quality gap to models like DVSM by ensuring the baked representation is really faithful to what they are trying to render.
Tom: Okay, so we’ve got better training signal, now let’s talk about how they handle those context tokens again. What else did they change in their setup?
Lu: They also made a change to how the queries and keys are created inside that fast weight layer. Instead of using an asymmetric query key projection, they used a shared linear projection to produce both the queries and the keys.
Jane: That's about making things more symmetric, which should help with stability during the baking process itself. It simplifies how those tokens interact within the MLP structure.
Meng: From an engineering standpoint, simplifying those projections is good for deployment because it reduces complexity in the inference path, even if it doesn't change the final speed much.
Tom: And they also added a layer to predict per token learning rates within that fast weight part. It’s like giving each part of the model a little nudge on how fast to learn during baking, which lets them control that process more finely.
Lu: That fine-grained control is what lets them manage the fixed capacity issue better, as you can tune how much influence each component has on the overall baked representation.
Jane: So to put it simply, they are tightening the screws on the training process—making it smarter about what information it’s baking and how fast each part learns—to get a higher fidelity result without needing massive amounts of context views at every step.
Tom: It sounds like they’re really pushing that boundary where baking and attention mapping start to work together more tightly than before.
Lu: Exactly. They're showing that the way you structure the inner loop objective and how you project those keys and queries matters a lot in making this baking strategy effective.
Jane: And this whole approach suggests that if you can bake complex attention mechanisms into a fixed size state using these targeted training signals, it opens up new possibilities for systems where an expensive attention module needs to be compressed.
Meng: That compression is key; we don't want to pay the computational price every single time we want a new view.
Conclusion: Tom: So we're wrapping up our look at TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models, and Jane, what's the final word on what this actually means?
Jane: To sum it all up, this paper shows how you can take a fast rendering technique and bake scene information directly from attention outputs during training, which gives you a quality boost while keeping rendering speeds very high.
Lu: It’s about creating a fixed-size scene representation that doesn't depend on the number of context views V, which is something we really need for practical deployment.
Meng: For me, it means if we can get this kind of memory footprint down to a few hundred megabytes, it becomes viable for running complex view synthesis models on standard hardware instead of needing massive GPU clusters just to handle the context window.
Lalam: I'm confident in my assessment; the paper is very solid. It proves that you can decouple the expensive attention mechanism from every single inference step by baking its useful knowledge into a small, baked structure.
Tom: It’s impressive how they managed to keep it at constant time per token rendering while still improving quality over models like DVSM, especially on those in-the-wild datasets where fine details matter.
Jane: And the fact that it recovers things like thin structures and small text that other methods blur out is a big win for visual fidelity.
Lu: The core idea is powerful because it lets you integrate complex attention into a fixed size state, which is exactly what we need when dealing with long context inputs or streaming sensor data.
Meng: Yeah, the capacity ceiling they mentioned still worries me though; if you push V too far, that fixed state might start limiting quality again.
Tom: And they use local-duplicate routing as a heuristic to manage that capacity problem, so it’s not just a theoretical concept but something they’ve actually implemented.
Jane: It really shows how much attention and rendering can be integrated in new ways beyond just simple view synthesis.
Lu: This baking perspective is definitely applicable beyond novel view synthesis; it could be useful in any domain where you need to compress an expensive attention module into a fixed state.
Meng: I just hope that when we move this into real-world traffic, the memory management doesn't become the bottleneck for our inference pipeline.
Lalam: I'm very sure about that; TTB achieves what it sets out to do with solid evidence, and it provides a much more efficient route to high-quality novel view generation.
Tom: So we’ve seen how repurposing existing components can yield big gains when you do it smartly in the paper titled "TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models."
Jane: It’s a really neat piece of work that shows how to get fast and high-quality results from these decoder-only view synthesis models.
Lu: We're looking forward to seeing how this fixed representation idea plays out in other areas where we need to amortize expensive attention modules.
Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
NVIDIA Department of Computer Science National Yang Ming Chiao Tung University
cs.CV
Submitted: 2026-11-12
Updated: 2026-11-12
Comments: NeurIPS 2026 (Spotlight). Project page: https://test-time-baking.github.io
Project page: https://test-time-baking.github.io
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry…
Key concepts
- Test-Time Baking (TTB)
- A strategy that learns global scene information from cross-view attention into a lightweight MLP during inference. Instead of just using pre-trained weights, TTB trains this small MLP on the fly using a specific loss function derived from ground truth novel views, allowing it to quickly decode high-quality novel views.
- Cross-View Attention Mapping
- The core idea is to use the fast weight MLP's learning process to explicitly learn how different context tokens relate across various input views. This mapping is learned by setting the inner-loop training target to the output of this attention mechanism, effectively making the model understand how different views influence each other for novel view generation.
- Modality Disalignment Resolution
- The paper addresses a mismatch where context tokens used for prefilling contain image content, while novel-view tokens are camera-only. This is fixed by separating context information into two distinct sets: 'context ground-truth tokens' (for mapping targets) and 'context rendering tokens' (for TTB input), disentangling the inputs for better learning.
- Local-Duplicate Inference
- To handle the fixed capacity of the fast weight MLP, this technique clusters context views into spatially-local groups. Each novel view only queries a corresponding copy of the baked MLP, ensuring O(1) per-token rendering cost while allowing quality to scale as more context views are introduced.
Terminology
Summary
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality The second key addresses a modality misalignment between context prefilling and novel-view rendering by disentangling context tokens into cameraonly context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target Our method achieves up to 23x rendering speedup with quality on par with the state-of-the-art across several benchmarks. p. 0 The first key
is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely,
achieving the same rendering efficiency as TTT while delivering substantially
higher quality. The second key addresses a modality misalignment between context
prefilling and novel-view rendering: we disentangle context tokens into cameraonly context rendering tokens to serve as TTB input, and context ground-truth
tokens with image content to serve as the TTB mapping target, ensuring the baked
MLPs are relieved of the modality generalization challenge between context and
novel-view processing.Where the paper says it
Pruning redundant reasoning tokens
The authors propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding This strategy is motivated by the idea of trading context-view processing time for faster and higher-fidelity novel-view rendering, achieving the Pareto frontier of quality and speed shown in Figure 1. Specifically, the inner-loop learning target becomes the output tokens from cross-view attention, instead of the raw value tokens used in TTT
How the training signal is built
The authors formalize their task as decomposing novel-view synthesis into two stages: Recon(I, P, K) and Rend(P′, K′, S). The model is trained end-to-end against ground-truth novel views I′ by the objective J = MSE(¯I, I′) + λ · Percep(¯I, I′). p. 4 As both Recon(·) and Rend(·) are differentiable, models
are trained end-to-end against ground-truth novel views I
′ by:
J = MSE(
¯I, I
′
) + λ · Percep(
¯I, I
′
), (2)
where Percep is the neural perceptual loss [27] and λ balances the photometric objectives. 2 Decoder-only View Synthesis Model
Our method builds on the Decoder-only View Synthesis Model (DVSM) [58], which uses a shared
Transformer decoder for both Recon(·) and Rend(·), and represents the scene S as KV caches. We call our strategy Test-Time Baking (TTB), which can be viewed as a generalized form of TTT. Formally, the fast-weight update rules of TTT and TTB are:
θ
(TTT) = θ − η∇θL(fθ(WkX),WvX), (6a)
θ
(TTB) = θ − η∇θL(fθ(WkX), g(X∗
)), (6b)
respectively, where θ is the fast weights of the MLP f(·), η and L are the inner-loop learning rate
and objective, Wk and Wv are linear projection matrices, and X is the input tokens. Note
that TTB falls back to TTT when X∗=X and g is a linear projection. In our case, we instantiate g(·)
as the CrossAttn(·) from DVSM, which shifts TTB beyond the linear-attention interpretation [36]
of TTT.Where the paper says it
Where the paper says it
Where the paper says it
Where the gains come from
The authors identify a modality misalignment issue between the reconstruction and rendering stages of MLP fast-weight prefilling, where context tokens X(ctx)0 used for prefilling contain image content, whereas novel-view tokens X(nv)0 are with camera-only information. This is resolved by pairing each context view into i) context ground-truth tokens and ii) context rendering tokens to disentangle the inputs. p. 0 The first key
is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely,
achieving the same rendering efficiency as TTT while delivering substantially
higher quality. The second key addresses a modality misalignment between context
prefilling and novel-view rendering: we disentangle context tokens into cameraonly context rendering tokens to serve as TTB input, and context ground-truth
tokens with image content to serve as the TTB mapping target, ensuring the baked
MLPs are relieved of the modality generalization challenge between context and
novel-view processing. The optimized TTB-MLP is then shared
between context rendering tokens and the downstream novel-view rendering tokens for retrieving
scene information, ensuring that their conditioning remains identical across all layers. Interestingly, TTB draws a natural parallel with traditional implicit radiance and light field reconstruction, where MLPs are optimized to map spatial coordinates to radiance values.Where the paper says it
Where the paper says it
The authors detail the TTB layer design by removing the asymmetric QK projection and instead using a shared linear projection to produce queries and keys. They adopt SwiGLU-MLP [50] as the fast-weight MLP architecture, and apply an additional linear layer to predict per-token learning rates. p. 5 We remove the asymmetric
QK projection and instead use a shared linear projection to produce queries and keys. We adopt
SwiGLU-MLP [50] as the fast-weight MLP architecture, and apply an additional linear layer to
predict per-token learning rates. The per-token learning rate is predicted by an additional
linear head with base learning rate of 0. The inner-loop objective maps context-rendering tokens to
the cross-view attention output of context-ground-truth tokens with the MSE loss.Where the paper says it
Where the paper says it
The authors show that swapping the inner-loop target from WvX to CrossAttn(X) broadens the effective view field of the baked fast-weight MLP, as CrossAttn(X) already aggregates across all V context views. The local-duplicate inference mechanism addresses the fixed capacity of the fast-weight MLP by clustering context views into K spatially-local groups via k-means on per-view camera poses. p. 6 4 Fast-weight local-duplicate inference
The baked fast weights have a fixed capacity, causing quality to plateau as the number of context views
V grows. We address this at inference time by clustering the context views into K spatially-local
groups via k-means on per-view camera poses, with K = V /B for a nominal budget of B views
per group. At rendering time, each novel view is routed to the group with the
nearest centroid and queries only the corresponding copy. Rendering thus remains O(1) per view and
each copy only has to absorb about B views, while the stored state grows linearly with K.Where the paper says it
Where the paper says it
The authors conclude that TTB matches DVSM [58] quality at a constant 23 FPS and consistently surpasses LaCT [77] TTB preserves thin and high-frequency structures such as handrails, window grilles, and small text that LaCT and LVSM [25] render with visible blur. The local-duplicate variant prevents the capacity ceiling of the single fast-weight MLP from causing quality to plateau as the number of context views V grows. p. 6 LVSM [25] Decoder-only LVSM [25] Efficient-LVSM [22] DVSM [58] Ours-- Ours
28. 4 Fast-weight local-duplicate inference
The baked fast weights have a fixed capacity, causing quality to plateau as the number of context views
V grows.Where the paper says it
The authors note that TTB is constrained by the fixed capacity of its fast-weight MLP, causing quality to plateau at large V, and the local-duplicate routing is a heuristic for mitigating this. The inner-loop cross-view attention also makes reconstruction quadratic in V, which is a drawback for single-shot scenarios. p. 8 TTB is constrained by the fixed capacity of its fast-weight MLP, causing quality to
saturate at large V ; the local-duplicate routing that mitigates this is a heuristic, and learning it during
training is left for future work. Its inner-loop cross-view attention also makes reconstruction quadratic
in V , a drawback for single-shot scenarios.Where the paper says it
The authors demonstrate that stacking all three components—context pairing, switching to MSE with T=3 inner steps (yielding TTB--), and replacing the linear-projection target with cross-view attention output—closes the quality gap to DVSM [58] while retaining TTT’s ∼50 FPS rendering. The scene representation size of TTB is a fixed-size set of fast-weight matrices whose footprint is independent of V. p. 9 5 Conclusion
We present Test-Time Baking (TTB), which retargets the TTT inner-loop optimization from raw
value tokens to cross-view attention outputs, turning the fast-weight MLP from a replacement for
attention into its collaborator. Combined with a context pair that resolves the reconstruction-rendering
modality misalignment, TTB bakes DVSM-level scene information into a fixed-size MLP while
retaining TTT-level rendering complexity.Where the paper says it
The paper suggests that the baking perspective offers a new way to integrate attention and TTT beyond view synthesis, potentially applicable beyond novel-view synthesis in any domain where a powerful but expensive attention module needs to be amortized into a state of fixed size. This is relevant for long-context language models, world models, and physical 3D perception systems that operate on streaming sensor inputs. p. 1 The t-SNE plots show that only (c) aligns the modalities from the first block. 1 Introduction
Our research studies the task of novel-view synthesis conditioned on a set of context images with the
powerful Large View Synthesis Models (LVSMs) [25].Where the paper says it
TTB overtakes DVSM after only 5–7 views at every V, while TTB-- leads from the first view at the quality cost reported in Tab. 1 The scene representation size of TTB is a fixed-size set of fast-weight matrices whose footprint is independent of V (324 MB across all L blocks in FP32). p. 9 5 Conclusion
We present Test-Time Baking (TTB), which retargets the TTT inner-loop optimization from raw
value tokens to cross-view attention outputs, turning the fast-weight MLP from a replacement for
attention into its collaborator. Combined with a context pair that resolves the reconstruction-rendering
modality misalignment, TTB bakes DVSM-level scene information into a fixed-size MLP while
retaining TTT-level rendering complexity. 5 Conclusion
We present Test-Time Baking (TTB), which retargets the TTT inner-loop optimization from raw
value tokens to cross-view attention outputs, turning the fast-weight MLP from a replacement for
attention into its collaborator. Combined with a context pair that resolves the reconstruction-rendering
modality misalignment, TTB bakes DVSM-level scene information into a fixed-size MLP while
retaining TTT-level rendering complexity. 5 Conclusion
We present Test-Time Baking (TTB), which retargets the TTT inner-loop optimization from raw
value tokens to cross-view attention outputs, turning the fast-weight MLP from a replacement for
attention into its collaborator. Combined with a context pair that resolves the reconstruction-rendering
modality misalignment, TTB bakes DVSM-level scene information into a fixed-size MLP while
retaining TTT-level rendering complexity. DVSM [58]
further introduces KV-cache to the cross-view attention layers, showing both quality and efficiency
improvement. The KV-cache serves as a global scene state for novel-view queries, but the rendering
time is still impractical under dense context view conditions due to the linear complexity per token
when attending to the KV-cache. DVSM [58]
further introduces KV-cache to the cross-view attention layers, showing both quality and efficiency
improvement. The KV-cache serves as a global scene state for novel-view queries, but the rendering
time is still impractical under dense context view conditions due to the linear complexity per token
when attending to the KV-cache. DVSM [58]
further introduces KV-cache to the cross-view attention layers, showing both quality and efficiency
improvement. The KV-cache serves as a global scene state for novel-view queries, but the rendering
time is still impractical under dense context view conditions due to the linear complexity per token
when attending to the KV-cache. DVSM [58]
further introduces KV-cache to the cross-view attention layers, showing both quality and efficiency
improvement. The KV-cache serves as a global scene state for novel-view queries, but the rendering
time is still impractical under dense context view conditions due to the linear complexity per token
when attending to the KV-cache.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The authors confirm that the rendering speedup is independent of context view count, confirming that the rendering speedup is independent of context view count TTB-duplicate tracks DVSM’s quality scaling while preserving O(1) per-frame rendering. The capacity scaling analysis shows that widening the fast-weight MLP closes the gap as well and reaches DVSM at 16× width. p. 5 More analysis. We empirically validate that swapping the inner-loop
target from WvX to CrossAttn(X) broadens the effective view field of
the baked fast-weight MLP. A naive remedy for LaCT is to initialize from pretrained decoder-only model weights, which does
improve quality, but the gap to the pretrained model remains significant. We argue that this quality gap can be minimized by repurposing the fast-weight MLP inner-loop
optimization as “baking” the information from cross-view attention, rather than replacing cross-view
attention entirely as in TTT. Its inner-loop cross-view attention also makes reconstruction quadratic
in V , a drawback for single-shot scenarios. Finally, TTB is built on reconstruction rather than
generation, interpolating within context views without extrapolating to unobserved viewpoints.Where the paper says it
Where the paper says it
Where the paper says it
The authors report that TTB overtakes DVSM after only 5–7 views at every V, while TTB-- leads from the first view at the quality cost reported in Tab. 1 The scene representation size of TTB is a fixed-size set of fast-weight matrices whose footprint is independent of V (324 MB across all L blocks in FP32). This compact, V-independent state is what enables TTB’s O(1) per-token rendering and is also a practical advantage for deployment scenarios with limited GPU memory. The paper's results on in-the-wild datasets show that TTB recovers fine details such as small text and thin structures that LaCT [77] and LVSM [25] blur out. The results on few-view lower-resolution settings show that TTB can faithfully reproduce the cross-view attentions results, as reflected by the comparable results with our pretrained DVSM. p. 8 Results are in Tab. Our
TTB can reproduce DVSM cross-attention quality in this setup. Results are in Tab. Our
TTB can reproduce DVSM cross-attention quality in this setup. Results are in Tab. Our
TTB can reproduce DVSM cross-attention quality in this setup. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a
lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key
is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely,
achieving the same rendering efficiency as TTT while delivering substantially
higher quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a
lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key
is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely,
achieving the same rendering efficiency as TTT while delivering substantially
higher quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a
lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key
is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely,
achieving the same rendering efficiency as TTT while delivering substantially
higher quality.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
The authors state that they use AdamW [38] with cosine annealing and perceptual loss weight λ = 0.2, training on DL3DV-10K [35] dataset with 64 A100 GPUs. p. 6 We
use AdamW [38] with cosine annealing and perceptual loss weight λ = 0. Models are trained
on DL3DV-10K [35] dataset with 64 A100 GPUs. Models are trained
on DL3DV-10K [35] dataset with 64 A100 GPUs. We first train at 340p for 100K iterations with
learning rate 4×10−4
, then finetune at 480p, 720p, and 960p for 10K iterations each with learning
rate 1×10−5
.Where the paper says it
Where the paper says it
The authors report that TTB overtakes DVSM after only 5–7 views at every V, while TTB-- leads from the first view at the quality cost reported in Tab. 1 The scene representation size of TTB is a fixed-size set of fast-weight matrices whose footprint is independent of V (324 MB across all L blocks in FP32). This compact, V-independent state is what enables TTB’s O(1) per-token rendering and is also a practical advantage for deployment scenarios with limited GPU memory. The results on in-the-wild datasets show that TTB recovers fine details such as small text and thin structures that LaCT [77] and LVSM [25] blur out. p. 7 5 confirms these trends qualitatively, where TTB
preserves thin and high-frequency structures such as handrails, window grilles, and small text that
LaCT and LVSM [25] render with visible blur. In-the-wild. 5 confirms these trends qualitatively, where TTB
preserves thin and high-frequency structures such as handrails, window grilles, and small text that
LaCT and LVSM [25] render with visible blur. In-the-wild.Where the paper says it
Where the paper says it
The authors conclude that TTB matches DVSM [58] quality at a constant 23 FPS and consistently surpasses LaCT [77] The scene representation size of TTB is a fixed-size set of fast-weight matrices whose footprint is independent of V (324 MB across all L blocks in FP32). This compact, V-independent state is what enables TTB’s O(1) per-token rendering and is also a practical advantage for deployment scenarios with limited GPU memory. The results on in-the-wild datasets show that TTB recovers fine details such as small text and thin structures that LaCT [77] and LVSM [25] blur out. The results on few-view lower-resolution settings show that TTB can faithfully reproduce the cross-view attentions results, as reflected by the comparable results with our pretrained DVSM. The efficiency analysis shows that TTB overtakes DVSM after only 5–7 views at every V, while TTB-- leads from the first view at the quality cost reported in Tab. 1.
The authors mention that Large language models were used as writing and editing aids during manuscript preparation, and Claude Code was used as a coding assistant during the development of their training and evaluation pipeline. The work was supported by NVIDIA Taiwan AI Research & Development Center (TRDC) and funded by the National Science and Technology Council, Taiwan. p. 9 Acknowledgments and Disclosure of Funding
This work was supported by NVIDIA Taiwan AI Research & Development Center (TRDC). This
research was funded by the National Science and Technology Council, Taiwan, under Grants NSTC
112-2222-E-A49-004-MY2, 113-2628-E-A49-023-, 115-2628-E-A49-024-, and 111-2628-E-A49-
018-MY4.Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Repurposing Test-Time Training layers to learn from cross-view attention outputs enables fast and high-fidelity novel-view decoding by learning
the cross-view attention mapping explicitly from its input-output pairs.
This achieves the same rendering efficiency as Test-Time Training while delivering substantially higher quality, sitting on the Pareto frontier of PSNR vs. FPS. -
Disentangling context tokens into context ground truth tokens and camera only rendering tokens resolves a modality misalignment between reconstruction and rendering stages, ensuring
the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing.
This pairing facilitates feature interpolation across layers that was previously hindered by this mismatch. -
Employing local-duplicate inference clusters context views into spatially-local groups, allowing each novel view to query only a corresponding copy of the baked fast weight MLP, which
recovers the constant-complexity rendering and is also a practical advantage for deployment scenarios with limited GPU memory.
This maintains O(1) per-token rendering cost while scaling quality with the number of context views V. -
Utilizing a convergence-oriented inner loop, specifically training the TTB fast weight MLP via MSE loss over T steps, yields better results than unnormalized dot-product loss used in prior Test-Time Training methods, as
learning the residual to the mapping target is easier to bake than learning the target directly.
This approach ensures thatthe rendering loss reflects the exact dependence of the final fast weights on the outer-loop parameters.
Abstract
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Sources
- Long-LRM++: Preserving Fine Details in Feed-Forward Wide-Coverage Reconstruction
- Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention
- Test-Time Training with KV Binding Is Secretly Linear Attention
- GLU Variants Improve Transformer
- DINOv3
- DVSM: Decoder-only View Synthesis Model Done Right
- tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction
- MeshLRM: Large Reconstruction Model for High-Quality Meshes
- Test-Time Training Done Right
- E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-training
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs