DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

arXiv:2606.12402 · cs.RO, cs.AI, cs.CV · Submitted 2026-06-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?".

Dev: Detailed Research Summary of DIRECT: Dynamic Compute Allocation for Vision-Language Models in Embodied Planning As a diligent researcher,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: We've seen that DIRECT is about dynamically allocating compute per prompt instead of picking a fixed model, and now we need to look at the mathematical structure behind that decision-making process.

Dev: Right, they formalize it with this utility function notation where k star i equals arg max over k U of q i k c i k. It basically says for every task demand q i inferred from context c i you want to pick the planner k that maximizes that utility function U of the task, planner, and context.

Taro: So what does that actually mean when we're talking about matching capability profiles? How do they define those candidate planners and how does the utility function really capture reasoning depth versus model size versus memory history?

Rosa: The paper states that this allows DIRECT to be agnostic to the specific form of the utility function, meaning it doesn't care if you’re prioritizing one thing over another as long as you match capability profiles to inferred needs.

Dev: It's about matching what the task requires—like deeper reasoning or longer-horizon memory—to what each planner inherently provides in terms of compute configuration.

Taro: So, if a task demands deep spatial reasoning, the router should infer that and route it toward a planner that handles those subtle physical constraints well, even if it's slightly more expensive than a simpler model.

Rosa: Precisely. It’s about the inference of task demands from scene and instruction context driving which planner gets invoked for that specific query.

Dev: And they mention that this whole routing framework is trained offline over a fixed pool of candidate planners, so the router itself doesn't have a huge inference cost at all during deployment.

Taro: That’s good for real-time loops because we don't want the router adding another significant latency spike to our control loop.

Rosa: It keeps the decision-making process fast while ensuring that you get that quality-cost tradeoff we talked about in the first place.

The paper's summary: Dev: Now let's look at how they summarized their findings in "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?"

Rosa: They summarize that test-time compute is not free in robotics because it compounds across a hierarchical stack, so costs from chain-of-thought, larger models, or denser memory can multiply the base model latencies.

Dev: And they showed that cutting compute uniformly lowers latency but risks failure on harder tasks, so the solution is to study where compute actually yields gains across three axes: reasoning depth, model size, and memory history.

Taro: So they are looking at those three specific areas because that’s where the real trade-off happens in embodied planning compared to just trying to cut everything equally.

Rosa: They examine chain-of-thought reasoning on VLABench and find that high-level planners with deliberate thinking chains can reason through subtle physical, semantic, and spatial constraints, but this reasoning comes at a heavy cost in token generation latency.

Dev: And they also look at model size and find that model size governs the breadth of skills a planner can reliably command; larger models offer greater breadth but introduce unnecessary overhead for simpler tasks.

Taro: So it’s not just about which model is biggest, but whether that extra size is actually needed for the specific skill command required by the task.

Rosa: They also looked at memory context and found that memory-oriented compute is only valuable when a task genuinely depends on distant history, otherwise paying for recall overhead across all tasks is wasteful.

Dev: So the summary boils down to this: effective allocation requires routing policies that infer task demands from scene and instruction context and match them to the compute configuration with the best quality–cost tradeoff.

Taro: That really frames it as an intelligent way to navigate the success–cost frontier instead of blindly scaling up compute.

Rosa: It means we move away from a one-size-fits-all approach to test-time compute allocation in embodied planning and toward something context aware.

The paper's improvements: Dev: Moving on to the proposed improvements in "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?"

Rosa: One main improvement they suggest is deploying DIRECT as a routing framework that uses multimodal scene context to allocate compute per prompt, which is designed to improve the success–cost Pareto frontier over fixed model selection.

Dev: And they show that this dynamic allocation allows embodied agents to match or exceed a stronger model’s success rate at up to sixty-five percent lower average latency.

Taro: That sixty-five percent reduction in latency is what makes it really compelling for real-world deployment because it shows we can get frontier performance without the usual massive compute penalty.

Rosa: They also suggest improving task performance by implementing per-step allocation on long-horizon tasks, which means DIRECT decides at each step whether routing to the thinking model is worth its cost.

Dev: And that per-step decision making showed highest progress success in physical validation results and recovers the Thinking model’s success at much lower latency for those critical steps.

Taro: So they aren't just allocating compute once, but deciding dynamically at each step whether the cost of routing to a specific planner is justified for that particular step.

Rosa: And regarding skill command across varied capabilities, they enable dynamic scaling based on task demands, which means model size primarily governs the breadth of skills a planner can reliably command.

Dev: So instead of deploying the largest models everywhere, you can reserve them only for tasks that explicitly demand that increased capacity for complex skill commands.

Taro: That lets you have a system commanded on more skills without having to deploy every single model at its maximum potential capability all the time.

Rosa: And finally, they improve memory utilization for long-horizon tasks by employing task-aware memory architectures, which identifies that memory-oriented compute is only worth its cost when a task genuinely depends on distant history.

Dev: So instead of paying the overhead on every single prompt for historical context, you can allocate it selectively based on whether the task actually requires that distant history.

Conclusion: Rosa: So to wrap up, this paper "DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?" suggests that naively scaling test-time compute is fundamentally wasteful.

Dev: It provides a principled path to achieving frontier-level embodied planning performance at a significantly reduced computational cost by treating the high-level planner as a dynamic variable inferred from scene and instruction context.

Taro: The implication for us is that we should stop thinking about just scaling up compute and start thinking about matching capability profiles to task demands.

Rosa: It means we can achieve much better performance, matching the strongest model at up to sixty-five percent lower latency while keeping the system viable for real-world deployment.

Dev: The key is using DIRECT to intelligently distribute test-time compute based on inferred demands from scene and instruction context so we can navigate the success–cost Pareto frontier effectively.

Taro: It’s about being adaptive, knowing when to spend more compute and when to save it for the next step.

Rosa: So that's what DIRECT is all about: a dynamic inference router for embodied planning that makes sense of where we should allocate our test-time compute.

Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn

Stanford University · University of Waterloo

cs.RO, cs.AI, cs.CV

Submitted: 2026-06-10

Updated: 2026-10-13

Comments: Conference on Robot Learning (CoRL) 2026

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 81/100

The gist: Detailed Research Summary of DIRECT: Dynamic Compute Allocation for Vision-Language Models in Embodied Planning As a diligent researcher, I have thoroughly analyzed the provided excerpts from the

Key concepts

Test-Time Compute Scaling: This is the practice of increasing the computational resources (like FLOPs or token usage) used by a model during inference to try and improve its output. The paper argues this scaling is often inefficient, leading to high latency without guaranteeing better results.
Dynamic Routing Framework (DIRECT): DIRECT is a system that uses information from the scene and instructions to dynamically select the best planning model for every single step of an agent's plan. It acts as a smart traffic controller, sending tasks only to the planner whose specific skills match what is needed at that moment.
Routing Efficiency Score ($\eta(\beta)$): This metric measures how well DIRECT allocates compute. It balances two competing goals: achieving a high task success rate ($q_s$) while keeping the latency cost ($c_s$) low. A higher score means the system is making smarter, more efficient choices about which planner to use.

Terminology

Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes
Vision-language models — 1× in this paper · explained in 2 episodes

Summary

Detailed Research Summary of DIRECT: Dynamic Compute Allocation for Vision-Language Models in Embodied Planning

As a diligent researcher, I have thoroughly analyzed the provided excerpts from the paper describing DIRECT (Dynamic Routing for Efficient Computation). This framework addresses a critical bottleneck in deploying Vision-Language Models (VLMs) as high-level planners for embodied agents: the inefficient and often wasteful scaling of test-time compute.

The core premise of DIRECT is that test-time compute allocation is not uniform; increasing it indiscriminately leads to diminishing returns across different planning dimensions, resulting in increased latency, token usage, and FLOPs without corresponding gains in success rate. DIRECT solves this by dynamically routing each task to the planner whose inherent capability profile best matches its specific demands.

Core Problem and Motivation

The paper identifies that scaling test-time compute is not a linear benefit across all planning dimensions. The key insights derived from experimental analysis are:

Chain-of-Thought (CoT) Depth: Helps significantly on tasks possessing implicit semantic, physical, or spatial constraints.
Model Size: Governs the breadth of skills a planner can command; larger models offer greater breadth but introduce unnecessary overhead for simpler tasks.
Memory Context: Memory-oriented compute is only beneficial when a task genuinely depends on distant history; otherwise, paying the recall overhead is wasteful across all tasks.

The DIRECT Framework: Dynamic Routing Mechanism

DIRECT introduces a routing framework that uses multimodal scene context (derived from the scene observation and language instruction) to infer the specific computational demands of an incoming task. This inference dictates which planner should be invoked for that prompt, effectively allocating compute per prompt rather than relying on a fixed model selection.

The decision-making process is formalized by a scalar utility function criterion:

k⋆i = k U(q i, k, c i, k)

where q i represents the task demand inferred from context c i, and k represents a candidate planner. This allows DIRECT to be agnostic to the specific form of the utility function, focusing instead on matching capability profiles to inferred needs.

The framework is trained offline over a fixed pool of candidate planners. The router itself has negligible inference cost (seconds to tens of seconds per query), making it practical for deployment.

Key Contributions and Experimental Results

DIRECT's contributions are multifaceted:

Diagnostic Analyses: Providing deep insights into the trade-offs between task difficulty and planner capability across embodied settings.
The DIRECT Framework: A dynamic allocation system that intelligently distributes test-time compute based on context inference.
Performance Gains: Demonstrating that DIRECT can match the performance of the strongest model at up to 65% lower latency compared to naive scaling approaches, while achieving near-oracle success rates across various hardware (VLABench, RoboMME, Franka DROID).

The efficacy of DIRECT is demonstrated across three dominant scaling axes:

Chain-of-Thought Test-Time Compute Axis: DIRECT exploits the CoT headroom on tasks with implicit constraints. For instance, in physical validation using Qwen3.5-9B, DIRECT routes to the Thinking model only when necessary for tasks requiring complex reasoning (like spatial reasoning or common sense object selection), recovering performance matching the Thinking model while incurring substantially lower latency than running it universally.
Model Size Test-Time Compute Axis: Instead of scaling up to the largest model, DIRECT converts an erratic scaling curve into a monotonic improvement by intelligently discarding the weaknesses of overly large models and reserving them only for tasks that explicitly demand that increased capacity (e.g., complex skill command).
Memory Context Test-Time Compute Axis: The framework proves that memory-oriented compute is only valuable when task requirements dictate it—specifically, when a task genuinely depends on distant history (e.g., tracking objects across multiple steps). It allocates historical context selectively, avoiding the overhead of paying for recall on every single prompt.

Evaluation Metrics and Routing Efficiency

The success of the routing mechanism is quantified by the routing efficiency score (eta(beta)):

eta(beta) = (1 + beta) q s (1 - c s)

where q s is task success rate, c s is planner latency cost, and beta controls the trade-off emphasis.

The evaluation suite includes:

VLABench: Testing language-conditioned manipulation tasks, where DIRECT successfully identifies which subset of tasks benefits from CoT reasoning across different model families.
Physical Validation (RoboMME/Franka DROID): This hardware testing confirms that DIRECT recovers near-oracle success at a fraction of the cost. For example, in physical routing results (Table 23), DIRECT achieves the highest routing efficiency (eta) by strategically selecting between cheap and expensive planners based on task demands.

Conclusion for Researchers

The overarching conclusion is that naively scaling test-time compute is fundamentally wasteful. DIRECT provides a practical, principled path to achieving frontier-level embodied planning performance in robotic systems at a significantly reduced computational cost. By treating the high-level planner as a dynamic variable inferred from scene and instruction context, DIRECT effectively navigates the success–cost Pareto frontier.

The detailed ablation studies (e.g., Table 29 showing complete router sweeps across various embedding fusion methods like concat/linear, normalize concat/linear, etc.) confirm that the choice of multimodal feature encoding and fusion strategy is critical to achieving optimal routing efficiency (eta). The framework successfully balances the need for high capability with strict latency constraints by making compute allocation task-specific.

Improvements for AI systems

Improve high-level planning efficiency by deploying DIRECT, which is a routing framework that uses multimodal scene context to allocate compute per prompt, improving the success–cost Pareto frontier over fixed model selection. This allows embodied agents to match or exceed a stronger model’s success rate at up to 65% lower average latency.
Enhance task performance by implementing per-step allocation on long-horizon tasks, as shown in the physical validation where DIRECT decides at each step whether routing to the thinking model is worth its cost, which shows highest progress success and recovers the Thinking model’s success at much lower latency.
Optimize skill command across varied capabilities by enabling dynamic scaling based on task demands, as evidenced by the finding that model size primarily governs the breadth of skills a planner can reliably command, leading to a system that can be command[ed] on more skills without uniform deployment.
Reduce computational waste by replacing fixed model selection with an adaptive strategy, as DIRECT demonstrates that naively scaling test-time compute is wasteful, allowing systems to allocate compute based on inferred demands from scene and instruction context.
Improve memory utilization for long-horizon tasks by employing task-aware memory architectures, as the framework identifies that memory-oriented compute is only worth its cost when a task genuinely depends on distant history, enabling selective allocation of historical context rather than paying the overhead on every task.

Abstract

Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, we observe that doing so increases latency, token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where embodied agents can be deployed. We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world, and we can budget inference at the planning level. We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute at each step of a closed-loop agent, selecting a planner for every subtask, improving the success--cost Pareto frontier over fixed model selection. Across three dominant scaling axes, namely chain-of-thought depth, model size, and memory history, our experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes yield qualitatively distinct capability gains. We validate these insights on a physical Franka arm in a DROID setup spanning zero-shot manipulation and long-horizon chaining, where our router matches or exceeds a stronger model's success rate at up to 65% lower average latency. Ultimately, our results show that naively scaling test-time compute is wasteful, and DIRECT can provide frontier-level embodied planning at a fraction of the cost. Project page can be found at jadee-dao.github.io/direct/.