Does Depth Survive Composition, and Can a Router Use It? Adaptive Compute in Latent World Models

arXiv:2607.10203 · cs.LG, cs.AI · Submitted 2026-07-11 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Does Depth Survive Composition, and Can a Router Use It? Adaptive Compute in Latent World Models".

Jane: The gist The inversion phenomenon, where shallow rollouts beat full-depth rollouts, is caused by training that makes early exits routable rather than being an intrinsic property of task dynamics.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: They are really digging into the idea that if we have a predictor that takes different amounts of time per step, we can decide whether to use a shallow or a deep rollout based on what the data is telling us.

Lu: The main point is they test if this depth routing strategy works across different tasks and training setups, specifically looking at whether the per-step precision holds up when those predictions are composed autoregressively.

Meng: If it works, that’s a big win for deployment because we don't have to run the most expensive models on every single step just to be safe. But how do you even measure that depth relationship?

Lalam: They use this shallow penalty, which is defined as the error of the shallowest-exit rollout divided by the error of the full-depth rollout, and that number is their diagnostic tool.

The paper's summary: Tom: So they summarize their findings by looking at nine DeepMind Control tasks and dividing them into three regimes based on that shallow penalty, rho.

Jane: They found three outcomes: some tasks show an intrinsic tradeoff where more depth is better, where two-ninths of the tasks show an inversion where the shallow exits actually beat the full stack, and one task is flat.

Lu: The most interesting part for me is that they show this inversion isn't just a fluke; it’s created by how they train the model to make early exits routable, which they call the "routability catch-twenty-two."

Meng: So it sounds like the mechanism that causes shallow exits to win isn't just a property of the task itself, but something we intentionally built into the training process.

Lalam: They show that if you only apply this specific deep supervision at the very first rollout step, you erase this robust inversion effect entirely, proving it’s tied to a specific training setup rather than some universal rule.

The paper's improvements: Tom: The authors suggest three main improvements. First, they propose that we should be able to explicitly identify when planning benefits from depth by measuring that shallow penalty rho.

Jane: That means if rho is greater than one, it suggests that more compute depth buys better rollout quality, which justifies routing deep for those specific steps.

Lu: And they also suggest training strategies need to be better at distinguishing between a real intrinsic tradeoff and this self-inflicted inversion by using causal ablation on the early exit supervision.

Meng: From an engineering standpoint, that’s useful because it helps us design training pipelines that don't accidentally create these performance differences just because of how we supervise the model.

Lalam: They also propose developing out-of-sample regime predictors for novel tasks, so we can predict whether a new task will exhibit extreme extrapolation based on its dimensionality alone.

Conclusion: Tom: So to wrap up, they’re saying that whether more compute helps isn't just about the task itself; it depends entirely on the full operating configuration of the world model.

Jane: They confirm that the intrinsic versus non-intrinsic dichotomy is stable across seven out of nine tasks when you look at both metric and horizon, which means we have a fairly predictable core mechanism here.

Lu: The inversion phenomenon in "Does Depth Survive Composition, and Can a Router Use It? Adaptive Compute in Latent World Models" is shown to be mechanism-backed by the catch-twenty-two architecture-robust because it reproduces under a transformer model, and predictable from task dimensionality.

Meng: So for deployment, this means we can use a calibrated router to turn that diagnostic signal into a favorable compute return operating point on at least some tasks. It’s about making the compute choice smarter rather than just running everything deep.

Lalam: The paper "Does Depth Survive Composition, and Can a Router Use It? Adaptive Compute in Latent World Models" shows that what compute helps with really depends on the specific setup you're using, not some universal law.

Tom: It started as a failed hypothesis for them but held to their pre-registration, it became this taxonomy of what "does compute help?" actually depends on. That’s a lot of structure to take away from that paper.

Achyuthan Sivasankar

New York University

cs.LG, cs.AI

Submitted: 2026-07-11

Updated: 2026-10-14

Comments: 26 pages, 12 figures, 12 tables. v4: major revision. Adds equal-compute routing comparisons (oracle, step-index schedules, three trained routers), equal-compute planner runs and a supervision-matched ablation control; narrows several v3 claims accordingly; corrects four perturbation-baseline numbers in v3

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 88/100

The gist: The gist The inversion phenomenon, where shallow rollouts beat full-depth rollouts, is caused by training that makes early exits routable rather than being an intrinsic property of task dynamics.

Key concepts

Adaptive Compute World Models: These models use predictors that change their depth on a step-by-step basis. The goal is to make planning cheaper by using fewer computation blocks when possible, relying on the assumption that deeper predictors yield better results.
Inversion Phenomenon: This occurs when shallow rollouts (less depth) perform better than full-depth rollouts. The paper shows this isn't a natural task property but is caused by training that makes early exits predictable and routable, creating a 'catch-22'.
Routability Catch-22: This mechanism creates the inversion. It involves deep supervision and distillation of shallow exit predictions at every rollout step. This training teaches the model to favor shallower rollouts over deeper ones in certain scenarios.

Terminology

World Model — 3× in this paper · explained in 7 episodes
Autoregressive Rollout — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Deep Supervision — 1× in this paper · explained in 2 episodes
Latent World Models — 1× in this paper · explained in 3 episodes
Transformer — 1× in this paper · explained in 2 episodes

Summary

The gist The inversion phenomenon, where shallow rollouts beat full-depth rollouts, is caused by training that makes early exits routable rather than being an intrinsic property of task dynamics.

Adaptive Compute in Latent World Models

Adaptive compute world models use early-exit or mixture-of-depths predictors that spend variable depth per step to make planning cheaper. This approach relies on the assumption that more predictor depth buys better predictions and that this depth can be routed adaptively. The study investigates whether per-step depth precision survives autoregressive composition using the shallow penalty ρ = err(shallowest-exit rollout)/err(full-depth rollout) as a measure.

Taxonomy of Depth Regimes

The study partitions nine DeepMind Control (DMC) tasks into three regimes based on the shallow penalty ρ.

inversion (2/9; shallow beats deep).
flat (1/9).

The Routability Catch-22

The inversion regime is created by the training that makes early exits routable, which is described as the routability catch-22. This mechanism involves deep supervision and distillation of shallow exits at every rollout step, which trains them to outroll the full stack. An ablation where this per-step deep supervision is only applied at the first rollout step erases cheetah’s robust inversion. This confirms that the mechanism is specific to robust inversion and not universal.

Configuration Dependence

The regime depends on several configuration axes. The taxonomy is measured in one configuration (latent-space error) but the fine-label agreement with the canonical read is 0.67. The out-of-sample regime predictor correctly predicts 2/3 genuinely novel held-out tasks’ regimes. The control-level inversion is direct evidence that ρ is a real diagnostic rather than a latent-metric curiosity. A calibrated router can turn the signal into a favorable compute–return operating point on at least some intrinsic tasks.

Conclusion

Whether more compute helps is not a task property but a property of the full operating configuration—with a stable, predictable, mechanism-backed core. The intrinsic-vs-non-intrinsic dichotomy is stable on 7/9 tasks across metric and horizon. The inversion phenomenon is mechanism-backed (the catch-22), architecture-robust (reproduces under a transformer), and a priori predictable from task dimensionality. The result began as a failed hypothesis and, held to its pre-registration, became a taxonomy, a mechanism, a predictor, and a map of what “does compute help?” actually depends on.

How it works

The study uses matched twins that differ only in the training horizon K: K=1 (single-step) and K=4 (latent overshooting). The shallow penalty ρ measures the quality side of the FLOP–quality tradeoff, being exactly the factor by which spending 5.6× fewer blocks per step changes rollout error. The causal mechanism for inversion stems from deep supervision and distillation applied to every shallow exit at every rollout step.

Key Findings

The intrinsic regime is the majority: for most DMC tasks depth genuinely matters for rollout quality.
Inversion is created by training: the inversion is induced by the training that makes early exits routable—the routability catch-22.
The phenomenon is architecture-robust: the inversion reproduces under a transformer predictor.
The regime shifts with data distribution: retraining cheetah and cartpole from competent planner-collected data erases both the inversion and the strong intrinsic tradeoff, collapsing both toward flat—even though the loss is unchanged.

Improvements for AI systems

Adaptive compute world models can be configured to explicitly identify whether planning benefits from depth by measuring the shallow penalty ρ = err(shallowest-exit rollout)/err(full-depth rollout). This allows deployment decisions to be made based on the regime: if ρ > 1, depth buys rollout quality (a genuine compute–quality tradeoff), justifying routing; if ρ < 1, planning should avoid deep computation.
Improve model training strategies by distinguishing between intrinsic tradeoffs and self-inflicted inversions. By implementing a causal ablation supervising the shallow exits only at the first rollout step, one can erase cheetah’s robust inversion, ensuring that observed performance differences are due to task dynamics rather than the routability catch-22.
Develop out-of-sample regime predictors for novel tasks. A frozen classifier trained on task dimensionality alone can predict whether a held-out task will exhibit an extreme extrapolation (dog-walk, obsdim 223), enabling proactive architectural decisions during model development rather than relying solely on in-sample performance metrics.
Create configuration-aware evaluation protocols for latent world models. Evaluation must account for the operating environment by measuring error across different axes, such as metric space dependence (latent vs. observation error) and representation dependence (encoder type), preventing the misinterpretation that latent prediction error, the default JEPA metric, can misstate whether compute buys quality.
Implement task-adaptive planning thresholds for routing decisions. Instead of a fixed depth gate, a router should dynamically select depth based on the measured regime; for instance, a calibrated router can clear performance bars by picking the highest-savings threshold meeting 0.9× on calibration, transforming the diagnostic signal into a favorable compute–return operating point on certain tasks.