When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization

summary

Video file (mp4)

The gist

The gist: Dynamic Tensor Rematerialization (DTR) exhibits non-monotone feasibility holes in its reference simulator, meaning memory budgets that succeed for both smaller and larger budgets can fail

In short

Dynamic Tensor Rematerialization (DTR) trains neural networks by evicting tensors when memory limits are hit. This study found 'feasibility holes': memory budgets where DTR fails, even though slightly smaller and larger budgets succeed. This non-monotone behavior occurs because the eviction policy's cost-based score lacks a necessary mathematical property.

Key concepts

Dynamic Tensor Rematerialization (DTR)
A method used to train neural networks by discarding tensors during training and recomputing them only when needed. It makes online decisions about which tensors to evict based on a calculated score involving compute, size, and staleness.
Feasibility Hole
A specific type of failure where DTR runs out of memory at a certain budget level, but slightly smaller and slightly larger budgets allow the training to complete. It represents a local failure inside an otherwise feasible region.
Monotonicity
The expectation that increasing a resource budget will always lead to better or equal performance. This paper shows that DTR's cost-based eviction policy violates this property, meaning feasibility is not guaranteed by budget size alone.

Terminology used across episodes

This episode discusses

The paper

When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization · Read on arXiv

Mahesh Reddy Pagadala

Aston University

Dynamic Tensor Rematerialization (DTR) trains neural networks under a memory budget by evicting tensors and recomputing them on demand. A natural expectation is that a budget large enough to complete training stays sufficient when it is increased. Using DTR's reference simulator, simrd, to replay the public DTR training traces, we show that this expectation fails. We call the exceptions feasibility holes: memory budgets at which an eviction policy runs out of memory although both a smaller and a larger budget complete, a form of non-monotone feasibility. Under a written sweep protocol with dated amendments, a 0.01-step budget grid shows no such hole for DTR on any of six confirmatory traces, although unresolved timeouts prevent it from establishing monotonicity. Finer sampling finds holes on InceptionV4 (three regions) and U-Net (two regions), in addition to one on the ResNet-32 development trace. Every hole is reproduced by three repeats and by unmodified simrd. For one InceptionV4 pair (0.2343 versus 0.235 of peak memory) we trace the failure to individual allocations. The larger budget avoids one eviction at an operator where the smaller budget was 173,479 bytes short. The extra data it carries forward forces a later eviction of a tensor needed three operators afterwards, and rebuilding that tensor triggers a recomputation cascade that cannot fit. Keeping that one tensor resident under the same budget prevents the failure. The two simple score changes we tested do not remove holes across traces; both add failing budgets on some traces. All results are from simulation, with no allocator model; behaviour on GPU hardware is untested.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "When More Memory Fails".

Jane: The gist: Dynamic Tensor Rematerialization (DTR) exhibits non-monotone feasibility holes in its reference simulator, meaning memory budgets that succeed for both smaller and larger budgets can fail for DTR <ref:2610.14369#pg6>.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To wrap up, the paper "When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization" highlights how DTR can encounter memory budget failures that aren't predictable just by increasing the budget size > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg1.

Jane: The authors found these exceptions feasibility holes using their reference simulator, simrd, and they showed they are reproducible with unmodified code > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg2. It’s important to remember that the study is limited because it's simulation only; simrd doesn't model things like allocator fragmentation or alignment on real hardware > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg8.

Lu: The paper asks a big question about whether these feasibility holes transfer to real hardware and if we can find a policy that guarantees monotone feasibility at an acceptable cost > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg1. It leaves the door open for real-world testing.

Meng: For us on the shipping side, it means we need to be cautious about relying solely on increasing memory budgets without understanding these hidden failure modes in our DTR algorithms > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg6.

Lalam: The finding that simple score changes don't fix the holes, and that a variant called TwoPhase actually has more failing samples, gives us a clear direction on what to look out for when designing our next memory management layer > When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization #pg7.

Conclusion: Tom: So, we're looking at this paper about Dynamic Tensor Rematerialization, DTR, and these feasibility holes in its simulator. It basically shows that if you just crank up the memory budget to a really big number, it doesn't guarantee success for DTR.

Jane: That's right. The title is "When More Memory Fails: Hidden Feasibility Holes in Dynamic Tensor Rematerialization." It sounds like the system thinks it can handle any amount of memory, but it hits a wall unexpectedly.

Lu: The authors found these holes by testing their reference simulator, simrd. They showed that there are specific memory budgets where the DTR process just runs out of space for one tensor even when both a smaller and a larger budget would have worked fine.

Meng: From an engineering standpoint, that's worrying. It means we can’t just blindly increase our allocation limits assuming everything will work smoothly under the hood. There are these hidden failure modes in the way it manages its working memory.

Lalam: I see what they did here is trace a specific failure down to individual allocations and show how an intervention—like stopping one particular tensor from being offered as an eviction candidate—can prevent a crash even under that same budget. It’s not just about the total number; it’s about the sequence of events.

Tom: Exactly, Lalam. And they tested different ways to adjust the scoring function, but those changes didn't fix anything; they actually added more failing budgets on some traces.

Jane: So what does this mean for us who are actually building these things? It suggests that relying on a fixed memory budget isn't enough; we need a smarter way to handle when things get tight during the training process.

Lu: The big question they leave is whether these holes show up when you move from the simulation to actual hardware, and if we can design a policy that guarantees success at an acceptable cost. That's where the real research needs to go.

Meng: I'm focused on what breaks in production, so I'm watching that part closely. If this behavior transfers to real hardware, it’s a significant risk for our deployment pipeline.

Lalam: And that opens up the next big thing: figuring out if we can build a memory management layer that is truly robust and predictable across different scales of training.

More episodes

← Home