IC-ThermBench: An Open, Progressive Benchmark for Generalizable 2.5D/3D-IC Thermal Learning

summary

The gist

IC-ThermBench introduces an open and progressive benchmark that combines established 3D-IC steady-state, transient…

In short

IC-ThermBench is an open benchmark testing thermal prediction models for 2.5D/3D integrated circuits across different generalization scopes. It combines standard tasks with a new 2.5D chiplet extension to evaluate how well models handle increasing physical variations and transfer knowledge to entirely new package systems.

Key concepts

Generalization Scopes (S1-S5)
These define the difficulty levels of the thermal prediction tasks. S1 is within-design prediction, while S2-S4 test within-family variation (layout, material changes). S5 tests cross-package Out-of-Distribution (OOD) transfer to unseen systems.
Physical Formulation G: (M, z) ↦→ T
This mathematical concept describes how the thermal process follows a heat equation. It defines the thermal solution operator 'G' that takes system inputs like power and geometry as input and outputs the resulting temperature distribution (T).
Within-Family Generalization (S2-S4)
This scope evaluates model performance when physical variations are introduced systematically within established system families. It measures the model's ability to learn patterns from gradually changing inputs like layout or boundary conditions while keeping the underlying package type consistent.
Cross-Package OOD Transfer (S5)
This scope tests if a model can predict temperatures accurately for a completely new, unseen package system that was not part of the training data. This measures true generalization beyond known physical configurations.

Terminology used across episodes

This episode discusses

The paper

OpenTherm: A Unified and Progressive Benchmark for 2.5D/3D-IC Thermal Prediction · Read on arXiv

University of Technology Sydney · School of Information Science and Technology, ShanghaiTech University

Standardized benchmarks are fundamental to reliable progress in AI for EDA, including learning-based thermal modeling. However, existing thermal prediction studies often rely on different datasets, simulators, data splits, preprocessing pipelines, and metrics, while most datasets and implementations remain unavailable, making fair and reproducible comparison difficult. We introduce IC-ThermBench, an open and progressive benchmark that combines established 3D-IC steady-state, transient, and industrial package tasks with a new 50,000-sample 2.5D chiplet extension designed to evaluate progressively broader represented physical variation and cross-package OOD transfer. Five Generalization Scopes cover 3D-IC fixed-design prediction, Within-Family Generalization under layout, material, and boundary-condition variation, and Cross-Package OOD transfer to unseen package systems. We evaluate eight representative baselines under common data, splits, labels, and metrics. Performance degrades gradually from S2 to S4 as represented physical support broadens, but Cross-Package OOD produces a much sharper degradation: the best RMSE and MAE increase from 1.216 and 0.938~K at S4 to 15.99 and 15.00~K at S5, respectively. With only 10 labeled samples per OOD case, target-domain adaptation reduces the best MAE to 2.60~K. IC-ThermBench further provides a unified generation, training, inference, and evaluation pipeline, enabling reproducible and fair comparison of existing and new thermal predictor.

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "IC-ThermBench: An Open, Progressive Benchmark for Generalizable 2.5D/3D-IC Thermal Learning".

Jane: Here's the gist of the paper. IC-ThermBench introduces an open and progressive benchmark that combines established 3D-IC steady-state, transient, and industrial package tasks with a new 50,000-sample 2.5D chiplet extension designed to evaluate progressively broader represented physical variation and cross-package OOD transfer This benchmark provides a unified generation, training, inference, and evaluation pipeline to enable reproducible and fair comparison of existing and new thermal predictors Benchmark Structure IC-ThermBench organizes thermal prediction into five deployment-oriented Generalization Scopes rather than a universal difficulty ranking S1 evaluates within-design prediction, while S2–S4 progressively broaden the represented physical support over the same observed system families through layout, material, and boundary-condition variation S5 evaluates case-disjoint transfer to previously unseen package systems The distinction between S2–S4 and S5 is central to the benchmark, with S2–S4 being Within-Family Generalization and S5 being Cross-Package OOD Physical Formulation and Inputs The thermal process follows the heat equation, which can be abstracted as a thermal solution operator G: (M, z) ↦→ T For steady-state analysis, the temporal term vanishes, whereas transient tasks predict the temperature evolution over a specified time horizon The model input expands cumulatively with the Generalization Scope, leading to inputs such as XS4/S5 =

P, x, y, k, Tamb, h, Rconv: For S4 and S5 tasks, ambient temperature is sampled uniformly from

25, 55: ◦C and the convection coefficient is sampled log-uniformly from

500, 20000: W/(m2K) Generalization Findings The results show that broader represented physical variation remains learnable across S2–S4, as prediction errors increase gradually without the abrupt degradation seen under cross-package transfer Cross-package transfer produces a sharp performance break from S4 to S5, with the best RMSE increasing from 0.933 K at S4 to 15.51 K at S5 Limited target-domain adaptation can reduce the best MAE to 2.25 K in OOD cases Model and Cost Analysis The Therm-FM family leads the evaluated baselines in global-field accuracy across S1–S5, but its advantage is pronounced on S1 and remains clear, though generally smaller, on S2–S5 The best performing variant changes with the Scope, indicating that the configuration that best absorbs represented variation need not be the one that transfers to unseen packages Furthermore, scaling larger models does not consistently improve accuracy across different scopes Adaptation Effectiveness Target-case adaptation is effective but distinct from zero-shot generalization, as limited target-domain supervision can recover a substantial fraction of the zero-shot accuracy loss However, adapted performance no longer measures frozen cross-package transfer, and the label budget per target case must be reported explicitly The complete tables reinforce two points hidden by RMSE alone: the best model changes with both label budget and metric, and adaptation is comparatively insensitive to optimization settings IC-ThermBench establishes a common foundation for developing and comparing thermal predictors beyond narrowly defined design distributions The benchmark demonstrates that while broader physical variability is learnable, transfer to unseen package systems exposes a much larger generalization gap By providing shared data, fixed splits, input semantics, metrics, generation tools, and reproducible baselines, IC-ThermBench allows for fair comparison of thermal predictors The benchmark results are directly comparable on the primary benchmark only when they use the same dataset version and evaluation protocols The paper concludes that zero-shot generalization and target-domain sample efficiency should be evaluated separately, with the label budget per target case reported explicitly The benchmark provides a common foundation for developing and comparing thermal predictors beyond narrowly defined design distributions The benchmark results are directly comparable on the primary benchmark only when they use the same dataset version, immutable splits, input and label definitions, preprocessing, normalization, and reference metrics The paper concludes that zero-shot generalization and target-domain sample efficiency should be evaluated separately, with the label budget per target case reported explicitly The benchmark provides a common foundation for developing and comparing thermal predictors beyond narrowly defined design distributions

Title and authors: Jane: Let’s talk about who wrote this stuff for a minute. The authors are Hang, Yang, Ding, Xin and Wei. They come from some big tech universities across the US and China.

Tom: Yeah, they’ve got experience in both the theory and the practical application of these thermal models. It tells me we’re looking at a solid foundation here because they bring different angles to it.

Lu: What's interesting is that they are combining established tasks like three dee-IC steady-state with these new 2 point 5D chiplet evaluations, which shows an understanding of the current state of the field <ref:2608.23977#pg1>.

Meng: From my side, I’m interested in whether this benchmark is just theoretical or if it can actually be used to test models we deploy on real hardware without needing perfect simulators.

Lalam: Based on what I see in the literature, these standardized benchmarks are fundamental for reliable progress in AI for EDA, including learning-based thermal modeling.

Tom: So, they are trying to fix the problem of not having a universal setup so that we can actually compare models on a consistent footing.

Jane: The implications here is that if we use this benchmark, our thermal predictors will be much more reliable when applied to new chip designs because we know what kind of variation they’ve been tested against.

The paper's summary: Tom: Okay, so the core summary is about structuring thermal prediction into five different generalization scopes instead of just one universal difficulty ranking.

Jane: That’s a key structural change. Scope S1 is within-design prediction, and then S2 through S4 progressively broaden the represented physical support by changing layout, material, and boundary conditions over the same system families.

Lu: The distinction they draw between S2 through S4 being within-family generalization and S5 being cross-package out-of-distribution transfer is central to how they evaluate model capability.

Meng: So if a model can handle those variations in layout or material within the same family, that’s one thing, but testing it on a completely new package system, which is S5, is the real stress test for deployment.

Tom: And this leads into the results where they show that broader represented physical variation stays learnable across scopes S2 through S4 because errors increase gradually.

Jane: But then there’s this sharp performance break between S4 and S5, where the best root mean square error jumps from zero point nine three three Kelvin at S4 to fifteen point five one Kelvin at S5 <ref:2608.23977#pg2>.

The paper's improvements: Lu: The paper points out that while broader physical variation is learnable in scopes S2 through S4, the cross-package transfer to unseen packages shows a much larger generalization gap than within-family variations.

Tom: And they also found that limited target-domain adaptation can reduce the best mean absolute error to two point two five Kelvin in those out-of-distribution cases, which is a decent recovery but it’s not the same as zero-shot generalization.

Meng: I gotta ask about this adaptation thing—how much of that recovery is real deployment potential versus just getting a temporary boost on a specific set of new packages?

Jane: The paper makes it clear that target-case adaptation is effective, but it’s different from zero-shot generalization because you have to report the label budget per target case explicitly.

Tom: And another thing they pointed out is that the best model changes depending on the scope you are testing and even with a fixed set of labels, adapting performance isn't always better than just sticking to the best zero-shot result.

Conclusion: Jane: So to wrap up, IC-ThermBench establishes this common foundation for comparing thermal predictors across much wider distributions than before. It shows that while broader physical variability is learnable, transfer to unseen package systems exposes a much larger generalization gap.

Tom: The key point they make is that zero-shot generalization and target-domain sample efficiency should be evaluated separately, with the label budget per target case reported explicitly in those cases.

Lu: The benchmark provides a common foundation for developing and comparing thermal predictors beyond narrowly defined design distributions because it standardizes the inputs and evaluation protocols.

Meng: For us in production, this means we need to decide which scope matters most for our use case—is handling material variation within a family more important than surviving a complete package transfer?

Lalam: The benchmark results are directly comparable on the primary benchmark only when they use the same dataset version, immutable splits, input and label definitions, preprocessing, normalization and reference metrics.

Tom: So we get this powerful tool for comparing thermal predictors across these five scopes. We’ll be looking at how models perform when they face those specific challenges in the next few episodes.

More episodes

← Home