Coordinated Cooling and Compute Management for AI Datacenters
summary
The gist
The paper proposes a novel hierarchical control framework that jointly coordinates computing and cooling resources to achieve energy-efficient Large Language Model inference because this approach…
In short
A novel hierarchical control framework was developed to jointly manage computing and cooling resources for Large Language Model inference. This system uses four levels of decision-making, from cluster activation to per-job frequency scaling, guided by workload forecasts and thermal models. The approach successfully reduced energy consumption by 24.2% in compute and 31.2% in cooling while maintaining service quality.
Key concepts
- Hierarchical Control Framework
- A multi-level system that coordinates different management tasks across various scales, from large clusters down to individual jobs. It ensures that decisions made at one level, like activating a GPU cluster, are coordinated with decisions at lower levels, such as adjusting cooling or individual GPU speeds. This structure allows for balanced optimization of compute and thermal resources.
- Thermal Model
- A mathematical model describing how heat moves within a server rack. It uses energy conservation laws to track the temperature across three zones: cold air, server exhaust, and hot areas. This model helps predict where hotspots will occur based on the workload and cooling setup, allowing for proactive thermal management.
- Model Predictive Control (MPC)
- A control strategy used at the cooling level to regulate airflow and inlet temperatures every 60 seconds. MPC looks ahead at future conditions to optimize cooling actions, aiming to maintain thermal stability while minimizing energy use. It is more effective than traditional methods like PID control for this purpose.
- DVFS Scaling
- Dynamic Voltage and Frequency Scaling applied at the lowest level, adjusting the GPU's operating speed based on immediate needs. This scaling is tailored per job and considers both Service Level Objectives (SLOs) for latency and current thermal conditions, ensuring GPUs run at the optimal power level needed.
Terminology used across episodes
This episode discusses
- Coordinated Cooling and Compute Management for AI Datacenters · Paper Radio
- Carbon Emissions and Large Neural Network Training
- Language Models are Few-Shot Learners
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
The paper
Coordinated Cooling and Compute Management for AI Datacenters · Read on arXiv
Nardos Belay Abera, Yize Chen
University of Alberta · Natural Sciences and Engineering Research Council of Canada
The AI datacenters are currently being deployed on a large scale to support the training and deployment of power-intensive large-language models (LLMs). Extensive amount of computation and cooling required in datacenters increase concerns about the energy use and carbon emissions of AI datacenters. Although current state-of-the-art has examined the energy efficiency of LLM inference, most prior research focused on optimizing compute-side scheduling without considering thermal objectives or constraints. Since GPU-intensive inference generates substantial heat that can degrade datacenter performance, ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving. To fill this gap, we profile the characteristics of GPU servers under varying cooling and AI jobs, and develop a joint cooling and computing modeling approach for AI datacenters. Built upon such workload and thermal dynamics models, a novel hierarchical control framework is proposed to co-optimize computing and thermal management by identifying the optimal GPU parallelism, frequency (DVFS), and cooling control knobs. Using real Azure inference traces and detailed GPU profiling, our model balances serving latency and thermal constraints in AI datacenters while significantly improving AI datacenters' energy efficiency.
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Coordinated Cooling and Compute Management for AI Datacenters".
Dev: Here's the gist of the paper. The paper proposes a novel hierarchical control framework that jointly coordinates computing and cooling resources to achieve energy-efficient Large Language Model inference because this approach significantly improves AI datacenters’ energy efficiency while balancing serving latency and thermal constraints.
Rosa: Problem Motivation The current state-of-the-art often focuses only on optimizing compute-side scheduling without considering thermal objectives or constraints.
Dev: Ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving because GPU-intensive inference generates substantial heat that can degrade datacenter performance through thermal throttling.
Rosa: Load and temperature imbalances also cause sub-optimal scheduling and cooling in datacenters, leading to cooling units reacting only after temperature increases occur.
Paper summary: Rosa: So this paper we’re looking at is called "Coordinated Cooling and Compute Management for AI Datacenters". It’s about how to make these big AI systems use less power while keeping them running smoothly.
Dev: Right, the core idea here is that most past research only focused on scheduling the computing part—how you put GPUs together—but they ignored how hot it gets.
Rosa: Exactly. They argue that ignoring the heat from these GPU-intensive tasks means you just increase total energy use and mess up performance because of thermal throttling.
Taro: And they point out that when the load and temperature are all shifting, traditional cooling systems lag behind, which leads to hot spots where things get really bad for the hardware.
Dev: So the paper proposes a way to fix that by making computing and cooling work together instead of separately. It sets up this kind of hierarchical control framework across different levels of decision-making.
Rosa: That sounds complex, but essentially they break it down into four main parts: cluster level, window level, cooling level, and per-job scaling.
Taro: I’m interested in the lowest level part because that’s where the actual hardware interaction happens. They talk about dynamic voltage and frequency scaling on a job-by-job basis to match the required speed with what the temperature is currently doing.
Dev: That low level control, that DVFS scaling, has a specific mechanism they introduced. It’s not just scaling based on how much work there is; it jointly considers that workload intensity and the actual GPU temperature when picking the right operating frequency to meet those service level objectives or SLOs.
Rosa: So it’s trying to be smart about power draw while keeping latency fine, even when things get hot.
Taro: It sounds like a way to handle those real-world failure modes where the system is pushed hard and needs immediate adjustments. What they also did was build a model for thermal dynamics that describes how the temperature changes across different zones in the rack using mass conservation principles.
Dev: And on top of that, they have this cooling power model that sums up everything—the power needed for the cold source plus the power used by the fans—and they use it to control things like supply air temperature and airflow rate.
Paper summary: Rosa: That means they’re not just reacting to a high temperature; they're actively managing the flow of air based on what’s happening in the compute cluster.
Taro: But how do they predict what’s coming? Because you can't manage cooling if you don't know when the workload is about to spike. They use a sequence-to-sequence long short term memory network, an LSTM, trained on data from thirty minute intervals to predict token demand at the cluster level <ref:2601.08113#pg3>.
Rosa: That prediction feeds right into the highest level of control. The cluster level controller uses that forecast to decide how many GPU clusters should be activated based on workload predictions, which helps maximize utilization and cut down on idle power.
Dev: So it’s a predictive loop where they look ahead at the demand to proactively manage the hardware resources before the heat builds up too much.
Taro: I wonder how robust this is when things go completely off script, like a massive, unexpected surge in requests that the LSTM didn't see coming. What happens when you test this framework outside of a perfectly predicted environment?
Rosa: That’s where my job comes in. I’m thinking about testing if this works outside the lab, not just on clean data sets or simulated traces. We need to know how long it lasts and what happens when the real world misbehaves.
Dev: The paper does show results from Microsoft Azure LLM inference traces, and they report some solid numbers on energy savings. Specifically, they achieved twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings while still keeping the service level objectives met.
Taro: Those are pretty big numbers for efficiency gains in a datacenter setting.
Rosa: And it’s not just about saving power; they also showed improvements in thermal stability, lowering the average GPU temperature by seventeen point zero percent, keeping everything under that maximum limit of fifty degrees Celsius.
Dev: That reduction in average temperature is significant when you’re dealing with high-density AI workloads.
Taro: But what about latency? Because if you’re constantly changing things up—scaling clusters, adjusting parallelism—doesn't that introduce some kind of overhead that hurts the actual speed of inference?
Dev: They specifically checked for that, and they found that the overall scheme maintained negligible impact on inference latency, meaning the end to end latency followed what we expected from our baseline tests.
Paper summary: Rosa: That’s a big win because it means you get the energy savings without making customers notice any slowdown in how fast their AI is responding.
Taro: So, in simple terms for someone just listening to this show, this paper suggests that we can finally manage the heat and the power consumption of large language models by coordinating the hardware scheduling with a predictive cooling system.
Dev: The title itself, "Coordinated Cooling and Compute Management for AI Datacenters," really captures what they did—they linked those two traditionally separate domains together.
Rosa: It’s about taking something that used to be just a scheduling problem and making it an integrated thermal management problem instead.
Taro: It’s interesting because the authors talk about how this hierarchical control demonstrates effective coordination between the compute and cooling layers even when demand conditions are changing a lot, which is what happens in real AI usage.
Dev: I think the most important thing is that they moved beyond just optimizing one part of the system to solving it holistically across all those levels.
Rosa: It shows how an LSTM prediction model can drive cluster activation, which then feeds into parallelism adjustments, which finally informs the cooling parameters and the DVFS scaling.
Taro: I’m still thinking about that limitation they mentioned in their work. They noted that they built this framework upon detailed models of LLM workloads and key cooling control parameters, and that extending it to include advanced liquid or hybrid cooling schemes is the next step for better efficiency.
Rosa: Right, so the current model is solid for air-cooled setups with these specific components, but pushing it toward liquid cooling opens up another area where we can see even more gains in energy efficiency and stability.
Dev: So to wrap up this part of our discussion on "Coordinated Cooling and Compute Management for AI Datacenters," the authors show a way to use an LSTM predictor to drive a four-level control system that balances compute needs with thermal constraints, resulting in about twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings while keeping inference performance stable.
Taro: It’s a practical approach for managing the massive power demands of modern AI deployments.
Rosa: Yeah, it gives us a concrete way to think about how to handle the heat generated by these systems without just throwing more power at it hoping for the best.
Conclusion: Rosa: So we've been talking about this paper that's titled "Coordinated Cooling and Compute Management for AI Datacenters." Basically, it shows how you can link the scheduling of computing power with managing the cooling systems in an AI datacenter to save energy while keeping things stable.
Dev: Right, it’s about taking something that used to be a separate problem—how much compute you want versus how hot things get—and making them work together across different layers of control.
Rosa: And the authors use this four-level system, from deciding how many GPU clusters to turn on down to adjusting the individual chip speeds for each job.
Taro: It’s interesting because they aren't just looking at one thing; they're trying to manage the entire environment with a predictive model that forecasts future AI workload demands.
Dev: That LSTM network, the sequence-to-sequence one, predicts what kind of token demand is coming in thirty minute intervals, which lets them proactively adjust how many GPU clusters are active.
Rosa: It suggests that instead of just reacting to heat when it’s already high, you can decide ahead of time how much power to use and how the cooling system needs to ramp up.
Taro: But I wonder about the real world stuff, Rosa. If the workload spikes way harder than that prediction model saw coming, does this framework handle that sudden shock gracefully?
Dev: That’s a valid concern, Taro. The safety case for me is in how fast the cooling controller reacts when things go off script, and they show their MPC cooling formulation performs much better than older proportional integral controllers under those changing conditions.
Rosa: And on the results side, they're claiming real savings: twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings without sacrificing service level objectives.
Taro: Those are solid numbers, but I’m still thinking about the next big step for this kind of work.
Dev: Exactly. The authors themselves noted that they built this framework based on specific models for air cooling and certain hardware parameters, so extending it to include advanced liquid or hybrid cooling schemes is the natural next goal for improving efficiency further.
More episodes
- 2608.13438-ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- 2610.34593-Future Prediction as Training-Only Video Supervision for Pretrained VLA Policies
- 2610.35385-EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know
- 2610.36378-Data-Driven Hamiltonian for Neural Optimal Control
- 2610.37275-A Heuristic for Scheduling Autonomous Vehicles at Intersections
- 2610.37288-Dynamic Coordination of Autonomous Vehicles at Intersections
- 2610.37632-Real-Time Coordination of Autonomous Vehicles at Intersections
- 2611.13779-Softness is something you also need! Why compliance helps control: information metrics of a passive walker with compliant ankle
- 2611.13411-Residual-Corrected Physics-Informed Neural Network KKL Observers with Input-to-State Stability Guarantees
- 2611.14488-Is Your Robot's Memory Just a Clock? Temporal Shortcuts in Memory-Augmented VLA Policies