Coordinated Cooling and Compute Management for AI Datacenters
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: I'm Rosa.
Dev: And I'm Dev, here with Taro.
Rosa: We're looking at the paper: "Coordinated Cooling and Compute Management for AI Datacenters".
Dev: Here's the gist of the paper. The paper proposes a novel hierarchical control framework that jointly coordinates computing and cooling resources to achieve energy-efficient Large Language Model inference because this approach significantly improves AI datacenters’ energy efficiency while balancing serving latency and thermal constraints.
Rosa: Problem Motivation The current state-of-the-art often focuses only on optimizing compute-side scheduling without considering thermal objectives or constraints.
Dev: Ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving because GPU-intensive inference generates substantial heat that can degrade datacenter performance through thermal throttling.
Rosa: Load and temperature imbalances also cause sub-optimal scheduling and cooling in datacenters, leading to cooling units reacting only after temperature increases occur.
Paper summary: Rosa: So this paper we’re looking at is called "Coordinated Cooling and Compute Management for AI Datacenters". It’s about how to make these big AI systems use less power while keeping them running smoothly.
Dev: Right, the core idea here is that most past research only focused on scheduling the computing part—how you put GPUs together—but they ignored how hot it gets.
Rosa: Exactly. They argue that ignoring the heat from these GPU-intensive tasks means you just increase total energy use and mess up performance because of thermal throttling.
Taro: And they point out that when the load and temperature are all shifting, traditional cooling systems lag behind, which leads to hot spots where things get really bad for the hardware.
Dev: So the paper proposes a way to fix that by making computing and cooling work together instead of separately. It sets up this kind of hierarchical control framework across different levels of decision-making.
Rosa: That sounds complex, but essentially they break it down into four main parts: cluster level, window level, cooling level, and per-job scaling.
Taro: I’m interested in the lowest level part because that’s where the actual hardware interaction happens. They talk about dynamic voltage and frequency scaling on a job-by-job basis to match the required speed with what the temperature is currently doing.
Dev: That low level control, that DVFS scaling, has a specific mechanism they introduced. It’s not just scaling based on how much work there is; it jointly considers that workload intensity and the actual GPU temperature when picking the right operating frequency to meet those service level objectives or SLOs.
Rosa: So it’s trying to be smart about power draw while keeping latency fine, even when things get hot.
Taro: It sounds like a way to handle those real-world failure modes where the system is pushed hard and needs immediate adjustments. What they also did was build a model for thermal dynamics that describes how the temperature changes across different zones in the rack using mass conservation principles.
Dev: And on top of that, they have this cooling power model that sums up everything—the power needed for the cold source plus the power used by the fans—and they use it to control things like supply air temperature and airflow rate.
Paper summary: Rosa: That means they’re not just reacting to a high temperature; they're actively managing the flow of air based on what’s happening in the compute cluster.
Taro: But how do they predict what’s coming? Because you can't manage cooling if you don't know when the workload is about to spike. They use a sequence-to-sequence long short term memory network, an LSTM, trained on data from thirty minute intervals to predict token demand at the cluster level <ref:2601.08113#pg3>.
Rosa: That prediction feeds right into the highest level of control. The cluster level controller uses that forecast to decide how many GPU clusters should be activated based on workload predictions, which helps maximize utilization and cut down on idle power.
Dev: So it’s a predictive loop where they look ahead at the demand to proactively manage the hardware resources before the heat builds up too much.
Taro: I wonder how robust this is when things go completely off script, like a massive, unexpected surge in requests that the LSTM didn't see coming. What happens when you test this framework outside of a perfectly predicted environment?
Rosa: That’s where my job comes in. I’m thinking about testing if this works outside the lab, not just on clean data sets or simulated traces. We need to know how long it lasts and what happens when the real world misbehaves.
Dev: The paper does show results from Microsoft Azure LLM inference traces, and they report some solid numbers on energy savings. Specifically, they achieved twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings while still keeping the service level objectives met.
Taro: Those are pretty big numbers for efficiency gains in a datacenter setting.
Rosa: And it’s not just about saving power; they also showed improvements in thermal stability, lowering the average GPU temperature by seventeen point zero percent, keeping everything under that maximum limit of fifty degrees Celsius.
Dev: That reduction in average temperature is significant when you’re dealing with high-density AI workloads.
Taro: But what about latency? Because if you’re constantly changing things up—scaling clusters, adjusting parallelism—doesn't that introduce some kind of overhead that hurts the actual speed of inference?
Dev: They specifically checked for that, and they found that the overall scheme maintained negligible impact on inference latency, meaning the end to end latency followed what we expected from our baseline tests.
Paper summary: Rosa: That’s a big win because it means you get the energy savings without making customers notice any slowdown in how fast their AI is responding.
Taro: So, in simple terms for someone just listening to this show, this paper suggests that we can finally manage the heat and the power consumption of large language models by coordinating the hardware scheduling with a predictive cooling system.
Dev: The title itself, "Coordinated Cooling and Compute Management for AI Datacenters," really captures what they did—they linked those two traditionally separate domains together.
Rosa: It’s about taking something that used to be just a scheduling problem and making it an integrated thermal management problem instead.
Taro: It’s interesting because the authors talk about how this hierarchical control demonstrates effective coordination between the compute and cooling layers even when demand conditions are changing a lot, which is what happens in real AI usage.
Dev: I think the most important thing is that they moved beyond just optimizing one part of the system to solving it holistically across all those levels.
Rosa: It shows how an LSTM prediction model can drive cluster activation, which then feeds into parallelism adjustments, which finally informs the cooling parameters and the DVFS scaling.
Taro: I’m still thinking about that limitation they mentioned in their work. They noted that they built this framework upon detailed models of LLM workloads and key cooling control parameters, and that extending it to include advanced liquid or hybrid cooling schemes is the next step for better efficiency.
Rosa: Right, so the current model is solid for air-cooled setups with these specific components, but pushing it toward liquid cooling opens up another area where we can see even more gains in energy efficiency and stability.
Dev: So to wrap up this part of our discussion on "Coordinated Cooling and Compute Management for AI Datacenters," the authors show a way to use an LSTM predictor to drive a four-level control system that balances compute needs with thermal constraints, resulting in about twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings while keeping inference performance stable.
Taro: It’s a practical approach for managing the massive power demands of modern AI deployments.
Rosa: Yeah, it gives us a concrete way to think about how to handle the heat generated by these systems without just throwing more power at it hoping for the best.
Conclusion: Rosa: So we've been talking about this paper that's titled "Coordinated Cooling and Compute Management for AI Datacenters." Basically, it shows how you can link the scheduling of computing power with managing the cooling systems in an AI datacenter to save energy while keeping things stable.
Dev: Right, it’s about taking something that used to be a separate problem—how much compute you want versus how hot things get—and making them work together across different layers of control.
Rosa: And the authors use this four-level system, from deciding how many GPU clusters to turn on down to adjusting the individual chip speeds for each job.
Taro: It’s interesting because they aren't just looking at one thing; they're trying to manage the entire environment with a predictive model that forecasts future AI workload demands.
Dev: That LSTM network, the sequence-to-sequence one, predicts what kind of token demand is coming in thirty minute intervals, which lets them proactively adjust how many GPU clusters are active.
Rosa: It suggests that instead of just reacting to heat when it’s already high, you can decide ahead of time how much power to use and how the cooling system needs to ramp up.
Taro: But I wonder about the real world stuff, Rosa. If the workload spikes way harder than that prediction model saw coming, does this framework handle that sudden shock gracefully?
Dev: That’s a valid concern, Taro. The safety case for me is in how fast the cooling controller reacts when things go off script, and they show their MPC cooling formulation performs much better than older proportional integral controllers under those changing conditions.
Rosa: And on the results side, they're claiming real savings: twenty-four point two percent computing energy savings and thirty-one point two percent cooling energy savings without sacrificing service level objectives.
Taro: Those are solid numbers, but I’m still thinking about the next big step for this kind of work.
Dev: Exactly. The authors themselves noted that they built this framework based on specific models for air cooling and certain hardware parameters, so extending it to include advanced liquid or hybrid cooling schemes is the natural next goal for improving efficiency further.
Nardos Belay Abera, Yize Chen
University of Alberta · Natural Sciences and Engineering Research Council of Canada
eess.SY, cs.DC, cs.SY
Submitted: 2026-01-13
Updated: 2026-11-13
Comments: Accepted at IEEE Transactions on Cloud Computing
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 92/100
The gist: The paper proposes a novel hierarchical control framework that jointly coordinates computing and cooling resources to achieve energy-efficient Large Language Model inference because this approach…
Key concepts
- Hierarchical Control Framework
- A multi-level system that coordinates different management tasks across various scales, from large clusters down to individual jobs. It ensures that decisions made at one level, like activating a GPU cluster, are coordinated with decisions at lower levels, such as adjusting cooling or individual GPU speeds. This structure allows for balanced optimization of compute and thermal resources.
- Thermal Model
- A mathematical model describing how heat moves within a server rack. It uses energy conservation laws to track the temperature across three zones: cold air, server exhaust, and hot areas. This model helps predict where hotspots will occur based on the workload and cooling setup, allowing for proactive thermal management.
- Model Predictive Control (MPC)
- A control strategy used at the cooling level to regulate airflow and inlet temperatures every 60 seconds. MPC looks ahead at future conditions to optimize cooling actions, aiming to maintain thermal stability while minimizing energy use. It is more effective than traditional methods like PID control for this purpose.
- DVFS Scaling
- Dynamic Voltage and Frequency Scaling applied at the lowest level, adjusting the GPU's operating speed based on immediate needs. This scaling is tailored per job and considers both Service Level Objectives (SLOs) for latency and current thermal conditions, ensuring GPUs run at the optimal power level needed.
Terminology
Summary
The paper proposes a novel hierarchical control framework that jointly coordinates computing and cooling resources to achieve energy-efficient Large Language Model inference because this approach significantly improves AI datacenters’ energy efficiency while balancing serving latency and thermal constraints. p. 0 Built upon such workload and
thermal dynamics models, a novel hierarchical control framework
is proposed to co-optimize computing and thermal management
by identifying the optimal GPU parallelism, frequency (DVFS),
and cooling control knobs. Using real Azure inference traces
and detailed GPU profiling, our model balances serving latency
and thermal constraints in AI datacenters while significantly
improving AI datacenters’ energy efficiency.Where the paper says it
Problem Motivation
The current state-of-the-art often focuses only on optimizing compute-side scheduling without considering thermal objectives or constraints. Ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving because GPU-intensive inference generates substantial heat that can degrade datacenter performance through thermal throttling. p. 0 Although current state-of-the-art has examined the
energy efficiency of LLM inference, most prior research focused on
optimizing compute-side scheduling without considering thermal
objectives or constraints. Since GPU-intensive inference generates
substantial heat that can degrade datacenter performance,
ignoring thermal effects can increase total energy consumption
and reduce the efficiency of LLM serving. Load and temperature imbalances
also cause sub-optimal scheduling and cooling in datacenters. Therefore, neglecting thermal dynamics could result in cooling
units reacting only after temperature increases occur [14]. This creates irregular compute
demand and rapid heat generation patterns across GPU racks. This heterogeneity breaks the assumption of the traditional
datacenter control system and overwhelms cooling systems,
whose slow response leads to temperature overshoots, local hot
spots, and inefficient use of cooling energy [16].Where the paper says it
Where the paper says it
Where the paper says it
Hierarchical Control Framework Design
The proposed hierarchical control framework operates across multiple levels to co-optimize computing and thermal management. The framework consists of four levels of decision-making, including a cluster-level controller, a window-level mixed-integer linear program (MILP) for TP selection, a cooling MPC formulation, and per-job DVFS scaling. p. 5 This profile-anchored
sizing translates the forecasted workload into a robust GPU
budget, which is subsequently passed to the window-level
mixed-integer linear program (MILP) for fine-grained allocation
across TP modes. Window-Level TP Selection
At the start of each control window, denoted as w (5 minutes),
the system verifies the optimal TP configuration mix m ∈ M,
where M is the set of potential GPU parallelism settings,
e.Where the paper says it
-
At the higher level, the cluster-level controller determines how many GPU clusters to activate based on workload forecasts obtained from an LSTM prediction model. p. 4
Where the paper says it
HIERARCHICAL COOLING AND COMPUTING CONTROL In this Section, we introduce the design of our hierarchical control architecture, which integrates four levels of decision making to jointly manage computing and cooling parameters in LLM inference. At the upper layer, a cluster-level controller determines how many GPU clusters to activate based on workload forecasts obtained from an LSTM prediction model, TABLE I: Performance metrics with different configurations of TP units for varying levels of total tokens, with 8×Tesla V100 GPUs (16 GB), on the Llama-2 7B model inference workload.
-
At the intermediate level, the window-level controller adjusts GPU parallelism (TP configurations) every 5 minutes while incorporating a novel thermal constraint to prevent overheating. p. 5
Where the paper says it
58 43 updated every 30 minutes. In the second layer, the TP configurations are adjusted every 5 minutes to meet workload demands, while incorporating a novel thermal constraint to prevent overheating.
-
At the cooling level, the MPC controller is updated every 60 seconds to regulate airflow and inlet temperature for thermal stability. p. 5
Where the paper says it
In the second layer, the TP configurations are adjusted every 5 minutes to meet workload demands, while incorporating a novel thermal constraint to prevent overheating. In the third layer, the cooling control is updated every 60 seconds to regulate airflow and inlet temperature for thermal stability.
-
At the lowest level, per-job DVFS scaling is applied to adjust GPU operating frequencies according to SLOs and thermal conditions. p. 1
Where the paper says it
On the lower level, dynamic voltage and frequency scaling (DVFS) is adopted to adjust the frequencies of GPUs according to the SLOs of workloads and thermal conditions. To achieve this, we introduce a novel temperature-aware DVFS mechanism that jointly considers workload intensity and GPU temperature when selecting the optimal operating frequency.
Key Modeling Components
The framework is built upon detailed models of LLM workloads and key cooling control parameters.
(i) Thermal Model:
The thermal dynamics per-rack are built based on energy conservation laws, describing the temperature field across three zones (cold, server exhaust, and hot) using multi-node state-space form governed by mass conservation principles.
(ii) Cooling Power Model:
The total cooling power is written as the sum of the cold-source consumption and the fan consumption. The RCU supply temperature and airflow rate are controlled by this model to remove heat from the GPU rack. p. 2 III-A and Sect. III-B develop
the thermal model, which includes (i) a temperature-field model
that describes rack-level heat transfer and airflow dynamics and
(ii) a cooling power model that links supply temperature, airflow
rate, and chiller/fan consumption.Where the paper says it
(iii) Workload Forecasting Model:
Precise workload prediction is achieved using a sequence-to-sequence long-short-term memory (LSTM) network trained on 30-minute interval data to predict token demand at the cluster level.
Performance and Results
The comprehensive evaluation using Microsoft Azure LLM inference traces demonstrates significant improvements in energy efficiency and thermal stability.
(i) Energy Savings:
The proposed approach achieves 24.2% computing-energy savings and 31. p. 1 Main Contribution
Building upon LLM serving characteristics and opportunities
of co-management of GPU cooling and computing, this work
makes the following key contributions:
• We propose a novel hierarchical compute–thermal control
framework for the AI datacenter that provides methodology for jointly optimizing computing resources (GPU
parallelism and frequency) and cooling resources (supply
air temperature and inlet airflow rate). A novel temperature
constraint is integrated to guaranty thermal safety while
enabling significant energy savings. • We perform a comprehensive evaluation using Microsoft
Azure LLM inference traces in combination with detailed
thermal and power profiling of the GPU [23]. Simulation
and experimental results demonstrate that the proposed
approach reduces both IT-side and cooling-side power consumption while preserving SLOs.Where the paper says it
Where the paper says it
(ii) Thermal Improvement:
The controller lowers the average GPU temperature by 17.0%, maintaining temperatures below the maximum allowable limit of 50°C. p. 5 Designed to minimize the energy
consumption of the GPU (12a). The constraint (12b) guaranties
that the aggregate capacity of all active pools Cm,w is adequate
to support the expected token demand; The constraint (12c)
applies the constraint budget of the GPUs Gmax (total available
GPUs), while (12d) will ensure that the resulting temperature
θGP U,w from the thermal dynamics in (11) remains below the
maximum allowable limit, i.Where the paper says it
(iii) Latency Preservation:
The overall scheme maintains negligible impact on inference latency, with end-to-end latency following the baseline well.
Control Strategy Effectiveness
The hierarchical control demonstrates effective coordination between compute and cooling layers under varying demand conditions.
(i) Cluster-Level Control:
This layer dynamically manages the number of active GPU servers every 30 minutes based on the LSTM workload forecast, effectively balancing resource allocation and energy efficiency.
(ii) Cooling MPC Performance:
The MPC controller significantly outperforms the PID controller in terms of energy savings by approximately 28% while maintaining the return-air temperature closer to its set point than the PID baseline.
(iii) Per-Job DVFS:
The DistilBERT-based job classifier enables class-specific latency and thermal constraints, allowing frequency selection to be formulated as an optimization problem that is token length–aware and ensures compliance with SLOs.
Conclusion
The proposed hierarchical design integrates LP-driven resource allocation, MILP-based LLM scheduling, and MPC cooling control supported by an LSTM workload predictor to tackle the energy management challenge. Extensive simulations and real system experiments validate that the framework can efficiently reduce total energy costs while maintaining reliable inference performance. p. 9 Table VI shows that the proposed controller
has a reduction in energy in both the cooling and the computing
subsystems. The result suggests that the proposed hierarchical
controller can efficiently reduce total energy costs while
maintaining reliable inference performance. 2% cooling energy savings, and 17% decrease
in mean GPU temperature, resulting in consistent total energy
reduction without latency degradation. Future research will
extend the scope further to include advanced liquid and
hybrid cooling schemes with the objective of enhancing energy
efficiency and temperature stability.Where the paper says it
Where the paper says it
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Hierarchical Control Framework Implementation: The proposed framework integrates LP-driven resource allocation, MILP-based LLM scheduling, and MPC cooling control to jointly manage computing and cooling resources. This system can achieve
coordinated control within various timescales, both the long-term workload variability and the fine-grained system changes.
-
Workload Forecasting for Proactive Allocation: By using an LSTM network trained on 30-minute interval data to predict token demand, the AI system can
proactively adjust the cooling supply, schedule workloads to ensure that temperature and latency constraints are satisfied.
-
Dynamic GPU Parallelism Adjustment: The window-level MILP optimization can dynamically select TP configurations based on forecasts, allowing the system to
reduce overall power consumption while (i) meeting the anticipated token demand Lˆw, (ii) adhering to the specified GPU budget Gmax and (iii) maintaining compliance with thermal regulation.
-
Thermal-Aware DVFS for Per-Job Optimization: The introduction of a DistilBERT-based job classifier enables frequency selection formulation that is
token length–aware and ensures compliance with SLOs,
allowing the system tominimize the power of the GPU by selecting the optimal operating frequency f for each job j, subject to several constraints.
-
Coordinated Cooling Regulation: The MPC cooling controller regulates supply air temperature and airflow rate by solving an optimization problem that
minimizes the total cooling power on the prediction horizon
while enforcing thermal safety constraints like keeping return-air temperature below its set point.
Abstract
The AI datacenters are currently being deployed on a large scale to support the training and deployment of power-intensive large-language models (LLMs). Extensive amount of computation and cooling required in datacenters increase concerns about the energy use and carbon emissions of AI datacenters. Although current state-of-the-art has examined the energy efficiency of LLM inference, most prior research focused on optimizing compute-side scheduling without considering thermal objectives or constraints. Since GPU-intensive inference generates substantial heat that can degrade datacenter performance, ignoring thermal effects can increase total energy consumption and reduce the efficiency of LLM serving. To fill this gap, we profile the characteristics of GPU servers under varying cooling and AI jobs, and develop a joint cooling and computing modeling approach for AI datacenters. Built upon such workload and thermal dynamics models, a novel hierarchical control framework is proposed to co-optimize computing and thermal management by identifying the optimal GPU parallelism, frequency (DVFS), and cooling control knobs. Using real Azure inference traces and detailed GPU profiling, our model balances serving latency and thermal constraints in AI datacenters while significantly improving AI datacenters' energy efficiency.
Sources
- Carbon Emissions and Large Neural Network Training
- Language Models are Few-Shot Learners
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Related papers
- Industrial Load Modeling and Optimization for Market-Based Interaction with Power Systems
- Exact and Approximate Convex Reformulation of Linear Stochastic Optimal Control with Chance Constraints
- Certified Set Convergence for Piecewise Affine Systems
- Symbolic Control for Stochastic Systems via Finite Parity Games
- Multiobjective optimization-based design and dispatch of islanded, hybrid microgrids for remote, off-grid communities in sub-Saharan Africa
- Physiology-informed layered sensing for intelligent human-exoskeleton interaction