Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

The paper investigates how reinforcement learning (RL) controllers for residential heat pumps can be designed to reduce compressor cycling, which is the main cause of compressor wear. The authors add a levelised compressor-wear term to the control reward, monetizing each compressor start at 0.0133 euros, and study how the learned policy depends on the RL algorithm. Training Soft Actor–Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, they find that "SAC learns a continuous modulation policy that keeps the compressor permanently engaged—the operating principle of an inverter-driven heat pump—achieving zero start-ups per day, whereas PPO collapses to bang–bang control that cycles more than the baseline. On the BOPTEST emulator, the SAC policy cuts thermal discomfort by up to 90.7 % for an 11.5 % cost increase, while eliminating all baseline cycling."

The reward is the negative weighted sum of three integrated costs over ∆t: thermal discomfort (in kelvin-hours), electricity cost (in euros), and the novel compressor-wear term. The wear term charges each rising edge of the binary duty signal (on/off state) with a levelised cost, obtained by amortising a representative compressor replacement (∼2000 e) over its rated ∼150 000 starts. A key property is that "a controller can drive J wear to zero in two ways: by never running the compressor, or by running it continuously so that no rising edge ever occurs. The first is excluded by the discomfort term (the zone would freeze); the second is exactly the inverter-modulation regime."

Training uses a fast SINDy surrogate plant, and policies are transferred to the genuine BOPTEST emulator for evaluation. Results show SAC records zero switches and zero starts on both scenarios, eliminating the 1.07 to 1.50 daily starts of the baseline, while PPO cycles 2 to 4 times more often than the baseline. SAC's commanded signal "varies substantially (non-zero TV in Table 1) yet never drops to the off state: uHP ∈ [0.007, 0.93] with 0.0 % of steps below the duty threshold on the peak day, and [0.001, 0.85] with again 0.0 % off on the typical day—the compressor turns on once and then modulates continuously, the signature of inverter operation. PPO instead spends 60.6 % (peak) and 82.3 % (typical) of steps fully off and saturates at the uHP = 1 rail—a bang–bang controller."

On BOPTEST KPIs, SAC cuts thermal discomfort from 8.38 to 0.78 K h (a 90.7 % reduction) for an 11.5 % increase in operating cost, and reduces it by 33.1 % on the typical day. The cost premium is explained by SAC drawing 11.0 % more source energy than the baseline, almost exactly its 11.5 % cost increase, so the premium buys capacity, not tariff mistiming. PPO's low cost is an artefact of under-delivery: it leaves the zone cold and cycles to do so.

The authors explain the algorithm-class effect: "SAC’s maximum-entropy objective values a broad, stochastic action distribution and penalises collapse onto the deterministic rails, while its off-policy replay smooths the critic; interior, continuously-modulated actions thus become the path of least resistance, and the agent converges to the benign always-on optimum. PPO, lacking a comparable entropy floor and updating on-policy, finds it easier to raise the average duty by mixing fully-on and fully-off actions, landing in the bang–bang basin. They conclude that for equipment whose lifetime is governed by switching, entropy-regularised off-policy learning is thus structurally aligned with the smoothness the hardware requires." Limitations include the permissive duty threshold ε = 10−3, the small absolute per-start cost, and training on an uncalibrated surrogate plant.

Improvements to AI systems:

1. Add a wear-aware reward shaping module that explicitly penalizes state transitions (rising edges) in the binary actuator signal, with a tunable levelised cost per start (e.g., 0.0133 EUR/start). This forces the agent to learn policies that minimize mechanical fatigue, not just energy cost and comfort.

2. Integrate an entropy-regularized off-policy RL backbone (e.g., SAC) as the default controller for any discrete-action or continuous-action HVAC equipment with wear constraints, replacing on-policy methods like PPO that collapse to bang–bang control. The system will automatically discover continuous modulation (inverter-style) policies that eliminate cycling.

3. Add a policy smoothness diagnostic layer that monitors the commanded actuator signal's total variation (TV) and the fraction of time spent at rail values (0 or 1). The AI system can flag policies that exhibit high TV or rail-saturation as high-risk for mechanical wear, and trigger retraining or a safety override.

4. Implement a dual-objective evaluation harness that reports both standard KPIs (cost, discomfort) and a cycling KPI (number of starts per day). This ensures the AI system is optimized for equipment longevity, not just operational cost, and prevents under-delivery artifacts (e.g., PPO's low cost from leaving the zone cold).

5. Use a surrogate plant model (e.g., SINDy) for fast policy iteration, but include a transfer-validation step on a high-fidelity emulator (like BOPTEST) to confirm that learned policies retain their cycling-reduction properties in realistic conditions.

What the improved AI system can do:

• Operate a heat pump (or similar switching-limited equipment) with zero or near-zero daily starts, extending compressor lifespan by orders of magnitude.

• Achieve up to 90.7% reduction in thermal discomfort while accepting only ~11.5% higher operating cost, with the cost premium transparently attributed to increased capacity delivery, not tariff inefficiency.

• Automatically distinguish between benign continuous modulation (e.g., uHP ∈ [0.001, 0.93], never off) and harmful bang–bang cycling (e.g., 60–82% of steps fully off), and reject the latter during policy selection.

• Provide a cost-benefit trade-off slider: the user can set the per-start wear cost (e.g., 0.0133 EUR) and the system will re-optimize to balance comfort, energy, and longevity.

• Generalize to other switching-limited actuators (compressors, valves, relays) by simply redefining the rising-edge penalty, without changing the RL algorithm or reward structure.

On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.