Operating heat pumps in residential complexes via reinforcement learning

A German research group has proposed to use reinforcement learning (RL) to optimize the utilization of heat pumps (HPs) for space heating in big apartment complexes.

RL is a type of machine learning that teaches agents to make optimal decisions in dynamic environments. RL agents learn from the environment by receiving rewards or penalties.

“A crucial step in designing an RL approach for building control is the selection of a meaningful reward function,” the scientists explained. “This is mandatory to give the agent the needed feedback to optimize its behavior. For space heating control, this reward function is often designed using the internal room temperature together with comfort bands of the residents.”

The research team created an environment modeling of a residential heat network including a hot water tank (HWT) as heat storage without requiring a building model and modeled a 100 kW heat pump based on a temperature-dependent coefficient-of-performance (COP) curve. “The agent learns to operate under perfect foresight conditions as well as relying on demand predictions,” it further explained.

The RL-based operational management approach is said not to compromise comfort level and to meet the heat demand of residents at all times. It was trained via a proximal policy optimization (PPO) algorithm, which is commonly used in the field of reinforcement learning by training a computer agent’s decision function to accomplish difficult tasks. “It applies a ‘proximal' approach by introducing a clipping mechanism into the objective function,” the team emphasized.

Their analysis considered parameters such as total electricity costs, electricity costs due to heat demand, electricity costs due to heat loss, total electricity consumption of the HP, number of on/off state changes of the HP, average operating electrical power of the HP, maximum operating electrical power of the HP, average SOC of the HWT, and maximum SOC of the HWT.

They assumed the experimental setup to be deployed in an apartment complex comprising 100 residential living units and 7,000 m2 of living space at a space heating demand of 25-28 kWh/qm per year.

The scientists said their assessment showed that the proposed setup can reduce electricity costs by approximately 35%, as it takes advantage of the variable electricity price, while also reducing the total energy consumption of the heat pump by up to 15%.

“Additionally, we show that the intelligent operation of HPs does not use the full installed nominal electrical power and could therefore reduce investment costs,” they added. “The robustness of the approach is shown by running ten independent training and testing cycles for all setups with reproducible results.”

The novel technique was introduced in the paper “Forecast-based and data-driven reinforcement learning for residential heat pump operation,” published in Applied Energy. The research group included scientists from the DLR-Institute for Software Technology, the Carl von Ossietzky University Oldenburg, and the Fraunhofer Institute for Manufacturing Technology and Advanced Materials (Fraunhofer IFAM).

“Improvements for further studies could be to increase the complexity of the environment,” the academics said, referring to the future direction of their work.

Popular content

Solarwatt presents new residential battery

22 November 2024 German manufacturer Solarwatt says its new battery can be flexibly configured as an AC or DC system. It also features an emergency power function and...