Energy Demand Response in a Food Processing Plant: A Deep Reinforcement Learning Approach

1. Introduction

Industrial food processing facilities are energy-intensive [1], where up to 50% of the food production costs are energy-related [2]. One effective approach to reduce energy costs, lower emissions, or enhance energy efficiency is DR for load profile adjustments [3]. The enormous DR potential of the industrial sector is shown in Siddiquee et al. [4]. In the food industry, this DR potential exists due to the high energy demand for heating and cooling food products [5]. One promising approach to use this load shifting potential is via thermal energy storages (TESs) [6]. In this study, a food processing plant with a TES is optimized using RL. The first part of the introduction focuses on DR in food processing plants using thermal capacities as TES followed by RL as a general optimization method and some DR applications of RL in various fields. Finally, the research gap is highlighted and the contributions are shown.

1.1. Thermal Capacities for Demand Response

The effectiveness of food products as TES has been proven by several studies [7,8,9,10,11] investigating supermarkets. In these studies, refrigerated food products are used as TES for load shifting by allowing a controlled temperature range. Transient models of food products can be physics-based via transient energy balances or data-driven, and a variety of optimization algorithms is used for both of them [7,10,11]. While [7,8,9,10,11] focuses on commercial buildings, [13,14,15,16,17] explores industrial settings, particularly refrigerated warehouses. The thermal capacity of such warehouses is mainly due to the stored products [18]. Case studies show event-based DR potential in two American refrigerated warehouses [13]. Akerma et al. [14] present a detailed model for load shedding during 2 or 4 hour DR events, and a follow-up paper [15] confirms that this 2 hour load shifting does not significantly impact stored food. Ma et al. [16] investigated multiple warehouses, using non-convex optimization for cooperative DR to reduce energy costs and consumption by varying set point temperatures. Khorsandnejad et al. [17] use MILP to analyze a cold warehouse with various compressors and a TES, achieving up to 50% reduction in electricity costs, 25-30% in CO₂ emissions, and about 50% in peak load. Besides existing work on the DR potential in food storage warehouses, multiple studies [19,20,21,22,23,24,25] have examined DR applications in food processing plants. For instance, Chen et al. [19] developed an energy hub model to simulate the electricity demand of a food processing procedure within an industrial park. They implemented a two-stage robust co-optimization approach using MILP to minimize electricity costs under a time-of-use (TOU) pricing scheme. The model included a battery energy storage system and a TES for flexibility. Giordano et al. [20] examined the energy supply of a milk production plant. They modeled the energy supply using fixed load profiles and incorporated an energy storage for flexibility. A rule-based operation optimized energy consumption, reducing energy demand and carbon emissions. Pazmiño-Arias et al. [21] optimized the energy supply of a dairy factory using an energy hub model with fixed load profiles for heat, cooling, and electricity. They applied nonlinear optimization to minimize total energy costs under a TOU pricing scenario. Cirocco et al. [22] investigated the DR potential of combining industrial TES with a photovoltaic (PV) system in an Australian winery. Using historical load profiles, the authors utilized a TES to boost PV self-consumption. A nonlinear optimal control algorithm reduced electricity costs by 22% compared to a non-optimized case without PV. Saffari et al. [23] conducted a case study in a dairy factory producing yogurt and cheese. The work investigated the combination of TES and PV using a MILP formulation based on load profiles. Simulations showed cost reductions from 1.5% to 10%, depending on the scenario. In an earlier study conducted by the authors of this paper [24,25], an Austrian food processing plant is modeled and its operation is optimized in a simulation study. The study uses a detailed model of the food production process for DR. Flexibility options like thermal mass or a chilled water buffer are compared. The DR optimization problem is solved via MILP, achieving reductions in electrical power consumption by up to 18%, electricity costs by up to 24%, and peak load by up to 36%. In summary, the literature shows a clear potential for DR in the food industry.

1.2. Reinforcement Learning for Demand Response

Literature reviews such as Zhang and Grossmann [26] from 2016 and the literature from SubsectionSection 1.1 show, that MILP is the standard method for optimization in industrial DR problems. In Zhang and Grossmann [26], 36 out of 42 industrial demand side management (DSM) papers used MILP. While MILP is computationally efficient for small problems and is capable of finding the global optimum, a high number of binary variables can drastically increase the computation time. Furthermore, a downside of MILP is that the problem has to be solved for every different time window separately in model predictive control (MPC). In contrast, machine learning (ML) techniques like RL require training only once, resulting in a computational efficient application afterwards. Therefore, the potential of RL in a food processing plant is investigated in this study.

Recently, promising technologies like Deep Q-Learning (DQL) have emerged. DQL, introduced by Mnih et al. [27], replaces the Q-table from classical Q-learning [28] with a neural network, enabling it to handle high-dimensional state spaces. DQL showed enormous potential, being capable of achieving great scores in over 2600 Atari games [27]. A limitation of the DQL network is its tendency to overestimate state values and moving targets. To solve this problem, van Hasselt et al. [29] introduced double deep Q-learning (DDQL). They adapted the DQL network by using one neural network for choosing the best action and one for estimating the value the next action. Thereby, they achieved better or similar scores in the Atari games compared to the DQL. RL’s application scope has rapidly expanded, making it a key research focus in various fields. Vazquez et al. [30] investigated the application of traditional RL algorithms such as Q-learning or Monte Carlo for DR and concluded that these RL algorithms are capable in DR to help integrate renewable energies into the power grid. Deep reinforcement learning (DRL) such as DQL and DDQL for smart building energy management are investigated in a review paper [31], showing that nearly all model-free DRL-based building energy optimization methods are still not implemented in practice. Furthermore, DRL can be used in buildings to simultaneously reduce energy cost, peak load, and occupant dissatisfaction degree. A notable example of RL in practice is Google DeepMind’s achievement of a 40% reduction in data center cooling costs using RL techniques [32]. In the following part of the literature review, the classification from OpenAI [33] is used where model-free RL is categorized into policy-based aka policy optimization, value-based aka Q-Learning, or methods based on both. There, DQL and DDQL are listed as deep Q-networks (DQN). In DR literature, value-based RL is applied in residential [34,35,36,37,38] or commercial buildings [39,40,41], smart grids [42,43], or in microgrids [44]. The applied RL algorithms are based on QL [34,42,43] or DQN [35,36,37,38,39,40,41,44] and used to improve the energy efficiency or to reduce the energy costs. Value- and policy-based optimization is applied in commercial building such as offices [45,46,47], in a laboratory setup [48], or in an industrial warehouse [49]. The applied RL algorithms are based on soft actor-critic (SAC) [45,46,47], deep deterministic policy gradient (DDPG) [48], or random augmented search [49]. SAC was successfully implemented in a real office building in showing a decrease of temperature violation by 68% while maintaining a similar energy consumption as the reference case [47]. Policy-based RL is applied in a commercial building [50] or a university [51], where proximal policy optimization (PPO) is applied in both. Policy-based RL algorithms are well-suited for continuous action spaces, but value-based RL like DQN is preferred for discrete action spaces for its lower computation costs and greater sample efficiency. Therefore, DQN has potential for application in energy management, because heating, ventilation, and air conditioning (HVAC) systems often use digital controllers with a discrete set point.

1.3. Contributions

Summarizing, RL shows great potential in various application fields. DDQL has not yet been applied to optimize the set point temperature of a refrigerated warehouse in a food processing plant for DR using real-time pricing (RTP) and compared with a state of the art MPC controller based on MILP. We investigate an onsite warehouse in a food processing plant. This warehouse is influenced not only by ambient temperature and the HVAC system, but also by the food production process, which significantly impacts the system’s dynamics. We use the thermal capacity of the ware and the building as TES and apply DR by optimizing the set point temperature of the warehouse’s proportional integral (PI) controller. Our main contributions are:

• We optimize the set point temperature of a PI controller rather than directly controlling the cooling power. This approach enhances stability and simplifies practical implementation.
• We apply DDQL - a state-of-the-art RL algorithm - for load shifting to reduce energy costs in an RTP scenario.
• We formulate the problem as MILP to compare RL with a state-of-the-art MPC controller.
• We investigate the energy cost savings and the computation time of RL and MILP.

The remainder of this study is organized as follows: The system model for RL and MILP, the simulation study concept, the RL algorithm, and the MILP are described in SectionSection 2. In SectionSection 3, we present the optimization results including an analysis of the RL training process and the influence of the price signal on the DR potential. Finally, conclusions are drawn in SectionSection 4.

2. Methods

In this study, we investigate a food processing plant, where cheese is produced and stored. The focus of the paper is the industrial warehouse in the plant, which is used for cooling and storing cheese. The products of multiple months is stored in a high-bay racking. Cheese enters the warehouse at temperatures between 80°C and 130°C and is cooled to approximately 0°C for storage until sold. An industrial refrigeration system handles this cooling process, and the thermal mass of the warehouse, including the stored cheese products, functions as a TES. By allowing a defined temperature range between 0°C and 5°C, this TES provides flexibility for optimization. The goal of the optimizer is to shift the electrical load of the refrigeration plant to periods with lower electricity prices, reducing overall energy costs. Instead of directly controlling the cooling power, the optimizer adjusts the set point temperature of an existing PI controller. We compare two optimization approaches: dynamic optimization using RL and classical MILP. The optimization is incentivized using hourly RTP. For comparison, we use a realistic, currently employed scenario where the set point temperature remains constant at 2.5°C as a reference case. The study uses data from an industrial plant, including production data (the mass flow of product in the warehouse due to production ${\dot{m}}_{in}$), ambient temperature ($T_{\infty }$), the energy efficiency ratio of the refrigeration system ($\beta$), and approximations of heat losses to the ambient (${\dot{Q}}_{l}oss$). Historical data from May 2020 to April 2023 is available. The hourly spot market prices from Energy Exchange Austria (EXAA) [52] are used as RTP for the optimization process. The data is split into two sets: two years of data are used to train the RL agent, while one year is reserved for testing, either via MILP or by the RL agent. The simulation study is conducted in Python.

2.1. Industrial Warehouse Model

FigureFigure 1 shows an overview of the system. The industrial warehouse is modeled with the thermal capacity $C_H$ of the complete warehouse and a transient energy balance. The thermal capacity is the sum of the internal thermal capacities of the stored cheese products and the thermal mass of the building, where the steel of the high-bay racking is a relevant factor. This is modeled by

$$\begin{array}{cc}\hfill C_H\left(t\right)& =c_{cheese}{m}_{cheese}\left(t\right)+c_{building}{m}_{building}\hfill \end{array}$$

(1)

where E is the energy, $c_{cheese}$ and $c_{building}$ are the specific thermal capacities of cheese and the building, and $m_{cheese}$ and $m_{building}$ are the masses of the cheese and the building, respectively. The temperature of the industrial warehouse $T_H$ can be calculated via a transient energy balance:

$$\begin{array}{cc}\hfill \frac{dE}{dt}& =C_H\left(t\right){\dot{T}}_H\left(t\right)={\dot{Q}}_{gain}\left(t\right)+{\dot{m}}_{in}\left(t\right)h\left(T_{in}\right)-{\dot{m}}_{out}\left(t\right)h\left(T_{out}\right)-{\dot{Q}}_{cool}\left(t\right)\hfill \end{array}$$

(2)

where ${\dot{Q}}_{gain}\left(t\right)$ is the heat gain from ambient, ${\dot{m}}_{in}\left(t\right)$ is the mass flow rate of hot cheese in the warehouse, ${\dot{m}}_{out}\left(t\right)$ is the mass flow rate of cold cheese from the warehouse, $h\left(T\right)$ is the enthalpy of the cheese, and ${\dot{Q}}_{cool}\left(t\right)$ is the cooling power of the chiller. The chiller is modeled via an average energy efficiency rate $\beta$:

$$\begin{array}{cc}\hfill {\dot{Q}}_{cool}\left(t\right)& =P_{el}\left(t\right)\beta \hfill \end{array}$$

(3)

Assuming that ${\dot{m}}_{in}\left(t\right)$ and ${\dot{m}}_{out}\left(t\right)$ are equal, $C_H$ is constant. As a further simplification, $T_{in}$ and $T_{out}$ of the cheese are assumed to be constant. Thereby, the heat flow rates can be simplified to:

$$\begin{array}{c}\hfill {\dot{Q}}_{heat}\left(t\right)={\dot{Q}}_{gain}\left(t\right)+{\dot{m}}_{in}\left(t\right)h_{in}-{\dot{m}}_{out}\left(t\right)h_{out}\end{array}$$

(4)

Applying all simplifications mentioned, this leads to:

$$\begin{array}{cc}\hfill {\dot{T}}_H\left(t\right)& ={\displaystyle \frac{1}{C_H}}\left({\dot{Q}}_{heat}\left(t\right)-P_{el}\left(t\right)\beta \right)\hfill \end{array}$$

(5)

Applying discretization to the inputs (${\dot{Q}}_{heat}\left(t\right)$,$P_{el}\left(t\right)$) for a specified time interval i of duration $\Delta t$, the transient thermal energy balance can be simplified to:

$$\begin{array}{cc}\hfill T_{H,i+1}& =T_{H,i}+{\displaystyle \frac{1}{C_H}}\left({\dot{Q}}_{heat,i}-P_{el,i}\beta \right)\Delta t\hfill \end{array}$$

(6)

Equation(6) can be used to calculate the system behavior.

The cooling power ${\dot{Q}}_{cool}\left(t\right)$ is controlled via a basic time-discrete PI controller with saturation:

Algorithm 1:PI controller with saturation

1: $u_i=k_P(T_{\mathrm{H},i}-T_{\mathrm{set},i})+(k_I\Delta t-k_P)(T_{\mathrm{H},i-1}-T_{\mathrm{set},i-1})+u_{j-1}$  2: if$P_{\mathrm{el},i}>P_{max}$then  3:     $P_{\mathrm{el},i}=P_{max}$  4: else if$P_{\mathrm{el},i}<P_{min}$then  5:     $P_{\mathrm{el},i}=P_{min}$  6: else  7:     $P_{\mathrm{el},i}=u_i$   8: end if

where $T_{set,i}$ is the set point temperature, $k_P$ is the proportional factor, $k_I$ is the integral factor, $P_{max}$ is the maximum power, and $P_{min}$ is the minimum power, $u_i$ is the output signal of the PI controller before saturation, and $P_{\mathrm{el},i}$ is the output signal after saturation that is used to control the electrical power of the chiller. In the reference case, $T_{set,i}$ is constant. In the scenarios RL and MILP, the $T_{set,i}$ can be between a lower boundary $T_{lb}$ and an upper boundary $T_{ub}$. The resulting flexibility in temperature associated to the thermal capacity $C_H$ is used for load shifting. The model parameters are shown in TableTable 1.

The specific thermal capacity $c_{cheese}$ is taken from literature [25], the energy efficiency rate $\beta$ is a yearly average based on historical data, the temperature band is defined from 0°C to 5°C, and the remaining parameters are estimated based on internal data of the company. The building model as environment for the RL algorithm was implemented in Python using Gymnasium [53].

2.2. Reinforcement Learning

FigureFigure 2 shows the general principle of RL.

In RL, the problem is framed as a learner (agent) interacting with an environment. The agent interacts with the environment by selection of actions and then receives a reward and a state from the environment. The agent only is aware of the state and a set of possible actions and, based on these, selects an action according to its policy. One iteration of this complete process is called step and all steps together are called episode. The cumulative rewards over the complete episode are called return. Maximizing the expected return is the goal of the agent. In Q-Learning the agent estimates the action-value function denoted as $Q\left(S,A\right)$. This function gives the expected return when starting in a state S, taking an action A and following a policy $\pi$ thereafter. Estimating the action-value function $Q\left(S,A\right)$ is done via bootstrapping in Q-learning and is trained via:

$$\begin{array}{c}\hfill Q\left(S,A\right)\leftarrow Q\left(S,A\right)+\alpha \left[R+\gamma \underset{a}{max}Q\left(S^{\prime },a\right)-Q\left(S,A\right)\right]\end{array}$$

(7)

where $\alpha$ is the learning rate and $\gamma$ is a discount factor. In DQL, the action-value function is approximated via a neural network $Q(S,A;\theta )$ with parameter set $\theta$. Known challenges in using function approximation via neural networks in Q-learning are moving targets and maximization bias. These issues are circumvented in DDQL, where two neural networks are used to stabilize the training process. The policy network $Q_{\pi }$ is used to select the best action given the current state while the target network $Q_t$ is used for estimating the value of the following action. The training process is based on mini-batch learning using memory replay [27]. This technique helps to break correlations of state-action sequences while increasing stability [27]. In contrast to Mnih et al. [27] or Hasselt et al. [29], who use periodical copies of the policy network as target network or update the networks symmetrically by switching the roles, respectively, the presented algorithm uses soft updates such as Lillicrap et al. [54]. The applied DDQL algorithm based on [27,29] is shown in Algorithm2.

Algorithm 2:Double Deep Q-Learning - Training

1: Initialize Policy Network $Q_{\pi }$ with random weights ${\theta }_{\pi }$   2: Initialize Target Network $Q_t$ with random weights ${\theta }_t$   3: Initialize Replay Buffer M as empty buffer   4: loop for each episode:   5:     Reset environment, observe S   6:     loop for each step of episode:   7:         With probability $ϵ$ select a random action A   8:         Otherwise select $A=\arg {max}_a{Q}_{\pi }(S,a;{\theta }_{\pi })$   9:         Decrease $ϵ$   10:         Take action A, observe R, $S^{\prime }$   11:         Store transition (S, A, R, $S^{\prime }$) in M   12:         Sample random mini batch (s, a, r, $s^{\prime }$) of transitions from M   13:         Set $y=\left\{\begin{array}{c}R,if{S}^{\prime }isaterminalstateR+\gamma Q_t(S^{\prime },arg{max}_{a^{\prime }}{Q}_{\pi }(S^{\prime },a^{\prime };{\theta }_{\pi });{\theta }_t),else\hfill \end{array}\right.$   14:         Perform a gradient descent step on the policy network $\nabla L_{\delta }(y,Q_{\pi }(S,A))$   15:         Soft update the weights of the target network $Q_t$   16:         $S\leftarrow S^{\prime }$   17:     end loop   18: end loop

First, the neural networks are initialized with random starting weights (${\theta }_{\pi }$ and ${\theta }_t$) and an empty replay buffer M is created. At the start of every episode, the environment needs to be reset. In lines $7-9$ exploration (selecting a random action) is performed with a probability of $ϵ$. Otherwise, exploitation, which means applying the best action according to the current policy (defined by the policy network), is applied. To improve the policy $\pi$, exploration is essential. As a consequence, the learning process starts with a high exploration rate and later refines the policy using less exploration. The exploration rate is scheduled according to:

$$\begin{array}{cc}\hfill ϵ& ={ϵ}_{e}nd+({ϵ}_{s}tart-{ϵ}_{e}nd)e^{-{\displaystyle \frac{N_{s}teps}{d_{ϵ}}}}\hfill \end{array}$$

(8)

where $ϵ$ is the current threshold for exploration, ${ϵ}_{s}tart$ is the start exploration rate, ${ϵ}_{e}nd$ is the end exploration rate, $d_{ϵ}$ is the decay rate, and $N_{s}teps$ is the number of steps done already. In line 10 the action is performed in the environment and the reward R and the next state $S^{\prime }$ are observed. The transition $(S,A,R,S^{\prime })$ is stored in the replay buffer and random samples from the replay buffer are used to train the policy network $Q_{\pi }$. As a measure of error in the training of the policy network’s weights, the Huber loss $L_{\delta }$ is used:

$$\begin{array}{c}\hfill L_{\delta }(y,f\left(x\right))=\left\{\begin{array}{cc}{\displaystyle \frac{1}{2}}{(y-Q_{\pi }(S,A))}^2\hfill & \mathrm{for}|y-Q_{\pi }(S,A)|\le \delta \hfill \\ \delta \left(|y-Q_{\pi }(S,A)|-{\displaystyle \frac{1}{2}}\delta \right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(9)

where $\delta$ is a parameter of the Huber loss function. The Huber loss acts as the mean squared error for small errors $(y-Q_{\pi }(S,A))$ and as the mean absolute error for larger errors, and, therefore is more robust to outliers. This loss is used to perform a stochastic gradient descent step using the Adam optimizer to update the weights of the policy network $Q_{\pi }$. The weights of the target network $Q_t$ are updated via soft updates:

$$\begin{array}{cc}\hfill {\theta }_t& =\tau {\theta }_{\pi }+(1-\tau ){\theta }_t\hfill \end{array}$$

(10)

where $\tau$ is the target network’s update rate.

This standard RL algorithm can be applied to any problem that can be abstracted in an agent and an environment, where the set of actions is discrete and the state can be discrete and continuous. The state should fulfill the Markov property, so it should include all information about all aspects of the past agent-environment interaction that make a difference in the future. For physical systems in a linear state space representation, the Markov property is fulfilled.

In the following paragraphs, this algorithm is applied to the DR problem in an industrial warehouse. Here, the controller of the refrigeration systems is the agent and the industrial warehouse is the environment. As an action, the agent can control the state of charge ($SOC$) of the TES, which is reciprocal to the set temperature of the industrial warehouse. The possible actions are discrete $(0-100)$, where 0 represents the lowest set state of charge and 100 is the highest set state of charge of the TES controlled by the PI controller. The operation is always optimized for a day, so one episode consists of 24 time steps ($\Delta t_{step}=1$h). The state of the environment is given by a set containing the state of charge, the number of remaining time steps of the day, the electricity price of the next 24 steps, and the predicted thermal load ${\dot{Q}}_{heat,i}$ of the next 24 time steps. The electricity price and the thermal load are assumed to be known for one day, so for 24 steps. Electricity spot-market prices are usually published one day in advance, so this is a realistic scenario. After the first step, only the 23 next steps are known, and so forth. Unknown values are set to 0. To stabilize the learning process, the variables defining the state are scaled. The temperature is used to calculate the state of charge of the energy storage in %:

$$\begin{array}{c}\hfill SO{C}_i=100{\displaystyle \frac{T_{ub}-T_i}{T_{ub}-T_{lb}}}\end{array}$$

(11)

The prices p, which are used to calculate the energy costs and the reward, are scaled between one and 10 on a daily basis using:

$$\begin{array}{c}\hfill p_i^{*}=9{\displaystyle \frac{p_i-min\left(p\right)}{max\left(p\right)-min\left(p\right)}}+1\end{array}$$

(12)

Scaled values are indicated with *. Prices are scaled due to three reasons: 1) the values of the state should all be in a similar range; 2) electricity prices fluctuate strongly during the year and the neural network should learn on the relative prices during a day; 3) therefore, the RL agent can handle price signals significantly higher or lower than already seen during training. The scaled prices are always greater than or equal to one, resulting in a positive energy price. If the price could be 0€/(kWh), the RL agent could use the energy during these time slots for increasing the energy consumption for free.

The thermal load is scaled to a load in percentage of the energy storage via:

$$\begin{array}{c}\hfill {\dot{Q}}_{heat,i}^{*}=100{\displaystyle \frac{{\dot{Q}}_{heat,i}\Delta t}{C_m(T_{ub}-T_{lb})}}\end{array}$$

(13)

Combining Equations(11-13) with ($24-i$) to indicate the remaining time steps of a day/episode, this results in a state $S_i$:

$$\begin{array}{c}\hfill S_i=\left(SO{C}_i,(24-i),p_i^{*},\cdots ,p_{i+23}^{*},{\dot{Q}}_{heat,i}^{*},\cdots ,{\dot{Q}}_{heat,i+23}^{*}\right)\end{array}$$

(14)

Note that if the electricity price and the thermal load for the forecast are not known, a value zero is used. The action space is discrete $(0-100)$ representing the resulting state of charge ($SO{C}_{set}$) for the PI controller. The set point temperature can be calculated from the set value of the state of charge by:

$$\begin{array}{c}\hfill T_{set}=T_{lb}+{\displaystyle \frac{(100-SO{C}_{set})}{100}}(T_{ub}-T_{lb})\end{array}$$

(15)

The environment is implemented using Equation(6) in combination with the PI controller. Note that while there are 24 steps for a day, the PI controller logic is executed multiple times (60 times per time step) during a single simulation step of the environment. Then, the average electrical power $P_{el,i}$ during that step is used to calculate the reward R:

$$\begin{array}{c}\hfill R_i=-P_{el,i}{p}_i^{*}\end{array}$$

(16)

After 24 steps, the terminated flag is set to true and the episode is finished. The algorithm2 described is used to train the agent. During training, a random initial state of charge is used in combination with a randomly selected price signal and a randomly selected load signal from the training set. 2000 training episodes are used. During operation, the greedy action (exploitation) is always applied.

The parameters of the RL algorithm can be seen in TableTable 2.

The RL algorithm was implemented in Python using PyTorch [55].

2.3. Mixed Integer Linear Programming

To validate the performance of RL, MILP is used as a benchmark. MILP is ideally suited for this purpose as benchmark because it the global optimum of an underlying linear problem can be determined. The objective is to minimize the energy costs and the decision variables are the set point temperatures of the warehouse. The model of the warehouse from SubsectionSection 2.1 and the PI controller from Algorithm1 are formulated as constraints in the MILP. While the model of the warehouse is linear, the PI controller has a non-linear effect given by the saturation that limits the controller’s output signal. Modeling this effect introduces binary variables to the formulation and, thereby, drastically increases the computation time. Also, the PI controller reacts every minute to changes in the warehouse temperature leading to an optimization problem for one day with 7201 continuous and 2880 binary variables. The MILP problem was implemented in Python using Gurobi 11.0 [56]. The complete MILP formulation and its detailed description can be found in AppendixAppendix A.

3. Results and Discussion

The trained RL agent is compared with a MILP optimization for the test set, which spans from May 2022 to April 2023. Both, RL and MILP, are used for MPC, always optimizing the operation fo a full day at midnight. The currently employed PI controller is simulated and serves as a reference scenario. The electricity price and the predicted heating load are assumed to be known for the current day. The results of this case study are shown in Table 3.

As expected, MILP has the highest cost reductions, as it leads to the global optimum. While MILP assumes to have perfect knowledge of the complete system dynamics, RL is model-free. In light of this, RL proves to be a promising solution, as it reaches cost reductions of 17.57% as compared to 18.65% reached by MILP. These results are especially good considering that RL is model-free, while the MILP assumes to have perfect knowledge of the complete model. The detailed optimization for three exemplary days can be seen in FigureFigure 3.

Both - RL and MILP - use low energy prices for the precooling of the warehouse and operate close to the upper boundary $T_{u}b$ of the set temperature. The use of the thermal capacity as energy storage can be seen in the second subplot. The capacity of the industrial warehouse including the stored cheese is so high, that the complete temperature band from $T_{l}b$ to $T_{u}b$ is not fully used in these exemplary days. The results of the complete year can be seen in FigureFigure 4.

Comparing the weekly savings from FigureFigure 4a with the electricity price from FigureFigure 4c shows that higher prices lead to a higher potential for cost savings. In 2023, where the energy prices where significantly lower than in 2022, the weekly absolute energy savings also were lower. Nevertheless, FigureFigure 4b shows that the relative energy savings are high during the complete test period. The only relevant outlier is the Christmas week, where there is no production, a low cooling demand due to cold temperature, and low energy prices. Therefore, the energy costs are low and an average reduction with low energy costs results in high relative (FigureFigure 4b) and low absolute (FigureFigure 4a) energy savings. All in all, FigureFigure 4 shows that both algorithms work for high and low energy prices as well as high and low fluctuations. FigureFigure 5 shows the energy price for the test and the training period. There it can be seen that in the training period the prices are significantly different, in particular lower with less fluctuations, than the prices from the test period. As seen the presented RL algorithm also works for prices differing from those in the data set for training, which can be explained via the scaling in the method. This shows the ability of RL to adapt to new states, encountered by the algorithm before.

3.1. RL Training Process

In this subchapter, the RL training process is investigated. The training process for the parameters from TableTable 2 is shown in FigureFigure 6.

The RL agent is quickly improving with decent results after 250 training episodes. High fluctuation of the results of single episodes is caused by the random initialization of the state of charge. In some episodes the energy storage starts full and ends empty, so no additional energy is needed resulting in a reward of 0€. Still, for training different start values are beneficial. To further investigate the training process, different values for the training episodes and the exploration decay rate $d_{ϵ}$ are compared and shown in FigureFigure 7. The number of training episodes and $d_{ϵ}$ are tested with values from 100 to 10000.

Figure 7 shows that the training process converges and, after 250 episodes, good results can be achieved. However, using longer training periods on average improve the results even further. While 250 episodes average 16.94%, 10000 training episodes average a cost reduction of 17.87% and the deviation of the results is much smaller.

3.2. Computation Time

MILP is capable of finding cost-optimal solutions; however, it can be computationally intensive due to the large number of variables involved. The original problem formulation contains 7201 continuous variables and 2880 binary variables. After applying Gurobi’s presolve, the number of continuous variables is reduced to 5070, while the number of binary variables remains unchanged. Despite this reduction, the average computation time per optimization step is 19s, which is reasonable for complex control tasks. In contrast, the RL agent operates significantly faster, with an average computation time of 2ms per optimization step (excluding the training phase). Training the RL model requires 306s (5min) for 2000 episodes. As a result, the total runtime, including both training and operation, is 307s, which is substantially less than the total runtime for MILP, which is 6963s (116min). While MILP provides competitive results with a manageable average optimization time of 19s, the RL approach offers significantly faster performance, being approximately 23 times faster than MILP. Note that all runtime tests were conducted on a 2022 MacBook Pro M2.

3.3. Applicability in Practice

The proposed RL approach is designed to be versatile, making it suitable for a wide range of DR applications, such as managing energy storage systems like batteries, TES systems, and boilers. However, in practical scenarios, collecting the data for multiple months, at least hundreds episodes each representing a full day, can be inefficient and impractical. To overcome this, transfer learning can be employed. Through this method, the agent is pre-trained using simulated environments and later deployed to real-world applications. This not only speeds up the learning process but also enhances safety and system stability by allowing early training phases to exclude extreme actions and critical states.

Another key aspect of the RL framework is its integration with existing control systems. Instead of directly controlling the cooling power, the RL algorithm adjusts the set-point temperature of an existing PI controller. This indirect control improves overall system stability and makes the solution easier to implement. Directly controlling chiller power is often impractical, making this approach more feasible for real-world applications.

The RL algorithm’s computation is efficient, requiring only 2ms per optimization. This makes it ideal for edge computing environments, such as Internet of Things (IoT) devices, where fast decision-making is essential. Its compatibility with microcontrollers that support Python, combined with the use of open-source libraries like PyTorch, further reduces implementation costs by eliminating the need for expensive proprietary software licenses. In contrast, traditional approaches like MILP would demand high-performance computing resources and expensive commercial solvers, such as Gurobi, making RL a more accessible and cost-effective alternative for DR applications.

4. Conclusions

We proposed a machine learning-based method for load shifting in a food processing plant. The food processing plant is modeled based on a transient energy balance and the thermal capacity of the food stored and the building are used as flexibility for load shifting of the industrial refrigeration system. The load is shifted via optimizing a PI controller’s set point temperature and, thereby, safe states of the system are ensured. To optimize load shifting, a RL algorithm is compared with a state-of-the-art MPC controller based on MILP. As a RL algorithm, DDQL is applied. The simulation study results show cost savings in an RTP-driven scenario of 17.57% via model-free RL, compared to 18.65% via MILP, which assumes perfect knowledge of the model. This demonstrates, that RL is capable of providing good solutions without prior knowledge of the model. Also, RL proves to be computationally highly efficient. Even when accounting for both training and operation, the RL agent is still 23 times faster than the MILP method used for comparison. Training the agent only needs to be done once, its operation being fast, with an average computation time of only 2 ms, compared to 19 seconds for solving the MILP. Additionally, the RL agent is based on open-source code, not requiring any expensive, commercial licenses.

The computational efficiency and independence of expensive licensing make RL a promising tool to optimize energy management problems, especially for edge computing on IoT devices. Due to the adaptive and self-learning capabilities of RL, this method can also be transferred to various energy management applications.