Heat Conduction Control using Deep Q-Learning approach with Physics-Informed Neural Networks

1. Introduction

Differential equations serve as the primary representation of complex interactions that are primarily non-linear in the study of real-world phenomena [1]. Although it is possible to model those systems, the analytical solutions of the resulting equations are typically only available for simple cases that are primarily of interest in academic settings. Complex cases that do not have an analytical solution often remain unresolved for a significant period, making it challenging to be used in a real application to make timely decisions to apply certain control measures. Often, surrogate models and numerical solutions are utilised to address this issue, leading to alternative methods for finding the actual solution.

Some research studies employ data-driven models to facilitate the capture of knowledge from spatial and temporal discretizations. For instance, the research conducted by Iakovlev et al. [2] highlights the capacity of a trained model to effectively handle unstructured grids, arbitrary time intervals, and noisy observations.

One way of modelling data is through the employment of neural networks [3,4], which are computational models that approximate the behaviour of a system and are composed of multiple layers to represent data with multiple levels of abstraction [5]. This approach has achieved superior results across a wide range of applications [6], such as visual object recognition, video classification [7] and voice recognition [8].

The typical process for training a neural network involves having a set of data that contains true inputs and their corresponding true outputs. Then, the neural network parameters, weights, and biases are initialised with random values. After such initialization, the goal resides in minimising the error between true known data and the prediction generated by a neural network model. The error is minimised by modifying the neural network parameters using an optimisation strategy.

The objective of standard neural network training is to reduce the error between the model’s predicted values and real values. However, there are cases where gathering the true resulting values is not possible, thereby preventing the application of this training process. To address such an issue, an alternative approach for training neural network models, as proposed by Lagaris et al. [9], states that the training can be performed using a set of diferential equations.

The objective is to approach to zero the error generated by these equations when evaluating a given set of data, collocation points, that are contained in the differential equations domain. This set of data also involves initial and boundary conditions for such differential equations. This methodology does not rely on training using expected results; instead, it employs evaluation data exclusively. This leads to quicker training procedures compared to methods that involve running simulations or conducting experiments before or during the training phase.

This training approach has been defined as Physics-Informed Neural Networks (PINNs). Some work has been successfully done in the fields of robotics [10], power transformers [11] and control strategies [12].

Despite the conceptual simplicity of PINNs, their training process can be challenging [3]. In addition to configuring Neural Network hyperparameters, such as number of layers, nodes per layer, activation functions, and loss functions, PINNs demand balancing multiple terms of the loss function, partitioning of results, and time marching to prevent convergence to undesirable solutions [13,14,15,16,17,18].

On the other hand, a system model defined using a PINN strategy can be used as a digital twin to provide information to a decision-making system that could also take information delivered by sensors [19], or coupled with a control system to define actuation actions in real-time situations [20]. In the same sense of digital twin, Liu et al. [21] proposed a multi-scale model utilising PINNs to predict the thermal conductivity of polyurethane-phase change materials foam composites.

In terms of control of nonlinear dynamical systems, the implementation of digital twins in the form of neural networks has been increasingly employed. Those digital twins are employed to estimate the state of the system at any given location and time, acting as virtual sensors. An example of such implementation to estimate flow around obstacles, stabilise vortex shedding, and reduce drag force has been presented in the work of Fan et al. [22] and Déda et al. [23]. One way to define control strategies is through the implementation of reinforcement learning algorithms, which is a machine learning area able to iteratively improve a policy within a model-free framework [24].

One of the reinforcement learning algorithms is the Deep Q-Learning algorithm. Let’s suppose a discrete time controller that uses a Markov decision process to define an action according to a structure of events. Once an action is defined and executed, this action changes the state of the system. The system states are compared against a reference. The result of such a comparison generates a reward related to such an action. The system then defines another possible action, executes it, and generates another related reward. This process is repeated until an objective is achieved. Those rewards are successively appended and used to train the process model. The Deep Q-Learning algorithm is a reinforcement learning algorithm that uses the Bellman equation to evaluate future incomes (Q-value) [25], starting from the current state, and a neural network (Deep) [26] to simultaneously evaluate the Q-values to all possible actions at some stage.

This work presents a strategy for controlling the central point temperature of a rod in which a heat source is applied at one end and at the other end it has free convection to ambient air as a heat transfer phenomenon. The control strategy is trained with the Deep Q-learning approach using a Physics-Informed Neural Network to evaluate the state of the system and forecast near-future states. This approach is also compared with a standard control strategy. The problem is presented as a uni-dimensional and continuous medium problem, which implies the control strategy has to deal with the delay effects caused by the control action and the heat loss to the ambient.

2. Deep-Q Learning Framework

In this section, the Deep-Q Learning framework is presented by introducing the physics-informed neural network model and control model employed further in this work.

2.1. System Control

A simple control system can be represented by a few components with information travelling between them (see Figure 1). One of the components represents a set of actuators which are initially configured with some possible values. The action coming out from those actuators result in an impact on the environment that is composed out of the system and its bounds interactions, and which states are measured by sensors. Another component is the control unit, that uses the information obtained by the sensors to update the system settings to obtain an objective.

In this system, the two components that are more relevant to this work are: the environment and the control system. Regarding the environment, it can be replaced by a simulator, enabling one to predict its behaviour for some system settings and therefore allow one to train the control beforehand. The control system needs to be defined, i.e., one needs to define a policy that decides the best system setting for an environmental state, represented by the sensor’s measurements.

In this work the environment behaviour will be modelled with a PINN strategy, and the definition of the control unit will be made with a Reinforcement Learning strategy (Deep Q-Learning).

2.2. PINN

Artificial Neural Networks have been developed since the 1950’s inspired by rats cortex functioning [27,28]. A Neural Network is a set of simple computational units, perceptrons (see Figure 2), where a set of inputs are multiplied by weights and its sum with a bias is sent to an activation function. This activation function can be used to polarize the result, i.e., the image is close to a minimum (corresponding to without a property) or close to a maximum (corresponding to having a property).

In Neural Network a set of inputs can feed a set of perceptrons, forming a layer. A feedforward Neural Network is composed of a series of layers, with the outputs of those layers being used as inputs for the following layer (see Figure 3).

The number of inputs and outputs of a Neural Network is defined by the data that is being modelled; however, the number of layers, number of nodes per layer and activation functions are not so easily defined. Data scientists usually use previous experience to set an initial estimate that is improved in the training stage.

On "standard" Neural Networks, the results estimated by the Neural Network $Y_{NN}$ (see Figure 4) are compared with known results Y (e.g. obtained by numerical simulations or laboratory measurements) resulting in a loss, ${\mathcal{L}}_{data}$. This loss can be calculated in several ways, e.g., in this work it was considered the mean square error.

In Physics-Informed Neural Networks, the previously referred data can be used to estimate a total loss, or the train can be performed without any data, making this process much faster, especially when data comes from time-consuming processes.

During PINN training, in addition to estimative $Y_{NN}$, its derivatives are also estimated and replaced on the differential equations to assess the resulting error, ${\mathcal{L}}_{PDE}$. The boundary conditions and initial conditions are assessed, leading to the losses ${\mathcal{L}}_{BC}$ and ${\mathcal{L}}_{IC}$, respectively. A total loss, ${\mathcal{L}}_{TOTAL}$, is calculated with the several partial losses multiplied by weights. In this work, the weights were defined as the inverse of the cardinality of each set.

At Table 1 is presented the PINN training algorithm to determine the Neural Network $Y_{NN}$ that receives the coordinates X, time t, control u as well the initial values $T\left(X,t=0\right)$ and returns the $T\left(X,t=t\right)$. During this training, is required the definition of a finite number of possibilities of each of these parameters. In this work was not considered partitioning of results and time marching to avoid convergence.

2.3. DQL

The objective of the control unit is to decide the best action for a given environment state, i.e., the action that maximizes the reward: at some stage taking an action from a given state, or during an entire episode considering the expected future rewards available from the current state up to the episode end when the goal is achieved.

With a Reinforcement Learning strategy, each state, action taken and reward obtained are recorded in a table to support future decisions. When the set of possible states is too big or even infinite, one needs to make some kind of generalization, and Neural Networks can be used to perform this task. At Deep Q-learning (DQL), instead of a function that looks at a table of previous results to decide the better action for a given state, an NN is used to evaluate the expected future reward (Q-value) as the sum of future rewards weighted with a discount factor, ${\sum }_{t>0}{\gamma }^t{r}_t$, for a set of possible actions [28].

The use of such a supervised learning algorithm enables one to progressively improve the model (see Figure 5).

Q-learning algorithm is based on the Bellman equation (Eq. 1)

$$Q_{new}\left(s_t,a_t\right)=Q_{main}\left(s_t,a_t\right)+\beta \left(r_t+\gamma \underset{a}{max}{Q}_{target}\left(s_{t+{\delta }_t},a\right)-Q_{main}\left(s_t,a_t\right)\right),$$

(1)

where $\beta$ is the learning rate, $r_t$ denotes the current reward obtained taking the action $a_t$ from state $s_t$, $\gamma$ is the discounting rate, a value in $\left]0,1\right[$ used to set the importance of immediate rewards compared with future ones. During the DQL training algorithm the estimate of $ma{x}_{a}Q\left(s_t,a_t\right)$ leads to systematic overestimation introducing a bias in the learning process. A solution to avoid this overestimation is to use two different estimators, $Q_{main}$ and $Q_{target}$, trained at different stages [29]. Whereas $Q_{main}$ is trained periodically every pre-determined number of iterations, $Q_{target}$ is updated getting the values of $Q_{main}$ less frequently.

At Table 2 is presented the DQL algorithm to determine the Neural Network Q that receives the environment state s and returns the reward Q-values (total discouted future rewards r) for each of available actions a.

3. Case Study

In this section, we present the details of the problem used to test the proposed methodology, that aims to utilise a dimensionless model, compatible with neural networks.

3.1. Geometry, Boundary Conditions, and Mesh

The scenario under examination involves a 1D rod of length L, where natural convection occurs at its left end and a controlled heat source is present at its right end (see Figure 6).

The goal of this case is to reach a specific temperature in the centre of the geometry, at $x=L/2$. The governing equation is then the Energy Conservation equation (2)

$$\frac{\partial T}{\partial t}=\alpha \frac{{\partial }^{2}T}{\partial x^2},$$

(2)

where T is the temperature, t the time, $\alpha$ the thermal diffusivity, and x the spatial coordinate.

The natural convection equation (3) relies on the heat transfer coefficient h, which characterises the heat convection condition at the surface of the rod, the thermal conductivity property of the rod material k, and the external temperature $T_{ext}$.

$$-k{\left[\frac{\partial T}{\partial x}\right]}_{x=0}=-h\left(T_{\mathrm{ext}}-{\left[T\right]}_{x=0}\right)$$

(3)

At the opposite end, $x=L$, the heat source is controlled by a u function, leading to the condition shown in equation (4).

$$-k{\left[\frac{\partial T}{\partial x}\right]}_{x=L}=u$$

(4)

As initial conditions, we consider an initial temperature $T_{ini}$.

The above governing equation (Eq. 2) can be adimentionalized by the following replacement of variables as presented in (5), (6) and (7).

$$\theta =\frac{T-T_{\mathrm{ext}}}{T_{\mathrm{ini}}-T_{\mathrm{ext}}},$$

(5)

$$\xi =\frac{x}{L},$$

(6)

$$\tau =\frac{\alpha }{L^2}t,$$

(7)

obtaining the equivalent equation (8).

$$\frac{\partial \theta }{\partial \tau }=\frac{{\partial }^2\theta }{\partial {\xi }^2},$$

(8)

Alternatively, it was considered the normalised values directly on Eq. 2, thermal diffusivity $\alpha =1$, heat transfer coefficient $h=1[\mathrm{W}·{\mathrm{m}}^{-2}{\mathrm{K}}^{-1}]$, $T\in \left[T_{\mathrm{ext}},T_{\mathrm{ini}}\right]$ with $T_{\mathrm{ext}}=0$ and $T_{\mathrm{ini}}=1$, and $x\in \left[0,L\right]$ with $L=1$. The time step was defined as ${\delta }_t=0.1\mathrm{s}$.

For the space discretization several meshes were considered, with 5, 11, 21 and 41 nodes.

3.2. Solution

For validation purposes, the analytical solution was a approximate for a numerical method, the Finite Volume Method, i.e., by integration on space $\left({\delta }_x\right)$ and time $\left({\delta }_t\right)$ at each mesh cell, leading to ${\delta }_x\left(T_i-T_i^{old}\right)={\delta }_t{\sum }_f\alpha \frac{\partial T}{\partial \mathbf{n}}$. The FVM discretization was performed considering a zero volume elements at both domain ends to tackle with boundary conditions, and the remaining elements with length ${\delta }_x$.

3.3. PINN Parameters

In this case study, several PINN configurations were tested until acceptable results were achieved. A deeper analysis and optimization of such a model is not a goal of this work.

The $x\left[\mathrm{m}\right]$ values considered in this work were in $\left[0,1\right]$ with increment $0.1$, leading to 11 possible points to the reference case. The mesh studies were performed with 5, 11, 21 and 41 points with the corresponding increments. The initial values were defined considering a degree 3 polynomial defined with the conditions: ${\left[T\right]}_{x=0}=T_0$, ${\left[T\right]}_{x=L}=T_L$, ${\left[\frac{\partial T}{\partial x}\right]}_{x=0}=T_0-T_{ext}$ and ${\left[\frac{\partial T}{\partial x}\right]}_{x=L}=u$, where $T_0$ and $T_L$ are the temperatures at $x=0$ and $x=L$, respectively, with values in $\left[0,1\right]$ with increment $0.1$, and the a random perturbation $rnd$ in $\left[-0.005,0.005\right]$. The time values, $t\left[\mathrm{s}\right]$, were set in $[0,2]$ with increment $0.05$ and $u\in \left\{0,0.25,0.5,0.75,1\right\}$. Therefore, a training set with $11\times 21\times 5\times 11\times 11=139755$ rows of eleven $T_x$ values was set, and $10\%$ of its values randomly chosen were used to train the PINN.

Regarding the Neural Network, the input parameters are 14: the corrdinate x, the time t, the control u and the eleven initial tamperatures ($T(x=0,t=0)$, $T(x=0.1,t=0)$, ..., $T(x=1,t=0)$). The output is just one value, $T(x,t)$. The Neural Network was defined with 5 inner layers with $2\times 14$, $3\times 14$, $2\times 14$, 14 and 7 nodes, respectively. As activation functions, the hyperbolic tangent was chosen to activate the hidden layers except the last layer, which was set with a linear activation function. The optimizer algorithm employed was Adam, and the train was performed with 200 iterations per each of the learning rates: ${10}^{-3},{10}^{-4},{10}^{-5},{10}^{-6},{10}^{-6}$.

3.4. DQL Parameters

As stated before, the input to DQL is a state s of the system at a given time t that should be enough to decide the action to take. However, it’s possible to increase this information to feed the control with the data of previous time steps. Therefore, this study considered sending the temperature at each coordinate x not only at time t but the last $N_{s_t,h}$, and evaluating the difference control response.

The DQL neural network was set with $11\times N_{st,h}$ inputs for the defined number of previous time steps considered, multiplied by the number of temperatures of each time step (eleven). The 4 hidden layers with $2\times 11N_{st,h}$, $4\times 11N_{st,h}$, $2\times 11N_{st,h}$ and $1\times 11N_{st,h}$ nodes each, and sigmoid activation function. The last layer with activation function was set with softmax to return a probability distribution of the decision to make. The optimizer algorithm was set Adam with a learning rate of ${10}^{-3}$ and categorical cross-entropy as a loss function.

The DQL was trained with 400 episodes, with a 20 maximum steps per episode, saving the results in a 16 long batch before each training, with 4 iterations (epochs), copying $Q_{main}$ to $Q_{trarget}$ every 64 steps. The random actions were taken with a initial probability $1.0$, decreasing $1\%$ every steps until achiving a minimum probability $0.1$.

The learning rate $\beta$ was set as $0.9$, discount factor $\gamma =0.1$ and the space of actions (set of possible values of u) in $\{0,0.25,0.5,0.75,1\}$.

As initial values were considered linear distribution between both ends with temperatures in $[0,1]$ plus a random perturbation in $[0,0.01]$.

Several possible $N{t}_{fw}$ future values were considered to evaluate the control system, to take into account the delay between the action and the system response.

4. Results

4.1. PINN Results

The training of the PINN was performed in an Intel^® Core^™ i7-7700K CPU at $4.20\mathrm{GHz}$ (4 cores, 8 threads) and $64\mathrm{GB}$ of RAM, and took approximately ${17}^{\prime }$ (with 5 elements), $2h{6}^{\prime }$ (with 11 elements), $11h{57}^{\prime }$ (with 21 elements), and $89h{21}^{\prime }$ (with 41 elements).

The prediction using the PINN is obtained almost instantaneously.

The PINN quality was evaluated through the analysis of its prediction to the temperature distribution for all (5, 11, 21 or 41) points homogeneously distributed along the range $\left[0,1\right]$, from the initial condition (at $t=0\mathrm{s}$) up to $t=2\mathrm{s}$ with a time step $0.1\mathrm{s}$.

Due to the integration interval, the FVM was employed considering previous time step (old) values to evaluate the following one, whereas PINN model is not limited by this issue.

When the initial conditions are defined at $t=0$, one can predict the evolution of the temperature at three points ($x=0\mathrm{m}$, $x=0.5\mathrm{m}$ and $x=1\mathrm{m}$) along the time (see Figure 7). The temperature decreases faster on the boundary with natural convection ($x=0\mathrm{m}$) whereas the two other places cool slower. The comparison of results obtained with PINN model with those obtained with FVM, enables one to note a good agreement, and consequently validate the PINN model (see Figure 7).

Regarding the mesh influence, from Figure 7 it can be noticed that similar results were obtained, with just some perturbation on the right boundary (red lines) temperatures obtained with PINN.

A similar analysis can be performed considering the evolution of the temperature distribution on the domain along the time (see Figure 8).

The comparison of the temperatures predicted by the PINN model and those obtained with FVM (see Figure 8) are similar with a slight difference on values to $t=0$, where the model seems to be affected by the boundary condition with natural convection. Several approaches were tried, namely different weights to losses and more training iterations, leading to continuous improving of these results.

4.2. DQL Results

With the PINN defined, i.e., trained, it was possible to train the policy model, with the Deep Q-learning algorithm. The training was performed on the same processor already referred and took about $18″$, $14″$, $11″$ and $11″$, respectively for 5, 11, 21 and 41 nodes, using the FVM, whereas the PINN model took $51″$, $36″$, $27″$ and $27″$, respectively.

The simulation to test the DQL control took around $0.67″$ with FVM whereas with the PINN took around $1.11″$.

To set a baseline, a bang-bang controller, i.e. the heating is turned on allways than the temperature at $x=0.5$ is equal to or lower than the goal temperature $T=0.5$. The evolution of the temperatures at three points ($x=0$, $x=0.5$ and $x=1$) were monitored, and from this evolution with FVM versus PINN model (see Figure 9) it seems that PINN enables a more stable control when coarser meshes are used with FVM improving its performance with mesh refinement. However, it should be noted that the PINN hyperparameters can be changed with a deeper study to improve this behaviour.

Aiming to evaluate the performance of the controller, the temperature evolutions along time at points $x=0$, $x=0.5$ and $x=1$ were represented in Figure 10, and it can be seen that the transition from the initial state to the oscillatory final state is smother when the controller defined with DQL was used. From the comparison of FVM and PINN several hyperparameters of PINN were tested with better results obtained when the control considered a forecast on time 11 time steps in the future than the current value. However, it should be noted that these results can be improved with a deeper study which is not the main goal of this work.

The evolution of the temperature on the domain along the time are presented in Figure 11 and Figure 12, respectively to Bang-bang controller and DQL controller, where the natural convection in the left boundary is identified by the characteristic temperature gradient and a progressive evolution to the final almost stationary state.

Since Bang-bang controller isn’t able to learn the time evolution of the temperature, cannot produce a stable solution with strong variations in temperature distributions (see Figure 11). On the other hand, the control defined with DQL strategy enables us to obtain much more stable solutions (see Figure 12).

Finally, the temperature of the control point (at $x=0.5$) obtained with the different meshes was analysed, considering its mean value and its standard deviation on the last 150 (of 200) time steps. These values (mean and standard deviation) are represented in Figure 13, against the mesh element size $dx$, i.e. lower values (when compared with Bang-bang control strategy) to finner meshes, and it can be seen that the Deep Q-Learning applied to PINN model enables one to obtain mean values closer to the objective as the mesh is finner alongside the standard deviation diminishing. Regarding the PINN versus FVM model, it can be noted that the PINN enables one to obtain values closer to the goal and with lower variations.

5. Conclusions

In this work, was trained a Physics-Informed Neural Network to enable its use in edge computing hardware instead of more resource-demanding alternatives. This model was validated by comparison with a Finite Volume Method solving. Next, this implementation was used on a Deep Q-learning algorithm to find a control policy aiming to attain a predefined temperature at a specific point. Such a control strategy was compared with the Bang-bang control, analysing a 1D problem and the results obtained enable one to show that this strategy is an interesting alternative implementation to use in more complex problems where the need of fast evaluations are required to make decisions.