Reinforcement Learning Based Speed Control with Creep Rate Constraints for Autonomous Driving of Mining Electric Locomotives

1. Introduction

Mining electric locomotives have the transportation function of materials, equipment, and people in roadway. Safe driving of mining electric locomotives is crucial. However, the method of underground mining is often used in the Chinese coal mining industry [1]. In deep underground confined spaces, there are unfavorable conditions for driving in coal mine roadway, such as slippery, muddy, dusty, foggy, and complex human behavior, which lead to frequent accidents. Reducing personnel participation and ensuring the safe operation of production and transportation equipment are necessary to ensure the safety production of coal mines. At present, the intelligent development of coal mine equipment has become an inevitable trend [2]. Unmanned transportation of coal mine equipment can fundamentally solve the problem of personnel participation and reduce casualties in the event of inevitable accidents. We have conducted relevant research on autonomous mining electric locomotives [3]. The mining electric locomotive has achieved functions based on RL and improved $\epsilon$-greedy such as autonomous and efficient operation on speed limited sections, maintaining a safe distance from vehicle in front, and avoiding obstacles.

However, problems such as slipping of autonomous vehicles and wheel idling caused by slippery roadway in deep mines have not been addressed in a targeted manner. Manually driven mining electric locomotives rely on the driver’s experience to sprinkle sand to increase wheel rail friction. But this method is a remedial measure taken when slipping/idling occurs during driving. And there is no quantitative evaluation standard for actions judged by human subjectivity. Moreover, actions based on human subjective judgment cannot be used as a quantifiable evaluation criterion applicable to machine autonomous decision-making. In the autonomous driving control process of mining electric locomotives, it is necessary to make decisions that can actively control the interaction between wheels and rails to prevent slipping, idling, and other phenomena caused by external environment.

For rail vehicles, the destruction of the adhesion between the wheel and rail is the basic reason of wheel slip/idling [4]. One of the reasons for the traction and braking forces of wheelsets is the presence of contact and friction between the wheel and rail [5]. This kind of wheel rail interaction force is not pure friction, but a phenomenon called creep caused by deformation after wheel rail contact [6]. Reasonably utilizing the creep rate can improve the efficiency and safety of vehicle operation [7,8]. Nowadays, to prevent train wheel slipping or idling, scholars have conducted research on algorithms related to adhesion control. According to different control methods, adhesion control algorithms are generally divided into re-adhesion control and optimized adhesion control [9]. Re-adhesion control is a method that can quickly adjust the motor torque to achieve balance with the current adhesion conditions, and avoid wheel idling when determining whether the locomotive’s wheels are idling/slipping. The research on this method is relatively mature [10,11,12,13], but it belongs to passive adhesion control method, with low adhesion utilization rate and long algorithm response time. The purpose of optimizing adhesion control is to achieve the peak point of the optimal adhesion utilization, which is of great significance for the operation control of trains under wet, muddy and emergency braking conditions. Mehmet Ali Çimen et al. [14] analyzed the input and output phase shift dynamic characteristics of the traction system. They also proposed an adaptive control method that effectively controls the adhesion utilization rate of the traction system. Song Wang et al. [15] proposed an adhesion control method based on optimal torque search for high-speed trains, which can achieve stable operation of trains in the optimal adhesion state under changes of track surface and high-speed driving conditions, effectively reducing the idle rate of the wheel and improving train adhesion utilization. Shuai Zhang et al. [16] proposed a sliding mode control method, which used recursive least squares method based on enhanced forgetting factor to solve the problem of wheel anti lock on heavy-duty trains. This algorithm can obtain the optimal creep rate and construct a PI closed-loop observer to estimate the unmeasured adhesion torque, enabling the locomotive to adjust to the optimal creep rate when the contact of wheel and rail changes. Although the optimized adhesion control algorithm can suppress the sliding and idling of the wheel, the maximum adhesion point in the contact area of wheel and rail is at the junction of creep and sliding, making it difficult to ensure that the contact state of wheel and rail is always creep during the actual control process. Therefore, this method still belongs to passive adhesion control. In addition, there are certain differences in the utilization of adhesion capacity among the axles of the train. The existing methods have insufficient control accuracy, as most of the adhesive characteristic curves relied on are empirical formulas obtained through experiments.

Therefore, this paper proposes a creep control method for mining electric locomotives based on RL. This method converts the impact of complex driving environments on mining electric locomotives into reward feedback values obtained by the electric locomotives from the environment. The impact of creep conditions that are difficult to quantify and evaluate on vehicles has been particularly considered in this method. The optimal range of creep rate is tried to achieve after training the mining electric locomotive to adjust the driving torque of the wheelset in this method. To achieve maximum driving efficiency under safe operating conditions is the goal of this method.

The overall structure of this paper is shown in Figure 1. Firstly, the autonomous control method of mining electric locomotives based on improved $\epsilon$-greedy is analyzed from the theoretical perspective. By changing environmental variables and replicating algorithms, the important factor affecting the operation of mining electric locomotive on wet and slippery tracks is identified. Secondly, a mining electric locomotives autonomous control algorithm based on creep control is proposed, which can effectively improve the safety and reliability of autonomous driving of mining electric locomotives. Finally, based on the three-dimensional dynamic model of mining electric locomotives modeled using Simpack, the feasibility of the proposed algorithm is verified using a multi software co-simulation platform.

Compared to [3], this paper identifies the creep rate, a key factor affecting the autonomous operation of mining electric locomotives, through theoretical verification of the algorithm and practical analysis of operating conditions. This paper also proposes a creep control method suitable for mining electric locomotives, which improves the accuracy of autonomous operation control for mining electric locomotives.

2. Analysis of Autonomous Control Algorithm for Mining Electric Locomotives

In this section, an important factor is found through theoretical analysis and simulation, which was not particularly considered in the previous algorithm design but affects the autonomous driving of mining electric locomotives. First, in reproducing the autonomous control algorithm of mining electric locomotives based on RL, we found that changing the friction between the wheels and rails can lead to poor control performance. Second, this algorithm has been theoretically proven to be reasonable and convergent. Mining electric locomotives can stably achieve autonomous driving under the control of this algorithm. Third, the relationship between creep rate and adhesion coefficient of rail vehicles is analyzed. It is necessary to take the creep rate between the wheel and rail into the autonomous control algorithm. Finally, we conclude that the improved control objective is to achieve the optimal control range of creep rate.

2.1. Introducing Problems

In reference [3], RL was adopted to solve the autonomous control problem of mining electric locomotives, and the $\epsilon$-greedy strategy was improved to balance the relationship of exploration and exploitation better. This control method is reproduced in this paper on the condition that the friction of wheel and rail is adjusted from 0.4 to 0.3. Under this working condition, the autonomous control state curve of the mining electric locomotive shown in Figure 2 is obtained. It can be seen that mining electric locomotive operates safely and efficiently on speed limited sections, and can maintain a safe distance from obstacles when using the autonomous control method based on the improved $\epsilon$-greedy strategy. However, it was found that mining electric locomotive is unable to control the acceleration duration correctly when accelerating and reaching the maximum speed limit, and the running speed of the electric locomotive would slightly exceed the maximum speed limit. Moreover, the mining electric locomotive cannot control the deceleration duration to reach the maximum speed on the next speed limit section when decelerating.

2.2. Theoretical Analysis

In order to find out the reasons for the problem above, we first verify whether the algorithm structure is designed reasonably from the perspective of algorithm theory. If the structural design is reasonable, the results of the algorithm will tend to converge with the training process of the agent. This section will demonstrate the convergence of the algorithm from two aspects: RL and improved $\epsilon$-greedy strategy.

2.2.1. Q-Learning

RL is a method for studying how an agent maximizes its reward in a complex and uncertain environment [17,18]. Figure 3 shows the process of RL. The agent interacts with the environment, obtains the current state $S_t$ and reward $R_t$ at moment t, and takes action $A_t$ based on certain strategies. After the agent takes action $A_t$, the environment obtains the latest state $S_{t+1}$ and reward $R_{t+1}$ at moment t+1, and passes them to the agent. This interaction process can be represented by the Markov Decision Process (MDP).

The MDP adds a decision layer to the commonly used Markov Process / Markov Reward Process, which is the action a shown in the Figure 4. This means that when the agent is in the state $S_t$ at time t, it must first decide on a specific action a to take in order to reach the middle layer, which is the black node in the diagram. After reaching the black node, the state of the agent at the moment t+1 also depends on the probability distribution.

RL, as a trial and error learning method, continuously repeats the above interaction process between agents and the environment to find a mapping that can maximize the cumulative sum $G_t$ of benefits $R_t$ over time.

$$G_t=\sum _{i=0}^{T-i-1}{\gamma }^i{R}_{t+i+1}.$$

(1)

where, T is the total time, $\gamma$ is the discount factor.

However, the sum of cumulative returns, also known as value return $G_t$, is not easily obtained. To solve this problem, researchers propose two value functions for estimating the sum of cumulative returns: the state value function $V_{\pi }\left(s\right)$ and the action value function $Q_{\pi }(s,a)$. The state value function $V_{\pi }\left(s\right)$ is the expected value of the sum of the reward functions under $\pi$ strategy and s state. The action value function $Q_{\pi }(s,a)$ represents the expected value of the sum of the benefit functions of action a under the $\pi$ strategy and s state.

$$V_{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[G_t\right|S_t=s]=\sum _{a\in \mathcal{A}}\pi \left(a\right|s)·Q_{\pi }(s,a).$$

(2)

$$Q_{\pi }(s,a)={\mathtt{E}}_{\pi }\left[G_t\right|S_t=s,A_t=a]=\sum _{r,s_{t+1}}p(s_{t+1},r|s,a)·[R+\gamma V_{\pi }\left(s_{t+1}\right)].$$

(3)

where, $\mathcal{A}$ is the set of action a.

According to the value function, the Bellman equation is derived as follows:

$$V\left(s\right)=R\left(s\right)+\gamma \sum _{s_{t+1}\in \mathcal{S}}p\left(s_{t+1}\right|s)·V\left(s_{t+1}\right).$$

(4)

where, $\mathcal{S}$ is the set of state s.

The Bellman equation for the Q-function is as follows:

$$Q(s,a)=R(s,a)+\gamma \underset{a_{t+1}}{max}Q(s_{t+1},a_{t+1}).$$

(5)

We adopt Theorem-Cauchy’s convergence Test to verify the convergence of RL we design to apply in the field of autonomous control of mining electric locomotives.

Theorem-Cauchy’s convergence Test: The necessary and sufficient condition for the convergence of a sequence is that for any positive number $\epsilon$, there must be a positive integer N that satisfies the following equation under the condition of $n>N$ and $m>N$:

$$|x_n-x_m|<\epsilon .$$

(6)

We refer to $\left\{x_n\right\}$ that satisfies this condition as a Cauchy sequence. So the theorem can be expressed as: the necessary and sufficient condition for the convergence of sequence $\left\{x_n\right\}$ is that the sequence is a Cauchy sequence.

Define H as the contraction operator, the following formula will be easily obtained:

$$HQ(s,a)=R(s,a)+\gamma {\mathtt{E}}_{s_{t+1}\sim p(·|s,a)}\left[\underset{a_{t+1}}{max}Q(s_{t+1},a_{t+1})\right].$$

(7)

Assuming that the optimal value function $Q^{*}$ is a fixed point of H, it means:

$$Q^{*}=H{Q}^{*}.$$

(8)

That is, $Q^{*}$ remains at its original value after any multiplication with the H operator.

$$\left(Hq\right)(s,a)=\sum _{s_{t+1}\in \mathcal{S}}{p}_a(s,s_{t+1})[r(s,a,s_{t+1})+\gamma \underset{a_{t+1}\in \mathcal{A}}{max}q(s_{t+1},a_{t+1})].$$

(9)

Next, the convergence of the Q function is proved:

$$\begin{array}{cc}\hfill \left|\right|H{q}_1-H{q}_2{\left|\right|}_{\infty }& =\underset{s,a}{max}\gamma \left|\sum _{s_{t+1}\in \mathcal{S}}{p}_a(s,s_{t+1})[\underset{a_{t+1}\in \mathcal{A}}{max}{q}_1(s_{t+1},a_{t+1})-\underset{a_{t+1}\in \mathcal{A}}{max}{q}_2(s_{t+1},a_{t+1})]\right|\hfill \\ \hfill & \le \underset{s,a}{max}\gamma \sum _{s_{t+1}\in \mathcal{S}}{p}_a(s,s_{t+1})|\underset{a_{t+1}\in \mathcal{A}}{max}{q}_1(s_{t+1},a_{t+1})-\underset{a_{t+1}\in \mathcal{A}}{max}{q}_2(s_{t+1},a_{t+1})|\hfill \\ \hfill & \le \underset{s,a}{max}\gamma \sum _{s_{t+1}\in \mathcal{S}}{p}_a(s,s_{t+1})\underset{a_{t+1}\in \mathcal{A}}{max}|q_1(s_{t+1},a_{t+1})-q_2(s_{t+1},a_{t+1})|\hfill \\ \hfill & =\gamma \left|\right|q_1-q_2{\left|\right|}_{\infty }.\hfill \end{array}$$

(10)

According to the Theorem-Cauchy’s convergence Test, any $q_1$ function can ultimately converge to $Q^{*}$ when $q_2$ is the optimal function of $Q^{*}$.

2.2.2. Improved $\epsilon$-Greedy

In reference [3], in order to balance the relationship between exploration and exploitation, the traditional $\epsilon$-greedy algorithm was improved by changing the value of $\epsilon$ in formula 11 to the value of $\epsilon$ in formula 12.

$${\epsilon }_1=({\epsilon }_{\mathrm{initial}}-{\epsilon }_{\mathrm{final}})·(1-episode/max\_episodes)$$

(11)

$${\epsilon }_2=\{\begin{array}{c}({\epsilon }_{\mathrm{initial}}-{\epsilon }_{\mathrm{final}})·(0.5+\sqrt{0.25-{(episode/max\_episodes)}^2}),(episode/max\_episodes)\in [0,0.5]\hfill \\ ({\epsilon }_{\mathrm{initial}}-{\epsilon }_{\mathrm{final}})·(0.5-\sqrt{0.25-{(1-episode/max\_episodes)}^2}),(episode/max\_episodes)\in (0.5,1]\hfill \end{array}$$

(12)

In order to evaluate the balance effect of exploration and exploitation, regret $l_t$ is defined to represent the average possible loss at each step:

$$l_t=\mathbb{E}[V^{*}-Q\left(a_t\right)]$$

(13)

where,$V^{*}=Q^{*}\left(a^{*}\right)=\underset{a\in \mathcal{A}}{max}{Q}^{*}\left(a\right)$ represents the expected return value corresponding to the optimal action.

Total regret $L_t$ is defined to represent the total loss:

$$L_t=\mathbb{E}[\sum _{\tau =1}^t{V}^{*}-Q\left(a_{\tau }\right)]=\sum _{a\in \mathcal{A}}\mathbb{E}\left[N_t\left(a\right)\right](V^{*}-Q\left(a\right))=\sum _{a\in \mathcal{A}}\mathbb{E}\left[N_t\left(a\right)\right]{\Delta }_a$$

(14)

where, ${\Delta }_a=(V^{*}-Q\left(a\right))$ represents the gap between the value of the action and the optimal action, and $N_t\left(a\right)$ represents the number of times that the action a has been selected. The goal of maximizing cumulative returns is actually equivalent to minimizing total regret, and an excellent algorithm is hoped to reduce the number of choices for actions with large gaps. For such an evaluation system, the most crucial issue is that the gap in practical problems is difficult to obtain, as $V^{*}$ is unknown and $Q^{*}\left(a\right)$ also needs to be estimated. However, this evaluation system can still have guiding significance for the evaluation of strategies. It is usually assumed that ${\Delta }_a$ is a known constant value when discussing.

If the total regret $L_t$ increases linearly with the increase of iteration, it indicates that the probability of selecting each action in the algorithm does not change at each time interval. That is, the regret $l_t$ at each step does not change, and the information obtained from exploration is not better utilized. A strategy that can better balance exploration and exploitation should have a sublinear total regret. As the iteration progresses, the regret $l_t$ for each time interval gradually decreases, and the selection of algorithms will gradually abandon those that are likely to achieve lower returns.

Lai and Robbins has proven that for all total regrets that may become an optimal strategy, a asymptotic lower bound in the form of logarithmic growth is required [19,20,21]:

$$\underset{t\to \infty }{lim}{L}_t\le 8lnt\sum _{a|{\Delta }_a}\frac{{\Delta }_a}{KL\left(R^a\right|\left|R^{a^{*}}\right)}$$

(15)

The strategy of $\epsilon$-greedy strategy can ensure that all actions are sampled infinitely as the number of iterations increases. Thereby the convergence of $Q\left(a\right)$ is ensured. This means that the probability of selecting the optimal action can converge to greater than 1 - $\epsilon$, which is close to certainty. However, the probability of selecting the optimal action is only approaching a value, and its effectiveness cannot be guaranteed in practice. Obviously, the probability of each choice of action by $\epsilon$-greedy is more than $\frac{\epsilon }{\mathcal{A}}$. So the regret $l_t$ is more than $\frac{\epsilon }{\mathcal{A}}\sum _{a\in \mathcal{A}}{\Delta }_a$, which is a linear total regret. However, although $\epsilon$-greedy is not the optimal method in theory, Kuleshov and Precup [22] have demonstrated through extensive experimental data that $\epsilon$-greedy can often achieve better results than some complex methods in practice.

For the improved $\epsilon$-greedy, the selection of actions follows the following rules:

$$a_t=\left\{\begin{array}{c}arg\underset{a}{max}Q(s_{t+1},a),forprobability1-{\epsilon }_t\hfill \\ randomfrom\mathcal{A},forprobability{\epsilon }_t\hfill \end{array}\right.$$

(16)

We can obtain the probability distribution of whether an action is the optimal action:

$$\pi \left(a\right|s)=\left\{\begin{array}{c}1-\epsilon +\frac{\epsilon }{\left|\mathcal{A}\right(s\left)\right|},ifa=arg\underset{a}{max}Q(s,a)\hfill \\ \frac{\epsilon }{\left|\mathcal{A}\right(s\left)\right|},ifa\ne arg\underset{a}{max}Q(s,a)\hfill \end{array}\right.$$

(17)

Set $episode=t$, $max\_episodes=T$. We will write the value of $\epsilon$ in the improved $\epsilon$-greedy as follows:

$${\epsilon }_t=\left\{\begin{array}{c}(0.5+\sqrt{0.25-{(t/T)}^2}),(t/T)\in [0,0.5]\hfill \\ (0.5-\sqrt{0.25-{(1-t/T)}^2}),(t/T)\in (0.5,1]\hfill \end{array}\right.$$

(18)

To demonstrate the convergence of the improved $\epsilon$-greedy strategy, we calculate the regret of the strategy:

$$l_t=\pi \left(a\right|s)\sum _{a\in \mathcal{A}}{\Delta }_a=\left\{\begin{array}{c}(1-\epsilon +\frac{\epsilon }{\left|\mathcal{A}\right(s\left)\right|})\sum _{a\in \mathcal{A}}{\Delta }_a,ifa=arg\underset{a}{max}Q(s,a)\hfill \\ \frac{\epsilon }{\left|\mathcal{A}\right(s\left)\right|}\sum _{a\in \mathcal{A}}{\Delta }_a,ifa\ne arg\underset{a}{max}Q(s,a)\hfill \end{array}\right.$$

(19)

$$\begin{array}{cc}\hfill L_t& \approx \underset{t}{\int }{l}_t\mathrm{d}t=\underset{t}{\int }\frac{\epsilon }{\left|\mathcal{A}\right|}\sum _{a\in \mathcal{A}}{\Delta }_a\mathrm{d}t\hfill \\ \hfill & =\left\{\begin{array}{c}\underset{t}{\int }\frac{\sum _{a\in \mathcal{A}}{\Delta }_a}{\left|\mathcal{A}\right|}(0.5+\sqrt{0.25-{(t/T)}^2})\mathrm{d}t,(t/T)\in [0,0.5]\hfill \\ \underset{t}{\int }\frac{\sum _{a\in \mathcal{A}}{\Delta }_a}{\left|\mathcal{A}\right|}(0.5-\sqrt{0.25-{(1-t/T)}^2})\mathrm{d}t,(t/T)\in (0.5,1]\hfill \end{array}\right.\hfill \\ \hfill & \to \left\{\begin{array}{c}\frac{\sum _{a\in \mathcal{A}}{\Delta }_a}{\left|\mathcal{A}\right|}(\frac{t}{2}+\frac{1}{8}Tarcsin\left(\frac{2t}{T}\right)+\frac{t}{4}\sqrt{1-4{\left(\frac{t}{T}\right)}^2}),(t/T)\in [0,0.5]\hfill \\ \frac{\sum _{a\in \mathcal{A}}{\Delta }_a}{\left|\mathcal{A}\right|}(\frac{t}{2}+\frac{1}{8}Tarcsin\left(2(1-\frac{t}{T})\right)+\frac{T}{4}(1-\frac{t}{T})\sqrt{1-4{(1-\frac{t}{T})}^2}),(t/T)\in (0.5,1]\hfill \end{array}\right.\hfill \end{array}$$

(20)

Plot the trends of $L_{\mathrm{t}}$ and its derivative $L_{\mathrm{t}}\text{’}$ as the number of iterations increases. In Figure 5, it can be seen that $L_{\mathrm{t}}\text{’}$ is 0 in the later stage of iteration, and the value of $L_{\mathrm{t}}$ no longer increases, indicating that the improved $\epsilon$greedy tends to converge.

In summary, RL and improved $\epsilon$-greedy strategy applied to autonomous control of mining electric locomotives is convergent. The autonomous control method for mining electric locomotives based on the improved $\epsilon$-greedy strategy is feasible.

2.3. Adhesion and Creep

After the analysis, we will focus on the constraints considered in the algorithm design process. When designing the algorithm, conditions were taken into consideration, such as speed limits, road obstacles, and safe following of mining electric locomotives . However, the road surface of the coal mine roadway is very slippery and muddy, and the contact condition between the wheels and rails is also one of the factors that needs to be carefully considered.

Two conditions must be met simultaneously to enable rail vehicles to run along the track [23]. One is that the rotating torque is applied to the moving wheel, and the other is that the contact between the moving wheel and the steel rail has frictional effect. As shown in Figure 6, with the wheel as the force analysis object, the moving wheel is in contact with the track due to the vertical force P applied by the vehicle body, and generates a motion trend under the torque M transmitted by the transmission device [24]. Assuming that there is static friction between the wheel and rail, the force F’ generated by the moving wheel on the track is equal in magnitude to the force F generated by the track on the moving wheel, but in opposite directions. For the mining electric locomotive, rim traction force F is the driving traction force. But the wheels of rail vehicles mostly have conical tread, which can cause elastic-plastic deformation when the wheels come into contact with the rail. The mining electric locomotive is subjected to shock and vibration during operation. When the wheel set rolls on the rail, it is accompanied by longitudinal and transverse sliding. There is no pure static friction state between the wheel and rail, but rather ‘slight movement in stillness’ and ‘slight sliding in rolling’. This phenomenon is creep, also known as adhesion between wheel and rail. The maximum longitudinal horizontal force in the adhesive state is the adhesive force, and the ratio of adhesive force $F_{\mu }$ to vertical load P is the adhesive coefficient $\mu$. According to Hertz elasticity contact theory [25], the contact area between the wheel and rail is approximately elliptical. The area where the wheel and rail are relatively stationary and do not slide under the action of positive pressure is called the adhesion zone. And the area where slight sliding occurs is called the creep zone. As the driving torque M increases, the creep zone becomes larger and the adhesion zone decreases to 0, and the wheels are in a sliding state. Therefore, when the vehicle is in traction or braking conditions, if the traction or braking force is greater than the adhesion between the wheels and rails, the wheels will idling or slip. This phenomenon can cause damage to the wheel rail tread and even affect the safety of rail vehicles.

According to the International Union of Railways (UIC) definition of creep rate, the creep rate is represented as follows:

$$\left\{\begin{array}{c}{CR}_{longitudinal}=\frac{{({VW}_{longitudinal}-{VT}_{longitudinal})}_{Atthecontactpatch}}{Nominalforwardvelocity}\hfill \\ {CR}_{lateral}=\frac{{(Wheellateralvelocity-Tracklateralvelocity)}_{Atthecontactpatch}}{Nominalforwardvelocity}\hfill \\ {CR}_{spin}=\frac{{(Wheelangularvelocity-TrackWheelangularvelocityvelocity)}_{Atthecontactpatch}}{Nominalforwardvelocity}\hfill \end{array}\right.$$

(21)

Based on the operating results when the friction between the wheel and rail is 0.3 (Figure 2), it can be observed that there is a sudden change in the interaction of wheel and rail during vehicle acceleration, and the creep rate value is abnormal.

2.4. Control Objectives

As the main auxiliary transportation tool in the coal mine roadway, the mining electric locomotive is greatly affected by the humid, muddy and other harsh environment during driving. When there are pedestrians or obstacles, the tram can’t bypass to avoid them, but only take emergency braking. Wet and slippery road surface may lead to insufficient wheel/rail adhesion of mining electric locomotive. When the vehicle is in traction condition and the traction force is greater than the wheel/rail adhesion, the wheel will idling. When the vehicle is in braking condition and the braking force is greater than the adhesion, the wheels will slip. The occurrence of these situation will cause the wheel tread and rail surface to be scratched, which will seriously affect the safety and stability of the vehicle. Therefore, the most critical control factor for the safe running of mining electric locomotive on the track is the value of the driving/braking torque applied on the axle. In this way, we can directly control the driving acceleration and speed of the vehicle by controlling the torque to avoid driving accidents such as vehicle braking failure and vehicle rollover.

As shown by the adhesion characteristic curves under dry and wet rail surface conditions in Figure 7, when the creep rate is within the range of -10% to 10%, the absolute value of the adhesion coefficient between the wheel and rail rapidly increases under braking/traction conditions [26]. When the creep rate is within the range of -30% to -10% and 10% to 30%, the absolute value of the adhesion coefficient between the wheel and rail reaches its optimal value under braking/traction conditions. When the creep rate is around ±20%, the wheel rail adhesion coefficient will reach its peak, and at this time, the vehicle can obtain maximum ground braking/traction force, which can minimize the braking distance. Therefore, controlling the creep rate within the range of ±10% to ±30% is an ideal control goal.

3. Creep Controller Model

In this section, an RL speed control method for mining electric locomotives based on the optimal creep rate is proposed. For mining electric locomotives, there are many turns and narrow sections in coal mine roadway, and there are many emergencies during the operation of the locomotives. This method can fully utilize the advantage of RL that does not rely on models. The impact of complex mine environments on vehicle driving processes, especially the impact of slippery wheel/rail conditions that are difficult to quantify and evaluate on vehicle safety and autonomous operation, is converted into reward feedback obtained by vehicles from the environment. This method simplifies the modeling process of vehicle driving conditions and improves the efficiency of intelligent algorithm calculation. There are three major elements in RL, including state, reward, and strategy for selecting actions. When using RL method for creep control of the mining electric locomotive, the mining electric locomotive is considered as an intelligent agent, and the algorithm components are designed as follows.

3.1. State and Action

According to the starting or stopping of the mining electric locomotive, whether the driving section is a curve or a straight road, whether there are obstacles, whether the operating speed reaches the speed limit of the corresponding section, and whether the electric locomotive reaches the destination, the state of the RL creep control algorithm for the mine electric locomotive is set to 16 as shown in the Table 1. In addition, the torque applied on the axle has three forms: positive, negative, and zero, corresponding to the three actions of vehicle acceleration, deceleration, and uniform speed.

During the execution of the algorithm, the real-time position and speed of the mining electric locomotive are recorded every 0.01 seconds to determine the changes of electric locomotive state during the sampling interval. At the same time, the algorithm will also timely provide the actions that should be applied within the sampling time of 0.01s, ensuring the timeliness of control.

It is worth pointing out that JB/T 4091-2014 ’Technical Conditions for Explosion-proof Special Battery Electric Locomotives in Coal Mines’ stipulates that mining electric locomotives should pass through the curve radius at a 50% of hourly speed. For the CTY1.5/6 electric locomotive used in this case, the maximum speed of the vehicle is 2.83m/s (10.2km/h), and the hourly speed of the vehicle is 1.80m/s (6.5km/h). Therefore, when the mining electric locomotive passes a curve with a radius of curvature greater than 4m, the minimum speed of the vehicle is 0.90m/s.

3.2. Features of Reward Function

The appropriate definition of the reward function can directly determine whether the learning process of RL algorithms can efficiently and quickly achieve the desired training objectives. The main task of RL is to maximize rewards by selecting the best action at each sampling time. The ultimate goal of control algorithm for the electric locomotive is to achieve the optimal creep rate utilization in uncertain and harsh roadway environments, reduce the occurrence of slip and idling phenomena, and achieve the best operating efficiency under the premise of safe operation. Therefore, for this case, the reward function is directly related to the control effect of the mining electric locomotive’s driving speed and creep rate under different states.

Whether the mining electric locomotive has encountered obstacles and reached the maximum speed of the corresponding road section is classified during the state setting. When setting the reward function, whether the vehicle is accelerating, decelerating, or running at the uniform speed in the current state should be considered. The vehicle changing at a faster speed to achieve the desired control purpose will present a better training effect. The following speed reward conditions is defined based on repeated experiments. The main way to measure the size of the reward value is to calculate the speed change that occurs during the unit sampling interval for the vehicle to achieve control objectives.

$$rewar{d}_{\mathrm{speed}}=c_{\mathrm{speed}}*(v_{\mathrm{record}}-v_{\mathrm{current}})$$

(22)

where, $c_{\mathrm{creep}}$ represents the speed reward coefficient, which can be obtained through multiple experiments.

According to the control goal proposed in section 2.4 of the paper to control the creep rate within the range of ±10% to ±30%, the creep control reward conditions are set. We have comprehensively considered the creep rate of four wheels, and the evaluation criteria for the creep rate reward are set as follows.

When the absolute values of the creep rate of the left front wheel $|{\xi }_{\mathrm{FL}}|$, right front wheel $|{\xi }_{\mathrm{FR}}|$, left rear wheel $|{\xi }_{\mathrm{BL}}|$, and right rear wheel $|{\xi }_{\mathrm{BR}}|$ are all approximately equal to the optimal creep rate of 0.2 when taking two decimal places, the reward value can be set to:

$$rewar{d}_{\mathrm{creep}}=c_{\mathrm{creep}\_\max }$$

(23)

In other cases, the reward value is set to:

$$rewar{d}_{\mathrm{creep}}=\frac{c_{\mathrm{creep}\_\max }}{\sqrt{\left(\right|{\xi }_{\mathrm{FL}}{|-0.2)}^2+\left(\right|{\xi }_{\mathrm{FR}}{|-0.2)}^2+\left(\right|{\xi }_{\mathrm{BL}}{|-0.2)}^2+\left(\right|{\xi }_{\mathrm{BR}}{|-0.2)}^2}}$$

(24)

where, $c_{\mathrm{creep}\_\max }$ and ${\mathrm{c}}_{\mathrm{creep}}$ represent the reward coefficient for optimal creep rate and the reward coefficient for creep rate respectively, which can be obtained through multiple experiments.

We add the evaluation results of speed and creep with ratio of 1:1 to obtain the final reward function formula:

$$reward=rewar{d}_{\mathrm{speed}}+rewar{d}_{\mathrm{creep}}$$

(25)

3.3. Structure of the Creep Control Method

After setting up the intelligent agent, state, action, and reward function, a complete creep control framework for mining electric locomotives should be designed. As shown in the Figure 8, the mining electric locomotive serves as an intelligent agent to determine the current state $S_t$ based on the conditions of the environment at the current moment t. Then the intelligent agent uses an improved $\epsilon$-greedy method to select the actions to be executed, obtain the current action $A_t$, and apply it to the axle. Finally, the intelligent agent obtains the corresponding reward value $R_{t+1}$ based on the current environment and state. This reward is used to update Q-table.

4. Simulation Verification and Results

In this section, we varified the proposed RL based creep control method for autonomous operation of mining electric locomotives through simulation experiments. A dynamic model of the CTY1.5/6 mining electric locomotive is built through Simpack. And the algorithm framework based on the Python language environment is designed. MATLAB was used as the intermediate carrier for connecting the algorithm and model, mainly responsible for the collection of dynamic model data and the output of algorithm operation results. Therefore, in this paper, the validation of the algorithm will be carried out through a co-simulation platform built by Simpack, MATLAB, and Python.The simulation platform is used to varify the effectiveness of the creep control method for autonomous operation of mining electric locomotives, and compared it with traditional $\epsilon$-greedy algorithms to highlight the advantages of the improved greedy algorithm in terms of learning efficiency.

4.1. Simulation Platform and Setup

The Windows 11 Professional 64 bit system is used as a simulation platform operating system, equipped with an Intel (R) Core (TM) i9-9900K CPU @ 3.60GHz CPU and NVIDIA GeForce RTX 2070 SUPER GPU. The composition software of the co-simulation platform is PyCharm Community Edition, MATLAB R2014b and Simpack 2018.1. The language environment of Python 2.7 and its equipped third-party libraries such as Matlabengine for Python R2014b and Openpyxl 2.6.4 provide the foundation for the operation of simulation platform algorithms.

We use the RL algorithm based on the improved $\epsilon$-greedy mentioned above to train the creep control of the CTY1.5/6 mining electric locomotive. The number of iterations is 500, with a maximum runtime of 45s and a sampling interval of 0.01s. The learning rate of this algorithm is set to 0.2, and the Q-learning discount rate is 0.8. The initial and final values of $\epsilon$ are set to 0.01 and 1, respectively. The driving section of the mining electric locomotive is composed of a combination of 20m straight road-20m curved road-20m straight road. In general, the friction between the rail and the wheel is 0.2-0.4. Therefore, the friction coefficient on the track surface is set to 0.3.

4.2. Simulation Results

The reward values for creep control of the mining electric locomotive obtained through simulation using traditional $\epsilon$-greedy and improved $\epsilon$-greedy algorithms are shown in the Figure 9. It can be seen that under the condition of training 500 times, the agent obtains lager reward values in the first half of the training using traditional $\epsilon$-greedy. The increasingly stable creep control reward value is obtained in the later stages of training during the use of improved $\epsilon$-greedy algorithm for autonomous driving simulation in mining electric locomotives. This indicates that the improved $\epsilon$-greedy algorithm we have designed can fully utilize the experience gained from exploration in the first half of the training, while the exploration of traditional $\epsilon$-greedy in the first half of the training is clearly limited. And towards the end of the training stage, the reward value tends to stabilize to ensure the reliability of the training results, allowing the agent to repeatedly train and verify the optimal value.

We take the results every 125 iterations and plot the real-time position, speed, and creep rate of each wheel of the mining electric locomotive. From the 125th iteration to the 500th iteration (Figure 10 and Figure 11), the mining electric locomotive can achieve their destination. However, the speed change is different. It can be easily observed that the mining electric locomotive cannot fully achieve the maximum speed limit for straight or curved roads during the initial training stage, and the vehicle cannot stop at its destination. As the training progresses, the mining electric locomotive acceleration increases and it can operate efficiently on speed limited sections, with accurate braking at the destination. On the first straight section of the road, the mining electric locomotive can maintain a safe distance when encountering obstacles.

In terms of controlling the creep rate, it can be seen that the creep rate fluctuates greatly during vehicle acceleration, which can easily lead to wheel spin. Meanwhile, when the vehicle is driving in a bend, there is also a fluctuation in creep rate, indicating the possibility of some wheels slipping at certain times during the turning process. After training the agent, it can be seen that as the number of iterations increases, the idle time during the vehicle acceleration process significantly decreases, almost eliminating the phenomenon of the vehicle slipping on curves. When the intelligent agent is trained 500 times, the creep rate of the vehicle during acceleration can be controlled within the range of 0.1 to 0.4. This indicates that the creep rate between the wheels and rails is well controlled during vehicle operation and is well utilized to drive wheel acceleration. This proves that the design of our learning algorithm is reasonable and effective, which can better utilize the creep rate and achieve the goal of safe and efficient creep control.

5. Conclusions

In order to reduce the occurrence of idling or slipping caused by wet track surface and improve vehicle safety when mining electric locomotive is driving, an improved $\epsilon$-greedy of creep control strategy for mining electric locomotives based on RL algorithm is designed in this paper. By reproducing and proving the autonomous control method of mining electric locomotives based on improved $\epsilon$-greedy, the analysis of the operating conditions of mining electric locomotives shows that creep rate is an important influencing factor on the operation of electric locomotives in wet and slippery tunnels. An RL based creep control method is proposed, which considers the efficiency of vehicle driving speed and the effectiveness of creep control in the algorithm design process. Finally, simulation verification is conducted based on a co-simulation platform that can simulate the dynamic process of mining electric locomotives underground. The results show that this method can achieve safe operation of mining electric locomotives in complex and ever-changing mine environments, including being able to drive with maximum efficiency on speed limited sections, follow at a safe distance, and reduce the occurrence of dangerous phenomena such as slipping and idling during vehicle driving.